Blog 4 Jordan Day

Visitor of the day


  • You
    from

Brag Stats

  • Comments:25,004
  • Articles:2,000
  • Article Hits:12,459,805
  • Unique Visitors:2,000,438
  • Rss Subscribers:3,052
  • Comment Subscribers:2,530
  • Spammers:136,315
  • Generated :757,671 spams
  • Monitoring:3,942,477 spam IPs
Powered by Qwaider Shield

Recent Comments

Check out the latest pictures on Sweetestmemories

« Double blogging!How to say Mab3oos in other languages! »

Reputation based anti-spam


    • Currently 4.7/5 Stars.
    • 1
    • 2
    • 3
    • 4
    • 5

    Rated 4.7/5 stars (93 votes cast) Thanks for your vote!

    As a new addition to the anti-spamming system that I'm using exclusively on this blog. I decided to incorporate a reputation based system that looks deeper into the reputation of the person submitting a comment before deciding whether or not that comment is more "likely" to be good or bad, then assigns a score for it.

    The idea is really simple, it's based on fending off attacks by depending on social engineering techniques, here's how it works:

    The user submits a comment.
    The system evaluates the comment and IP address and decides whether it's spam or not. (this is old), a minor change was added to "score" each stage
    The system looks at the identity of the person who commented. The identity is a union of the user's name, email, website, IP address and the history of names, and IP addresses. THIS is where it gets interesting

    By looking back at the history of an IP, weighing the good vs the bad, we can know if the IP is more likely to be generating SPAM, that's part one.

    Update:
    For a limited time, you can see your comment SpamScore when you comment here. Red=Bad, Green=Good. But Unfortunately, I don't have these values for all the comments around here. So apologies if you don't see it

    Now by looking at the Name, email, Website, and IP in a historical fashion. We get to know if this "identity" and deal with it as such. Or more specifically, by looking at the number of valid comments with those parameters, we can add a probability score of a specific comment being spam or not.

    Of course the greater the number of valid comments the better the overall results for that specific identity. However, this will need a user to have previously commented. But, I guess that goes without saying since we started this whole article with the word, "reputation". In other words, history.

    The question that arises here is, what if the spammers fake specific previously known identities? wouldn't that side track this whole mechanism, even worse, use it to get better spamscore than before using it?

    Very good and valid question. In fact, this is one side that I considered. The elements I considered to create a distinguished Identity is Email. Email, is never communicated or disclosed. Although, it's not a big secret, but it's STILL too much work for a spammer to do (figure out the name, email AND website) a specific user is using to comment. But it's possible and therefore EVEN with well known identities other measures will provide additional spam values that will outweigh such an attack

    So surface of attack is: New users with little or no history. Attacker who knows the combination of Name, Email and Website of users with good scores (like say, Hani Obaid on this blog).

    The hard part to figure out, is how to translate spam-score to an actionable item

    Thoughts? issues? concerns? Criticism? let me know what you think. Oh and ask me about your scores if you're interested :)

    Other Memories Documented on November 25
    « Double blogging!How to say Mab3oos in other languages! »

    Memories....

    • #1
    • Summer
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 8:56:48 PM
    • SpamScore=[54]
    this is a test....lets see if it will work out!
    • #2
    • Qwaider
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 9:01:16 PM
    • SpamScore=[-47.45]
    Summer, you scored 54, that's pretty high! I don't think I've seen this score so far. I think you need to be careful while you're on this network. Run the latest OS, Latest Anti-Virus, and enable firewall

    (thanks for testing :))
    • #3
    • Summer
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 9:28:11 PM
    • SpamScore=[53.43]
    how is this going? more testing!
    • #4
    • Maioush
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 10:11:58 PM
    • SpamScore=[-2.28]
    o ana o ana o ana :D
    • #5
    • hamede
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 10:38:13 PM
    • SpamScore=[-1.31]
    test
    • #6
    • mab3oos
    • Ubuntu OS Firefox Browser
    • Said
    • On: 11/25/2008 10:57:12 PM
    • SpamScore=[-1.29]
    spam shpam. As If..
    • #7
    • Qwaider
    • Windows Internet Explorer
    • Said
    • On: 11/25/2008 11:20:29 PM
    • SpamScore=[-37.82]
    As if what mr Mat3oos?
    • #8
    • Hani Obaid
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 1:11:08 AM
    • SpamScore=[-5.65]
    cheap halitosis pills $20 a pack... click here
    • #9
    • Hani Obaid
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 1:14:01 AM
    • SpamScore=[-5.67]
    how did I go from -.03 to -5.65 in 1 comment
    • #10
    • Qwaider
    • Windows Internet Explorer
    • Said
    • On: 11/26/2008 1:22:44 AM
    • SpamScore=[-47.49]
    You didn't! You moved from -5.65 to -5.67

    Besides, the system is still in early beta :)
    • #11
    • whisper
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 1:25:22 AM
    • SpamScore=[-0.03]
    hope it's greeeeen :S
    • #12
    • Faisal
    • Windows Internet Explorer
    • Said
    • On: 11/26/2008 3:48:59 AM
    • SpamScore=[1.04]
    maaan !! This is like credit history! There could be decent guys who have no "history", but they cannot get a credit card ( comment ) !

    This is a very interesting field. You might wanna check out " collaborative filtering" or " recommender algorithms"

    or the forums of www.netflixprize.com if you have time you could win a million dollars !
    • #13
    • Nizar
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 4:40:04 AM
    • SpamScore=[-0.41]
    I hope mine turns green not like Summer's :P
    • #14
    • Noura
    • Windows Internet Explorer
    • Said
    • On: 11/26/2008 5:40:23 AM
    • SpamScore=[-0.04]
    My curiosity got the best of me :)
    let's see ......
    • #15
    • kinzi
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 7:59:42 AM
    • SpamScore=[3.68]
    Me too!!
    • #16
    • kinzi
    • Windows Firefox Browser
    • Said
    • On: 11/26/2008 8:02:39 AM
    • SpamScore=[-1.31]
    Did mine get lost somewhere?
    • #17
    • KJ
    • Windows Chrome  Browser
    • Said
    • On: 11/27/2008 10:22:33 AM
    • SpamScore=[-0.15]
    Order NOW v14gr4 and get FREE Moogle plushies!
    • #18
    • KJ
    • Windows Chrome  Browser
    • Said
    • On: 11/27/2008 10:28:22 AM
    • SpamScore=[0.4]
    ewa ewa free moogles eltelli
    • #19
    • za3tar
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:27:22 PM
    • SpamScore=[0.5]
    Uno
    • #20
    • Summer
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:27:54 PM
    • SpamScore=[5.5]
    Dos
    • #21
    • za3tar
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:28:25 PM
    • SpamScore=[3.43]
    Tres
    • #22
    • za3tar
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:30:05 PM
    • SpamScore=[-1.56]
    Quatro
    • #23
    • Summer
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:31:17 PM
    • SpamScore=[0.01]
    Cinco
    • #24
    • za3tar
    • Suse OS Firefox Browser
    • Said
    • On: 11/27/2008 8:41:30 PM
    • SpamScore=[-1.58]
    First of all, my apologies to Summer for the impersonation .. this was only a one time thing for the sake of science ;-p.

    I think using the IP address along with the name/email/website info is a great idea because it will protect bloggers from each other. While it is simple for bloggers to impersonate other bloggers (we know each other's info from our blogs' comments), it is considerably harder to spoof the *correct* IP address without easy detection.

    I left 5 comments (Uno -> Cinco). Two of them got caught for moderation. However, clearing the site cookies between comments fixed the problem easily. I also noticed that all the scores that i got were pretty similar. Which i think is a good sign that your spam checker is working correctly.

    Additionally, i got a good score when i commented as Summer (Cinco), and i think that is attributed to the change of IP address.

    So all in all, it seems that the IP address info trumps everything else in your system. I think this is a great thing. However, i wonder what would happen to legitimate people who use a compromised IP address.

    Great job on this system!
    • #25
    • Qwaider
    • Windows Internet Explorer
    • Said
    • On: 11/27/2008 10:01:06 PM
    • SpamScore=[-46.15]
    The whole idea is to make sure that bad people (spammers) don't get through, while the good ones do get through. Spammers don't have time to socially know who my friends are, and then get their emails and after that submit comments with their identity. It's too much work
    I already made sure that my registered users don't face such issue (no one can comment with the name Maioush for example on this blog) Even if they used the right user/email/website combination.
    However, Summer (in particular) opted out of this (she used to be protected with this system, but didn't like logging in)
    Anyway, I have the system in "observation" mode right now. It gives me recommendations ONLY without taking any action because. Frankly, I don't know how to score things yet. I mean, What does it mean? How can I give positive marks for certain things and negative for others. Then have the stuff cancel out for good users in bad networks :)

    What you didn't know is that for the single score that you're seeing, there are 4 others, and additional details that I'm capturing. One for the combination: IP, NAME, EMAIL, website, Valid comments. One for combination: Name, Email, Website, another for: Name, Email, Website valid comments, and total comments. These all participate in the spam score, but there are almost 28 other factors at play in the spam score :(

    Thanks for testing Za3tar, I really appreciate it.
    • #26
    • za3tar
    • Suse OS Firefox Browser
    • Said
    • On: 11/28/2008 12:57:26 AM
    • SpamScore=[-1.63]
    Yes yes, i agree with you Qwaider that my tests were not indicative of normal spammer attacks because i had leverage of extra pieces of information. I also think that your system actually did well under my comments (i.e., it seemed to respond to the fact that i am not really Summer although i have impersonated her), so all in all i think it is a good system you got there buddy.

    I have to admit, i am getting sick of the Captchas and will be counting for the day i can remove it with my mind put on ease.. and you are demonstrating that that day is drawing ever more closer.
    • #27
    • Qwaider
    • Windows Internet Explorer
    • Said
    • On: 11/28/2008 1:13:41 AM
    • SpamScore=[-46.17]
    Everyone in the security world knows what's going on in the sweatshops in the Philippines. Captchas are getting cracked left and right with real humans at the helm. Even new and improved OCR that is able to solve the captchs (granted, they fault levels are still 100 times higher than humans. But they're closing that gap and quickly)
    What people don't seem to realize is that some of the world most brilliant minds are working to break these rudimentary methods of false security and I have no doubt in my mind that they will succeed. For god's sake we landed a man on the moon with processing power that wouldn't power a modern entry level cellphone.

    Thanks again Za3tar, you always help me keep my brain cells busy, like today I saw a greater need to implement my new Volume based anti-spamming mechanism which I'm only thinking of. It goes something like this. If a commentor sends few comments in a row, this might mean that he's a type of a commentor that is probing the system. Therefore, there needs to be an upper limit for the number of comments from a [user, ip] tuple, ip, and user. Throttled to within a specific period of time.
    You too can have your Memories Documented

    Country:

    HTML has been disabled but if you wish to add any hyprlinks or text formating you can use any of the following codes: [B]bold text[/B], [I]italic text[/I], [U]underlined text[/U], [S]strike through text[/S], [URL]http://www.yourlink.com[/URL], [URL=http//www.yourlink.com]your text[/URL]

    Whisper (your comment will not be displayed)

    Please refer to Commenting policy


    Notify me of follow-up comments by email
    « Double blogging!How to say Mab3oos in other languages! »
    Read by:
  • Guest-
  • |
  • Guests(6)-
  • |
  • KJ-
  • |
  • Secret Window-
  • |
  • Guests(2)-
  • |
  • Guests(4)-
  • |
  • Mariam Ayyash-
  • |
  • Summer-
  • |
  • Guests(39)-
  • |
  • Guests(2)-
  • |
  • Guest-
  • |
  • Guests(481)-
  • |
  • Guests(11)-
  • |
  • Guests(563)-
  • |
  • Guests(9)-
  • |
  • Guests(10)-
  • |
  • Guests(7)-
  • |
  • Guests(8)-
  • |
  • Guests(21)-
  • |
  • Guests(17)-
  • |
  • Guests(191)-
  • |
  • Guests(96)-
  • |
  • Guests(9)-
  • |
  • Guest-
  • |
  • Guests(8)-
  • |
  • Guests(5)-
  • |
  • Guests(11)-
  • |
  • Guests(34)-
  • |
  • Guests(16)-
  • |
  • Hani Obaid-
  • |
  • Princess N-
  • |
  • whisper-
  • |
  • Guests(25)-
  • |
  • Guests(2)-
  • |
  • Guests(6)-
  • |
  • jessyz-
  • |
  • Guests(4)-
  • |
  • Guests(2)-
  • |
  • Guests(18)-
  • |
  • Guest-
  • |
  • Guests(12)-
  • |
  • Guests(233)-
  • |
  • Guests(37)-
  • |
  • Guest-
  • |
  • Guests(7)-
  • |
  • Guests(5)-
  • |
  • Guests(115)-
  • |
  • Guests(4)-
  • |
  • Guests(20)-
  • |
  • Nizar-
  • |
  • Guests(8)-
  • |
  • Guests(2)-
  • |
  • Guest-
  • |
  • Guests(7)-
  • |
  • Guests(9)-
  • |
  • Guests(65)-
  • |
  • Guests(3019)-
  • |
  • hamede-
  • |
  • kinzi-
  • |
  • Krystal-
  • |
  • mab3oos-
  • |
  • Maioush-
  • |
  • Noura-
  • |
  • Summer-
  • |
  • Verbal Alchemy-
  • |
  • za3tar-
  • |
  • za3tar-
  • |
  • Guest-
  • |
  • Guest-
  • |
  • Guests(13)-