March 17, 2007

TechCrunch has 15,000 Spam Comments Per Day

Michael Arrington

76 comments »

On January 4 we reported that the Akismet filter had stopped a million spam comments from reaching TechCrunch. At that point we’d been using it for about nine months.

The number of blocked spam comments is now two million, just ten weeks later. That works out to about 15,000 spam comments hitting TechCrunch every day.

If we did not have Akismet, we couldn’t allow anonymous commenting here on TechCrunch. We used to go through all spam comments to pick out the occasional false positive and accept it. Now, there are just too many to go through. All comments marked by Akismet as spam get deleted almost immediately.

We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. This will significantly reduce the overall amount of spam that hits the blog, so I don’t expect to see this total spam number continue to increase exponentially.

  • Sphere It

Trackbacks/Pings (Trackback URL)

  1. Ryan McDonnell - » Stop comment spam on a Wordpress blog
  2. BlogAjuda » Como resolver o problema dos spams?
  3. MMISoftware » Blog Archive » Spam, spam, spam..
  4. TechCorner » Extending WordPress - Plugins Used
  5. Akismet - I think I love you! | Bookmark Bliss
  6. adult japan tv
  7. mesh ferrari 360
  8. Spam Comments | moverton.com

Comments

RSS feed for comments on this post.

  1. Bill D'Alessandro

    Wow, and I thought I had a lot of spam when I got 800 in one day.

    Thank god for Akismet!

  2. wayne

    WOW, that is crazy, I hate spam, I had to put in Spam Karma 2 along with Math Comment Spam Protection to block spam from my blog, can you name the recommend wordpress plug-in?

    Thanks

    Wayne

  3. Bill D'Alessandro

    Wayne,

    I think Michael may be referring to the Comment Timeout plugin.

    - Bill

  4. Morgan

    Yeah it’s defintely a good idea with the timeout.

    I had always wondered how there wasn’t a bunch of spam on here, now I know. That’s insane.

  5. Michael Arrington

    We used a plugin originally that didn’t scale properly, too many db calls and it took us down for a couple of hours. The new plugin is custom code that I believe Wordpress has developed internally.

  6. Brett Tabke

    This isn’t a spam issue as much as an administration issues.

    Why not impliment a simple captcha? Or the newer:

    “prove your a human - what is 3+4= ….”

    Both systems virtually eliminate robotic spam based comments.

  7. Michael Arrington

    Brett…because Akismet takes care of it for us. I’m not asking for a solution, we have one.

  8. _ryan

    What’s the wordpress recommended plugin you’re using?

    _r

  9. Zaid

    Captcha would be nice but big as TC is, it might have to address captcha usability problems for those with various disabilities.

    I guess key point here is to know if your comment doesn’t appear, you should edit and re-submit instead of waiting for admins to approve it.

    –Zaid

  10. Leftblank

    Interesting to see, that numbers make my 20+ daily spamcomments vanish. I wonder if TechCrunch is also using some sort of plugin such as Bad Behavior that will actually disallow malicious visitors (or actually robots) from even loading the pages of your website. This plugin has helped me to reduce the spam from 100’s to just 10’s a day, which are then caught by Akismet, a wonderful duo!

  11. Jonathan Jiang

    @ Brett

    That won’t stay infallible for long

    “prove your a human - what is 3+4= ….” computers can add too you know :P

    Something more along the lines of random questions that any human can answer with ease like,

    “What is the color of the ocean?” With answers like blue, green, maybe for some crazy people turquoise.

    Of course, there _are_ some people who never have seen the ocean before… so.. that may be discriminatory.

    Agh, stick with the [)(U2RS00WFG] “type this out please to prove that you are indeed, breathing”

  12. Andy

    Which explains why my comment - with a couple of relevant links to the BBC website, on the BBC Jam story - got deleted a few days ago. I now presume Akismet deleted it. I do agree with Brett here, surely you still need to review your systems, as valuable community members could be confused if their messages disappear.

  13. Chris Paton

    You would save yourself $50/month by implementing a captcha.

    Wordpress really should have a captcha as standard but then Akismet wouldn’t exist.

  14. Joel Johnson

    There has been a sharp uptick in WP spam comments on Dethroner lately. While our traffic has been increasing, it’s been disproportionate to growth. We’re getting over a thousand a day on one tiny little site.

  15. Michael

    To everyone recommending a CAPTCHA: I highly disagree. CAPTCHAs have been shown to be highly ineffective, especially on a website to the scale of TC. The idea of asking questions (like the color of the ocean or how to add 2 and 4) is also a really bad idea. Even with 200 questions (which would take a while to create), it wouldn’t be long before all of the questions were harvested and the bots could spit out the answers.

    Plus, anyone heard about the porn webmasters who make users answers CAPTCHAs from other sites in order to enter? Then the computer knows the answer to the CAPTCHA and can get in! This would be even more slick for questions because they would only have to do it once and they could keep it in a DB.

    Akismet is REALLY good and TechCrunch is a great example of how well it works.

  16. David Mackey

    Pretty amazing. Akismet is great.

  17. BlogReader

    Chris Paton You would save yourself $50/month by implementing a captcha.

    Yeah, save yourself a latte a day and implement something that will only partially work and will be annoying to 80% of the people that post on here.

    Why are you offering a solution that risks a lot and gains only $50/month?

  18. kyle

    click here to buy my stuff

  19. pankaj

    I am interested to know if TechCrunch Forums are also hit by Spam? If not then why not continue commenting on forums, say after 1 day?

  20. wayne

    Ok, thanks for the info, since I’ve started using Spam Karma 2 my spam has cut down almost 95% cutting down the spam from showing up in my comments, does any of the other solutions work better?

  21. www.directhealthseach.com

    Mike, can you list the plugins you use for TechCrunch ?

    Also, I have always wondered what is correct, TechCrunch or Techcrunch.

    I ask because the title of your home page is Techcrunch, but everyone else, including titles of the sub pages, it’s TechCrunch.

  22. El Guapo

    This is spam. I am selling viagra. It’s only slightly used. Any takers? Oh, and buy my stock too. It can’t miss.

  23. Devon Young

    I’ve seen a huge increase in comment spam over the past 2 weeks myself. It used to be about 3-5 per day. Now it’s easily around 40 per day, and my blog hasn’t been around that long yet to get much attention. Weird thing about it is, 99% of them are advertising medication. My site’s all about tech stuff. I’m very curious what plugins you’re using.

  24. Jeff

    The trouble with not correcting Akismet on the false positives is that you’re not helping to build the strength of the system and may actually be hurting it by allowing it to think legitimate posts are spam.

    I’d recommend trying to find those false positives - the more the system learns from everyone the better the tool becomes.

  25. anonymous

    I’m curious, what percentage of spam comments contain html links in them? I’m guessing that the intent of the spammers is either to game pagerank, and/or to draw traffic to their sites. While adding the rel=”nofollow” tag would resolve the former problem, I image banning external links altogether would be an even more effective solution.

    Has anyone experimented with the efficacy of banning external links from comments?

  26. Alan

    Wow that’s a huge amount of spam. Akismet also saved our bacon over at Hacked Gadgets. After trying so many spam plugins I was thinking to have comments enabled meant dealing with hundreds of spam comments for every real one. That was moderation hell…

  27. Theo Tonca

    Devon: I’ve also noticed an exponential increase in spam comments on my blog in the past week or so as well. What gives?

  28. Survival Shows

    Akismet has saved all my blogs from getting overloaded w/spam it also saves us a lot of time :)

  29. Deepak

    Yep, the Wordpress folk need some kind of super award for how good Akismet is. It’s amazing how good it is. I’ve also noticed what kind of posts get spam. 90% of the spam is due to 10% of my posts.

  30. Al

    Well, by some measures you are the most popular blog on the web. 15k a day is still mind boggling.

    My little corner of the web is a fraction of that, but I recently installed the “Bad Behavior” plugin and it seems to have cut down on quite a bit of spam.

  31. Spud

    Puts the minimal amount of spam that I receive to shame. Well done guys for keeping the site spam free.

  32. Michael Mahemoff

    “We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. ”

    Mike, I appreciate the problems of spam are serious and frustrating, but it’s a pity to have to take this measure. If Akismet is really effective (it is, in my experience), why do it? Long tail says all those old posts still have their uses. I wonder how much free content (in the form of insightful and uplifting comments like this one) you’re going to miss out on with this policy. It could actually be measured easily by running a query on old blog posts.

    Plus, google likes old pages more when they receive new comments.

  33. E O Fall

    Yo , TECHCRUNCH has a new CEO , Heather Harde. Did i skip a post or is it just rumor?

  34. Ad Detective

    Shouldn’t this post be titled “Thanks to TechCrunch sponsors?”. It makes one wonder how much TC makes on such posts.

  35. Chris Lea

    Please note all that the $50/month is only if you purchase the Enterprise API key, which is what they ask you do to if you are using Wordpress for a “Corporate Blog or Blog Network”. Essentially, if you’re a company making money in part, or in whole, because of Wordpress, they ask you to pay for it.

    Additionally, if you are a pro-blogger, which by their definition means you are making $500 per month via your blog, they ask that you pony up for the $5/month pro-blogger key.

    These requests are completely reasonable. If you’re making money off Wordpress, they ask that you support the effort by putting a little back into Automattic by paying for Akismet.

    You can read their own words about the subject on the Akismet web site.

  36. Mik

    90% of everything is spam - be it myspace messages, emails, or posts to blogs.

  37. Tony Chen

    Michael, is there a way to check if my past comments got through with me having to check each post?

    Thanks,
    Tony

  38. Stephen

    Re Comment Timeout.

    Before y’all go installing that, you might want to do a thorough log analysis. In our case 90% of our traffic comes in through Google to the archives. We have some very active comment threads in old posts, and guess what? Those posts just keep rising and rising in the Google results (you can use Google’s webmaster tools to check this).

    Commenters just keep adding more and more keywords, many times things we’ve never heard of.

    We use the Spam Karma II WordPress plugin, and it works perfectly, with no need to turn off comments for old posts.

    Trackbacks are another thing: turn those suckers off!

  39. Rex

    Yeah, those are some crazy numbers! 15,000 SPAM! Nuts. Comment spam does seem to go in spurts, like the past few days, I have seen a definite increase also. Then there are some days, there are hardly any. I just deleted 50 about 20 minutes ago. That was from like only a few hours earlier, when I deleted about 40. Of course those numbers are nowhere near the thousands you are getting.

    Akismet - they definitely are one of the more worthwhile plugins you can have for your blog!

    Rex

  40. Brian Laks

    For people experiencing a surge in comment spam, check the referrer, your blog might be listed in a directory which the spammers use as a home base.

  41. awesomo

    Inteersting that my blog on blogspot never gets any spam and it uses CAPTCHA. I wonder if they also combine that with a spam detector similar to Akismet. The numbers for TechCrunch are very interesting though; 15,000 a day!

  42. Eric

    We use Akismet as well, though it is a lot easier for me to go through it each day. I suppose the level of spam you end up snaring in that filter is a sign of popularity. Perhaps someone needs to make a “Spam Rank” toolbar.

    Still, it is quite awesome - I’ve yet to actually see a legitamate comment in there that got filtered on accident. But it sure would be nice if there was a, “block this IP” feature in WordPress.

    P.S.: El Guapo, you’re hilarious.

  43. MacShout

    I had a blog that was getting between 500 and 700 spam comments a day, and I installed Dr. Dave’s Spam Karma II. It has caught every one, and not one good comment has been put in there. After about 3 weeks of not getting one spam comment through, they have pretty much stopped, I am now only getting a few per day.

  44. Steven

    The word “bespoke” does not mean what you think it does.http://www.asf-converter.net

  45. Dmitry Shechtman

    It’s been proposed to collect spam links using a filter (e.g. Akismet) and subsequently attack those sites (i.e. each blog that has been spammed would query an offending site upon each page load).

    You may read more about the phpBB modification based on this principle here:

    http://blog.phpbb.cc/2007/03/0.....link-spam/

  46. Owen

    *grin* .. you can almost measure your blog’s popularity by the amount if spam you receive per day.

    I think Askimet does a brilliant job. Wish I had something similar for my email!

  47. Petar Pavlovic

    Hm… Interesting.. When you click on “Mail (will not be published) (required)” label, you get focus on field on top of the page. :)

  48. Zoli Erdos

    Mike, I still wonder what the value of allowing *anonymous* comments is…

  49. Jeremy Wright

    Wow, I hadn’t looked at ours (from memory we were at about 2M when you hit 1M Mike)…

    But we’re at 5.5M spam comments blocked. Insane!

  50. Danny Sullivan

    Let’s see if Akismet still blacklists any comment from the searchengineland.com domain the the URL.

  51. Danny Sullivan

    Hurray — finally fixed :)

    Main main concern about Akismet is while it is catching so much spam, it can and does grab good comments as well — a long standing issue I’ve had when commenting in particular. Because site owners often seem to let it grab everything, rather than doing stuff like CAPTCHAs, I think that still leaves it vulnerable to the admittedly small amount of good stuff getting buried in the spam.

  52. Jesse Harris

    CAPTCHAs break usability. In the quest to end spam, we cannot and should not ever resort to challenges like CAPTCHAs and “reply if you’re not spam” e-mails. They frustrate users and discourage participation while being no more effective than a good collaborative reporting system like Akismet.

    When I run a mail server with Spam Assassin, I end up with a very low rate of false-positives and false-negatives without making users jump through hoops. Google’s spam filtering is even more effective without making users do the hokey pokey in reverse or something equally as inane.

    Spam is inconvenient and can take a lot of work to defeat, but CAPTCHAs are a solution worse than the problem.

  53. Matt

    If anyone is interested in the plugin Mike alluded to it’s available on our new plugin directory here:

    http://wordpress.org/extend/pl.....old-posts/

  54. Gareth Heyes

    Hi,

    You might want to try my plugin it requires no captchas and we haven’t had a single spam comment on our site. Try it out:-
    http://www.thespanner.co.uk/2007/02/12/spambam/

    Cheers

  55. Don Singleton

    I don’t get anywhere near as many as you do, but I certainly get a lot. We need to encourage search engines like Google to set up a way where we can forward them unwanted spam comments, and if they get a certain number of them they will blacklist the domain that the spammers are hoping to increase the page rank for.

  56. the1gofer

    I get 400-500 every day, and I don’t get a fraction of even that number of visitors. I wish their was something to be done. It seems like a societal problem more than anything else. Those people responsible aren’t concerned about anyone other than them selfs and the money they are making. And societal problems are much harder to fix than computer problems.

  57. Elias

    Oh my god, this is really many spam. On my site i get 50-100 spam comments or trackbacks. But, Akismet works fine. ;)

  58. datter

    Can I interest anyone here in some v.1.a.g.ra? Just checking… you never know.

  59. Wes

    I think the amount of spam just depends on how big the site is. I used to get at 50 a day one blog and then around 10 on another but 15,000 a day is really big.

  60. Advertisespace

    I couldn’t have comments on my blog without it.
    Thanks Akismet!

  61. How to start a clothing line from scratch

    we get a whopping 50 spam comments a day!

  62. Ulf (ULE)

    Am I the only one lamenting the two week timeout? I kind of like how some unique or especially illuminating old blog posts continue to live their own life in the comments and on Google. That’s part of the internet and blog magic for me, to see things take on their own life, and morph, and spawn new ideas and discussions that weren’t intended.

    The old blog post about “post secret” that You (TechCrunch) linked to a few weeks back - don’t you regret missing out on that kind of continous commenting over the years?

    Even if it doesn’t happen all too often, you must have at something like 10-100 blogposts that continue to get well deserved traffic through google long after their debut and among those, there must be some legitimate continuing of some kind of discussion?

  63. TechYob

    Bad Behaviour is another comment-spam filtering tool that can prevent a site from being accessed at the IP level.

    I understand the author worked on Akismet in the early days.

  64. Stephen

    As an able bodied user, I have no problem with Captcha. I think claims to the contrary are overreaching, perhaps exageration by advocates for the disabled.

  65. Ayal Rosenthal

    I guess my curiousity is how many anonymous comments does TechCrunch receive, and what does it consider anonymous?

  66. Rick

    I think my comments seem to get sucked up by Akismet I guess, I posted a comment about the BBC article from last week but it never showed up. :(

    Who knows if this comment will show up either!

  67. Ryan McDonnell

    Rename wp-comments-post.php to something else.

    Create a blank wp-comments-post.php file (just so that the spammers still get a 200 HTTP status code)

    Edit the “comments” page in template:
    Leave the “action” attribute pointing to the default wp-comments-post.php page. Add some Javascript to the FORM tag’s “onSubmit” method. Something like:

    onsubmit=”this.action=’location-of-new-wp-comments-post.php’; return true;”

    Anyone with Javascript enabled (nearly 100%) will be able to post comments but the spam bots (which do not parse Javascript) will not.

    You can even obfuscate the Javascript code further by throwing in some random concatenation…

    onsubmit=”this.action=’location’ + ‘-of-new-wp-comments-post.’ + ‘php’; return true;”

    … or setting the URL in a variable and referencing the variable instead.

  68. Baxter Tocher

    Apologies if I’ve missed it (I can only see two green “boxes” above from the author, and I can’t see it there), but what IS the recommended WordPress plug-in?