On January 4 we reported that the Akismet filter had stopped a million spam comments from reaching TechCrunch. At that point we’d been using it for about nine months.
The number of blocked spam comments is now two million, just ten weeks later. That works out to about 15,000 spam comments hitting TechCrunch every day.
If we did not have Akismet, we couldn’t allow anonymous commenting here on TechCrunch. We used to go through all spam comments to pick out the occasional false positive and accept it. Now, there are just too many to go through. All comments marked by Akismet as spam get deleted almost immediately.
We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. This will significantly reduce the overall amount of spam that hits the blog, so I don’t expect to see this total spam number continue to increase exponentially.
Wow, and I thought I had a lot of spam when I got 800 in one day.
Thank god for Akismet!
WOW, that is crazy, I hate spam, I had to put in Spam Karma 2 along with Math Comment Spam Protection to block spam from my blog, can you name the recommend wordpress plug-in?
Thanks
Wayne
Wayne,
I think Michael may be referring to the Comment Timeout plugin.
- Bill
Yeah it’s defintely a good idea with the timeout.
I had always wondered how there wasn’t a bunch of spam on here, now I know. That’s insane.
We used a plugin originally that didn’t scale properly, too many db calls and it took us down for a couple of hours. The new plugin is custom code that I believe Wordpress has developed internally.
This isn’t a spam issue as much as an administration issues.
Why not impliment a simple captcha? Or the newer:
“prove your a human – what is 3+4= ….”
Both systems virtually eliminate robotic spam based comments.
Brett…because Akismet takes care of it for us. I’m not asking for a solution, we have one.
What’s the wordpress recommended plugin you’re using?
_r
Captcha would be nice but big as TC is, it might have to address captcha usability problems for those with various disabilities.
I guess key point here is to know if your comment doesn’t appear, you should edit and re-submit instead of waiting for admins to approve it.
–Zaid
Interesting to see, that numbers make my 20+ daily spamcomments vanish. I wonder if TechCrunch is also using some sort of plugin such as Bad Behavior that will actually disallow malicious visitors (or actually robots) from even loading the pages of your website. This plugin has helped me to reduce the spam from 100’s to just 10’s a day, which are then caught by Akismet, a wonderful duo!
@ Brett
That won’t stay infallible for long
“prove your a human – what is 3+4= ….” computers can add too you know
Something more along the lines of random questions that any human can answer with ease like,
“What is the color of the ocean?” With answers like blue, green, maybe for some crazy people turquoise.
Of course, there _are_ some people who never have seen the ocean before… so.. that may be discriminatory.
Agh, stick with the [)(U2RS00WFG] “type this out please to prove that you are indeed, breathing”
Which explains why my comment – with a couple of relevant links to the BBC website, on the BBC Jam story – got deleted a few days ago. I now presume Akismet deleted it. I do agree with Brett here, surely you still need to review your systems, as valuable community members could be confused if their messages disappear.
You would save yourself $50/month by implementing a captcha.
Wordpress really should have a captcha as standard but then Akismet wouldn’t exist.
There has been a sharp uptick in WP spam comments on Dethroner lately. While our traffic has been increasing, it’s been disproportionate to growth. We’re getting over a thousand a day on one tiny little site.
To everyone recommending a CAPTCHA: I highly disagree. CAPTCHAs have been shown to be highly ineffective, especially on a website to the scale of TC. The idea of asking questions (like the color of the ocean or how to add 2 and 4) is also a really bad idea. Even with 200 questions (which would take a while to create), it wouldn’t be long before all of the questions were harvested and the bots could spit out the answers.
Plus, anyone heard about the porn webmasters who make users answers CAPTCHAs from other sites in order to enter? Then the computer knows the answer to the CAPTCHA and can get in! This would be even more slick for questions because they would only have to do it once and they could keep it in a DB.
Akismet is REALLY good and TechCrunch is a great example of how well it works.
Pretty amazing. Akismet is great.
Chris Paton You would save yourself $50/month by implementing a captcha.
Yeah, save yourself a latte a day and implement something that will only partially work and will be annoying to 80% of the people that post on here.
Why are you offering a solution that risks a lot and gains only $50/month?
click here to buy my stuff
I am interested to know if TechCrunch Forums are also hit by Spam? If not then why not continue commenting on forums, say after 1 day?
Ok, thanks for the info, since I’ve started using Spam Karma 2 my spam has cut down almost 95% cutting down the spam from showing up in my comments, does any of the other solutions work better?
Mike, can you list the plugins you use for TechCrunch ?
Also, I have always wondered what is correct, TechCrunch or Techcrunch.
I ask because the title of your home page is Techcrunch, but everyone else, including titles of the sub pages, it’s TechCrunch.
This is spam. I am selling viagra. It’s only slightly used. Any takers? Oh, and buy my stock too. It can’t miss.
I’ve seen a huge increase in comment spam over the past 2 weeks myself. It used to be about 3-5 per day. Now it’s easily around 40 per day, and my blog hasn’t been around that long yet to get much attention. Weird thing about it is, 99% of them are advertising medication. My site’s all about tech stuff. I’m very curious what plugins you’re using.
The trouble with not correcting Akismet on the false positives is that you’re not helping to build the strength of the system and may actually be hurting it by allowing it to think legitimate posts are spam.
I’d recommend trying to find those false positives – the more the system learns from everyone the better the tool becomes.
I’m curious, what percentage of spam comments contain html links in them? I’m guessing that the intent of the spammers is either to game pagerank, and/or to draw traffic to their sites. While adding the rel=”nofollow” tag would resolve the former problem, I image banning external links altogether would be an even more effective solution.
Has anyone experimented with the efficacy of banning external links from comments?
Wow that’s a huge amount of spam. Akismet also saved our bacon over at Hacked Gadgets. After trying so many spam plugins I was thinking to have comments enabled meant dealing with hundreds of spam comments for every real one. That was moderation hell…
Devon: I’ve also noticed an exponential increase in spam comments on my blog in the past week or so as well. What gives?
Akismet has saved all my blogs from getting overloaded w/spam it also saves us a lot of time
Yep, the Wordpress folk need some kind of super award for how good Akismet is. It’s amazing how good it is. I’ve also noticed what kind of posts get spam. 90% of the spam is due to 10% of my posts.
Well, by some measures you are the most popular blog on the web. 15k a day is still mind boggling.
My little corner of the web is a fraction of that, but I recently installed the “Bad Behavior” plugin and it seems to have cut down on quite a bit of spam.
Puts the minimal amount of spam that I receive to shame. Well done guys for keeping the site spam free.
“We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. ”
Mike, I appreciate the problems of spam are serious and frustrating, but it’s a pity to have to take this measure. If Akismet is really effective (it is, in my experience), why do it? Long tail says all those old posts still have their uses. I wonder how much free content (in the form of insightful and uplifting comments like this one) you’re going to miss out on with this policy. It could actually be measured easily by running a query on old blog posts.
Plus, google likes old pages more when they receive new comments.
Yo , TECHCRUNCH has a new CEO , Heather Harde. Did i skip a post or is it just rumor?
Shouldn’t this post be titled “Thanks to TechCrunch sponsors?”. It makes one wonder how much TC makes on such posts.
Please note all that the $50/month is only if you purchase the Enterprise API key, which is what they ask you do to if you are using Wordpress for a “Corporate Blog or Blog Network”. Essentially, if you’re a company making money in part, or in whole, because of Wordpress, they ask you to pay for it.
Additionally, if you are a pro-blogger, which by their definition means you are making $500 per month via your blog, they ask that you pony up for the $5/month pro-blogger key.
These requests are completely reasonable. If you’re making money off Wordpress, they ask that you support the effort by putting a little back into Automattic by paying for Akismet.
You can read their own words about the subject on the Akismet web site.
90% of everything is spam – be it myspace messages, emails, or posts to blogs.
Michael, is there a way to check if my past comments got through with me having to check each post?
Thanks,
Tony
Re Comment Timeout.
Before y’all go installing that, you might want to do a thorough log analysis. In our case 90% of our traffic comes in through Google to the archives. We have some very active comment threads in old posts, and guess what? Those posts just keep rising and rising in the Google results (you can use Google’s webmaster tools to check this).
Commenters just keep adding more and more keywords, many times things we’ve never heard of.
We use the Spam Karma II WordPress plugin, and it works perfectly, with no need to turn off comments for old posts.
Trackbacks are another thing: turn those suckers off!
Yeah, those are some crazy numbers! 15,000 SPAM! Nuts. Comment spam does seem to go in spurts, like the past few days, I have seen a definite increase also. Then there are some days, there are hardly any. I just deleted 50 about 20 minutes ago. That was from like only a few hours earlier, when I deleted about 40. Of course those numbers are nowhere near the thousands you are getting.
Akismet – they definitely are one of the more worthwhile plugins you can have for your blog!
Rex
For people experiencing a surge in comment spam, check the referrer, your blog might be listed in a directory which the spammers use as a home base.
Inteersting that my blog on blogspot never gets any spam and it uses CAPTCHA. I wonder if they also combine that with a spam detector similar to Akismet. The numbers for TechCrunch are very interesting though; 15,000 a day!
We use Akismet as well, though it is a lot easier for me to go through it each day. I suppose the level of spam you end up snaring in that filter is a sign of popularity. Perhaps someone needs to make a “Spam Rank” toolbar.
Still, it is quite awesome – I’ve yet to actually see a legitamate comment in there that got filtered on accident. But it sure would be nice if there was a, “block this IP” feature in WordPress.
P.S.: El Guapo, you’re hilarious.
I had a blog that was getting between 500 and 700 spam comments a day, and I installed Dr. Dave’s Spam Karma II. It has caught every one, and not one good comment has been put in there. After about 3 weeks of not getting one spam comment through, they have pretty much stopped, I am now only getting a few per day.
The word “bespoke” does not mean what you think it does.http://www.asf-converter.net
It’s been proposed to collect spam links using a filter (e.g. Akismet) and subsequently attack those sites (i.e. each blog that has been spammed would query an offending site upon each page load).
You may read more about the phpBB modification based on this principle here:
http://blog.php...-backlink-spam/
*grin* .. you can almost measure your blog’s popularity by the amount if spam you receive per day.
I think Askimet does a brilliant job. Wish I had something similar for my email!
Hm… Interesting.. When you click on “Mail (will not be published) (required)” label, you get focus on field on top of the page.
Mike, I still wonder what the value of allowing *anonymous* comments is…
Wow, I hadn’t looked at ours (from memory we were at about 2M when you hit 1M Mike)…
But we’re at 5.5M spam comments blocked. Insane!
Let’s see if Akismet still blacklists any comment from the searchengineland.com domain the the URL.