March 17, 2007

TechCrunch has 15,000 Spam Comments Per Day

Michael Arrington

76 comments »

On January 4 we reported that the Akismet filter had stopped a million spam comments from reaching TechCrunch. At that point we’d been using it for about nine months.

The number of blocked spam comments is now two million, just ten weeks later. That works out to about 15,000 spam comments hitting TechCrunch every day.

If we did not have Akismet, we couldn’t allow anonymous commenting here on TechCrunch. We used to go through all spam comments to pick out the occasional false positive and accept it. Now, there are just too many to go through. All comments marked by Akismet as spam get deleted almost immediately.

We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. This will significantly reduce the overall amount of spam that hits the blog, so I don’t expect to see this total spam number continue to increase exponentially.

Sphere It

This entry was posted on Saturday, March 17th, 2007 at 2:38 pm and is filed under Company & Product Profiles. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Trackbacks/Pings (Trackback URL)

Ryan McDonnell - » Stop comment spam on a Wordpress blog

March 19th, 2007 at 3:42 pm

BlogAjuda » Como resolver o problema dos spams?

March 21st, 2007 at 5:47 am

MMISoftware » Blog Archive » Spam, spam, spam..

April 14th, 2007 at 3:27 pm

TechCorner » Extending WordPress - Plugins Used

June 24th, 2007 at 4:32 pm

Akismet - I think I love you! | Bookmark Bliss

July 20th, 2007 at 3:01 pm

adult japan tv

September 5th, 2007 at 8:59 pm

mesh ferrari 360

October 19th, 2007 at 3:32 pm

Spam Comments | moverton.com

January 15th, 2008 at 8:41 pm

Comments

RSS feed for comments on this post.

Bill D'Alessandro

March 17th, 2007 at 2:49 pm

Wow, and I thought I had a lot of spam when I got 800 in one day.

Thank god for Akismet!
wayne

March 17th, 2007 at 2:50 pm

WOW, that is crazy, I hate spam, I had to put in Spam Karma 2 along with Math Comment Spam Protection to block spam from my blog, can you name the recommend wordpress plug-in?

Thanks

Wayne
Bill D'Alessandro

March 17th, 2007 at 2:59 pm

Wayne,

I think Michael may be referring to the Comment Timeout plugin.

- Bill
Morgan

March 17th, 2007 at 3:01 pm

Yeah it’s defintely a good idea with the timeout.

I had always wondered how there wasn’t a bunch of spam on here, now I know. That’s insane.
Michael Arrington

March 17th, 2007 at 3:08 pm

We used a plugin originally that didn’t scale properly, too many db calls and it took us down for a couple of hours. The new plugin is custom code that I believe Wordpress has developed internally.
Brett Tabke

March 17th, 2007 at 3:08 pm

This isn’t a spam issue as much as an administration issues.

Why not impliment a simple captcha? Or the newer:

“prove your a human - what is 3+4= ….”

Both systems virtually eliminate robotic spam based comments.
Michael Arrington

March 17th, 2007 at 3:10 pm

Brett…because Akismet takes care of it for us. I’m not asking for a solution, we have one.
_ryan

March 17th, 2007 at 3:15 pm

What’s the wordpress recommended plugin you’re using?

_r
Zaid

March 17th, 2007 at 3:23 pm

Captcha would be nice but big as TC is, it might have to address captcha usability problems for those with various disabilities.

I guess key point here is to know if your comment doesn’t appear, you should edit and re-submit instead of waiting for admins to approve it.

–Zaid
Leftblank

March 17th, 2007 at 3:25 pm

Interesting to see, that numbers make my 20+ daily spamcomments vanish. I wonder if TechCrunch is also using some sort of plugin such as Bad Behavior that will actually disallow malicious visitors (or actually robots) from even loading the pages of your website. This plugin has helped me to reduce the spam from 100’s to just 10’s a day, which are then caught by Akismet, a wonderful duo!
Jonathan Jiang

March 17th, 2007 at 3:54 pm

@ Brett

That won’t stay infallible for long

“prove your a human - what is 3+4= ….” computers can add too you know

Something more along the lines of random questions that any human can answer with ease like,

“What is the color of the ocean?” With answers like blue, green, maybe for some crazy people turquoise.

Of course, there _are_ some people who never have seen the ocean before… so.. that may be discriminatory.

Agh, stick with the [)(U2RS00WFG] “type this out please to prove that you are indeed, breathing”
Andy

March 17th, 2007 at 3:56 pm

Which explains why my comment - with a couple of relevant links to the BBC website, on the BBC Jam story - got deleted a few days ago. I now presume Akismet deleted it. I do agree with Brett here, surely you still need to review your systems, as valuable community members could be confused if their messages disappear.
Chris Paton

March 17th, 2007 at 3:57 pm

You would save yourself $50/month by implementing a captcha.

Wordpress really should have a captcha as standard but then Akismet wouldn’t exist.
Joel Johnson

March 17th, 2007 at 3:57 pm

There has been a sharp uptick in WP spam comments on Dethroner lately. While our traffic has been increasing, it’s been disproportionate to growth. We’re getting over a thousand a day on one tiny little site.
Michael

March 17th, 2007 at 4:12 pm

To everyone recommending a CAPTCHA: I highly disagree. CAPTCHAs have been shown to be highly ineffective, especially on a website to the scale of TC. The idea of asking questions (like the color of the ocean or how to add 2 and 4) is also a really bad idea. Even with 200 questions (which would take a while to create), it wouldn’t be long before all of the questions were harvested and the bots could spit out the answers.

Plus, anyone heard about the porn webmasters who make users answers CAPTCHAs from other sites in order to enter? Then the computer knows the answer to the CAPTCHA and can get in! This would be even more slick for questions because they would only have to do it once and they could keep it in a DB.

Akismet is REALLY good and TechCrunch is a great example of how well it works.
David Mackey

March 17th, 2007 at 4:21 pm

Pretty amazing. Akismet is great.
BlogReader

March 17th, 2007 at 4:24 pm

Chris Paton You would save yourself $50/month by implementing a captcha.

Yeah, save yourself a latte a day and implement something that will only partially work and will be annoying to 80% of the people that post on here.

Why are you offering a solution that risks a lot and gains only $50/month?
kyle

March 17th, 2007 at 4:46 pm

click here to buy my stuff
pankaj

March 17th, 2007 at 5:05 pm

I am interested to know if TechCrunch Forums are also hit by Spam? If not then why not continue commenting on forums, say after 1 day?
wayne

March 17th, 2007 at 5:07 pm

Ok, thanks for the info, since I’ve started using Spam Karma 2 my spam has cut down almost 95% cutting down the spam from showing up in my comments, does any of the other solutions work better?
www.directhealthseach.com

March 17th, 2007 at 5:10 pm

Mike, can you list the plugins you use for TechCrunch ?

Also, I have always wondered what is correct, TechCrunch or Techcrunch.

I ask because the title of your home page is Techcrunch, but everyone else, including titles of the sub pages, it’s TechCrunch.
El Guapo

March 17th, 2007 at 5:25 pm

This is spam. I am selling viagra. It’s only slightly used. Any takers? Oh, and buy my stock too. It can’t miss.
Devon Young

March 17th, 2007 at 5:40 pm

I’ve seen a huge increase in comment spam over the past 2 weeks myself. It used to be about 3-5 per day. Now it’s easily around 40 per day, and my blog hasn’t been around that long yet to get much attention. Weird thing about it is, 99% of them are advertising medication. My site’s all about tech stuff. I’m very curious what plugins you’re using.
Jeff

March 17th, 2007 at 5:51 pm

The trouble with not correcting Akismet on the false positives is that you’re not helping to build the strength of the system and may actually be hurting it by allowing it to think legitimate posts are spam.

I’d recommend trying to find those false positives - the more the system learns from everyone the better the tool becomes.
anonymous

March 17th, 2007 at 5:51 pm

I’m curious, what percentage of spam comments contain html links in them? I’m guessing that the intent of the spammers is either to game pagerank, and/or to draw traffic to their sites. While adding the rel=”nofollow” tag would resolve the former problem, I image banning external links altogether would be an even more effective solution.

Has anyone experimented with the efficacy of banning external links from comments?
Alan

March 17th, 2007 at 6:02 pm

Wow that’s a huge amount of spam. Akismet also saved our bacon over at Hacked Gadgets. After trying so many spam plugins I was thinking to have comments enabled meant dealing with hundreds of spam comments for every real one. That was moderation hell…
Theo Tonca

March 17th, 2007 at 6:06 pm

Devon: I’ve also noticed an exponential increase in spam comments on my blog in the past week or so as well. What gives?
Survival Shows

March 17th, 2007 at 6:06 pm

Akismet has saved all my blogs from getting overloaded w/spam it also saves us a lot of time
Deepak

March 17th, 2007 at 6:15 pm

Yep, the Wordpress folk need some kind of super award for how good Akismet is. It’s amazing how good it is. I’ve also noticed what kind of posts get spam. 90% of the spam is due to 10% of my posts.
Al

March 17th, 2007 at 6:16 pm

Well, by some measures you are the most popular blog on the web. 15k a day is still mind boggling.

My little corner of the web is a fraction of that, but I recently installed the “Bad Behavior” plugin and it seems to have cut down on quite a bit of spam.
Spud

March 17th, 2007 at 6:20 pm

Puts the minimal amount of spam that I receive to shame. Well done guys for keeping the site spam free.
Michael Mahemoff

March 17th, 2007 at 6:35 pm

“We’ve now implemented a Wordpress-recommended custom plugin that turns comments off on posts that are more than two weeks old. ”

Mike, I appreciate the problems of spam are serious and frustrating, but it’s a pity to have to take this measure. If Akismet is really effective (it is, in my experience), why do it? Long tail says all those old posts still have their uses. I wonder how much free content (in the form of insightful and uplifting comments like this one) you’re going to miss out on with this policy. It could actually be measured easily by running a query on old blog posts.

Plus, google likes old pages more when they receive new comments.
E O Fall

March 17th, 2007 at 7:01 pm

Yo , TECHCRUNCH has a new CEO , Heather Harde. Did i skip a post or is it just rumor?
Ad Detective

March 17th, 2007 at 7:01 pm

Shouldn’t this post be titled “Thanks to TechCrunch sponsors?”. It makes one wonder how much TC makes on such posts.
Chris Lea

March 17th, 2007 at 7:25 pm

Please note all that the $50/month is only if you purchase the Enterprise API key, which is what they ask you do to if you are using Wordpress for a “Corporate Blog or Blog Network”. Essentially, if you’re a company making money in part, or in whole, because of Wordpress, they ask you to pay for it.

Additionally, if you are a pro-blogger, which by their definition means you are making $500 per month via your blog, they ask that you pony up for the $5/month pro-blogger key.

These requests are completely reasonable. If you’re making money off Wordpress, they ask that you support the effort by putting a little back into Automattic by paying for Akismet.

You can read their own words about the subject on the Akismet web site.
Mik

March 17th, 2007 at 7:33 pm

90% of everything is spam - be it myspace messages, emails, or posts to blogs.
Tony Chen

March 17th, 2007 at 7:40 pm

Michael, is there a way to check if my past comments got through with me having to check each post?

Thanks,
Tony
Stephen

March 17th, 2007 at 7:42 pm

Re Comment Timeout.

Before y’all go installing that, you might want to do a thorough log analysis. In our case 90% of our traffic comes in through Google to the archives. We have some very active comment threads in old posts, and guess what? Those posts just keep rising and rising in the Google results (you can use Google’s webmaster tools to check this).

Commenters just keep adding more and more keywords, many times things we’ve never heard of.

We use the Spam Karma II WordPress plugin, and it works perfectly, with no need to turn off comments for old posts.

Trackbacks are another thing: turn those suckers off!
Rex

March 17th, 2007 at 9:13 pm

Yeah, those are some crazy numbers! 15,000 SPAM! Nuts. Comment spam does seem to go in spurts, like the past few days, I have seen a definite increase also. Then there are some days, there are hardly any. I just deleted 50 about 20 minutes ago. That was from like only a few hours earlier, when I deleted about 40. Of course those numbers are nowhere near the thousands you are getting.

Akismet - they definitely are one of the more worthwhile plugins you can have for your blog!

Rex
Brian Laks

March 17th, 2007 at 9:18 pm

For people experiencing a surge in comment spam, check the referrer, your blog might be listed in a directory which the spammers use as a home base.
awesomo

March 17th, 2007 at 9:48 pm

Inteersting that my blog on blogspot never gets any spam and it uses CAPTCHA. I wonder if they also combine that with a spam detector similar to Akismet. The numbers for TechCrunch are very interesting though; 15,000 a day!
Eric

March 17th, 2007 at 9:51 pm

We use Akismet as well, though it is a lot easier for me to go through it each day. I suppose the level of spam you end up snaring in that filter is a sign of popularity. Perhaps someone needs to make a “Spam Rank” toolbar.

Still, it is quite awesome - I’ve yet to actually see a legitamate comment in there that got filtered on accident. But it sure would be nice if there was a, “block this IP” feature in WordPress.

P.S.: El Guapo, you’re hilarious.
MacShout

March 17th, 2007 at 9:53 pm

I had a blog that was getting between 500 and 700 spam comments a day, and I installed Dr. Dave’s Spam Karma II. It has caught every one, and not one good comment has been put in there. After about 3 weeks of not getting one spam comment through, they have pretty much stopped, I am now only getting a few per day.
Steven

March 17th, 2007 at 10:23 pm

The word “bespoke” does not mean what you think it does.http://www.asf-converter.net
Dmitry Shechtman

March 18th, 2007 at 12:06 am

It’s been proposed to collect spam links using a filter (e.g. Akismet) and subsequently attack those sites (i.e. each blog that has been spammed would query an offending site upon each page load).

You may read more about the phpBB modification based on this principle here:

http://blog.phpbb.cc/2007/03/0.....link-spam/
Owen

March 18th, 2007 at 1:49 am

*grin* .. you can almost measure your blog’s popularity by the amount if spam you receive per day.

I think Askimet does a brilliant job. Wish I had something similar for my email!
Petar Pavlovic

March 18th, 2007 at 4:42 am

Hm… Interesting.. When you click on “Mail (will not be published) (required)” label, you get focus on field on top of the page.
Zoli Erdos

March 18th, 2007 at 6:27 am

Mike, I still wonder what the value of allowing *anonymous* comments is…
Jeremy Wright

March 18th, 2007 at 7:17 am

Wow, I hadn’t looked at ours (from memory we were at about 2M when you hit 1M Mike)…

But we’re at 5.5M spam comments blocked. Insane!
Danny Sullivan

March 18th, 2007 at 7:38 am

Let’s see if Akismet still blacklists any comment from the searchengineland.com domain the the URL.
Danny Sullivan

March 18th, 2007 at 7:40 am

Hurray — finally fixed

Main main concern about Akismet is while it is catching so much spam, it can and does grab good comments as well — a long standing issue I’ve had when commenting in particular. Because site owners often seem to let it grab everything, rather than doing stuff like CAPTCHAs, I think that still leaves it vulnerable to the admittedly small amount of good stuff getting buried in the spam.
Jesse Harris

March 18th, 2007 at 8:16 am

CAPTCHAs break usability. In the quest to end spam, we cannot and should not ever resort to challenges like CAPTCHAs and “reply if you’re not spam” e-mails. They frustrate users and discourage participation while being no more effective than a good collaborative reporting system like Akismet.

When I run a mail server with Spam Assassin, I end up with a very low rate of false-positives and false-negatives without making users jump through hoops. Google’s spam filtering is even more effective without making users do the hokey pokey in reverse or something equally as inane.

Spam is inconvenient and can take a lot of work to defeat, but CAPTCHAs are a solution worse than the problem.
Matt

March 18th, 2007 at 9:46 am

If anyone is interested in the plugin Mike alluded to it’s available on our new plugin directory here:

http://wordpress.org/extend/pl.....old-posts/
Gareth Heyes

March 18th, 2007 at 9:55 am

Hi,

You might want to try my plugin it requires no captchas and we haven’t had a single spam comment on our site. Try it out:-
http://www.thespanner.co.uk/2007/02/12/spambam/

Cheers
Don Singleton

March 18th, 2007 at 10:08 am

I don’t get anywhere near as many as you do, but I certainly get a lot. We need to encourage search engines like Google to set up a way where we can forward them unwanted spam comments, and if they get a certain number of them they will blacklist the domain that the spammers are hoping to increase the page rank for.
the1gofer

March 18th, 2007 at 10:46 am

I get 400-500 every day, and I don’t get a fraction of even that number of visitors. I wish their was something to be done. It seems like a societal problem more than anything else. Those people responsible aren’t concerned about anyone other than them selfs and the money they are making. And societal problems are much harder to fix than computer problems.
Elias

March 18th, 2007 at 11:22 am

Oh my god, this is really many spam. On my site i get 50-100 spam comments or trackbacks. But, Akismet works fine.
datter

March 18th, 2007 at 3:13 pm

Can I interest anyone here in some v.1.a.g.ra? Just checking… you never know.
Wes

March 18th, 2007 at 3:28 pm

I think the amount of spam just depends on how big the site is. I used to get at 50 a day one blog and then around 10 on another but 15,000 a day is really big.
Advertisespace

March 18th, 2007 at 3:58 pm

I couldn’t have comments on my blog without it.
Thanks Akismet!
How to start a clothing line from scratch

March 18th, 2007 at 4:22 pm

we get a whopping 50 spam comments a day!
Ulf (ULE)

March 18th, 2007 at 4:54 pm

Am I the only one lamenting the two week timeout? I kind of like how some unique or especially illuminating old blog posts continue to live their own life in the comments and on Google. That’s part of the internet and blog magic for me, to see things take on their own life, and morph, and spawn new ideas and discussions that weren’t intended.

The old blog post about “post secret” that You (TechCrunch) linked to a few weeks back - don’t you regret missing out on that kind of continous commenting over the years?

Even if it doesn’t happen all too often, you must have at something like 10-100 blogposts that continue to get well deserved traffic through google long after their debut and among those, there must be some legitimate continuing of some kind of discussion?
TechYob

March 18th, 2007 at 6:04 pm

Bad Behaviour is another comment-spam filtering tool that can prevent a site from being accessed at the IP level.

I understand the author worked on Akismet in the early days.
Stephen

March 18th, 2007 at 7:08 pm

As an able bodied user, I have no problem with Captcha. I think claims to the contrary are overreaching, perhaps exageration by advocates for the disabled.
Ayal Rosenthal

March 18th, 2007 at 9:20 pm

I guess my curiousity is how many anonymous comments does TechCrunch receive, and what does it consider anonymous?
Rick

March 19th, 2007 at 2:13 am

I think my comments seem to get sucked up by Akismet I guess, I posted a comment about the BBC article from last week but it never showed up.

Who knows if this comment will show up either!
Ryan McDonnell

March 19th, 2007 at 8:51 am

Rename wp-comments-post.php to something else.

Create a blank wp-comments-post.php file (just so that the spammers still get a 200 HTTP status code)

Edit the “comments” page in template:
Leave the “action” attribute pointing to the default wp-comments-post.php page. Add some Javascript to the FORM tag’s “onSubmit” method. Something like:

onsubmit=”this.action=’location-of-new-wp-comments-post.php’; return true;”

Anyone with Javascript enabled (nearly 100%) will be able to post comments but the spam bots (which do not parse Javascript) will not.

You can even obfuscate the Javascript code further by throwing in some random concatenation…

onsubmit=”this.action=’location’ + ‘-of-new-wp-comments-post.’ + ‘php’; return true;”

… or setting the URL in a variable and referencing the variable instead.
Baxter Tocher

March 19th, 2007 at 11:24 am

Apologies if I’ve missed it (I can only see two green “boxes” above from the author, and I can’t see it there), but what IS the recommended WordPress plug-in?