Google finally has an explanation for its giant fail whale earlier today. It was Asia’s fault. For some reason, Google re-routed some of its traffic to Asia and that caused a massive traffic jam. The company says about 14 percent of users were affected, and everything is back to normal now. From the Google Blog:
Imagine if you were trying to fly from New York to San Francisco, but your plane was routed through an airport in Asia. And a bunch of other planes were sent that way too, so your flight was backed up and your journey took much longer than expected. That’s basically what happened to some of our users today for about an hour, starting at 7:48 am Pacific time.
An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam. As a result, about 14% of our users experienced slow services or even interruptions. We’ve been working hard to make our services ultrafast and “always on,” so it’s especially embarrassing when a glitch like this one happens. We’re very sorry that it happened, and you can be sure that we’ll be working even harder to make sure that a similar problem won’t happen again. All planes are back on schedule now.
Yeah, imagine how pissed you’d be if you were on a plane to San Francisco and you landed in Hong Kong.
For “Some Reaons” Google “routed” the traffic
So its Google’s fault how come Asia’s fault? You guys will never leave praising google even when they are messing up!
They are blaming Asia, hence the title. It is called sarcasm.
I totally agree with Sagar.
Also, from the snippet that you posted, they are not blaming Asia they are admitting that the fault was in their system.
Title is seriously misleading.
Which part is actually “blaming Asia” ? Sounds like an attention-hoarding title to me.
Seriously. Please provide some facts showing Google “blames Asia” for this incident.
The traffic went through Asia, causing a traffic jam. It is clearly Asia’s fault (the whole continent).
It’s not Asia’ fault. Google should blame us users because we were searching on Google and created a traffic jam
It was Al Gore’s fault!
Nice try. Grab attention and then call it sarcasm. Next time my vista PC crashes, I’m going to blog saying “Techcrunch crashes PCs” and say I was being sarcastic.
Grow up, TC!
Get laid Chinmay.
@Simon Chan,
Shut up, idiot.
Agree with Chinmay – I hate this idea of saying whatever you like, regardless of facts, and then use the sarcasm excuse, like it can excuse anything. Bad journalism again.
I agree. Even sarcasm is made to look like Google didn’t screw up! Why is it that everyone thinks Google is invincible.. and when they do screw up, why are people still trying to make it look like someone else’s fault.
Google has some issues, but you don’t hear anyone talking about that.
And while I’m ranting, I might also add, GMail Spammers are BY FAR the biggest batch of spammers on the Internet. Why can Google not do something about that either? Is that Asia’s fault too?
GOD
DAMN
ASIAAAAAAA
You should get your sarcasm detector checked.
Yeah that’s still not much of an answer or transparency into the meat of the issue…
Sure it is, its called “we fucked up our BGP config”.. They’re just explaning that phenomenon to users in plain english – even some of the most technical of users don’t have a very good understanding of BGP and its a pretty good analogy.
BGP used to be one of my specialties. Thank god those days of drudgery are behind me.
Hah, someones fired. Who wants to estimate the dollar amount lost?
Ass. Do you really think it’s funny when someone loses their job? You’d be surprised how easy it is to make a mistake like this inside a large, complicated system. I think it’s unlikely Google would fire someone over a single incident like this.
Actually, being a developer I do, which was the meat the joke you didn’t get, but that’s fine.
I don’t care as long as it’s back. We lost it for the afternoon and you’d be amazed at how often I kept typing in searches that would never go anywhere…damn you Google built into FF!!!
the most disturbing part of the ‘flight re-routing’ to me personally was Google Voice. Hard to use that as a primary number when calls were both getting forwarded to incorrect numbers & some just getting dead air. On the Google Apps status page, there is no tracking info for GV and of course, like most Google services, no one to call or nowhere to turn for info. I’m going to have to re-think using this number.
There is a place to turn for info. It’s the Google Voice Help forum. Google employees monitor it and provide updates when there’s problems.
http://www.goog...m/p/voice?hl=en
Damnit, google has a we fart and they go and blame Asiatics! The nerve!
yup,I’m sure Google lost a couple of million dollars In advertising Income due to the Blip.
I suggest we all have an internetless practice drill once every six months (something like a fire drill). With Twitter down for maintenance yesterday and last Friday, a Google outage today and who knows what tomorrow, folks just won’t know what to do unless we are prepared.
Looks like Twitter is down – the whole site.
New post!
Some fire up the MG signal. Its like the bat signal, cept its just a big pair of glasses projected into the sky.
“Yeah, imagine how pissed you’d be if you were on a plane to San Francisco and you landed in Hong Kong.”
Less pissed, more confused. I mean, who routes that way?
Worlds largest internet asset screws up this big? I’ll have to check my AdWords account to see if I’m charged anything in the time it was down.
I was unable to even put up posts on http://www.worstpizza.com all morning due to this. I guess my google analytics also got held up. That sucks!
So much for being enterprise-ready. Hard to be held accountable for screwups when you slap the Beta moniker on everything…
I’ve actually heard that they have pin-pointed it to a small fishing village n Man Drang, South Korea.
Supposedly all the traffic ended up at 54-year old Nneng Ken, who used it to fill his 2TB storage with ‘Friends’ and ‘Frasier’ episodes before Google finally managed to shoot and injure enough 1 and 0’s to scare the rest into submission.
-’This is just what we fear when re-routing traffic through Asia. You just never know how the data will react to the foreign food’ a Google technician commented this afternoon.
Daniel
https://spideroak.com – Free Online Backup and File & Folder Sync
We use Google Apps for our mail. The error we were getting had to do with the server not supporting SSL. IF we were being routed thru Asia, we were being routed to the wrong system!
Who does one sent mail to when Goole prevents you from sending mail? It is good that when Google screws up the do a REALLY GOOD JOB of it and enough people scream that it does get fixed.
Actually it’s not Google’s fault at all. My source is an anonymous tech from Level 3. Level 3 goofed and routed all of that traffic to Asia, not Google. It’s Level 3’s fault.
Makes sense, my traceroutes were dying at level3.
Does this mean that Level 3 is now Level 2.99? I was frustrated this morning because couldn’t get any work done. Mad dash to catch up, and vi-oh-la, the poor dork who plugged in the wrong virtual network cable actually seems comical. They definitely shouldn’t fire the guy/gal, I bet this isn’t a mistake that he/she will repeat
.
Use Yauba.
“Yeah, imagine how pissed you’d be if you were on a plane to San Francisco and you landed in Hong Kong.”
Actually if the flight started in, say, Bangalore, it would be perfectly reasonable for it to land in Hong Kong. (There isn’t a non-stop flight between Bangalore and SF, according to ebookers.com).
Why do we trust they chose to reroute the traffic? Or even that an employee did it? Perhaps it was done by a 3rd party but they don’t want to admit it?
Also I don’t believe the 1-hour figure nor do I believe the 14% — everyone I know saw issues and I saw odd things with Google as early as last night.
Lastly, this impacted much more than Google.com given Google Analytics was frozen as well.
I dare say that this must put a black eye on Mr. Schmidt
This happens.
Basically, don’t choose to be a surgeon if you do not want to see blood. It is a fact of living on “The Internet Times”, and no one is infallible.
Kudos to Google for facing up and fixing it as quick as they could.
Now, for the critics: I do not dismiss this and I, on my post, try to avoid service interruptions like the plague; but again, servers will crash, routers will fail, and people will make mistakes. It is about staying cool, and reacting accordingly.
Why the hell is this Asia’s fault. May be you don’t read the google’s blog carefully and mix your own thinking in the original fact.
This post is completely ill-formed and illogical.
It’s always Asia’s fault!
lol.
Where in Asia?
Did millions of Americans just send their Gmail conversations through China’s internet surveillance systems?
Never buy Asian bandwidth. It’s completely toxic and been known to kill millions of sessions.
I’m just saying.
Pathetic attempt to get attention through a misleading title.
Grow up Erick.
this wasnt asias fault!!!
I’m in Asia and had no problem with my Gmail. I didn’t notice any traffic jams here.
how come some one like techcrunch, keep such a title to its blog? this is not acceptable.
Blaming complete continent for just a temporal company.
“It was Asia’s fault. For some reason, Google re-routed some of its traffic to Asia and that caused a massive traffic jam.”
Oh I see, so Google Asia rerouted traffic to Asia that caused the massive traffic jam? Or did Asia rerouted the traffic to Asia?
Nice informative article!
(-sarcasm-)
geez ppl calm down… why not just use a different search engine for a chane.. google is not the be all and end all of your enitre lives is it?
This shows how dependent on Google we all are!
Interesting article this is – hmmm.
I am surprised they are not blaming George Bush.
So if I make the mistake of routing all my traffic through TechCrunch’s servers, and you fail to pass it on, is it TechCrunch’s fault?
Good Point Alek!
it was way longer than an hour. it started at 7:30 a.m. EST and went until I went for lunch about 1pm. It was pretty much gone by 2pm
Yes, the entire Asian region teamed up and said “let’s bring down Google!”
Doesn’t affect me much. I use Yahoo! mail, and I’ve never ever faced a big outage. And well, for most searches, its search isn’t so bad either.
I think I did it. I was searching for stuff on google and It said i won a free meal at Apple Bee’s. I clicked it and all of sudden, the internet crashed. what should I do?