Comments on: Amazon Down For An Hour And Counting (Updated)

By: Website/SaaS downtime: bad planning or bad luck « Nickpoint

Website/SaaS downtime: bad planning or bad luck « Nickpoint — Fri, 17 Jul 2009 14:12:01 +0000

[...] and slowdowns. These problems can result in a loss of revenues, customers and even reputation. Amazon looses $310,000 for every 10 minutes of downtime!! Can downtime be avoided? Yes, it can. The root cause of most slowdowns or [...]

By: Selling Books On Amazon » Blog Archive » What to do when seller central goes down

Thu, 13 Nov 2008 12:45:20 +0000

[...] Amazon Down For An Hour And Counting [...]

By: IT’s About Uptime - The StackSafe Blog » Blog Archive » A Tale of Two Outages

Fri, 25 Jul 2008 16:40:44 +0000

[...] 4.59% stock drop, lost sales, and damage to its reputation. What is interesting in this case is how noticeable an Amazon downtime incident is to all of us. A long time ago, downtime was a cost of doing business online. Now it is front page news and can [...]

By: Jessy Schoss

Jessy Schoss — Wed, 25 Jun 2008 22:25:28 +0000

While it is interesting to speculate why the outage happened and what’s the associated cost, let’s think a bit about how to test such a complex application.
Existing SW testing tools give you plenty of features to verify how the app behaves in a TEST environment and under synthetic conditions.
But what about the behavior in a REAL environment? How do you know what is the NORMAL app behavior in a production environment? How to determine it behaves abnormally and do something about it before users are impacted?
Lastly, how to learn from past failures and apply it the next time you roll a new rev? How to compare the before and after behavior?
Till these questions are answered, app development and especially rolling out new revs to production will include elements of guess and prayer.
The solution is to use the app behavior in a REAL environment as a baseline to make predictive decisions and to create before-and-after scenarios that tell the app developers what is really happening.

By: Five Nines is Still Not Enough - Refresh the Net

Five Nines is Still Not Enough - Refresh the Net — Fri, 20 Jun 2008 18:16:24 +0000

[...] Apps, Salesforce, Twitter and others struggle with outages of various sizes and causes. Amazon’s recent outage lasted for more than an hour and Twitter seemed to make news for when it was actually [...]

By: Matt

Matt — Mon, 09 Jun 2008 21:26:37 +0000

It’s happening again today, 6/9

By: Laurent’s blog » Around the web this week

Laurent’s blog » Around the web this week — Sun, 08 Jun 2008 19:02:07 +0000

[...] Amazon.com (not the Web services) was down for nearly two ours on Friday. They should learn from Twitter how to communicate on [...]

By: DavidB

DavidB — Sun, 08 Jun 2008 13:44:27 +0000

All your PS3 are belong to us! F5 F5 F5

By: Nate Klaiber

Nate Klaiber — Sat, 07 Jun 2008 03:31:00 +0000

Amazon must have switched their architecture to Rails, because, you know, that’s the only possible explanation. *sigh*

By: Ben

Ben — Sat, 07 Jun 2008 00:44:32 +0000

Sean:

Wouldn’t the page be cached? I’d have expected the product page to be on an in-memory remote cache (e.g. memcached –> Dynamo). The time to check for availability would be done periodically on the product page, but then verified when ordering. At least, that would be the most sensible approach. The problem must be more complicated…

By: Chris Sundberg

Chris Sundberg — Fri, 06 Jun 2008 22:01:38 +0000

Arrington, is there really a reason to bring up Twitter’s troubles while reporting on the Amazon outage? I think you’re secretly in love with Twitter. See here: http://www.chri...-loves-twitter/

By: Alistair Croll

Alistair Croll — Fri, 06 Jun 2008 21:37:12 +0000

Just posted a more detailed set of speculations here: http://gigaom.c...t-means-to-you/

I have to say that it’s unlikely a single game could take them offline. They have elasticity in their processing (it’s why they got into the EC2/S3 business in the first place) and that worked fine. Also international sites were OK. But more importantly, I’m willing to bet that peak days like Thanksgiving, Christmas, and Valentines have way more traffic across the board.

It’s always possible that something triggered a capacity shortage. But that would often show itself as a gradual slow-down, or periodic TCP resets, rather than a hard failure across one site (http) while others (https) kept working.

By: Amazon.com is down…

Amazon.com is down… — Fri, 06 Jun 2008 21:17:05 +0000

[...] III Email from Amazon PR (from Techcrunch): The Amazon retail site was down for approximately 2 hours earlier today (beginning around 10:25) [...]

By: Sebastien Page

Sebastien Page — Fri, 06 Jun 2008 20:57:38 +0000

Amazon didn’t lose $3,286,000… they just didn’t make $3,286,000.

By: Sean

Sean — Fri, 06 Jun 2008 20:45:49 +0000

You guys denying this is related to the PS3 bundle don’t understand how people who want these things operate. I guarantee you this was the cause of the downtime.

This bundle is in VERY high demand. It was posted about on EVERY major video game blog. MOST people who want a PS3 and don’t have one already have been waiting for this bundle to come out, myself included.

There were probably literally hundreds of thousands of people on amazon.com around 9:45am or so, starting to hit refresh like mad, waiting for the item to be buyable. This type of refreshing is unlike any other type of normal traffic. It does not apply to things like Harry Potter where there are millions of copies available right away – no one feels like they need to be there the second it comes out or they won’t get it. No, this ps3 bundle is in VERY short supply, so everyone wanted to be there the second it was able to be bought.

When the PS3 was first launched on costco.com, it brought down their entire web site too because of this same type of behavior. Millions of people sitting on one page hitting F5, F5, F5 over and over again. This is just like a DDoS attack basically. Seriously, read this article:

http://digg.com..._Before_8AM_PST

I guarantee the PS3 bundle was the cause of the downtime, without a doubt.

By: DMo

DMo — Fri, 06 Jun 2008 20:45:39 +0000

I am getting redirected to their robots page, which gives me a pretty crappy user experience. The majority of people who see this page are going to be pretty annoyed (I would think).

“We’re sorry!

You have been denied access to this feature because we believe you violated the terms, conditions, rules, guidelines or policies of our site in the past. If you believe we have taken this action in error, you may contact us at ad-help-us@amazon.com.

We apologize for the inconvenience.”

By: Davood

Davood — Fri, 06 Jun 2008 20:43:33 +0000

seriously..boohoo they lost 3+ million bucks…. i can not even manage to get a loan for 25grand so I can go to school next year.. they make enough money that a losing 3+ million over a few hour period because of being down should not be too big of a deal.

By: Martin

Martin — Fri, 06 Jun 2008 20:15:42 +0000

I don’t believe any of these figures of loses of $2-4 million, most people would just come back later to complete their transaction. Amazon has a lot of loyal customers who don’t like to order from other sites.

By: TechCrunch Japanese アーカイブ » アマゾンが1時間以上ダウン

TechCrunch Japanese アーカイブ » アマゾンが1時間以上ダウン — Fri, 06 Jun 2008 20:12:34 +0000

[...] [原文へ] [...]

By: FVB > Amazon, Twitter and Operant Conditioning

FVB > Amazon, Twitter and Operant Conditioning — Fri, 06 Jun 2008 20:09:06 +0000

FVB > Amazon, Twitter and Operant Conditioning…

According to the laws of operant conditioning, any behavior that is consistently rewarded, every single time, will extinguish at a faster rate while intermittently reinforcing behavior leads to more stable rates of behavior that are relatively more r…..

By: amazon outage | Lasts information

amazon outage | Lasts information — Fri, 06 Jun 2008 20:08:32 +0000

[...] Amazon Down For An Hour And CountingThe Amazon outage, first noticed by Twitter users at as early as 10:24 am PST this morning (I’d like to Twitter except that it’s down), has gone on for an hour now. It’s not necessarily news when a startup has minor downtime, but Amazon …TechCrunch – http://www.techcrunch.com [...]

By: brooks

brooks — Fri, 06 Jun 2008 20:03:25 +0000

I’ve got a picture of a dog and the following message when I visit amazon.com:

“You have been denied access to this feature because we believe you violated the terms, conditions, rules, guidelines or policies of our site in the past. If you believe we have taken this action in error, you may contact us at ad-help-us@amazon.com. ”

What?!

By: Kevin Z

Kevin Z — Fri, 06 Jun 2008 19:53:09 +0000

According to AdAge, Amazon just lost $1.8 million.

http://adage.co...ticle_id=127593

By: Nag

Nag — Fri, 06 Jun 2008 19:50:50 +0000

@Alistair Croll

Wow! Good information.. Thanks Alistair..

Cheers, Nag

By: Dan

Dan — Fri, 06 Jun 2008 19:44:01 +0000

I wouldn’t be surprised if they see a pretty big spike in sales once the site is back up. If I went to Amazon to buy something (and it wasn’t terribly urgent) and the site was down, I’d probably just come back and buy it later. That’s not really a lost sale. Obviously not everyone’s gonna do that, but I doubt they end up losing as much as everyone’s saying.