Apparently a traffic accident is to blame for a major datacenter outage this evening. Rackspace’s Dallas datacenter lost electricity shortly after a traffic accident caused damage to a power transformer. Rackspace’s generators kicked in but, as we’ve seen before, lots of other things can then go wrong. In this case, two chillers within the data center failed to start back up, and a number of servers were taken offline to avoid damage from overheating.
Laughing Squid’s Scott Beale, who is affected by the outage, has been posting status updates on his blog. Since Rackspace doesn’t have any blog or status page for outages, Scott is our only direct source of information right now.
We’re tracking who’s offline. 37Signals is down. Who else?
Update: They’re back up. Total outage time was around 3 hours. Rackspace’s official response is here.
Yep my business got knocked offline. Based in NZ and rung up Rackspace and they told me “a truck has hit our datacentre”. DIdn’t turn out to be quite the case. My site has just come back online this moment though.
all our production servers at Rackspace are offline as are some email services we run through there. effects our blog as well.
Tom Fragala
Truston
yep, we’re back up now. We came back on-line about 20 minutes ago.
Total outage was probably about 2 hours.
It’s certainly not a commodity tonight, eh?
Downtime sucks, and what it shows is what we all forget every minute – it’s power that runs the Internet.
Our startup uses Rackspace pretty extensively for one reason: absolutely no downtime. At this moment we’re discussing plans to move from them (very expensive) to a local co-lo (much cheaper) where we can have more control over operations. I understand that this may not be their fault but no downtime is why we pay, literally, 3 times more than anyone else.
Mike it’s good you’re reporting this. It will certainly be interesting to see how Rackspace responds and deals with the situation.
2230 hours EST – 37Signals apparently still off-line; at least my Backpack services are not responding.
BTW I was worried about their data center infrastructure about 4 years ago when I started using Backpack. Love it. But … the reply they gave me then is safely tucked away, just for reference. You guessed it – on a BP page!
I have 3 clients that went down in this outage. Some longer than others, but very frustrating experience none the less. Yes, Rackspace does promise 100% uptime. The question is how will they respond? We are pretty close to recommending moving 25 servers away from them. They are not cheap, but if there’s no reason for paying a premium, why pay it!?!
Total downtime was about 2 hours. That’s why I pay double the going rate, for their service and their downtime history. I guess you can’t help a wayward truck. What will be interesting to see is how fanatical they are when they contact those affected by this.
We run a price comparison and we were down, but are back up. Matt, you’re right on the pricing– it’s still worth it to us for the great support (they did probably a 4 hour custom PHP/cURL compile that perhaps doubled our price lookup capabilities for no charge) but for folks that don’t use the support regularly, it might be hard to justify that premium. We treat it as an expert employee we don’t need to keep on full-time.
It’s still frustrating, when you read all the battery-backed-up and indefinite-diesel-generation capabilities you expect it will take a serious, prolonged outage before anyone is impacted. If Rackspace can’t do it properly, I don’t know who I would trust.
Man, I’d hate to work in support there right now.
My team is trying to finish a web app. project and lost access to a lot of files on BaseCamp. I thought that 37 Signals use Amazon S3?
Yup, we’re pretty upset about this extended downtime. It’s closing it on 2.5 hours now.
Rackspace has been generally wonderful about support and uptime, however. Their facilities are top notch and stable. But shit does happen and this time it hit a bunch of fans.
We’ll certainly be reevaluating our disaster preparedness in response to this situation. We apologize to everyone who was affected by this.
Our startup lost services both Sunday morning and Monday morning, but came back after some time. We recieved our first Rackspace email “Notification of Power Issues in DFW” on Sunday so wasn’t caused by that traffic accident. Despite this, I’ve been very impressed by Rackspace’s services, but looking to move to a “pay for what you use”/Rails focused service provider.
Downtime most definitely sucks. Whenever, I’ve seen a SLA that has 100% uptime I move on to the next vendor. 100% is IMPOSSIBLE to guarantee!!!
It’s simply a marketing ploy. Good luck!
The irony is that the Rackspace banner was flashing at the top of the TechCrunch page. I guess they don’t host there mission critical marketing servers on their own data centers……. I’m kidding as this is a third party marketing platform.
a car smashed into their transformer. this comes after two days of outages.
what is SHOCKING is that their generators cannot run the chillers AND servers. what is there was real power failure?
we are beyond pissed off at MOG.
the worst part is poor communication. my team thought we were hacked saturday night.
Arnold: We use S3 to host uploaded files, not our actual software, services or databases.
@ Jason
You guys should offer some backup service for your products especially for BaseCamp. Companies like mine are totally screwed without access to the project data!
Welcome to Rackspace. My name is Noelia and I am a Live assistant. How may I help you today?
Noelia: Welcome to Rackspace, how may I help you?
you: when will our servers be back online?
Noelia: An incident occurred involving a power transformer outside of our datacenter. The transformer feeding power to the DFW datacenter was damaged causing loss of power and cooling to the entire datacenter. As a result of the temperature increase, your server had to be shut down in order to prevent additional issues. We are currently working to rectify the situation and will provide updates via My.Rack as they become available.
Weird, I thought 37s was hosted in Chicago.
Arnold: You can export your data from Basecamp or Backpack in XML format whenever you’d like (click the Account tab, then export). Highrise allows you to export your contacts as often as you want (”Export” option in the sidebar) . Of course the services need to be available, but we do offer export from these products.
The high priority right now is disaster recovery. We’ll be looking into mirroring our facilities at another data center so we can “flip a switch” if one data center is rendered inoperable. Or at the very least we can maintain a reduced capacity but baseline level of service during a major outtage The “perfect storm” that hit Rackspace is incredibly rare, but we should be better prepared nonetheless.
We’ll be investigating options once we’re back up and settled in.
Given the ethos that 37Signals tend to extrude on their blog, they will probably just give out 6 months or maybe even 1 year credits to everyone affected. Gotta earn those customers back.
On second thought, I have yet found anything that matches Basecamp so maybe no one will dare complain…
Dont these guys have the slogan “bullet proof” hosting? Obviously not truck proof. I’ve never used them..insanely expensive, and now I definitely won’t
Threadless is also effected. And i can just copy and paste what Jason said:
“The high priority right now is disaster recovery. We’ll be looking into mirroring our facilities at another data center so we can “flip a switch” if one data center is rendered inoperable. Or at the very least we can maintain a reduced capacity but baseline level of service during a major outtage The “perfect storm” that hit Rackspace is incredibly rare, but we should be better prepared nonetheless.”
PR trick …. lol
Probably not, but either way it’s PR
I clicked on my first ever ad on TC today, TheItRoom.com, around 8:15 pm PST and their site was down. Our site is on rackspace and we are back up. That advertiser is wasting my precious attention!
I have three client servers at Rackspce, Dallas. None of them went down at all. Even though none of my servers went doen I received numerous emails from Rackspace over the last two days keeping me informed about the status – and then they called me by phone today to make sure I was in the loop.
I guess I am lucky I stayed up – but Rackspace also told me they have three generators, but needed only one to power the site. I think they have things under control.
Rob
I cannot see this a PR gimmick. Not good for business.
Just had to point out that Rackspace advertises itself as “The Zero-downtime Network”:
http://www.rack...kspace/network/
Of course, who could have forseen such a chain of events started by a truck (Ted Stevens perhaps?)
gome see hus
This should have never happened.
They keep blaming it on a truck hitting the transformer outside. What about the UPS system? Do they not have one in place? If they did, the cooling system and servers would have never shutdown and needed to be started back up.
Also, don’t they ever test these systems and have redundency for everything?
JamBase was down for about 2 hours tonight. Hope no one missed any shows.
I honestly love Rackspace for their fanatical support, and know they’ll probably make good on their promises. They were responsive on the phone and did a decent job keeping me updated via a support ticket I filled out. I imagine we’ll be seeing a big update from them about redundancy and fixes.
Mercury must still be in retrograde or something…
Michael, you forgot to add your disclosure you’re also in this market.
I use rackspace and had no down time, though things ran pretty damn slow for a while (and it had nothing to do with bandwidth usage).
http://www.zimbra.com also went down.
Mercury came out of retrograde Nov 1, so that can’t be it.
Our servers are hosted at Rackspace and while we received a support ticket notifying us that there was a critical issue, our website remained online.
37signals and our products are back. We’ve posted an update and summary of the downtime.
FreshBooks was down at the same time, so I’m guessing they were involved as well. I had a mini-stroke thinking that all my financials had just evaporated to the big DeadPool in the sky. Whew!
Man I was just about to talk to them about moving my customer hosting over to them. I wonder what the sales guy will say. I still feel that everyone goes down at some point. Rackspace still seems like a great place to host. I have been hosting site for 12 years, small co loc are ok, but not worth the headache.
Is it just me or TC’s page also loads painfully slow, especially comments and stuff on the right hand side…not sure why linkedin has been on and off a LOT lately.
Wow… this was such a useless post.. thanks for nothing
@37 Voices.com CEO
Dude, great url (voices.com). Must be nice knowing you have a 7 figure domain name….
Just got a personal call from the Rackspace CEO. He explained everything that happened and apologized for the mishap. Sounded like a terrible situation (truck accident, EMS, power shutdown cause the guy was still in his truck, etc). They did the best they could given a really unusual (im)perfect-storm situation.
Looks like Ron Paul got knocked out for a little while too. It’s a conspiracy!!!
I’m not a Rackspace apologist, but it’s pretty amazing this downtime is from a freak accident when other hosts I’ve been with have had downtime “because.” If it takes a truck accident to take out my clients’ sites for a few hours, then I’m sticking with Rackspace.
I’m still waiting for more details before I pass judgement. We don’t even know how many servers in the facility went offline. My servers were fine, and they are hosted at Rackspace’s DFW facility, so I know that the *whole* datacenter was *not* impacted. Seems to have only been a partial outage.
Surprising news, but not world ending. Look at every other major provider out there, compare their track records for stability and outages, and you’ll see that they still are worth the money. The first major outage in seven years? That’s still pretty phenomenal. Or rather, as they call it, Fanatical.
they don’t have a mirrored site? Ouch. With their very high prices I just assumed they did….
I want to say “sorry” to our customers who were affected by our downtime. We promised you no downtime and we failed you. We will make it right. We are determined to restore your faith in us.
To Amy Wilsch:
A mirrored site? Perhaps a magical mirror that harnesses the power of the universe to keep several thousand customer servers synced in real-time to exact replicas somewhere across the globe?
Get a clue, and stop spreading uneducated FUD.
Their response is going to be the true measure of the strength of their product. From the post above, it looks like they are taking this quite seriously.