Last week, Michael Jackson’s death caused sites to fail left and right. Today, it’s a very different problem. The hosting service Rackspace has been completely down for the past 30 minutes or so. Don’t believe us, just listen to Justin Timberlake or Michelle Malkin, both of which have sites on the service and took to Twitter to complain.
Apparently, it’s an entire network outage [Update below, while it was a massive outage, it wasn't a full outage, apparently.] and so the usually very responsive Rackspace team cannot even respond to emails or tweet (though I’m sure we’ll be seeing some updates from smartphones shortly). Along with sites like Timberlake’s and Malkin’s, the popular event site, EventBrite, is apparently down as well.
Update: Here’s the status from Rackspace via Twitter: “We are having an issue that is affecting part of our DFW data center. No details yet. Will update as we get more information.”
Update 2: It looks like a lot of the sites on Rackspace are finally coming back up — including Rackspace’s own site. Downtime looks to have been about an hour. And here’s the all-clear from Rackspace itself: “All power is restored to the DFW data center – all devices affected are starting to come on-line. Details to follow.”
Update 3: Rackspace officially says that only one of its nine servers were affected, but it still hit a huge number of sites. And it notes, “We owe better, and will deliver.” Well played.
why do you use twitter profiles of celebrities as proof instead of actual documentation of websites being down? we get it, you like to sensationalize twitter because its soooooo revolutionary, but using twitter for proof of everything doesnt work
Find a hobby, seriously ^
For once I have to agree here. I know a lot of people don’t like the amount of Twitter coverage that TechCrunch has, but quoting Justin Timberlake’s Twitter feed for a story about Rackspace going down is absolutely ridiculous.
Why? I bet his website gets a ton of traffic. Would it be more relevant if Kevin Rose made that tweet?
Is that the MTV VJ from the time of Kennedy?
Because Rackspace is typically assumed to be ‘the best’ in terms of reliability, the quotes simply reflect how it effects the nature of their high paying client base, as it rarely happens.
They’re probably the only major provider where celebrities actually Tweet about their hosting company being offline.
It’s examples of reaction.
You’d prefer a screenshot of an error page?
“but using twitter for proof of everything doesnt work”
Yes, it does. I’ll go tweet this article on Twitter, creating an endless loop.
You’d do it using BASIC too.
I think I actually lose intelligence reading this blog.
You were in minus numbers when you started, wouldn’t worry about it
He’s not the only one. It’s _negative_ numbers.
they already know that, that’s why you get stories about updates that say “we’re having an issue. ok, no more issue” well no shit sherlock.
Thankfully Slicehost seems to be unaffected. Looks to be a regional thing.
If I am right, Slicehost servers are still not in Rackspace datacenters. Someone correct me if I am wrong.
Many of Slicehost servers are still in st. Louis I believe, although some are indeed with Rackspace I think.
Downtime was <15 minutes, TheRegister.co.uk disappeared offline, but most sites are already back online.
i’m still not seeing any of the ones i’m checking. like rackspace.com. nor are most people
https://twitter...rch?q=rackspace
been about 45 minutes now.
Definately been online for UK traffic for at least 10 minutes, looking at Twitter they’re also reporting it as coming back online. Obviously depends if individual subnets came back online and/or carriers for the sites we monitor vs what others are checking, so obviously I can’t be 100%!
You were much closer to 0%.
“definately”
Every time a car hits something in DFW, half the known world freaks. After 15 years I say buy a damned vowel.
Most sites online? Except Rackspace themselves? http://www.down...t.com/rackspace
No they aren’t. And downtime is 30+ minutes and growing.
Their whole network? Wow, they’re screwed, uptime is pretty much what big sites care about, they’re going to lose customers through this regardless of duration. Unlucky for them, nice company. This will also affect slicehost, I assume.
Mosso/Slicehost were not down, that I saw.
Rackspace rarely goes down, if ever.
but, it did go down. today.
Cisco Systems warned last year that web traffic was going to quadruple by 2013
http://powerlis...nkmgr/?s=514b1c.
Don’t know the nature of problems at Rackspace, but maybe we can expect to see more of these..
what’s with the affiliate link, dickcheese?
and now you know why people pay the big bucks for a good CDN like akamai…
distributed nodes, redundancy… they work..
Site down for over 30 minutes. Was on the phone with Rackspace support…no ETA given. Our phones ringing off the hook. Business down completely. Not very fanatical support from Rackspace.
you’ll be happy with their apology for not having fanatical customer service…later.
To their credit…we did get a call from their management to personally apologize for the outage. It was much appreciated, although the damage was done. Why was there a single point of failure?
Rackspace is definitely the best when it comes to the day-to-day issues of managing servers. They do need to work on their crisis management skills. Customers need to be notified when things go down. ( I heard about this on cnet, and when I called, it turned out the monitoring servers were down, too. )
However, the fact of the matter is that datacenters are going to go down. If people are so concerned about 100% uptime, get a hot copy in another location.
We work with several data centers throughout the US that have tested and proven back up power and generating equipment. Testing is required on a regular basis to ensure that when needed…..IT WORKS! If you are looking for alternatives, call me at 817-529-9675.
They’re back up (at least for our site). Down for 36 minutes. Still waiting on some communication from them as to what happened.
Got a call back from our rep. He said there are some theories floating around about it being a power issue with some key networking hardware but he doesn’t know for sure. He promised a call back as soon as he knows what happened.
“I heard it had something to do with electricity.”
I received the incident report this morning. Yes, there was a power outage. Initially everything switched to generator power seamlessly, but then one of their generator banks experienced an “excitation failure” which escalated to the point where the generators were no longer able to maintain the electrical load. They then had an unspecified problem with their Pad Mounted Switch, which prevented them from switching to their secondary utility feeder, so at that point all servers on UPS clusters A, B, & E went down once the batteries died.
Seriously!. I will stop reading tech just because of its marketing for twitter. Anything and everything you write has twitter in that. If you want to be marketter for twitter so it be. But do not overwhelm your readers with so many millions of post about twitter. May be you found twitter to be to exciting not everyone. Sigh!
Then maybe you should stop reading. Newsflash: TechCrunch, and other sites, do not belong to you. They do not write what you want them to write. They write what they want. If they want to write about Twitter, good for them.
yes, kenny should piss off and go read another tech site. but your whole argument about techcrunch writing what they want, not what you want, blah blah is pretty lame. really, can you imagine any other business using this model?
Wait a minute. One of their selling points is 100% uptime isnt it?
nobody guarantees 100% uptime. It’s 99.99%, which is actually not that hard to do.
99.99% only gives them 52 minutes grace a year.
rackspace actually does guarantee 100% uptime
http://www.rack...sting/index.php
100% Network Uptime Guarantee
everybody can enjoy their pro-rated nickel.
Actually, they do guarantee 100% uptime.
The important thing to note is they’re not saying their network will be up 100% of the time, just that that is what they aim for.
If they fail to hit 100% they’ll compensate you for it.
I believe Rackspace pay something like double the value of down time.
Still down here in San Francisco. Mail is out as is rackspace.com
“A day in my life without twitter”. How about you write a cover story like this?. Atleast one day, just one day why don’t you allow us to read something else on tech crunch than twitter.
soooo bugging that rackspace went down on the day of our new video platform launch!! minutes after we sent out our newsletter to thousands of people. thanks rackspace!!!
Whatever. This just gives you wusses an excuse to go to Starbucks and post on your wussy blog.
Power failure?
‘All power is restored to the DFW data center – all devices affected are starting to come on-line. Details to follow.’
They had scheduled and necessary maintenance of their UPS system in the DFW datacenter. Apparently something went wrong.
Yeah, once again.
Third power related issue in DFW in the past month.
Getting ridiculous.
It’s been raining cats and dogs here in Dallas for the past few weeks. Still not a reason for power outages.
If they hit the lines?
I’m just saying and who dropped all the pets from the sky anyway.
Swords.com issuing a 10% discount coupon for customers affected by the Rackspace failure. Limit 100 Customers. Expires 07/02. Use coupon code “RACKSPACEFAIL”
What do swords have to do with RackSpace? Stop spamming.
maybe they started as a porn host
We were down for over 30 minutes. We’ve been on Twitter about this since it happened. Our customers were simply dropped – phone ringing off the hook. Nothing we could do. Business was dead. So yah, we need to take care of our customers.
yes, but this is not the place to offer your coupons. maybe email your customers. or post the coupon on your own forums. and on your twitter account. but not on techcrunch where there is no legitimate correlation – seems like common sense but anything to make a sale, right? …right???
FYI – We’re a Rackspace customer. Are you?
No, I’m HUNGRY.
This is by no means Rackspace’s only major outage within the past year. The effect of their downtime has been well chronicled by Tumblr, Vimeo and a handful of other companies. Increasingly, fully-managed hosting is failing to live up to a variety of its promises, and proving to be more complex than beneficial.
And this is why “the cloud” fails and will never replace proprietary infrastructures.
Mike: except proprietary infrastructures are still dependent on outside networking pipes so you can get on the internets. And unless you’re Verizon or Google, nobody can afford this stuff.
Managed hosting has a lot of issues, but uptime isn’t the biggest for the huge majority of users.
Any chance of articles like this for when other major hosts go down? My site (like TechCrunch) is hosted on MediaTemple and when they fell over last, I was a tad disappointed not to be able to get more info via TechCrunch. Having an expert give some insight into these things is something I do find interesting.
Basically; more of the same please
I don’t think anyone has the time to blog everytime that MediaTemple goes down. That is a full time job, in itself isn’t it?
I’m still down. This has to be related to the service outage a few days ago. That one revolved around power and the generators for the data center.
David, My firm works with several data centers throughout the US that have tested and proven back up power and generating equipment. Testing is required on a regular basis to ensure that when needed…..IT WORKS! If you are looking for alternatives, call me at 817-529-9675.
We experienced an interruption in power to a portion of our Dallas-Fort Worth data center facility.
Power has been restored and our DC engineers are working on devices that need to be manually brought back online as quickly as possible.
Further updates will be made as soon as new information is made available. Please monitor our MyRackspace customer portal as this is the quickest way we can get updates out to you.
save us scoblus!
I wonder how they will play this in terms of credits as compared to Media Temple and their big outage?
http://www.data...-outage-credit/
A year of credit might be enough to slow any bleeding from the platforms but then again, when you are “Rackspace” you wonder where someone would go as an alternative for the same price/feature/etc. if not uptime (today notwithstanding).
Does this mean Twitter is the new SLA monitor?
No, just the meter. TC will monitor.
Our Furniture website (furniturexo.com) is still down! Rackspace rarely goes down, but this is a fourth power related issue in DFW in the past few weeks.
We have several servers with RS and a few were affected by this outage. Yes, it stinks and we had to do some damage control–but I wouldn’t consider switching.
IMO, they are still the best managed hosting company around–and I have yet to find comparable customer service+tech support.
Try DataPipe. Just as good if not better tech support.
DataPipe sucks hard. I wouldn’t even recommend them to my enemies.
We use rackspace and were impacted. Still wouldn’t switch away from them though, they’re always very responsive when we need them.
amazon aws is better option than all managed hosting service provders
I’m sick and tired of Twitter…. it’s time is going to come, like MySpace. And to make this relevent to the post… I wish Twitter was hosted by Rackspace so it could have been down too.
Twitter goes down all the time. If they were hosted by Rackspace I believe their uptime would go up and Siegler would write three times as much twitter drivel on tc.
Timberlakes Tweet about bringing rackspace back is funny. It shows he is technical literate and able to laugh at himself.
This has not affected any of my servers in the San Antonio datacenter. Looks like a DFW only event.
I find it laughable that companies (such as this one) totally rely on their own network to server their own support and marketing sites.
You need a spook site for just said emergencies and then use a DNS switching service to move your traffic so you can push out updates to your paying customers when your main service is down.
It is great you have faith and confidence to host all of this yourself, however, you need to have a contingency plan (especially for critical customer communication like your support site) for events such as these. Regardless of who fault it is or who cut whichever cable.
Maybe it had something to do with the car chase in on 635. That car got tboned pretty hard.
holy shit, big deal. get over it people, everything breaks now and then. maybe you should switch to another hoster…ooops, they have downtime/failures too. things break. go read a fucking book.
Free consultation for those affected by Rackspace outage or any other co-location or managed services provider – We partner with Tier 1 co-lo and data center providers who ensure outages do not happen or impact business
Tweet, Email or Call!
why use a tier-1 partner when i can do that myself?
If you can afford the power/cooling costs with hosting in-house, that’s great.
Our message is aimed at companies/websites who rely on up-time for their business survival.
Bad news is good news. No news is bad news. Applies to stars and any web site, even Rackspace itself. It’s free advertisement.
This is not the first time that Rackspace servers have been down. The first time they went down for a bit I moved sites to another host.
Down again? Couldn’t register for event at eventbrite.com, it says service is down.
It’s still down, some of my sites with SSL came up briefly and then went back down.
How can Rackspace keep so many sites down so long, seems like it is bigger problem than simple power failure!
“both of which”
- both of whom
So this is twice in one year. Ouch!
HOLY CRAP…
thats the 2nd time this yr… and 30 min??
so much for 100% SLA guarantee
Well Twitmatic is back up, though there still some strangeness with the DNS machines to deal with. Hope everybody else gets back up soon, too!
Well, I am sure Rackspace would be perfectly happy if the nearly 400k people following JT never knew their name.
Now they do, and they can associate Rackspace with failing their most pop-a-licious customer.
Thanks to TC for reporting honestly about one of your sponsors. Good to know there is still integrity in the blog-o-sphere.
I’ve hosted ontri with RS for 2 years and (knock on wood) it has been a superb experience. Very responsive and very knowledgeable across a wide array of technologies. I’ve even got help from them on PHP code.
Has it been perfect? No. There have been a couple blips in the 2 years – seems like often related to power – but I’m happy to have them hosting my site.
Rackspace went down again today. I cannot believe a company like this does not have a backup plan
crackspace
Twitter is down now! Are they hosting on Rackspace too?
Big FAIL DAY for tons of sites today….
Dang it Scoble! Did you spill your coffee?
Seriously, it sucks when this type of thing happens but it happens to even the best of them. Glad everything is back up for their customers.
what kind of this site?