We will look back and laugh at the time SaaS ISVs got away with no or very weak SLAs. SaaS will improve if ISVs pay real cash to users for downtime….

We run our company on the 3s principal. Three main data centers each carrying 1/3 traffic. At least three load balanced web servers, three app., three sql. Any two systems can carry the full load if an emergency comes up.

Definitely RS failed with the hour turn around for hardware. If they are managing the FW and LBs they are responsible for a current configuration in another area. They have a ton of these in each data center and it literally takes minutes to reflash a FW or LB.

My company loves Basecamp and uses it frequently, but any sort of technical issue isn’t going to stop us from any critical day-to-day operations.

A single point of failure on a load balancer is just asking for it. This could have been avoided by paying a few hundred extra per month for stateful failover for the webmux, F5 or whatever gear you had in there.

Doesn’t matter what the hardware replacement time is, anything over 30 seconds of downtime will bother anyone.

Folks, 37signals has one million signups, not customers. It’s a small company with limited resources. That’s probably why they did the cost analysis and decided they didn’t want to pay for a second load balancer.

At the end of the day, ask yourself two questions:
1. How much do you pay for your 37signals service.
2. How much downtime have they had since you’ve started using them.
3. Would you pay double to halve the downtime.

I don’t feel too bad for the 37signals guys, though. They got so much free marketing and exposure through their hype. Since when should people care that the product was written in Ruby rather than in Java, .NET or PHP.

They *are* marketing geniuses. Our product has very few features *by design*. Brilliant.

1. If your web site is important or creates revenue, invest in redundancy up to the point any outage impact costs less than the architecture to prevent it. Even that may be to risky for consideration. (e.g Air Traffic Control) ((Would you want your crash rate to increase by even 1% because a budget did not allow for $000’s hardware purchase.

2. Consider using some of the ‘grid’ type services available. Even RackSpace has a spinoff group that offers this highly redundant type of hosting LAMP type applications.

3. Create a backup hosting environment with a different company that maintains a replication of your production environment. (Yes, it will be a pain to setup) However, the advantage is you can very quickly shift users to the alternate site while recovering the primary site.

I’m actually surprised to see 37signal this clueless about web services, given the hype and sometimes solid advice from the company.

Thats like saying “I had two flat tires, I better buy a new car”. Good thing you aren’t my financial adviser.

To my surprise, RackSpace was on top of this immediately. I received a 3 way phone call from the account director and from her Vice President, Jairo Romero (he was on his mobile – driving home in his car). They both assured me that the outage was not their fault (very diplomatically of course). They both re-assured me that RackSpace provided the best hosting service there was and absolutely guaranteed me that I would be happy with RackSpace if I hosted my site with them.

To me, this is pretty incredible. I would never expect an Australian company to go that extra mile – especially for a ‘potential’ customer half way across the world.

We’ve been seriously considering hosting our entire product with Amazon Web Services due to the low cost of establishment and the ability to scale dynamically as our customer base grows – we’ve been looking at SmugMug’s business model closely since we think it’s very similar to ours (different online service though).

But since the call this morning from RackSpace, I’m wondering if ‘customer service’ (which I doubt Amazon will provide in huge bucket loads) should be factored in our equation when looking for a hosting provider. With AWS I hadn’t considered it a huge issue since I assumed that “Hey, it’s Amazon – their big and can’t afford to have major downtimes or expansion issues – we’d be nuts not to take advantage of AWS” ….. maybe I’m wrong. Maybe having a team pf people who go to the trouble of calling you at 10am on a Saturday from half way round the world is important?

Pretty impressive.

It’s BS for 37signals to point the finger like this. If I was Rackspace I would drop them like a cheap suit.

If you want to run a business with SaaS at the core, you have to over-invest on the hosting and backups fronts. Not point the finger at someone else.

1 million customers and you have to scapegoat a vendor?

Lame, lame, lame.

i think 37signal has very good services

I’m a Lead Tech in the Intensive Team (Linux). And well, to be honest, hardware fails, and as such we stress to our clients, if it’s a core solution, have a HA setup. We have solutions that cater for almost every failure that we can (redundant power, network, storage, servers)! But again, that costs money, we regularly do redundant Firewall, and Load Balancer pairs, which are all Cisco kit, with some of the BEST cisco tech’s on the market working for us.

When we write a SLA (which all of our clients have) we regard them as promises, which we keep to the best of our ability, a promise we make, is a promise we keep. It’s unfortunate that the hardware took nearly 2 hours to replace and get back up and running, and for that, I can only but apologize, but as things go, even if we’d met the 1 Hour that was listed in the SLA, would that have been OK to 37 Signal’s clients?

It’s awesome to see so many customers coming to our rescue in the comments, and I’m sure if 37 signals went anywhere else, they wouldn’t get the attention and support they receive now. I would be more than glad to spend my personal time with 37 Signals to get their faith in Rackspace restored. (as would any other racker, no doubt). DHH, if you would like, you’re more than welcome to email me personally at p eefy net and i’d be more than happy to do my best to help you come to love rackspace.

P

“In our case, it took about two hours from reporting the problem to having it fixed. That’s not good enough.”

Maybe you guys should house your own racks. You could get Zed to configure the Mongrel install. He loves Rails.