I got the first alert as I was stepping towards the door to leave (it is always like that), and when I got back to my seat found that half the web seemed to be talking about it. The main Techcrunch site was still serving pages to most, due to our super-aggressive-mega-cache, but it seemed that the entire Dallas NOC was being rebooted.

From the status blog:

As of 12:35AM CST Rackspace Cloud engineers are seeing intermittent connectivity to our WC2 cluster in our Dallas – Fort Worth (DFW) and data center. We are working to resolve the issue as quickly as possible and will update the status post accordingly.

If you have any questions or concerns please contact our support via live chat or at 1-877-934-0407 international +1.210.581.040.

UPDATE: As of 1:15am CST, Rackspace Cloud engineers are still working to address the current connectivity issues. We are making significant progress and we will post another update here shortly.

UPDATE: As of 1:30am CST, service has been restored to the majority of our technology clusters in our WC2 cluster. Some sites may still be having performance issues, We are continuing to monitor and address the situation. Additional updates to follow.

From slicehost (who actually mention power outage):

DFW Interruption
November 3rd, 2009 @ 01:14 AM

UPDATE 1:16AM CDT: Power has been restored, however, we’re working to check all our systems and make sure everything comes back up correctly. Slices have not yet been restarted. We’ll try to keep you updated as much as possible.

We are currently experiencing a service interruption in our Dallas data center. Our engineers are currently working to restore connectivity. We will send an update as soon as information becomes available.

And from Scoble, on Twitter:

(the list he pointed to is actually a good one to follow if you are a Rackspace customer).

This will likely lead to many cursing the cloud, when in essence there is nothing about this problem that seems unique to being a ‘cloud problem’. What is more concerning is that the NOC seems to have run out of power (almost unimaginable) and then took so long to come back online.

So – how did you all spend the downtime? It seems most admins and devs from Rackspace hosted companies were just hanging out on Hacker News and IRC bitching about RS (first time I noticed that he shares initials with his employer).

As soon as we know what happen etc. or any more, we will be posting updates here

Update From Rackspace: from their site:

Rackspace has experienced a service interruption during tonight’s scheduled maintenance on UPS Cluster G. We were testing phase rotation on a Power Distribution Unit (PDU) when a short occurred and caused us to lose the PDUs behind this Cluster. The phase rotation allows us to verify synchronization of power between primary and secondary sources.

All power has been restored and devices are being brought back online. The PDUs were down for a total of about 5 minutes. We have aborted the maintenance for the remainder of the evening and will reschedule this for another date.

Service to Cloud sites has been restored and we are continuing to work with Cloud sites customers to bring them online.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

When Salesforce.com founder and CEO Marc Benioff launched his CRM platform in the cloud in 1999, he embarked on a “No Software” campaign to tout his “Software as a Service” agenda. Today, hosting service Rackspace is promoting a similar campaign with the launch of NoMoreServers.com, a site dedicated to the emergence of Computing-as-a-Service models (like hosting, cloud computing and SaaS) to power enterprise IT.

NoMoreServers.com is a rally cry of the computing-as-a-service era. The site seeks to empower businesses to acknowledge the decline of in-house computing and the rise of the All Cloud Enterprise (ACE). Covering hosting, cloud computing, SaaS, and the key vendors driving them (eg: Amazon, Google, Rackspace, Salesforce, etc), NoMoreServers.com will feature daily commentary explaining all things cloud computing. The site will include third-party content and news about hosting, cloud computing and will have a live community portal for visitors to engage on the topic of outsourcing computing.

Rackspace deliver hosting and cloud services to businesses and has more than 70,000 customers, including over 51,000 cloud computing customers. Hosting guru Andrew Schroepfer is taking lead of the new site. Schroepfer founded, led, and sold Tier1 Research, a research firm focused on the hosting and data center industry. Rackspace also recently partnered with Robert Scoble to launch Building 43, which aims to help businesses use modern technology and social sites to increase their exposure and the money they’re making by offering valuable advice.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

In addition, The Rackspace Cloud (formerly known as Mosso) has made available Cloud Files language bindings along with technical guidelines for Java, PHP, Python, C# and Ruby under the MIT license through GitHub. Rackspace aims to offer a reference implementation in Python soon and in a press release casually mentions it “is aware of Ruby, Perl, Java, and Twisted Python Cloud Servers bindings”, which are all in the process of being developed.

With the approach, the company hopes to compete better with cloud computing giant Amazon on its own turf – and also Microsoft with its upcoming Windows Azure service – by generally being more open to developers as far as their client-side tools go. In case you were not aware, Rackspace also recently released its Cloud API for Cloud Servers, which allows users to write code that detects a workload in the cloud and scales up the number of servers meeting it as needed, in public beta. The company is heavily trying to position itself as the best alternative to Amazon, which it acknowledges is bigger in size but lacking an open strategy towards the cloud and standards.

On a sidenote, we’re hearing the company is preparing the launch of a new iPhone application that will let customers manage their Rackspace cloud accounts from their iPhone devices. It should be arriving some time in the next few weeks.

Will Rackspace’s efforts in breaking open their cloud offering be enough of a differentiator to compete effectively in an increasingly saturated market? Time will tell, but judging by its stock performance investors are taking quite a liking into the hosting company and its growth strategy.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

For the second time in 8 days, a power outage interrupted service at one of its data centers. And again it was the Dallas center that was effected. This time however, Rackspace was able to get things up and running fairly quickly, and more importantly, communicated well through its blog and Twitter throughout the downtime.

Still, it raises the question, why do power outages keep taking down a service that so many rely on? They have backups in place, so what’s going on?

Last time, Rackspace blamed the failure on a series of events that began with a power failure, and eventually tripped up its backup system. It isn’t saying exactly what happened this time yet, but if it was the same issue, obviously that’s a problem.

The issue here is reliability. A lot companies run their services through Rackspace. If it goes down, even for just an hour, that’s lost business. Last time around, Rackspace coughed up as much as $3.5 million in credits to those who were affected. It will undoubtedly have to cough up money this time around as well.

The promise of reliability is presumably one of the reasons Google kept Gmail in beta for so long, finally removing it today after several years. Now, if something happens, there’s no leaning on that beta crutch anymore.

To be clear, Rackspace has a pretty good history when it comes to reliabilty. Before last week’s downtime, it was November 2007 that a major outage last occurred. And that’s why it’s so troubling that we’ve seen two outages in just about a week.

[photo: flickr/Gordon McDowell]

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

While Rackspace has backup systems in place, a series of events apparently caused those backups to fail, resulting in the servers going down. Here’s the key nugget:

The breaker on the primary utility feeder tripped, initiating a sequence of events that ultimately caused a power interruption in Phase I and Phase II of the data center. All systems initially came up on generator power without customer impact. The ‘A’ bank of generators, which support UPS clusters A and B in Phase I and UPS cluster E in Phase II, then experienced excitation failure which escalated to the point where the generators were no longer able to maintain the electrical load. Rackspace then attempted to switch to our secondary utility feeder, but was unable to do so due to an issue in the Pad Mounted Switch (PMS). At approximately 3:15pm CDT, power supply through UPS clusters A, B and E was lost when the batteries in those clusters discharged, and equipment receiving power through those clusters experienced an interruption in service.

The service says only one of its nine data centers were affected by this failure, but many high profile sites collapsed as a result, including EventBrite, Justin Timberlake’s site and Michelle Malkin’s popular political blog. As Rackspace noted yesterday that “We owe better, and will deliver.”

Below, find the full incident report.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Apparently, it’s an entire network outage [Update below, while it was a massive outage, it wasn't a full outage, apparently.] and so the usually very responsive Rackspace team cannot even respond to emails or tweet (though I’m sure we’ll be seeing some updates from smartphones shortly). Along with sites like Timberlake’s and Malkin’s, the popular event site, EventBrite, is apparently down as well.

Update: Here’s the status from Rackspace via Twitter: “We are having an issue that is affecting part of our DFW data center. No details yet. Will update as we get more information.”

Update 2: It looks like a lot of the sites on Rackspace are finally coming back up — including Rackspace’s own site. Downtime looks to have been about an hour. And here’s the all-clear from Rackspace itself: “All power is restored to the DFW data center – all devices affected are starting to come on-line. Details to follow.”

Update 3: Rackspace officially says that only one of its nine servers were affected, but it still hit a huge number of sites. And it notes, “We owe better, and will deliver.” Well played.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Fever is a hot new RSS reader that aims to cure “second inbox syndrome, unread item guilt, and unbold elbow.” In other words, the common plights of the modern RSS power user.

Besides offering a full-featured feed reader, the application attempts to create a personalized Techmeme by scanning a user’s feed list for popular (or hot) links. Fever then groups these links into stories and assigns each a “temperature.” This allows a user to quickly keep a pulse on what’s going on in his or her “slice of the web.”

The other refreshing feature of the app is its move away from email inbox-style unread counts. As a long-time Google Reader user, I always dreaded the experience of returning from an offline vacation only to find several thousand unread items in my reader. With Fever, the emphasis is on dividing subscriptions into two camps: must-reads (called Kindling) and everything else (Sparks). By moving the “hit-or-miss” feeds into the Sparks bin, Fever ensures that a user gets only the most relevant content.

I’ve been using the product for a little less than a week and it has yet to disappoint. I now feel like I’m always aware of the trending stories in my area of Internet interest. Furthermore, I’ve been able to subscribe to a number of high-volume feeds that I would have never added to my Google Reader. And since I added them as Sparks, they now help Fever’s algorithm better find the most interesting stories from my Kindling.

Fever is the newest product from designer/developer Shaun Inman: He is also the creator of Mint, a web site analytics suite (not to be confused with Mint, the financial site); Shortwave, a command line bookmarklet; and Horror Vacui, an 8-bit iPhone game.

Although Fever has fully replaced Google Reader as my everyday feed reader, there are two drawbacks to the app: its cost and its requirements. Fever costs $30 (there is no demo or trial available). It also requires self-hosting and self-installation. Ultimately, this will prevent widespread adoption.

A possible solution to increase mass appeal would be if a hosting company, e.g. Media Temple or Rackspace, were to offer a hosted version of Fever for a few dollars a month. Even better would be an ad-supported free version.

But in the end, Inman seems to be fine with a more targeted market:

The price for feed readers has bottomed out at free so anything more than that is going to turn certain people off. And I don’t mind the deterrent. Most products price to be inclusive, to make the most money possible. I designed Fever (like Mint) first and foremost for myself. Any money I make on top of the personal utility I get out of it is just icing on the cake.

I also support my customers personally. Anything I can do to keep that level of support manageable helps — especially with two commercial products.

Check out the demo video here.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Our roundtable on cloud computing is coming up next week. (Get tickets here via Eventbrite: $75 each based on availability). In addition to the previously announced speakers, I am happy to announce a few more very special participants: Amazon’s chief technology officer Werner Vogels, Ning CEO Gina Bianchini, Facebook VP of Engineering Mike Schroepfer, and Jon Engates, CTO of Rackspace.

An event on cloud computing wouldn’t be the same without Amazon, which is becoming the computing infrastructure of more and more startups through its various Amazon Web Services (winner of this year’s Crunchies Award in the enterprise category). And Rackspace, through its Mosso subsidiary, is diving headfirst into the business as well. But we also wanted to hear from consumer-facing companies such as Facebook and Ning that are creating their own cloud platforms. What happens when all of these various cloud platforms meet in sky? And how does the similarity in their back-ends result in similar applications for users, whether they be consumers or enterprise users? Do enterprise apps need to become more social, or is that simply a frivolous pursuit?

These questions and more will be put to everyone on the roundtable, which will be preceded by five cloud computing enterprise product demos which will be evaluated by a panel of judges TechCrunch50-style. We got so many good candidates we decided to expand the demo session and move the event up a half hour earlier.

There will also be a networking party afterward where more companies will be giving product demos. We couldn’t do this without our sponsors Microsoft and Ribbit. To find out about how to become a sponsor or get a demo table please email Jeanne Logozzo or Heather Harde.

Below is the complete list of Roundtable Participants

Roundtable Discussion
Werner Vogels, CTO, Amazon
Mike Schroepfer, VP of Engineering, Facebook
Gina Bianchini, CEO, Ning
John Engates, CTO, Rackspace
Marc Benioff, CEO, Salesforce.com
Vic Gundotra, VP Engineering, Google
Amitabh Srivastava, Corporate VP, Windows Azure
Lew Tucker, CTO, Cloud Computing, Sun Microsystems
Scott Dietzen, SVP Communications Products, Yahoo
Paul Buchheit, Co-founder, FriendFeed; creator of Gmail

Roundtable Moderators:
Erick Schonfeld, co-editor TechCrunch
Steve Gillmor, editor TechCrunchIT

Date: Friday, February 27, 2:30 – 6:30 pm

Location: Microsoft Mountain View Conference Center
1065 La Avenida St
Building 1
Galileo Auditorium
Mountain View, CA 94043

(more info here).

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Web hosting provider Rackspace has acquired JungleDisk, an online backup service, and Virtual Machine provider Slicehost in a deal designed to help bolster its offerings against top competitor Amazon Web Services. The announced acquisition price is $11.5 million in cash and stock, with the possibility of up to an additional $16.5 million depending on performance.

Jungle Disk is a file storage and backup service that up until now has relied on Amazon’s Simple Storage Service (S3). With the new announcement the company says that it will begin offering the service using Rackspace’s similar service Cloud Files, but will continue to support storage using Amazon with plans to support even more services in the future.

Slicehost offers developers “slices” in Xen-based virtual servers that are much cheaper and generally easier to use than a traditional dedicated server. The service is a direct competitor to Amazon’s Elastic Compute Cloud (EC2).

As part of today’s annoucement Rackspace also announced a partnership with CDN Limelight Networks to help distribute data as part of its Cloud Files service, and an agreement with Sonian to offer archiving for its Mailtrust Email service.

The additions help strengthen Rackspace’s cloud based services, collectively called Mosso, which the company launched in February.

This is all good news for developers. Amazon has been the dominant force in this space for some time, and competition will only decrease prices and (hopefully) lead to an arms race in features, stability, and performance.

Disclosure: Rackspace is a TechCrunch advertiser.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

And Rackspace is a solid company financially. But in this market, any IPO is a sign of hope (there were no VC-backed IPOs last quarter). Rackspace backed off from an IPO once before, in 2000. It’s been champing at the bit to go public for a long time. It looks like the shares are trading up. Let’s see where they end the day.

(Disclosure Rackspace is a TechCrunch advertiser).

Update: Rackspace ended its first day of trading at the same price where it opened: $10. Not a confidence builder for other IPO-aspirants.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

With the filing we also get a clearer picture of Rackspace’s business and financials. Its revenues grew 62 percent last year to $362 million, but it posted net profits of $17.8 million, which were down 10 percent from the year before. Cash flows from operations, though, remained healthy at $105 million last year, up from $61 million in 2006. (Click on the table below for a bigger image and more data):

The decline in profits was because the company spent a lot more staffing up and spending more on sales and marketing. About half of the $53 million increase in its cost of revenues last year was attributable to the fact that it nearly doubled the number of employees to 2,021 (of that, data center employees went from 576 to 994, and sales and marketing headcount went from 224 to 353). Servers, software licensing costs, bandwidth, power and rent made up most of the rest of the increase.

Another interesting tidbit: that truck accident that took down one of its data centers in Texas last November cost the company $3.4 million in credits to customers.

At the end of the year, it had 29,193 customers, compared to 12,677 the year before. But nearly all of that growth was due to its acquisition of Webmail.us (i.e., they are hosted e-mail customers, not hosted Website customers). Rackspace has 36,692 servers across seven data centers, 114,749 square feet of data center space, with a 61 percent utilization rate. The company makes $3,504 a year per square foot, a number that has been growing nicely, illustrating that Web hosting is a scale business with increasing returns the more servers that can be rented out.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

There are no filings yet with the SEC, but press releases touting its revenues have mysteriously been stripped from the company’s site. Here is one that’s been cached, announcing 2006 revenues of $224 million. Various reports put quarterly revenues for 2007 at $75 million, $84 million, and $96 million, respectively for each of the first three quarters, which suggests that full-year 2007 revenues were north of $350 million. Rackspace claims to be profitable, and has more than 15,000 customers. A major outage last November caused by a traffic accident near its Dallas data center was noticed across the Web.

Rumors of an IPO have been swirling recently. The company just hired a new chief financial officer on March 31. Last October, it acquired Webmail.us and it offers cloud computing services that compete with Amazon’s Web Services through its Mosso brand. In January, it shifted strategyto emphasize its utility computing business model.

It is time to pull the trigger.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Today, hosting provider Rackspace is offering a new cloud computing service through its subsidiary Mosso. (Disclosure: Rackspace is a TechCrunch advertiser). The service competes with Amazon’s Elastic Compute Cloud (EC2), although it doesn’t require any load balancing or other administration. It also competes with Joyent and Media Temple’s Grid Service. Pricing starts at $100 a month for:

—50 GB of storage
—500 GB of bandwidth for transferring data
—3 million HTTP requests.

From there additional capacity per month costs:

—$0.50/GB of storage
—$0.25/GB of bandwidth
—$0.03/1,000 HTTP requests

This is a bit more expensive than Amazon (which charges in a different way) but a lot cheaper than the $350 to $400 a month Rackspace charges to host a dedicated server for a Website.

Mosso bills itself as a Web app hosting service. Applications are hosted on redundant server clusters (although the data center is only in one location, so something could take the whole thing out—like, say, if a truck were to run into a nearby power transformer). Coders choose what technology stack they want their apps to run on and upload their code. Mosso supports both Windows and Linux, PHP, Ruby on Rails, .Net, Perl, Python, MySQL, and SQL Server. (Amazon, in contrast, does not support Windows). Mosso does not yet support Java applications, but it is working on that. The company actually has been testing the service for nearly two years and already runs 37,000 apps.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

37Signals is having a bad morning, according to their current home page image above. They’re pointing fingers at their service provider, which was (and we believe still is) Rackspace. Last November they suffered a three hour outage along with other Rackspace customers.

Update: It’s back up, total outage was about 2 hours. Per the comments, 37Signals doesn’t seem super duper happy with Rackspace these days.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0