« Previous post
Next post »

July 24, 2007

Key Sites Offline Today; Major SF Power Outage May Be To Blame

Michael Arrington

75 comments »

What Do Netflix, Technorati, Craigslist, Typepad, Vox and LiveJournal all have in common? They’re offline today, that’s what. A power outage in San Francisco has left at least 20,000 customers without electricity and might be partially to blame. These datacenters are supposed to have redundant power sources, so it may not be the entire story. If you have more info, please comment and we’ll update the post.

  • Sphere It

This entry was posted on Tuesday, July 24th, 2007 at 2:59 pm and is filed under Company & Product Profiles. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Trackbacks/Pings (Trackback URL)

  1. Anonymous

    July 24th, 2007 at 3:27 pm

  2. Zoli's Blog

    July 24th, 2007 at 4:19 pm

  3. Zoli’s Blog » Blog Archive » Web 2.0 Blackout

    July 24th, 2007 at 4:20 pm

  4. Ubertor Real Estate Blog » Blog Archive » Internet goes on vacation

    July 24th, 2007 at 4:48 pm

  5. Aside: SF Power Outage Clogs Intertubes - Evan Sims

    July 24th, 2007 at 5:03 pm

  6. Somewhat Frank

    July 24th, 2007 at 11:47 pm

  7. RobotSkirts » Blog Archive » Key Sites Offline Today; Major SF Power Outage May Be To Blame

    July 25th, 2007 at 1:15 pm

Comments

RSS feed for comments on this post.

  1. Miles Price

    July 24th, 2007 at 3:03 pm

    Wow — I just noticed this issue — how long will it take for these sites to get back up do you think? I don’t understand why they don’t have any other options for powering their datacenters, isn’t it a requisite to have serious power backups for when times like these happen?

  2. James

    July 24th, 2007 at 3:07 pm

    You can add Vox.com to that list. I just received this message from them:

    Dear Vox member,

    Vox is currently unavailable due to power issues in the San Francisco
    Bay Area that have impacted our co-location facility. This means that
    your Vox blog is not reachable at this time. This began at
    approximately 1:50 pm Pacific Daylight Time today, Tuesday July 24
    2007.

    We are working closely with our hosting provider to bring Vox back
    online as soon as possible. We sincerely apologize for the
    invonvenience this is causing, and we appreciate your patience. We
    will send another email update with more information as soon as
    possible.

    Thank you,
    Team Vox

  3. OMG

    July 24th, 2007 at 3:09 pm

    The main reason is it’s near the end of the world.

  4. Justin

    July 24th, 2007 at 3:09 pm

    bah it sucks to be one of the 20,000…

  5. CloudTap

    July 24th, 2007 at 3:11 pm

    “The main reason is it’s near the end of the world.”

    lol — rather dramatic but oh well, we really are nearing the end of the world, it seems so often that our future seems so screwed over. Oy

  6. Frank

    July 24th, 2007 at 3:11 pm

    Digg has been down too.

  7. John Tompson

    July 24th, 2007 at 3:12 pm

    Our IT folks at VOX say that it will take at least six hours to be fully restored.

  8. bob cobb

    July 24th, 2007 at 3:16 pm

    Seems to be a common occurrence for so many websites every summer because they’re hosted in California.

  9. Sean

    July 24th, 2007 at 3:21 pm

    Turn off your damn A/C’s, people :P

  10. Zoli Erdos

    July 24th, 2007 at 3:23 pm

    Digg is *up* from here…

  11. Mac

    July 24th, 2007 at 3:23 pm

    Netflix was down last night for at least an hour also.

  12. Browse

    July 24th, 2007 at 3:23 pm

    Six Apart owns both Vox and Moveable Type.

    NetFlix is back online. It would be nice to know which facility is down. All those sites have their own IP addresses, so it’s hard to determine without some sleuthing.

  13. Browse

    July 24th, 2007 at 3:26 pm

    Or maybe it’s 365 Main’s datacenter. =D

  14. Zoli Erdos

    July 24th, 2007 at 3:26 pm

    Sean, it’s centrally turned off in SF now :-(

  15. Ben

    July 24th, 2007 at 3:28 pm

    Yelp appears to be down too.

  16. Search*Engines WEB

    July 24th, 2007 at 3:32 pm

    30 years ago this week - New York City went through two days of blackouts

    This was pre web - so the business capital of the world really had to endure a hardship

  17. Nerd Forum

    July 24th, 2007 at 3:33 pm

    This may be a dumb question, but don’t data centers (I assume these sites are hosted on data centers) brag about having backup power?

    -Chris
    http://www.nerdcouncil.com

  18. Zoli Erdos

    July 24th, 2007 at 3:33 pm

    Netflix and Technorati are back now. Quote from Technorati:

    We are working with our co-location facility managers to assess why it is back-up power generators failed to provide the necessary back-up power to prevent our site going down.

  19. Jeffrey

    July 24th, 2007 at 3:34 pm

    Alexa is offline too.

  20. jim

    July 24th, 2007 at 3:36 pm

    fucking 365 Main colo center

  21. Danny Burkes

    July 24th, 2007 at 3:37 pm

    I know for sure that 365 Main had power issues- perhaps others in the area as well.

  22. Mike Koss

    July 24th, 2007 at 3:42 pm

    Alex is “up and down” - I’m thinking it might be because of their new toolbar they just made available for Firefox.

    Is there a good site on the internet that will display the current “uptime” stats for the top 1000 internet sites? Would be great to see effects of “storms” like this one.

  23. mainstreetreporter

    July 24th, 2007 at 3:43 pm

    My typepad based blog (www.consumerpassion.com) is down, got this note:

    Dear TypePad member,

    The TypePad service is currently unavailable due to power issues at
    our co-location facility. This means that the TypePad application and
    your TypePad blog are not reachable at this time. This began at
    approximately 1:50 pm Pacific Daylight Time today, Tuesday July 24
    2007.

    We are working closely with our hosting partner to bring TypePad back
    online as soon as possible. We sincerely apologize for the
    inconvenience this is causing, and we appreciate your patience. We
    will send another email update with more information as soon as
    possible.

    Thank you,
    The TypePad team

    So nearly two hours down time so far. That’s an amazing chunk of time, very troublesome. I can see a short burst of down time, but this … does not bode well.

  24. nerd hater

    July 24th, 2007 at 3:47 pm

    die nerds die.

  25. Jon

    July 24th, 2007 at 3:49 pm

    The only fault the websites have in this is trusting 365Main and not having redundancy at another location.

    The real humor here comes from the press release 365Main released just this morning before everything went down: http://www.prnewswire.com/cgi-.....amp;EDATE=

    They brag about 100% uptime for RedEnvelope, which made RedEnvelope decide to kill redundancy and move everything to 365Main. RedEnvelope had a message on their site earlier saying that it was in the process of updating the site rather than the truth that the site was down.

  26. Patrick

    July 24th, 2007 at 3:52 pm

    If it was 365 Main, it may have been a case of hubris. Posted earlier today:
    REDENVELOPE REPORTS TWO YEARS OF CONTINUOUS UPTIME AT 365 MAIN’S SAN FRANCISCO DATA CENTER

  27. Beatle Beatles

    July 24th, 2007 at 3:55 pm

    ROFL:

    the VERY second Digg came online - we were able to submit this story

    What an irony - getting diggs for them being offline :-D

    http://digg.com/tech_news/BREA.....wer_Outage

  28. Ferd

    July 24th, 2007 at 4:02 pm

    Ok, ok I’ll turn the power back on……sheesh why doesn’t everyone just relax!

  29. Smith

    July 24th, 2007 at 4:02 pm

    http://valleywag.com/tech/brea.....282053.php

  30. Kendall Kunz

    July 24th, 2007 at 4:12 pm

    …wait, Ferd, don’t throw that switch yet my finger is in the……- ZAP!

  31. Ryan

    July 24th, 2007 at 4:16 pm

    I just spoke to a K11 news reporter and she told me it was a throng of Harry Potter fans dressed in capes and brandishing wonds…that were the cause of the power outage.

  32. Andy Schworer

    July 24th, 2007 at 4:22 pm

    http://valleywag.com/tech/brea.....282021.php

  33. Steve

    July 24th, 2007 at 4:22 pm

    Don’t let Netflix off too lightly. I live opposite 360 Main and the power cut that affected them only hit around 2pm. The Netflix outage started last night at about 11pm…they had about 12 hours of downtime. Much longer than everyone else and completely unrelated to the Main street hosting power problems.

  34. trevo

    July 24th, 2007 at 4:22 pm

    @26: Re RedEnvelope Press Release:

    “The site, which specializes in all-occasion gift-giving, has also closed its redundant data center in the Midwest and moved its excess capacity to 365 Main”

    I guess, not such a great idea after all. LOL.

  35. Tylor

    July 24th, 2007 at 4:31 pm

    Quote from MSNBC article on the Netflix incident:

    “The breakdown didn’t appear to be related to San Francisco power outages that were blamed for temporarily knocking out several popular Web sites, including Craigslist, Technorati, Typepad and Livejournal.

    Service to Netflix’s site was finally restored around 3 p.m. PDT after Netflix’s engineers had missed several earlier estimated times for fixing the trouble.

    Netflix had been in the process of updating its computers to reflect price reductions that took effect Tuesday.”

    http://www.msnbc.msn.com/id/19932882/

  36. neal24

    July 24th, 2007 at 4:32 pm

    count hi5.com as another victim.

  37. Jason

    July 24th, 2007 at 4:34 pm

    I’ve been at 200 Paul during Power failures. No downtime and no problems. People using 365 main should switch over to 200 Paul. Digital Realty Trust seems to do a much better job.

    Our website is at 200 Paul and no problems today at all.

  38. WTF

    July 24th, 2007 at 4:35 pm

    Two words (kinda): George W.

    He and his vast right-wing, SF-hating co-conspirators are obviously to blame.

  39. douglas mcintyre

    July 24th, 2007 at 4:54 pm

    We use Typepad and have been down for over 2 hours. Our site is back up, but we cannot access typepad.com itself.

    The people at Six Apart are really irresponsible. There should be a back-up power source for a business that hosts tens of thousands of sites.

  40. Ed

    July 24th, 2007 at 4:55 pm

    http://www.geeked.info/power-o.....jor-sites/

    PG&E is reporting 30,000 now and that it was due to a fault in the line that caused an explosion. It interrupted a Fruit of the Loom commercial, as well as taking down above mentioned sites and Second Life as well. Rumor has it that all 15 SecondLife participants called Linden Labs.

  41. Jason Kolb

    July 24th, 2007 at 4:58 pm

    It looks like the Qwest backbone in California may be down - http://www.systemstatus.com/backbonereport.html Bad day for backbones…

  42. Mike

    July 24th, 2007 at 5:02 pm

    Now it’s obvious why Google bought those solar panels. Things like this would never happen to them. :)

  43. Lowell Cremorne

    July 24th, 2007 at 5:06 pm

    Second Life is also down - no logins allowed…

  44. Raj

    July 24th, 2007 at 5:15 pm

    Anyone read the posting on ValleyWag that says some drunk idiot wrecked the hosting computers at 365? Any confirmations out there??

  45. Austin Storm

    July 24th, 2007 at 5:17 pm

    Oh no, it’s like that The Onion video. I’d better start printing off e-mail forwards to hand out on the street once the crash comes.

  46. mainstreetreporter

    July 24th, 2007 at 5:22 pm

    Here’s what you get when you go to http://www.typepad.com now:

    http://status.sixapart.com/

    blow by blow reports of how crummy long this is taking to fix.

  47. mainstreetreporter

    July 24th, 2007 at 5:39 pm

    Planets are in orbit again … all is well at Typepad, after a mere four hours in kaputsville.

  48. Steve Hubbard

    July 24th, 2007 at 5:47 pm

    I’ve heard from an inside source at 365 Main that it was the iPhone. Too many users trying to access the above mentioned sites have caused the data centre to crash. Apple, Cisco and Duke University are now examining logs…

  49. tapster

    July 24th, 2007 at 6:04 pm

    @steve.. haha. guilty sir. The second I switched to EDGE on my iPhone, the lights came back on ;)

  50. Luis

    July 24th, 2007 at 6:07 pm

    Steve you rock!

  51. Ben

    July 24th, 2007 at 6:29 pm

    Are they gonna rebrand their company to 364Main?

    Just to be safe.

  52. Augmentalist

    July 24th, 2007 at 6:34 pm

    I was at 365 today until 11:45, the one day I have to leave early
    all the action goes down. Working in my clients cage most days, I
    know most of the employees over there I can almost 100% say nobody was recklessly drinking/blending, i don’t even think there is a blender in the break room. The outage, according to what I heard, affected several locations including AT&T Park.

  53. SutroStyle

    July 24th, 2007 at 6:35 pm

    I am just back from 365main- the most outrageous thing was that people had to wait in line for 45 minutes to get to their servers to fix the problems 365main caused, because their security checking was not able to handle so many people trying to get into the datacenter at once.

    The line to get into 365main was half a block long. We first thought that the whole web2.0 blew up under the AJAX load.

  54. David Mackey

    July 24th, 2007 at 6:55 pm

    Wow. Horrible. The colocation facility is going to be in some major hot water.

  55. Andrew Pass

    July 24th, 2007 at 7:13 pm

    I wonder how many data centers would have to go down to truly disrupt the Internet. The video on the crash of the Internet might be appropriate at some point, but I don’t think one person will cause it.

  56. CrunchBack

    July 24th, 2007 at 7:38 pm

    @Raj Valleywag is full of sh*t, and what’s more I think they know it and put the lie out there anyway. Drunk employee at Valleywag, perhaps?

    Still - no excuse for 365main going down. Ain’t what the diesels are for?

  57. Steve Pudont

    July 24th, 2007 at 8:18 pm

    I noticed this earlier. These are the main sites but this outage will adversely effect their supporting sites also. I bet that like Koork.com didn’t get much traffic today either.

  58. Dudeche

    July 24th, 2007 at 8:56 pm

    Doesn’t surprise me at all…..Colo 4 in 365 Main is warmest colo space I have ever been in. Used to work for a company in that space. Wearing shorts was definitely an option. My guess is…when the power went out they simply didn’t have the capacity to appropriately handle the already overloaded colo 4.

  59. DT

    July 24th, 2007 at 9:20 pm

    Feedburner is down too and throwing 500 errors now

  60. Travis Stoliker

    July 24th, 2007 at 9:38 pm

    Any update on this? Was it related to the power outage? Thats a pretty serious claim i’m wondering where the meet to this story is?

  61. Ouriel Ohayon

    July 24th, 2007 at 9:39 pm

    Mike, looks like the internet has crashed just like the in the video you published two days ago :)

  62. Apogee

    July 25th, 2007 at 4:50 am

    It was Homer Simpson and his Santa Clause toy again :p

  63. Rich Miller

    July 25th, 2007 at 5:27 am

    Several generators failed to start when the facility lost grid power, according to the initial incident report from 365 Main. They’re still trying to sort out why.

  64. Jason

    July 25th, 2007 at 8:30 am

    Companies need to plan for data disasters. Just like this article says.
    http://www.smartbrief.com/news.....54C82E6757
    There has to be enough redundancy in your infrastructure to allow your business to continue functioning. I bet a lot of webmasters are backing up sites and making plans for when it’s their turn. J

  65. KwangErn Liew

    July 25th, 2007 at 12:21 pm

    Aren’t we vulnerable. haha :)

  66. Matt Keegan

    July 25th, 2007 at 3:01 pm

    Time for the high tech companies to leave Frisco! Seriously, I guess this is bound to happen, but I wonder what the state of the internet would be if some major disaster, such as an earthquake felled networks for weeks on end? This reminds me to back up my sites on a frequent basis.

  67. Michael T. Halligan

    July 25th, 2007 at 5:38 pm

    This isn’t really surprising.. Let’s just say I’m very happy to have finished moving my production infrastructure out of 365main and into a datacenter in Seattle where the power grid is stable, and the power rates are actually affordable.

    In April, 2005 365main had an outage that affected all customers for 50 minutes due to a failed EPO valve. 365 handled that outage spectacularly, claling all of their customers within 15 minutes of the outage.

    In February, 2006 365main experienced a partial outage for 3 seconds that only affected some customers, but caused problems in their Telco spine, affecting connectivity.

    In October, 2006 365main had a backup generator fail, but supposedly no customers were directly affected, but customers were not allowed to enter the building between 3:29 PM and 4:40 PM.

    This isn’t that bad of a track record. That being said, 365main is the most expensive datacenter in California.

  68. DataGuy35

    July 25th, 2007 at 7:19 pm

    Phoenix is another great alternative to California data centers. Power is affordable, there is little risk of natural disasters, and some guys, like the guys at i/o Data Centers, have the right infrastructure to cover power outages. There was just a great article about it in the Phoenix business journal: http://www.bizjournals.com/pho.....ory15.html

« Previous post
Next post »