Twitter Downtime On the Upswing

Trackbacks/Pings (Trackback URL)

1. wojai
   

   December 20th, 2007 at 9:18 am

   Time for alternative solutions..

2. Intercon
   

   December 20th, 2007 at 9:21 am

   Did they eventually complete the move to a new datacenter?

   Traceroute before and after the move showed same path (except for a while in between)…
   % traceroute http://www.twitter.com
   …
   15 ae-5-5.car1.Oakland1.Level3.net (4.69.134.37) 195.965 ms
   4.71.164.91 (4.71.164.91) 195.958 ms
   ae-5-5.car1.Oakland1.Level3.net (4.69.134.37) 195.982 ms
   16 4.71.164.91 (4.71.164.91) 196.011 ms
   8.7.217.43 (8.7.217.43) 196.319 ms
   4.71.164.91 (4.71.164.91) 195.945 ms
   …

3. Adam
   

   December 20th, 2007 at 9:24 am

   Did they take into account the maintenance windows that Twitter took this month for their data center move? I would figure *that is *most of Twitter’s downtime this month.

4. Who Cares
   

   December 20th, 2007 at 9:28 am

   Who cares. It’s a free non-essential service.
   “Oh no…twitter is down… now nobody can see that I am currently making a PB&J sandwich!!!!

   I don’t see why people complain… there is no SLA…the service is ASIS - WHENEVERITIS.

5. bta
   

   December 20th, 2007 at 9:41 am

   Kinda makes you wonder how hard can it be to run the world’s most simple application. Incompetent developers, admins, or just poor architecture?

6. Jason M. Lemkin
   

   December 20th, 2007 at 9:44 am

   It’s interesting this is one area where business web services have innovated ahead of consumer web services … out of necessity. Most consumer web sites probably would not like the level of disclosure above, or at say trust.salesforce.com.

7. Sebastian Macias
   

   December 20th, 2007 at 9:51 am

   Rails can’t scale. It’s that simple. twitter needs to be ported to django

8. Cameron Barrett
   

   December 20th, 2007 at 9:52 am

   You know, for what is essentially a free service you people are complaining a bit too loudly. If 97-98% uptime to too low a number for something you ARE NOT PAYING FOR, then perhaps you should turn off your computer, put away your cell phone and go for a walk.

9. BlogSign
   

   December 20th, 2007 at 9:52 am

   >Most consumer web sites probably would not like the level of disclosure above

   except twitter.. which could plan and announce for a half-day totally-down “maintenance period” without even being slightly apologetic..

10. Amit Raman
    

    December 20th, 2007 at 9:57 am

    They need to run on Sun. RoR can’t scale. 6 days is inexcusable.

11. Intercon
    

    December 20th, 2007 at 10:00 am

    > you people are complaining a bit too loudly.
    > If 97-98% uptime to too low a number for something you ARE NOT PAYING FOR

    1) I didn’t and don’t even use twitter
    2) If you know how much twitter “will be” (if not is) worth because of “we people” who ARE NOT PAYING them — in fact, you are paying your cost — only mentally and virtually — more than a little few bucks they could possibly charge
    3) The problem is they set a very bad precedence for the general reputation of Web 2.0 service quality

12. Rich Miller
    

    December 20th, 2007 at 10:02 am

    There’s more Twitter downtime ahead, as they’re not completely done with the data center move yet:

    http://www.datacenterknowledge.....ikely.html

13. Intercon
    

    December 20th, 2007 at 10:08 am

    >They need to run on Sun.

    They are already on Sun — some 8 Sun X4100s, according to –
    http://highscalability.com/sca.....ent-faster

14. Tony Stubblebine
    

    December 20th, 2007 at 10:14 am

    bta, it’s only the “world’s most simple application” on the UI side. It’s actually got a huge volume of messaging going through a lot of channels (web, IM, SMS, API). That makes it a more complicated application than you experience as a user.

    As a Rails developer, I have to take the Rails doesn’t scale bait. It’s the wrong way to describe the Twitter situation. If you think Twitter is an application that runs at scale then Rails scales by definition because they’re still running on Rails. MySpace switched away from ColdFusion at one point, so you could use them as an example that CF doesn’t scale (although even then I think there are counter examples).

    You could make the case that Rails is hard to scale using Twitter’s downtime as an example, but I think that’s simplifying a lot of things:

    * No popular web platform scales without developer intervention.

    * Almost every popular web platform has an example of running at scale.

    Therefore, the most important factor in the Twitter downtime is the team. But we have almost no idea what tradeoffs they’re making. How have they prioritized uptime vs. rapid-development vs. keeping-a-small-team vs. cashflow?

15. Bas Bakker
    

    December 20th, 2007 at 10:17 am

    Nothing shocking, only March 2007!
    Ten hours Twitter off is not that scary, right?

16. Intercon
    

    December 20th, 2007 at 10:33 am

    >Therefore, the most important factor in the Twitter downtime is the team.

    Right, and the mentality.. if computing is becoming some electricity.. imagine electricity down for, how long? 1 sec? (video recording stops).. 2 sec? (pc reboots).. 3 sec? (you begin to yell)… and that’s at your home.. imagine.. in a building.. in a township.. whatever whatever.. meaning, do you think it’s easy to maintain the power grid’s uptime? yes it goes dark once in a (long) while, but whenever that happens, it’s almost always unacceptable (particularly due to poor management decisions).

    As we stated before, the twitter community is super-forgiving towards twitter the company.. even the company people become pretty arrogant sounding.. well, that might be a good thing for them.. love me, love my dog.

17. bta
    

    December 20th, 2007 at 10:48 am

    @Tony:

    >It’s actually got a huge volume of messaging going through a lot of channels
    > (web, IM, SMS, API). That makes it a more complicated application than you
    > experience as a user.

    Now, now, there are over 20 years of research into how to effectively deal with loads of volume and managing large amounts of data flow. And most companies do it very well (think of how much data your banking and other financial firms deal with — some of them even have to have end-to-end processing times [for all activities] in the milliseconds).

    Just because it’s a “web 2.0″ app, it doesn’t get away scot-free on good design and management.

    So, yes, just because it has “a lot of data flow” (and let’s be honest a high volume of 140-character messages isn’t exactly earth-shattering load) - doesn’t mean it isn’t simple. It does two things: take messages in, and send them out (with an absurdly simplistic UI to boot).

    Web-based email services like gmail and yahoo, have much higher volumes of data and do much more complex work with it - yet don’t quite have the downtime.

18. Intercon
    

    December 20th, 2007 at 10:49 am

    @Jason
    Solution for those who need better solutions.. which do not necessarily entail some problem.. or what’re the problems with this guy: http://timoreilly.jaiku.com (whose life I’ve no envy of..)

19. Intercon
    

    December 20th, 2007 at 10:55 am

    @Jason
    thanks for noting the amusement, obviously too cool to be correct -
    if computing is becoming some utility.. imagine electricity
    henceforth, no new water, just meta water!

20. Intercon
    

    December 20th, 2007 at 11:10 am

    And remember also that Twitter is not a nobody nomoney company -
    Twitter is co-founded by Evan Williams, who was the creator of Blogger, which was previously acquired by Google.

21. Lars Fischer
    

    December 20th, 2007 at 11:28 am

    Highrise by 37signals can scale and it’s fast (Rails). So it’s either poor design or a bad production setup.

22. Bob Warfield
    

    December 20th, 2007 at 11:50 am

    Personally, I wonder if Twitter would have fared better if it had been built on something like Amazon’s Web Services. The potential to scale up and down rapidly is there. They could even create a duplicate service pretty cheaply to try out new architectures.

    More on my blog:

    http://smoothspan.wordpress.co.....ons-cloud/

23. Jinc
    

    December 20th, 2007 at 3:23 pm

    their XMPP server can not scale and therefore they have an unstable service. As more and more twitters tweet, the more and more outages they will have. they need to reconsider their XMPP server.

24. Mickaël Rémond
    

    December 21st, 2007 at 4:46 am

    Hello Jinc,

    If you know a bit what you were talking about you would know that XMPP service is not the problem, being very marginal in Twitter architecture.
    And from what I know ejabberd has been rock solid for Twitter, unless you have other information

25. Maximilian Schulz
    

    December 25th, 2007 at 6:02 pm

    Well… I do not really follow the entire Twitter hype and therefore I have no knowledge about their potential cash flow. But I imagine the cash flows are not that good: who is responsible for the server setup. Hiring a team of professional can be pretty expensive for a startup with bad financing. I do not wanna support Twitter, but I know it from similar experiences, that having a good idea and finding the perfect technical solution are two very different things - especially if you cannot afford external support / knowledge.