Even after moving to a new data center, Twitter has been undergoing some growing pains. Royal Pingdom reports that the service has been down almost a total of six days since it began monitoring the service last February. Downtime so far in December has exceeded almost any preceding month, with nearly 11 hours of downtime, compared to 9 hours in November.
I guess everyone is trying to get onto that Tweeterboard.
Time for alternative solutions..
Did they eventually complete the move to a new datacenter?
Traceroute before and after the move showed same path (except for a while in between)…
% traceroute http://www.twitter.com
…
15 ae-5-5.car1.Oakland1.Level3.net (4.69.134.37) 195.965 ms
4.71.164.91 (4.71.164.91) 195.958 ms
ae-5-5.car1.Oakland1.Level3.net (4.69.134.37) 195.982 ms
16 4.71.164.91 (4.71.164.91) 196.011 ms
8.7.217.43 (8.7.217.43) 196.319 ms
4.71.164.91 (4.71.164.91) 195.945 ms
…
Did they take into account the maintenance windows that Twitter took this month for their data center move? I would figure *that is *most of Twitter’s downtime this month.
Who cares. It’s a free non-essential service.
“Oh no…twitter is down… now nobody can see that I am currently making a PB&J sandwich!!!!
I don’t see why people complain… there is no SLA…the service is ASIS - WHENEVERITIS.
Kinda makes you wonder how hard can it be to run the world’s most simple application. Incompetent developers, admins, or just poor architecture?
It’s interesting this is one area where business web services have innovated ahead of consumer web services … out of necessity. Most consumer web sites probably would not like the level of disclosure above, or at say trust.salesforce.com.
Rails can’t scale. It’s that simple. twitter needs to be ported to django
You know, for what is essentially a free service you people are complaining a bit too loudly. If 97-98% uptime to too low a number for something you ARE NOT PAYING FOR, then perhaps you should turn off your computer, put away your cell phone and go for a walk.
>Most consumer web sites probably would not like the level of disclosure above
except twitter.. which could plan and announce for a half-day totally-down “maintenance period” without even being slightly apologetic..
They need to run on Sun. RoR can’t scale. 6 days is inexcusable.
> you people are complaining a bit too loudly.
> If 97-98% uptime to too low a number for something you ARE NOT PAYING FOR
1) I didn’t and don’t even use twitter
2) If you know how much twitter “will be” (if not is) worth because of “we people” who ARE NOT PAYING them — in fact, you are paying your cost — only mentally and virtually — more than a little few bucks they could possibly charge
3) The problem is they set a very bad precedence for the general reputation of Web 2.0 service quality
There’s more Twitter downtime ahead, as they’re not completely done with the data center move yet:
http://www.datacenterknowledge.....ikely.html
>They need to run on Sun.
They are already on Sun — some 8 Sun X4100s, according to –
http://highscalability.com/sca.....ent-faster
bta, it’s only the “world’s most simple application” on the UI side. It’s actually got a huge volume of messaging going through a lot of channels (web, IM, SMS, API). That makes it a more complicated application than you experience as a user.
As a Rails developer, I have to take the Rails doesn’t scale bait. It’s the wrong way to describe the Twitter situation. If you think Twitter is an application that runs at scale then Rails scales by definition because they’re still running on Rails. MySpace switched away from ColdFusion at one point, so you could use them as an example that CF doesn’t scale (although even then I think there are counter examples).
You could make the case that Rails is hard to scale using Twitter’s downtime as an example, but I think that’s simplifying a lot of things:
* No popular web platform scales without developer intervention.
* Almost every popular web platform has an example of running at scale.
Therefore, the most important factor in the Twitter downtime is the team. But we have almost no idea what tradeoffs they’re making. How have they prioritized uptime vs. rapid-development vs. keeping-a-small-team vs. cashflow?
Nothing shocking, only March 2007!
Ten hours Twitter off is not that scary, right?
>Therefore, the most important factor in the Twitter downtime is the team.
Right, and the mentality.. if computing is becoming some electricity.. imagine electricity down for, how long? 1 sec? (video recording stops).. 2 sec? (pc reboots).. 3 sec? (you begin to yell)… and that’s at your home.. imagine.. in a building.. in a township.. whatever whatever.. meaning, do you think it’s easy to maintain the power grid’s uptime? yes it goes dark once in a (long) while, but whenever that happens, it’s almost always unacceptable (particularly due to poor management decisions).
As we stated before, the twitter community is super-forgiving towards twitter the company.. even the company people become pretty arrogant sounding.. well, that might be a good thing for them.. love me, love my dog.
@Tony:
>It’s actually got a huge volume of messaging going through a lot of channels
> (web, IM, SMS, API). That makes it a more complicated application than you
> experience as a user.
Now, now, there are over 20 years of research into how to effectively deal with loads of volume and managing large amounts of data flow. And most companies do it very well (think of how much data your banking and other financial firms deal with — some of them even have to have end-to-end processing times [for all activities] in the milliseconds).
Just because it’s a “web 2.0″ app, it doesn’t get away scot-free on good design and management.
So, yes, just because it has “a lot of data flow” (and let’s be honest a high volume of 140-character messages isn’t exactly earth-shattering load) - doesn’t mean it isn’t simple. It does two things: take messages in, and send them out (with an absurdly simplistic UI to boot).
Web-based email services like gmail and yahoo, have much higher volumes of data and do much more complex work with it - yet don’t quite have the downtime.
@Jason
Solution for those who need better solutions.. which do not necessarily entail some problem.. or what’re the problems with this guy: http://timoreilly.jaiku.com (whose life I’ve no envy of..)
@Jason
thanks for noting the amusement, obviously too cool to be correct -
if computing is becoming some utility.. imagine electricity
henceforth, no new water, just meta water!
And remember also that Twitter is not a nobody nomoney company -
Twitter is co-founded by Evan Williams, who was the creator of Blogger, which was previously acquired by Google.
Highrise by 37signals can scale and it’s fast (Rails). So it’s either poor design or a bad production setup.
Personally, I wonder if Twitter would have fared better if it had been built on something like Amazon’s Web Services. The potential to scale up and down rapidly is there. They could even create a duplicate service pretty cheaply to try out new architectures.
More on my blog:
http://smoothspan.wordpress.co.....ons-cloud/
their XMPP server can not scale and therefore they have an unstable service. As more and more twitters tweet, the more and more outages they will have. they need to reconsider their XMPP server.
Hello Jinc,
If you know a bit what you were talking about you would know that XMPP service is not the problem, being very marginal in Twitter architecture.
And from what I know ejabberd has been rock solid for Twitter, unless you have other information
Well… I do not really follow the entire Twitter hype and therefore I have no knowledge about their potential cash flow. But I imagine the cash flows are not that good: who is responsible for the server setup. Hiring a team of professional can be pretty expensive for a startup with bad financing. I do not wanna support Twitter, but I know it from similar experiences, that having a good idea and finding the perfect technical solution are two very different things - especially if you cannot afford external support / knowledge.