It doesn’t really matter if Twitter’s Chief Architect Blaine Cook was fired or resigned. The important thing is that he’s gone now, and this gives Twitter the opportunity to hire someone (or a team) who may actually be able to scale the nearly two year old service and keep it live.
Cook was directly responsible for scaling Twitter, and he very much failed in his job. A year ago he spoke at the Silicon Valley Ruby Conference about scaling Rails applications. His presentation suggested Twitter’s problems were behind them, but in fact some of their biggest stumbles hadn’t occurred yet. Note in particular slide 9 of that presentation, where Cook says about scaling Rails apps like Twitter: “It’s Easy. Really.” Whether Twitter’s woes were all on Cook’s shoulders or not, he should not have been boasting about solving the problem last year.
Meanwhile, Twitter has made at least three key hires this year on the technical side. Lee Mighdoll joined as VP Engineering and Operations in January. And this week they hired two scaling experts – John Kalucki and Steve Jenson (”known for his work scaling Blogger and Blogspot”).
hoping to see better uptime for twitter with new hires.
It is a bit baffling that it takes them so long to establish an infrastructure that scales properly given that they know of their success for a while now and that sufficient funding is available. However, you never know how flaky the idea was conceived in the first place and how much the system suffers from it’s experimental stage heritage.
I hope Twitter gets on top of its reliability problems. Too many things are counting on it now. Wait until the next earthquake when traffic will spike 1000x above where it is right now.
Really hoping this fixes twitter’s issues, as you said, Mike, we’ve become dependent.
Got a dumb question.
How does Twitter make money?
It should be said, thanks to Blaine and team for building something we’re all addicted to. I do tap my hat to you and hope you do something interesting in your career path again. We’ll be watching.
DumbQ – everybody asks that, and nobody knows. No doubt their biz model will become clear before much longer. (though I remember saying that about Napster a few years back)
And given their time in operation, with no money making in sight, why exactly do we expect Twitter to be able to scale to support everything we want?
Is Twitter anything more than a grand experiment in chitchat? I think not.
Wow, someone certainly has an axe to grind.
I echo Scoble’s sentiment re: Blaine and team. Many of us have become addicted to Twitter thanks to their hard work.
I hope that with the upscaling, they retain the functionality that Twitter already has, much less adding new function. It would be a shame to see things taken away because they can’t handle the load.
@dumbquestion – Twitter doesnt make money. Except for Twitter Japan which has ads. Which probably wont earn them much seeing as the majority of twitter activity happens over their API.
I think Twitter should scheduled downtimes so you addicted folks will stop spamming stuff no one cares about.
Is Ruby on Rails the problem and not the guy?
http://antonioc...ails-criticism/
Have to agree with Patrick – you guys sound venomous.
Personally, I think Blaine did a terrific job scaling something they never expected would grow like it did. He also built an amazing platform that millions love, hundreds have built apps off of and that works 95% of the time. It was Blaine’s idea to make the messages available via IM and by other means through the API. I don’t know about you, but I certainly couldn’t handle following 800+ people through my SMS.
I think smudging Blaine’s skills is an unfortunate misrepresentation of the hard work, long hours and multiple weekends he spent trying to keep us all happy.
FWIW…he left Twitter last week. This last weekend? Let me tell you, it would have been fixed in hours if he and @meangrape had been around (Jay was out of town).
So what’s up? Burying the twitter disaster thread?
Anyway it could be the rails curse…
So, what you’re saying is:
“I have no new information. I do have a scapegoat, though I have no concept of the technology I’m railing against. So let’s jump on the bandwagon and complain about someone, because it’s convenient.”
Blaine is an excellent engineer who’s put out some really innovative stuff, and _you_ are the reason why many people still don’t consider bloggers journalists. You’re a gossip rag, nothing more.
You need different people at different stages of the product maturity. The article is a bit harsh about Blaine. Don’t forget he helped to get twitter all the way to this level. And I’m sure that they had to be rather innovative to keep up with the quick growth.
It might take some time to see the results of the new people. I’m sure it’s not just “Buy X more machines…”
“Apparently Mike Arrington’s deleting comments from the post about my departure. Classy.” really ? http://twitter....tuses/795211971
Blaine in April 2007, on the ease of scaling twitter:
Amateur Hour Over At TechCrunch? Yes!
Blaine is the main reasons Twitter’s scaling problems were actually contained and under control for so long. To try and link his comments on Rails scaling with what is happening at Twitter now is a true demonstration of ignorance and bad reporting. The Twitter engineering team, which up until very recently was just 3 (!) people, deserves a lot of credit for being able to accomplish so much with so little. Pointing fingers at the developers demonstrates your lack of understanding and the low quality of your reporting. And if only you got some of your other facts right…
I can vouch for that deleting comment.
Comments against Mike et al. are mysteriously disappearing into the ether.
Wow, this is the most unnecessarily vitriolic articles I’ve ever seen. Twitter and Blaine took Rails to places it hadn’t been before in terms of scale and traffic (just as other sites did with PHP, Java, ColdFusion, and so on), and so acting as if he’s some unskilled hippie that didn’t know what he was doing just makes you look like an idiot. Sure, they had some bumps along the way, but no one else had even gotten on the same road as them to encounter the bumps. Being forced to figure out things along the way because no one has done them before doesn’t make you an “amateur.”
Even further, what the heck qualifies you to criticize someone’s ability to scale a website or doing anything remotely technical for that matter? You’re out of your league. Please stick to talking about business or whining about someone not giving you ad money.
i met all kinds of new people due the absolute randomness of direct message distribution over the past few days, at least here in europe.
so it did help the social aspect, but networking really needs work, lots of it.
look what happened to pownce …people just left.
everyone and their brother went on twitter, it’s the default now.
Eran – right. he’s awesome. that’s why he left and twitter hired all these other guys to scale the application.
@scoble sure I hope he lands on his feet, lessons learned and all that.
But I think the bigger lesson is for the industry as a whole – the days of the throwing out a POC and figuring out scale issues later are probably long gone.
I posted on this yesterday.
@Arrington: I don’t know. Aren’t you supposed to be reporting here? Shouldn’t you know if you’re going to post about it?
Love this:
This last weekend? Let me tell you, it would have been fixed in
hours if he and @meangrape had been around (Jay was out of town).
HOURS?!
I’m not sure if that’s a compliment or criticism.
I have to second Tara’s comments. We were lucky enough to have Blaine speak at FOWA Miami back in February and he’s by far one of the nicest and most intelligent guys that I’ve had the pleasure of meeting.
Blaine and the team have done an amazing job scaling Twitter to it’s present form and let’s face it, whilst we might be quick to criticise any down time, can you imagine life without Twitter? No, thought not…
Good luck Blaine – hope rumours of you moving to the UK are true
The most important question is how they’re planning on making money. Doesn’t matter how much we enjoy the product/service if it’s unsustainable. Maybe they just get bought out buy someone else who wants access to the users?
I know form scaling several MVC frameworks including Rails that its NOT DAMN EASY IN RAILS!
Michael, from the SAI:
“I am moving to the UK so that my partner can pursue her career. We’re Canadian and her visa makes it impossible for her to work in the US,”
How hard is that to understand?
Geez Arrington this post seems kind of personal. What’d Blaine hit on your boyfriend?
Blaine is a very smart guy, a fine engineer, and a professional. Twitter would not be many of the good things it is today without Blaine’s excellent work and passion. I’m toasting Blaine’s awesomeness and I know he’ll land someplace where he’ll be able to make another significant impact.
That’s a bit harsh. You’ve taken co-incidence and got causality without any evidence that I can see. Blaine’s leaving? Best of luck to him. He’s a super-sharp guy; I’m sure he’ll have companies beating a path to his door. As a service, Twitter is growing at a phenomenal rate with finite human resources. Something has to give – them’s the breaks. Twitter will be fine. Blaine will be fine. Techcrunch will be fine. Let’s all hug.
Twitter down time is good for the world economy; it’s the only time any work gets done. Don’t knock Twitter down time. It’s not like it’s an essential service anyway. I feel Arrington is being a little harsh and graceless here; Scoble’s comments are more appropriate.
Time for an acquisition. Your product is hot right now and who knows if the scaling issues will ever be fixed.
Wow, what’s the deal with the venomous personal attack? Leave the character assassination to Valleywag, Arrington.
-Charlie
C’mon folks, this is all in “fun.”
Vitriolic? Sure. We’re human. Who ever said we’re supposed to be nice.
If you want to put yourself out in the public, and to put it mildly, be overconfident (”It’s easy, really.”), then you better have a thick skin.
Welcome to Hollywood.
Hmmm… I wonder if the “Cache the hell of everything” makes it harder sometimes to delete posts.
Mike – if you agree with me that twitter has had serious uptime issues, it’s kind of hard not to follow the dotted lines to the chief architect.
Fred – no, see comment 20. scaling rails is quite easy.
How is their infrastructure hosted these days? Are they still gambling on bleeding edge technology or have they moved to a more tried and true dedicated server solution?
I’m not doubting that they have general architecture issues that would persist regardless of virtual or physical infrastructure, but I think they had originally placed too much faith in virtualization hype and attention to architecture suffered as a result (this is pure speculation, take with a grain of salt).
@Arrington – the problem is, you have no clue about this story. You read it elsewhere, copied it, and made it into a personal attack. Your post is the same as blaming the TechCrunch graphic designer for your poor reporting.
The tone of this post is why I’ve grown to dislike Techcrunch over the past months, and why it will never grow beyond a certain point.
This post is bitchier than most found on Vallywag, and contains more opinion that news. Techcrunch seems to have replaced crunchnotes lately for Michael’s bitching and sideswiping.
For a site that’s so hard on traditional media, there’s certainly a lot of lessons that could be learned about what constitutes journalism.
TC has also technical problems : I read in the comments, 10 mn ago, a reference to Blaine’s explanations about his leaving “I am moving to the UK so that my partner can pursue her career…” as in 32. , but this comment disappeared.
Will the person in charge of the service at TC be fired too then?
Software architects are second guessed as much as NFL coaches.
I want to add my name to the list of people thanking Blaine. Getting a site like twitter to scale to the volume it now has on a shoestring budget is non trivial. I would feel lucky to work with Blaine in the future.
Michael Arrington: You’re successful enough of a blogger to not have to resort to this kind of public humiliation of folks to gain traffic. If you have a beef with Twitter’s architecture, I would prefer you leave names out of it. You don’t know what went on behind the scenes, so you can’t judge who the problems should be attributed to. Let’s all make the internet a more hospitable place.
Am i the only one who COULD live without twitter?
Like, happily.
And…easily.
@jimmy, Michael: I’d be fine with the criticism if it wasn’t coming form an uninformed dunce who knows about as much about scaling a high-traffic website as writing informative journalism. Which, by the way, isn’t much.
Michael, I don’t think you know what you’re talking about, and I certainly I don’t think you’re familiar with the ins and outs of Twitter’s technical details. Personally, I’ve had all three of Twitter’s engineers (at the time) explain their architecture to me over dinner at Naan N’ Chutney, and I can tell you that Blaine Cook has *not* been their problem (nor has Ruby or Rails or any of the other technical issues favored by armchair analysts). This baseless scapegoating is unprofessional and your readers deserve better, even if you are unable to give them that.