April 23, 2008

Amateur Hour Over At Twitter?

Michael Arrington

263 comments »

It doesn’t really matter if Twitter’s Chief Architect Blaine Cook was fired or resigned. The important thing is that he’s gone now, and this gives Twitter the opportunity to hire someone (or a team) who may actually be able to scale the nearly two year old service and keep it live.

Cook was directly responsible for scaling Twitter, and he very much failed in his job. A year ago he spoke at the Silicon Valley Ruby Conference about scaling Rails applications. His presentation suggested Twitter’s problems were behind them, but in fact some of their biggest stumbles hadn’t occurred yet. Note in particular slide 9 of that presentation, where Cook says about scaling Rails apps like Twitter: “It’s Easy. Really.” Whether Twitter’s woes were all on Cook’s shoulders or not, he should not have been boasting about solving the problem last year.

Meanwhile, Twitter has made at least three key hires this year on the technical side. Lee Mighdoll joined as VP Engineering and Operations in January. And this week they hired two scaling experts - John Kalucki and Steve Jenson (”known for his work scaling Blogger and Blogspot”).

  • Sphere It

Trackbacks/Pings (Trackback URL)

  1. Why Big Brands Don’t Sponsor Blogs - Covering All That's Social All the Web
  2. Amateur Hour Over at TechCrunch | Stephen Tudor
  3. Blaine Cook leaves Twitter, takes the service with him » VentureBeat
  4. mathewingram.com/work | Twitter-storm: Blaine leaves, blame flies
  5. Amateur Hour Over at TechCrunch | Stephen Tudor
  6. RealityCrunch - TWITTER WATCH 2008!!
  7. Anyone hounding Arrington about RazorGator? You try scaling Twitter « Internet Marketing Observations
  8. STARTUP CHATTER » BIG PICTURE » Ning, Twitter Viral VC Ventures: Startup Loops That Kill
  9. Twitter techie Blaine Cook leaves the company « IT Spot
  10. Sazbean » Web 2.0 Expo - Real Time Web
  11. Twitter Scapegoat? | Technology Watch
  12. Twitter’s Lead Architect Leaves : The Last Podcast
  13. Buttcrunch < LostFocus
  14. Arrington (techcrunch) zbiera cięgi za wpis o odejściu głównego architekta twittera. at AntyWeb
  15. Joe@Nitobi » Blog Archive » Personal Attacks over Appplication Performance are not productive
  16. Blern
  17. The Road Ahead For Twitter - Rev2.org
  18. 10 Rules for Using Twitter for PR « PR for Pirates
  19. Tech World, I Am Your Master : Matt’s Homepage
  20. Twitter has issues. In other news, water is wet.
  21. Between the Lines mobile edition
  22. Lee Mighdoll Out At Twitter, Business Plan Still MIA
  23. Stop Flipping the Bird! « Seizing the Day
  24. Stop Flipping the Bird! « American Pai
  25. Enterprise Alley mobile edition
  26. Иван Комаров » Blog Archive » links for 2008-04-24
  27. Geogad Travel Shouts » Blog Archive » Microsoft’s Web 2.0 Part 2
  28. Twitter Said To Be Abandoning Ruby on Rails
  29. Reflections of a BizDrivenLife » Blog Archive » Does Ruby on rails Scale or Not>?
  30. Twitter: the decentralization debate » mathewingram.com/work |
  31. www.ubraniaroxy.pl » Blog Archive » VP Lee Mighdoll Out At Twitter, Business Plan Still MIA
  32. Blaine Cook Joins Today’s Gillmor Gang, Talks Twitter
  33. Blaine Cook on the Gillmor Gang « The Pursuit of a Life

Comments

RSS feed for comments on this post.

  1. PressReleasePoint

    hoping to see better uptime for twitter with new hires.

  2. Peter Urban

    It is a bit baffling that it takes them so long to establish an infrastructure that scales properly given that they know of their success for a while now and that sufficient funding is available. However, you never know how flaky the idea was conceived in the first place and how much the system suffers from it’s experimental stage heritage.

  3. Robert Scoble

    I hope Twitter gets on top of its reliability problems. Too many things are counting on it now. Wait until the next earthquake when traffic will spike 1000x above where it is right now.

  4. Will Johnston

    Really hoping this fixes twitter’s issues, as you said, Mike, we’ve become dependent.

  5. dumbquestion

    Got a dumb question.
    How does Twitter make money?

  6. Robert Scoble

    It should be said, thanks to Blaine and team for building something we’re all addicted to. I do tap my hat to you and hope you do something interesting in your career path again. We’ll be watching.

  7. johng

    DumbQ - everybody asks that, and nobody knows. No doubt their biz model will become clear before much longer. (though I remember saying that about Napster a few years back)

  8. Ken Camp

    And given their time in operation, with no money making in sight, why exactly do we expect Twitter to be able to scale to support everything we want?

    Is Twitter anything more than a grand experiment in chitchat? I think not.

  9. Patrick

    Wow, someone certainly has an axe to grind.

  10. Jay Virdy

    I echo Scoble’s sentiment re: Blaine and team. Many of us have become addicted to Twitter thanks to their hard work.

  11. Steven Van Tilburg

    I hope that with the upscaling, they retain the functionality that Twitter already has, much less adding new function. It would be a shame to see things taken away because they can’t handle the load.

  12. Jason

    @dumbquestion - Twitter doesnt make money. Except for Twitter Japan which has ads. Which probably wont earn them much seeing as the majority of twitter activity happens over their API.

    I think Twitter should scheduled downtimes so you addicted folks will stop spamming stuff no one cares about.

  13. Prokofy Neva

    Is Ruby on Rails the problem and not the guy?

    http://antoniocangiano.com/200.....criticism/

  14. Baker

    Have to agree with Patrick - you guys sound venomous.

  15. Tara Hunt

    Personally, I think Blaine did a terrific job scaling something they never expected would grow like it did. He also built an amazing platform that millions love, hundreds have built apps off of and that works 95% of the time. It was Blaine’s idea to make the messages available via IM and by other means through the API. I don’t know about you, but I certainly couldn’t handle following 800+ people through my SMS.

    I think smudging Blaine’s skills is an unfortunate misrepresentation of the hard work, long hours and multiple weekends he spent trying to keep us all happy.

    FWIW…he left Twitter last week. This last weekend? Let me tell you, it would have been fixed in hours if he and @meangrape had been around (Jay was out of town).

  16. Steve

    So what’s up? Burying the twitter disaster thread?
    Anyway it could be the rails curse…

  17. Kevin Clark

    So, what you’re saying is:

    “I have no new information. I do have a scapegoat, though I have no concept of the technology I’m railing against. So let’s jump on the bandwagon and complain about someone, because it’s convenient.”

    Blaine is an excellent engineer who’s put out some really innovative stuff, and _you_ are the reason why many people still don’t consider bloggers journalists. You’re a gossip rag, nothing more.

  18. Marco

    You need different people at different stages of the product maturity. The article is a bit harsh about Blaine. Don’t forget he helped to get twitter all the way to this level. And I’m sure that they had to be rather innovative to keep up with the quick growth.

    It might take some time to see the results of the new people. I’m sure it’s not just “Buy X more machines…”

  19. Pierre Carion

    “Apparently Mike Arrington’s deleting comments from the post about my departure. Classy.” really ? http://twitter.com/blaine/statuses/795211971

  20. Michael Arrington

    Blaine in April 2007, on the ease of scaling twitter:

  21. Eran Hammer-Lahav

    Amateur Hour Over At TechCrunch? Yes!

    Blaine is the main reasons Twitter’s scaling problems were actually contained and under control for so long. To try and link his comments on Rails scaling with what is happening at Twitter now is a true demonstration of ignorance and bad reporting. The Twitter engineering team, which up until very recently was just 3 (!) people, deserves a lot of credit for being able to accomplish so much with so little. Pointing fingers at the developers demonstrates your lack of understanding and the low quality of your reporting. And if only you got some of your other facts right…

  22. Yup

    I can vouch for that deleting comment.
    Comments against Mike et al. are mysteriously disappearing into the ether.

  23. Jeremy McAnally

    Wow, this is the most unnecessarily vitriolic articles I’ve ever seen. Twitter and Blaine took Rails to places it hadn’t been before in terms of scale and traffic (just as other sites did with PHP, Java, ColdFusion, and so on), and so acting as if he’s some unskilled hippie that didn’t know what he was doing just makes you look like an idiot. Sure, they had some bumps along the way, but no one else had even gotten on the same road as them to encounter the bumps. Being forced to figure out things along the way because no one has done them before doesn’t make you an “amateur.”

    Even further, what the heck qualifies you to criticize someone’s ability to scale a website or doing anything remotely technical for that matter? You’re out of your league. Please stick to talking about business or whining about someone not giving you ad money.

  24. westernworld

    i met all kinds of new people due the absolute randomness of direct message distribution over the past few days, at least here in europe.
    so it did help the social aspect, but networking really needs work, lots of it.
    look what happened to pownce …people just left.
    everyone and their brother went on twitter, it’s the default now.

  25. Michael Arrington

    Eran - right. he’s awesome. that’s why he left and twitter hired all these other guys to scale the application.

  26. Bob Lozano

    @scoble sure I hope he lands on his feet, lessons learned and all that.

    But I think the bigger lesson is for the industry as a whole - the days of the throwing out a POC and figuring out scale issues later are probably long gone.

    I posted on this yesterday.

  27. Kevin Clark

    @Arrington: I don’t know. Aren’t you supposed to be reporting here? Shouldn’t you know if you’re going to post about it?

  28. jimmy dean

    Love this:

    This last weekend? Let me tell you, it would have been fixed in
    hours if he and @meangrape had been around (Jay was out of town).

    HOURS?!

    I’m not sure if that’s a compliment or criticism.

  29. Mel Kirk

    I have to second Tara’s comments. We were lucky enough to have Blaine speak at FOWA Miami back in February and he’s by far one of the nicest and most intelligent guys that I’ve had the pleasure of meeting.

    Blaine and the team have done an amazing job scaling Twitter to it’s present form and let’s face it, whilst we might be quick to criticise any down time, can you imagine life without Twitter? No, thought not…

    Good luck Blaine - hope rumours of you moving to the UK are true :)

  30. Adam J. Schmidt

    The most important question is how they’re planning on making money. Doesn’t matter how much we enjoy the product/service if it’s unsustainable. Maybe they just get bought out buy someone else who wants access to the users?

  31. Fred Grott

    I know form scaling several MVC frameworks including Rails that its NOT DAMN EASY IN RAILS!

  32. John C.

    Michael, from the SAI:

    “I am moving to the UK so that my partner can pursue her career. We’re Canadian and her visa makes it impossible for her to work in the US,”

    How hard is that to understand?

  33. Tim G

    Geez Arrington this post seems kind of personal. What’d Blaine hit on your boyfriend?

  34. David Pollak

    Blaine is a very smart guy, a fine engineer, and a professional. Twitter would not be many of the good things it is today without Blaine’s excellent work and passion. I’m toasting Blaine’s awesomeness and I know he’ll land someplace where he’ll be able to make another significant impact.

  35. Mike Stenhouse

    That’s a bit harsh. You’ve taken co-incidence and got causality without any evidence that I can see. Blaine’s leaving? Best of luck to him. He’s a super-sharp guy; I’m sure he’ll have companies beating a path to his door. As a service, Twitter is growing at a phenomenal rate with finite human resources. Something has to give - them’s the breaks. Twitter will be fine. Blaine will be fine. Techcrunch will be fine. Let’s all hug.

  36. Charles

    Twitter down time is good for the world economy; it’s the only time any work gets done. Don’t knock Twitter down time. It’s not like it’s an essential service anyway. I feel Arrington is being a little harsh and graceless here; Scoble’s comments are more appropriate.

  37. Dave Litsky

    Time for an acquisition. Your product is hot right now and who knows if the scaling issues will ever be fixed.

  38. Charlie Wood

    Wow, what’s the deal with the venomous personal attack? Leave the character assassination to Valleywag, Arrington.

    -Charlie

  39. jimmy dean

    C’mon folks, this is all in “fun.”

    Vitriolic? Sure. We’re human. Who ever said we’re supposed to be nice.

    If you want to put yourself out in the public, and to put it mildly, be overconfident (”It’s easy, really.”), then you better have a thick skin.

    Welcome to Hollywood.

  40. Sabil Ardie

    Hmmm… I wonder if the “Cache the hell of everything” makes it harder sometimes to delete posts.

  41. Michael Arrington

    Mike - if you agree with me that twitter has had serious uptime issues, it’s kind of hard not to follow the dotted lines to the chief architect.

  42. Michael Arrington

    Fred - no, see comment 20. scaling rails is quite easy.

  43. Doug D

    How is their infrastructure hosted these days? Are they still gambling on bleeding edge technology or have they moved to a more tried and true dedicated server solution?

    I’m not doubting that they have general architecture issues that would persist regardless of virtual or physical infrastructure, but I think they had originally placed too much faith in virtualization hype and attention to architecture suffered as a result (this is pure speculation, take with a grain of salt).

  44. Eran Hammer-Lahav

    @Arrington - the problem is, you have no clue about this story. You read it elsewhere, copied it, and made it into a personal attack. Your post is the same as blaming the TechCrunch graphic designer for your poor reporting.

  45. Andrew

    The tone of this post is why I’ve grown to dislike Techcrunch over the past months, and why it will never grow beyond a certain point.

    This post is bitchier than most found on Vallywag, and contains more opinion that news. Techcrunch seems to have replaced crunchnotes lately for Michael’s bitching and sideswiping.

    For a site that’s so hard on traditional media, there’s certainly a lot of lessons that could be learned about what constitutes journalism.

  46. Pierre Carion

    TC has also technical problems : I read in the comments, 10 mn ago, a reference to Blaine’s explanations about his leaving “I am moving to the UK so that my partner can pursue her career…” as in 32. , but this comment disappeared.

    Will the person in charge of the service at TC be fired too then?

  47. Mark Madner

    Software architects are second guessed as much as NFL coaches.

    I want to add my name to the list of people thanking Blaine. Getting a site like twitter to scale to the volume it now has on a shoestring budget is non trivial. I would feel lucky to work with Blaine in the future.

    Michael Arrington: You’re successful enough of a blogger to not have to resort to this kind of public humiliation of folks to gain traffic. If you have a beef with Twitter’s architecture, I would prefer you leave names out of it. You don’t know what went on behind the scenes, so you can’t judge who the problems should be attributed to. Let’s all make the internet a more hospitable place.

  48. Raskin

    Am i the only one who COULD live without twitter?

    Like, happily.

    And…easily.

  49. Jeremy McAnally

    @jimmy, Michael: I’d be fine with the criticism if it wasn’t coming form an uninformed dunce who knows about as much about scaling a high-traffic website as writing informative journalism. Which, by the way, isn’t much.

  50. Coda Hale

    Michael, I don’t think you know what you’re talking about, and I certainly I don’t think you’re familiar with the ins and outs of Twitter’s technical details. Personally, I’ve had all three of Twitter’s engineers (at the time) explain their architecture to me over dinner at Naan N’ Chutney, and I can tell you that Blaine Cook has *not* been their problem (nor has Ruby or Rails or any of the other technical issues favored by armchair analysts). This baseless scapegoating is unprofessional and your readers deserve better, even if you are unable to give them that.

  51. Arun Vijayan

    Crazy! Dont kill him for his slides. Those are like bible for some of us. Any engineer will think multiple times for taking a responsibility like this. And the kind of stuff he had done with limited infra, time is amazing. I think mike, you gotta learn to respect talent.

  52. Scabr

    Maybe new twitter team will remove twitter bugs

  53. Michael Arrington

    Jeremy, I’ve got two rails developers sitting ten feet away from me who seem to have their finger on the pulse of the rails community. And trust me, I toned the post down significantly. but you’re right, i don’t know much about journalism and even less about rails. but i do know that a for profit company that can’t get their shit together two years after launch deserves some criticism.

  54. yah

    Wow, TC has got over its love of Facebook. Can anyone say TC loves Twitter ?

    How many stories in a day do we need about Twitter ?

    Lets all sing , Techcrunch and Twitter sitting in a tree , K—i—S–S I–N–G.

    Dude enough of the Twitter love ,

  55. Scott

    only us geeks have ever heard of twitter anyway

  56. Michael Arrington

    coda - ok, what exactly in the post do you disagree with? the fact is cook said all their problems were solved a year ago. they’ weren’t. he looks a little dumb now. doesn’t he?

  57. Mike Stenhouse

    @Michael

    You _could_ follow that dotted line… but he might just be tired of working nights and weekends. Or maybe he’s got a startup idea of his own to pursue. Or perhaps he’s eloping with the missus and then joining the circus. Who knows? It’s speculation, not news.

  58. Eran Hammer-Lahav

    Re: 41 - So stability has nothing to do with management and operations? I don’t have the full story of why Twitter has some stability issue and can’t even tell if they are issues of scale or something else. The way you connect the dots is nothing more than wishful thinking. It might be true, but you *are* just guessing. Again, quality reporting!

  59. jro

    Someone can’t scale Rails, and that makes them an amateur? Wow, talk about uninformed writing. Some might even say “amateurish”.

  60. Christopher Herot

    Let’s face it - most sites never become popular enough to have this kind of scaling problems. And few have pushed RoR this far. I think we should thank Blaine for giving us Twitter and be grateful for the chances he took opening up the API, and wish the next guy well.

  61. Michael Arrington

    what else you guys got?

  62. Andrew

    “Mike - if you agree with me that twitter has had serious uptime issues, it’s kind of hard not to follow the dotted lines to the chief architect.”

    That’s a surprisingly ignorant statement.

    You can be the best architect, or anything, in the world but factors outside your control can change the end result in many ways.

    Maybe Blaine just became fed up at to the level of staff/software/hardware support he received while trying to scale and support such a massive platform as twitter?

    I don’t know if this is true, but I do know your statements are needlessly bitchy and unprofessional.

    As for the slides - pfft. Talk about taking things out of context.

  63. Nick

    @Michael

    If Twitter is such a hack-y piece of software, perhaps you should stop using it. No sense in supporting the work of an ‘amateur’ developer like Blaine.

    I know it might be hard… you know… not hearing you complain about stuff every 10 minutes on Twitter, but the rest of us (non-250 folks) will survive. Don’t you worry.

  64. Mike M

    I think Blaine’s done an excellent job with twitter. Getting RoR scaling this much is a mean feat. I don’t envy his successor.

    I believe Blaine could be right on horizontal scaling. Adding more servers can solve the problem. Perhaps the folks at procurement may have had cash issues? Perhaps they couldn’t find the cash to improve their infrastructure?

    One can only guess. But I don’t fault his choice of platform. (even though I’m a PHP guy).

  65. Nick Taylor

    Yea - Twitter in it’s entirety is a masterful bit of communicaion though - it’s like a tamagotchi crossed with a web-app. It’s small and cute and anthropomorhisable - and the fact that it wobbles sometimes doesn’t stop us loving it. Tamagotchi do wobble. They’re kindof supposed to.

    And the Blaine Cook? Never met him - but he seems like a tinkerer, and a project-doer and an all-round geek’s geek… whereas the writer of this blog most certainly does not.

    Which is my 2C why this article is such an own-foot-shoot.

  66. jimmy dean

    From No. 61:
    > what else you guys got?

    Now THAT’S the kind of reply Blaine should have posted here.

    Instead, we get the righteous defenders, “We love you Blaine,” which reminds me of the moms in the parks with their kids doing the “Good job!” Sickening.

  67. nick halstead

    Wow, this what happens when you annoy the Ruby/Rail’s community in any shape or form (but that is a different rant) - this is not about Ruby Scaling in anyway, although
    they have certainly suffered from not rebuilding from scratch when they should have at the outset. This is about the many-to-many problem in databases, and frankly given the timescales they have had the scaling/reliability issues should not have been an issue even if they had programmed the bloody thing in BASIC. So get over the ruby thing - this is just about not understanding scaling in general - as rule 1) performance DOES NOT equal scalability - so should he have gone? yes - about 6 months ago.

    sent from: fav.or.it [FID327972]

  68. maximo

    @Eran: You consider TechCruch real reporting? Please!! This is juicy tech gossip at best. Just read and enjoy it!

  69. Ben

    Mike, this isn’t valleywag. And you’re not very good at trolling.

  70. Brick and Click

    Every fifteen years or so, there’s a three foot ‘Nor’easter’ blizzard in the Washington DC region. Happened in ‘79 and ‘96. Shuts everything down for a week: Gov’t, schools, many businesses. You can’t drive until the plow truck comes, which may be five days for secondary streets. A lot of folks freak out at first, then acceptance sets in. Cross country skis and Flexible Flyers and Snow Shoes break out, and parents play with their kids again (rare opportunity to build world class snow forts and igloos). You re-meet the neighbors, shovel drives and sidewalks together, borrow supplies if needed, and pop Advil like candy because you worked long-forgotten muscle groups moving several thousand pounds of snow. And you take a zillion pictures of the most amazing winter landscape you’ve ever seen.

    Big snow in Twitterville. Enjoy life for a while.

  71. Scott M.

    @Prokofy Neva:
    Aww, c’mon, Prok. Can’t you keep your uninformed and deliberately inflammatory grandstanding/griefing to the Second Life world?

  72. John Lynn

    @Coda Hale You said “I can tell you that Blaine Cook has *not* been their problem (nor has Ruby or Rails or any of the other technical issues favored by armchair analysts)”

    If you know this is the case, then let Mike and the rest of us know what or who is responsible so he can update the post. Otherwise, your comments seem more baseless than Mike’s.

  73. Dan

    Speaking of amateur hour…

  74. michael air

    Put it on Google Appengine. Problem Solved :)

  75. Loren Feldman

    ROR Sucks {seesmic_video:{”url_thumbnail”:{”value”:”http://t.seesmic.com/thumbnail/La1XPF4uMI_th1.jpg”}”title”:{”value”:”R