January 31, 2008

Twitter and Joyent Split Amidst Downtime Travails

Mark Hendrickson

37 comments »

Update: According to an ARIN lookup, Twitter appears to be hosted by Verio now.

Update 2: Twitter has come out on their blog to say that they are now hosted by NTT America.

According to Joyent’s corporate blog, the company stopped hosting Twitter late last night:

Twitter has been officially off Joyent since 10PM last night. This may come as a surprise to some after yesterday’s posts here and here regarding the two companies working together. Those of us at Joyent appreciate the opportunity we had to work with the talented folks at Twitter. It is a great service. We wish Twitter every continued success.

As I mentioned yesterday, Joyent is standing ready with excess free infrastructure to support Twitter through this transition in the event that they need it.

The news comes amidst frequent outage problems that have plagued Twitter. Just last night, Twitter went down again, this time for a “planned maintenance project” that went “far beyond [their] planned time window”. The service has also recently suffered downtime during the State of the Union and Steve Jobs’ keynote at Macworld.

Despite all of these problems, just yesterday both companies were showing strong support for each other on their respective blogs. Both wrote posts (here and here) describing how they were working together to prepare for the Super Bowl this coming Sunday.

When reached over the phone, Joyent’s CEO David Young preferred not to comment on Twitter’s stability issues in particular. He did emphasize that Joyent has free infrastructure on standby should Twitter want to use it again. He also wished Twitter the best of luck, saying the team is amazed at their “great service”.

Biz Stone, co-founder of Twitter, responded to an email inquiry about the situation as such:

We’re still very much engaged in our efforts to bring solid reliability to Twitter. Achieving our goals is a sustained effort, not an overnight fix. Performance is our most important measure of success and we appreciate both the patience and frustration folks are sharing with us.

With regard to discussing technical specifics about last night’s efforts, we’ll be more keen to do that once we have a chance to come up for air and cover it with some perspective.

Given that both companies are reluctant to go into details about the break-up, we’ll just have to see whether more information comes out in time.

  • Sphere It

Trackbacks/Pings (Trackback URL)

  1. Bare Feet Studios
  2. GoGrid Blog
  3. Twitter Testing Advertising In Twitter Streams
  4. pligg.com
  5. Twitter Having Problems? Say It Ain’t So!
  6. Lee Mighdoll Out At Twitter, Business Plan Still MIA
  7. As Twitter Service Woes Continue, Japanese Money Looks Likely

Comments

RSS feed for comments on this post.

  1. Sabas

    Hopefully things work out. On the other hand, perhaps this is what Pownce needs to get some traffic.

  2. Oh no...

    Omg. Twitter founder can foresee bankruptcy.
    They heading to deadpool soon.

  3. Alan Wilensky

    The scaling novices strike again. You can’t buy your way out of a poorly designed back-end. The best technical presentations come from Brand Fitzpatrick of danga.com.

    Even with clustered databases, write saturation is a real issue, and needs experienced, creative talent that has lived the scaling nightmare firsthand.

    On Twitter’s blog they wrote something to the effect of’…working with Joyent to emplace more CPU’s, RAM, etc…..” that’s a rough quote. All they needed was a trip to Brad’s site to see into the near-term future of what throwing hardware at badly designed, over-trafficked and stressed back end will get you.

    I highly recommend reading his slides for Linux world.

  4. Michael

    Who’s else beside joyent for PHP cloud-hosting? personally looking for one.

  5. Dennis Howlett

    @Michael - depending on your needs Amazon EC2 is good for pay per drink.

  6. twit er

    I guess this isn’t the joy of hosting.

  7. Alan Wilensky

    You need to find hosts that can think beyond the Relational model. Correct me if I’m wrong, but Twitter at least started as a Rails ORM application.

    Lots of small, fast writes, and virtual joins and user ID clustering. Very bad for MySQl. How do I know this despite being out of the actual code writing business for a decade? Because I have been paying to prototype, out of pocket mind you, a mobile dispatching system that has lots of small objects that do many fast writes, and clusters very dynamic user token data. whew!

    Simulations are great for stressing these systems because the arc of services spanning http request to file systems can be deterministically modeled.

    Load ‘em up and see what happens! RDBMS in my case broke very quickly due to write saturations, even with clusters. They need to synchronized, replicated, yechhhhhh.

    Answer: I won’t even elaborate - http://www.db4o.com/default.aspx

    I have not built it in yet - money…….but I hope the Twitter CTO is looking down this road.

  8. weeeee

    Slicehost rocks! (http://www.slicehost.com/)

  9. Alan Wilensky

    Here is Mr. Fitzpatrick’s excellent primer on scaling:

    http://conferences.oreillynet......k_brad.pdf

  10. Todd

    Definitive story of Twitters back end and how they are scaling:
    http://www.highscalability.com.....ent-faster

  11. Steven Hodson

    This makes the comment by one of the Joyent reps to a post I wrote today about Twitter all the more curious

    http://www.winextra.com/2008/0.....-restless/

    Mind you she didn’t say anything specific about Twitter - it was more about Ruby on Rails

  12. EH

    Now here’s a web2.0 business development I can get behind. Are they even using dedicated write-only and read-only mysql boxes via replication? MySQL has its problems, but there’s a lot of history and expertise out there to get them over this hump.

  13. Adam

    You guys really need to stop posting those shameless links - it gets really irritating.

  14. Bob

    That service is lame. Web2.0 lameness all the way. Die Web2.0 start-ups, die! =p

  15. twit

    Twitter, Seesmic etc. - party over in 2008!

  16. Jeff

    Never really understood all the excitment

  17. sd

    @Jeff,
    How can’t you understand the importance of this ground breaking news?

    image the supplier of the 7-11 down your street change their delivering company. 7-11 is everywhere! the delivering company is the backbone of the 7-11.

    wouldn’t you see such news in your local news paper? Why wouldn’t everyone so excited?

  18. gilltots

    does anybody know how twitter is able to send so many SMS messages? from what (little) research i’ve done it looks like a short code is $1000/month, and then another $1000/month for a hosting-type company that actually enables you to use your shortcode, and then on top of that, sending messges is still like 5 cents a message! how does twitter manage??

  19. Jamboree

    @gilltots

    VC money. LOL.

    YouTube was blowing $1Mil/month on bandwidth. Look where THAT got them.

  20. Jamboree

    @Update:

    Verio??? *shudder*

    That’s like a pre Web1.0 company

  21. irrelevant

    Who cares? I just switched gas stations from shell to another brand. Email me if you want to find out which one.

  22. Jeff the Great

    Twitter rumor mill says the split from their hosting company was due to Twitter being bought out….

  23. Alan Wilensky

    twitter is down, again at 8:40PM PST. What was that about splitting reads and writes over the cluster?

  24. Steek

    Sure, Twitter has problems but so does Joyent. They sure as hell overcharge for what they can deliver.

  25. Andrew

    they prob should recode service with php for starters. yeah, i know this is not a web service but whatever… they gotz issues!

  26. Henry

    Twitter down! Twitter down! It was up most of the day and (seemingly) working well. Now it’s down. Must be rough for those guys.

  27. Peter Juan

    Reports scattered around the web point more to Twitter’s scalability issue. If that is in fact the case, changing hosts won’t be enough to fix it. That said, I’m hoping that it isn’t true. I’m a big Twitter fan and so are millions of other people. A problem with their back end will simply mean that we will be Twitter deprived that much longer.

  28. rubu

    Simulations are great for stressing these systems because the arc of services spanning http request to file systems can be deterministically modeled.

  29. Larry Larrikin

    db4o sounds good but then object databases have sounded good for 15 years and never really delivered. Does anyone know how well it scales and replicates?

  30. Switcher

    I am currently researching a switch to another host and would like suggestions.

    Its time since the site has grown to hundreds of SMTP connections that are being opened simultaneously which is putting stress on the current shared hosting plan I am with ($39.00 per month).

    My site runs on Ruby on Rails. The quote from my current host is $100 to set up, and $205 per month to run a Virtual Private Server. Its not something they specialize in and I am worried after recently going through a lot of problems with them.

    It has been suggested by my programmer that I check out Joyent, however, my reading finds that I may land back in the same frying pan of down time - something we all dred! I have, alternatively, read a lot of good things about Slicehost.

    My programmer wants to end up with root priveledges and it looks like I will need at least :
    40gb disk
    768mb RAM, 1024 burstable
    700gb monthly bandwidth
    CPanel Control Panel
    Daily backups

    My goal is to end up with a SOLID Virtual Dedicated Server since a move to a dedicated server appears to just bee too much for now.