We’re hearing this from multiple sources: After nearly two years of high profile scaling problems, Twitter is planning to abandon Ruby on Rails as their web framework and start from scratch with PHP or Java (another solution is to stick with the Ruby language and move away from the Rails framework).
Former Chief Architect Blaine Cook famously said scaling Rails was “easy” in April 2007 (see image to right), but problems persisted after Cook claimed to have conquered the problem. The service most recently had a three day outage affecting their largest users.
Other massive Rails sites include Scribd, Hulu, and the popular Facebook app Friends for Sale. CrunchBase, our tech company database, is also built on Rails.
Switching off Rails may not solve all of Twitter’s problems. They have nearly two years of infrastructure built up and would face many more growing pains if they switched frameworks or rolled their own. As Twitter considers moving away from Rails, some companies are doing the opposite: last year, Yellowpages.com scrapped Java for Rails, and is now second on the unofficial Rails 100 wiki.
Rails has always bred controversy. Developers have argued that it is fundamentally flawed and unscalable; others have argued back saying the opposite (see here, here, and here). Earlier this year, one of the core community members and creator of the popular Rails web server Mongrel abandoned rails and trashed the community.
Update: Regarding Evan Williams’ statement here, all I can say is that multiple sources claim that Twitter is telling people they are planning on moving away from Ruby on Rails. This is not the first time a company has denied something that has turned out to be 100% true.
Partially agree with Matt. They should recode parts of their codebase in C and use some scripting language (Ruby’s ok) when appropriate.
Real Life Case of C rewrite: Rocketmail (way back in the day and became Yahoo Mail) was built in Python and then rewritten in C as things started to scale.
Twitter shouldn’t leave Rails. They’ve done a lot for the community by going through the growing pains that hopefully others (including myself) won’t have to. I hope they can keep it going and get rock solid with it.
Last time I looked at ActiveMQ it was all hype, not much implemented and buggy. Maybe it’s better now. I would try Joram.
On another note, I guess Bezos won’t be investing in Twitter. We all know how much of a Rails fan he is.
I’m not surprised about Twitter/Ruby. Simply because Ruby-on-Rails is non-scalable trendy piece of shit for weekend projects.
I know it’s matter of personal taste, but based on my big experience if I were Twitter’s VPE I’d create infrastructure on custom built C++ servers or on Java with layer of in-memory database.
This might be the opportunity Powce was waiting for to try a come back: my experience with large software projects and that they fail 75% of the time, special if they bring a new person in.
Outages can occur because of operations too. In fact in large scale com servers like twitter it should be expected. How many here work for a fortune 100 that has good exchange uptime? How do we know someone didnt accidently init 6 a production box thinking they were still ssh@ /dev? Every app & architecure design has its tradeoffs & it usually comes down to how much of a life or how much sleep your DBAs are willing to give up. You could have the best design that is so novel your dba doesn’t understand how to keep it humming. Sounds like they need ITIL more than a new interpreter or framework
This is nuts. This app is fundamentally a messaging problem, & a modest one at that.
Putting a DB into the flow IS THE PROBLEM. It’s not the language, it’s not politics, it’s not even about friendships or past contributions to a community.
It’s simply that you’re asking a DB to act like a router, something which it is pretty bad at doing.
Why be surprised when it doesn’t work?
Quite frankly I’m glad this has happened since I sometimes get the impression the Ruby on Rails community are better than everybody else which is simply not the case.
Ruby on Rails looks to me like it was written for all those unemployed Visual Basic programmers looking to be re-skilled in web development work.
I would laugh if after a couple of years the folks at Twitter decide to use something like Seaside (Smalltalk), because by then they will have realised that was probably the best tool for the job.
Personally I’m still promoting Ruby, since it’s got a lot more going for it longer term unlike Ruby on Rails IMHO.
With Yahoo! developers jumping ship, there will be no shortage of solid PHP developers with large-scaling experience!!
Good move on Twitter
ROR is a tool and it has it’s place, but it’s no where as mature as java. There are so much high technology stuff around java that Ruby is not going to catchup to it.
I think the java move is smart. One of the largest online poker site is on java and uses clustering very effectively
They have to switch canoes in the middle of the stream to send a message to the moneymen that they’re serious about building teh awesome.
Remove the DB and rewrite, using a messaging protocol using jabber / IRC.
There’s an O(n^n) problem here when users send messages to every other user through a database. If you replace that with the idea of subscriptions and use a decent messaging protocol, the problem goes away.
Relational databases are not the solution to all problems.
I agree with #56.
Was just thinking the opposite today. http://verni.wo...p-has-to-offer/
Surprise! Using a framework like Rails to do something as simple as Twitter didn’t work very well.
I agree with the poster who said this should be written in C. They could still use Rails to build out the frontend, or frankly anything else. The workhorses behind the scenes still needs to be something lower-level, faster, with greater opportunities for optimization.
This isn’t an indication of Rails’ failing. This is an indication of a team of people choosing the wrong tool for the job, plain and simple.
Look at Facebook. PHP frontend to a primarily C++ backend.
php or python is the way to go for content oriented sites. java is o.k but require development skills and more $.
“Readers tired of Twitter articles said to be abandoning TechCrunch.”
The Greatest Enemy of Knowledge is not Ignorance, but Illusion of Knowledge
– Stephen Hawking
That is what is happening here in these comments, everyone giving their opinion about how to scale twitter and what languages to use to make it better when they don’t even know what the real problem is.
Loren Feldman: Nice turban asshole. Welcome to the real world.
I doubt all those apps have the amount of database hits that twitter does. Maybe they should look at ironPython and go the .net route. If .net can power mySpace I am damn sure it can power twitter.
Why the change… old school hires… nuf said
“…We’re thrilled to announce two talented new team members. John Kalucki is an experienced distributed systems architect well versed in relational messaging as the former co-founder of San Francisco based SQLstream. Steve Jenson is a familiar face for a few of us here at Twitter HQ because he’s a former Google software engineer known for his work scaling Blogger and Blogspot–a service which tens of millions of people use on a regular basis…”
I cannot believe you idiots brought up Zed Freaking Shaw again.
Hasn’t that shrill halfwit gotten enough ink by now?
This is great…Finally Twitter surrender to PHP or Java…
Ruby On Rails—Write less, suffer more
Apparently this site can’t scale either.
That’s the 3rd connection timeout I’ve had in the past 10 minutes.
What’s wrong here?
Twitter can be done scalable in almost any platform. The only real problem with RoR is it’s relative lack of maturity, meaning any high-traffic site will have to find it’s own solutions. That learning process also includes regularly ending up with dead-end solutions and trying again. Not something everyone can afford.
And in general, the best strategy for a highly scalable site is ’small pieces, loosely joined’, leaving room for mixing and matching several technologies. This is why hanging a project like that on any monolithic framework (like Rails) is a not such a good idea.
The most likely solution for Twitter will be to step by step refactor pieces of the platform with better suited technologies, not completely ditching RoR.
Bad. fucking. idea.
@64 – your saying PHP and Python DON’T require any development skills?
Real professionals are technologically agnostic.
They pick the right tools for the right job. If it becomes apparent that they need to switch tools, then great, choose a better tool for your current scenario.
It’s not rocket science people, really…
Bashing a language or framework just highlights your ignorance.
You don’t have to use the same arrow every time, that’s what a quiver is for.
This is hilarious. What part of Rails, exactly, doesn’t scale?
Is it the database that doesn’t scale? The front-end web server? The IM connections? The db connection pool? The outgoing email servers? The API connections? The SMS gateway?
Rails is just a Ruby app. Which, in an application as simple as Twitter (big, but simple) is just like any other app. You can run as many instances of it as you like.
The scaling problems Twitter has are related to their data, the different gateways they connect to and the need to push new content instantly. PHP can’t send ten thousand updates per second over IM, SMS, email and HTML any faster than something written in Ruby.
You wanna stop messing about and run an Enterprise Java system on Sun UNIX. That’s what you wanna do.
Boy the FUD floodgates have really opened up on this one. Given how many different language and platform suggestions people have made on this thread, maybe there is one generalization we can make: it’s not the language’s fault. Scalability is not something inherent to a language. Your app is not going to scale because it’s written on Java, .NET, PHP, or C.
Also, many people on this thread appear to be confused as to the difference between scalability and performance. One language may have a linear performance benefit over another, but it does not make it any more scalable. Nobody is going to argue that C has better performance than Ruby. Because C is statically compiled and ruby is run time interpreted.
However rewriting twitter in C and keeping the same rough architecture is only going to allow them to reduce the # of servers by a linear factor. It is likely that there is something else wrong with their architecture on a conceptual level that is preventing easy scalability.
People, don’t you get it? This is just BS, Twitter is not giving up on Rails, at least not for the front end.
Wait and see who was right.
TechCrunch == FoxNews of the web.
-Matt
Using any sort of general framework for something at the scale of Twitter is insanity. It makes more sense to develop your own HTTP daemon and cut things down to the lowest level possible, especially for the backend / AJAX-y request side of things. Why get a big framework or bulky runtime in the way? It’s not so relevant now, but Feed Digest was architected to handle 1 billion requests a month on a single server (and made it to 500m at one point) following this low level model. It’s possible.
Twitter using Ruby on Rails is like NASA using Visual Basic. It’s nonsense.
Ruby On Rails – “Write less, suffer more.”
RoR losing a major account – it’s all mostly downhill from here for them, likely
@Loren Feldmen – have a more pleasant face, you look like shit
As much as I dislike Rails, and frameworks in general, if they have problems scaling it for something like Twitter, the problem isn’t Rails but their architecture.
The problem of scaling that kind of message distribution is “solved” and has been for many years. Heck, they could even do it by setting up a cluster of Jabber servers and a set of “dumb”, largely stateless frontends.
Facebook uses APC with PHP, works good and fast. Switch.
If the language would just be properly updated or provide a compiled solution or a better memory caching system everything would be fine. BUT NOOO. We have to have 100-200MB of memory for each rails app. lul.
For speed go for Django
OK we are just going to have to take predictions on this one. The market is opening at 60% that they announce they are dropping Rails by the end of July. Something makes me think that forecast is going down:
http://www.hubd..._July_6663/view
Wikipedia Architects: http://upload.w...-2005-04-12.png
Twitter is currently hiring (twitter.com/help/jobs). Check out some of qualities they want in their engineers:
- Experience with Ruby on Rails stack performance tuning
- Experience with JVM-based application stacks
- Code using primarily Java, Ruby, C/C++ and Scala
- Expert knowledge developing and debugging in Java and C/C++ on Unix
- Knowledge of python and ruby
I wouldn’t quite say they were “abandoning rails.”
@77. Of course you can add a new instance but surely it need money. The problem now is rails apps wasting a lot of resource. So, it is easy for your server suffer overload. PHP is not the only alternative as we have Java and many more.
Twitter is stupid and so is RoR fad.
Come on, twitter isnt exactly a complex bit of code. With a couple of million in the bank as well it’s not even a scaling problem. I see no one mentioning message queues which would be the heart of a decent messaging system. Anyone here actually used message queues?
It’s not the ‘framework’. that has nothing to do with scaling. it’s the way objects are marshaled. There is a different byte order for every persistent object in Active Record. Plus, the toy databases and the table structures, migrations (changes to the schemas) etc., all add to the problem.
People should read this to understand the problem: (There are better development stacks) http://www.avibryant.com/?p=16
I wonder if Apache ActiveMQ is part of the new mix (if not already)? That would be neat to read about in a soon-to-be high profile site that does heavy lifting service behind the scenes.
Twitter is dead.
@loren and others, plz share your computer science qualificaitons — graduating summa cum laude from egomaniac U doesn’t cut it
Ah, ad revenue down _again_ this week?
I can’t believe it.
If you want to know the real deal with Twitter, then why not ask them? I just asked two of their staffers and they said you’re full of crap. I can quote them on that and I’ll gladly put you in direct contact with them.
As for all of the asinine comments in this thread, please do learn something about web development, specifically these “superior alternatives” you’re suggesting. Saying Rails doesn’t scale isn’t fashionable anymore and it doesn’t make you look smart. I’ve been involved with massively scaling Rails on a number of sites and, though it’s been painful at times, it works.
As I said before: PHP went through this same trial. Java went through this same trial. Python is on the downside of this same trial. _Someone_ had to take it to a big enough scale before someone cared enough to make it scale that far. Twitter has done this for Rails: they’ve pushed it to the limits and taken it beyond them to new heights. They’ve motivated other people to do so too, as many of the new updates to Rails have greatly improved performance and scalability. As Twitter grows or another site surpasses it, we’ll see the same growth again. It just takes time.
@77.: I sort of agree with you. Twitter should be IO constrained, not limited by framework scaling issues that can’t be reasonably easily overcome. That said, scaling _Ruby_ to that level isn’t trivial – you do need to know far more about Ruby than the typical Rails developer will.
We learned that the hard way at Edgeio when we ran into “fun” Ruby quirks like the ridiculous performance hit you get with the combination of many sockets open and many threads running at the same time (solution: architect your app to use multiple processes instead), various memory leaks (yes, Ruby can leak like a sieve if you do the wrong things), lockups (the MySQL c-client lib can hang your Ruby script indefinitely if you don’t take care) and a number of other issues that you first start running into when you get to a certain size.
That said, we had a messaging middleware server written in Ruby, using multiplexing to handle IO that was mostly IO bound (the most we ever processed was around 2-3 million messages a day, and we never exceeded around 10% CPU usage on a single 2GHz Xeon core peak) that would’ve been trivial to scale by clustering them in a mesh, so it’s certainly doable.
As I said elsewhere, this isn’t a Rails issue, or a Ruby issue. Ruby is by far my preferred language, but if it’s easier for them to find people who can do it properly in another language than Ruby, then it might still be worth switching for them.
AOLserver (with TCL) FTW!