We received a number of tips early this morning that the majority of web servers at Twitter was exposing server and load-balancer status information to the public. The status page, which are an (often default) option in the open source Apache web server dump an output of all connections and state information for a particular server. The information is used by administrators to monitor servers, and the pages are often either removed entirely or locked down to prevent the information from being used for nefarious purposes.
At some point in the past 24 hours (I would more accurately guess 22 hours 28 minutes and 4 seconds ago, based on the status page itself), the Twitter web servers introduced a misconfiguration to expose this information to the public. The page includes overall server statistics along with every HTTP requests currently being handled by that server, with the full request URL. The server status page is usually accessed by requesting /server-status for a web server. In the case of Twitter, this exposure allows anybody to see requests that sometimes rely on being secret to remain secure, such as oAuth keys, which are used to authorize applications to access Twitter accounts.
News of the pages being open spread quickly through Twitter, with some calling it “great transparency” while others recognizing it for what it is – a little too much transparency, and unintentional. Twitter were very quick to respond and blocked all access to the page, and the vast majority of the information found is purely informational and can be deduced through other means. Your Twitter account is probably safe again, but that doesn’t mean we can’t geek out while we get a sneak peak at what Twitter looks like behind the curtain.
Screenshot of one such page below with some of the information cut out.
Too much Bing and Google?
seems to me that only!
Did Twitter actually say they did this for transparency? Did they make any official statement on the matter?
No – it was just a conclusion that some came to when the URL was being re-tweeted.
Most likely is that with the server reboot from 23h ago this was somehow introduced, a few people noticed and sent it out. Not sure if anybody emailed them (should have) or if they found out by seeing the Tweets
Why, then, is the title “Twitter: YOU say ….”?
Big security risk, but not bad stats.
Why the heck are they using this resource hogggg called apache?!
what’s another good alternative?
lighttpd
Nginx, lighttpd
nginx +1
They are using apache + passenger, although passenger does have an nginx module, so im not sure why they’re still on apache
My thoughts exactly. Apache?
I switched to nginx early this year after using apache for about a decade- and love it. It’s a fantastic web server and blazingly fast.
Never cared to seek out what Twitter was using to serve pages, but I was still surprised to see Apache.
Definitely too transparent. Maybe these two new deals with Google and Bing will give Twitter enough money to actually deal with this vulnerability and the uptime problems.
People love to point out what a security “hole” the Apache server status page is, and yet there is no good reason for it to be called a hole.
The source IP in this case is clearly a proxy server (10. internal IP), and all you can see if the URL of a GET request anyway. More sensitive POST data isn’t show).
Know that this particular node is using 312 workers is not exactly a security hole.
To me it raises more interesting questions:
- Why is each node only serving up to a max of ~300 clients
- Why apache??? Better tools have been out and are solid (nginx, lighttpd)
Depending on their apache modules and what specs each server instance has that may be all the memory they have available. Apache processes can use upwards of 25M each.
We’ve found running more low-powered web-tier servers works better for us [disclaimer: not Twitter] than fewer high-powered ones.
Looks like they’re using almost all of their capacity on that server though. Not good news for traffic spikes.
ok, open it on your servers then and paste the URL here
I’m pretty sure those pages belong to haproxy, not Apache. That url doesn’t exist out the box on an Apache server
Uber nerds, Apache is the Swiss-Army knife of the web.
Sure, lighttpd and nginx have their places, but there is nothing on the planet that Apache can’t do.
Alternatively, lighttpd and nginx are very basic web servers.
You guys may as well say run YAWS / Erlang which by the way would kick the snot out of everything in terms of performance and redundancy.
Apache is the bomb. The cat’s meow. It can absolutely do everything and anything.
Omelets folks. Just Omelets.
They may have just installed Cacti. Our server-admins recently set that up for one of projects, initially the URL is publically accesible then you can lock it out. Cacti has some nice real-time monitoring features. Makes sense for Twitter.
lol @ all the “apache? get outta heeeeer” didn’t (if not still does) facebook utilize apache?
Any other well know sites that display this data i.e. server-status page. The only one I could find was http://www.apac...g/server-status.
I believe that Twitter used http://www.ntt.com/ in 2008 for their hosting infrastructure. I dont know who or what they use now, but I believe they do not host their service in house.
This would imply that the slip was caused by their server host.
They used to use Joyent as well, http://www.joye...-joyent-update/
But this is a mod you have to explicitly enable in httpd.conf and I’m sure that even if they don’t own their boxes they don’t let people play around with their config files. That could be disastrous.
wow that does not look so great… over 300 workers at only 111 req /sec? 12 idle workers? i know rails takes it’s sweet-ass time to answer a request, but you’re running a little hot there…
Twitter should use nginx. Apache is like Microsoft Word – 1,000 options, but you need about 4.
nginx is nowhere near as bloated.