Twitter is living up to its promise to open up its data stream as much as possible to developers. While I was negotiating with Twitter cofounder Evan Williams to sit down and do a video interview at Foo Camp last weekend, Gnip founder Eric Marcoullier was hitting him up to give Gnip, and therefore everyone, Twitter’s XMPP “firehose.” Williams was obviously in a good mood, because I got my interview and, as I just found out today, Eric got his data feed.
What does this mean for the average Twitter user? It means that more third party services will start to work better. Today, other than a handful of services like Summize (which was just acquired by Twitter) and Friendfeed, third party apps must talk to Twitter via their normal APIs. Those APIs require applications to send Twitter a request and then get a response. The two way communication creates a big load on Twitter in the aggregate.
With XMPP Twitter just sends out all of their data in a constant stream, whether you ask for it or not. The third party, in this case Gnip, takes the data and parses it for further use.
Gnip acts as an intermediary between applications that create social content and those that consume it. They take the Twitter feed, which is a list of usernames, Twitter status URLs and time stamps, and make it available to any third party that requests it. Both Plaxo and MyBlogLog are already using the new feed, and more partners will add it immediately. And every third party that takes data from Gnip doesn’t have to take it from Twitter, easing the overall load on Twitter’s servers.
For now Gnip is only sending updates for requested users, not the richer data that some applications like Twhirl need to build a Twitter-like desktop environment. Twitter may give Gnip permission to send additional data, like @replies and direct messages, over time (if that last sentence doesn’t mean anything to you, it means you aren’t a crazy-heavy Twitter user, just disregard it).
What this means is that Twitter is taking yet another step towards openness and leaning on outside parties to help them with scaling issues.
Battle Over: Twitter Open Up To Gnip. Read more at TechcrunchIT >>
Its about time that Twitter has done something like this. Lets hope it really is faster
It’ll be interesting to see how the Twitter execs pondered about how open should they be. Every company has that on their minds these days. Facebook with the OpenID has some issues too. I think that even if it’ll offer some relieve on their servers, Twitter is replaceable, and as this article mentions: history tends to repeat itself. What history? Friendster, etc… first mover: first leaver.
Way cool. Do you think if they have already talked about GTalk integration?
We also have a simple REST-based API that let’s you host a list of users you care about at Gnip and we’ll push notifications to you in real-time. Full info is in our API doc (http://docs.goo...hvp8s_3hhwdmdfb).
Any idea how far away @replies integration might be? This is great news for my current project as Gnip alleviates the need for rate limiting/checking on my part…heading over now to sign up!
And you did explain it to me so I can understand… Thanks TechCrunch and thanks twitter for moving forward and not falling down anymore.
Just need to extend beyond user focus now!
On the roadmap
Eric, will you be beta testing extended featureset in the near future? With specific partners? Hint?
Way to go twitter! Now twitter users will see their tweets show up more quickly in sites like Plaxo Pulse, and it will lessen the load sites like Plaxo put on twitter to get those updates–a clear win win, and a great demonstration of the power of open standards and putting users in control of their data!
I agree, they need to go beyond the user focus.
I was sure Friendfeed was one of the few sites that had access to the XMPP feed, but whatever.
Way to read the article dipshit.
You’re right, I misread it, I fucked up. Bit late where it is where I am when I read it. But yeah, fuck you too mate.
Great development. Congrat, Gnip, and way-to-go, Twitter!
This is really good news for Twitter fans.
Gnip’s service is just the breather Twitter needs. Congrats on the feed guys and keep up the good work.
Could Twitter be the future backbone of the net!?!
Used as a life journal shared with each other, form of therapy, way to exchange money (@bankofamerica send $50 to @myfriend), get medical advice, research, the next Google, the possibilities are endless!
Side note – Would someone please make a Gmail Twitter app, if it does not exist already! Read stream and Twitter within Gmail!
Looks like techcrunch totally focused onto twitter for some reason. Move on…people doesn’t like you keep covering twitter twitter twitter all the time!
I second the Gmail Twitter app! Thanks for explaining all this good news.
Yo Eric,
I’m a little confused after reading the API docs. When I consume the twitter stream I don’t get the tweet itself but an url to the update html?
Any chance for a more direct and complete update – including the 140 chars?
Heya Jim – Rev one of Gnip is a notification server that lets applications know *when* a relevant user updates. It’s then up to the application to grab the information. In the near future we’ll be pushing full metadata alongside the notification. We’re currently working with a number of services to ensure that we do it in a way that supports everyone involved (data producers and consumers alike).
Sure – I can appreciate the ecosystem dynamics. Keep us in the loop as you evolve this. We’ve got got our eye on you!
Never understood what all the fuss about the ‘firehose’ was about. I asked for it a few months ago and was given it within a day. It was a huge amount of data to parse/process, and I ran in to some problems with how OpenFire server deals with logs, but I had full access to everything. For free. Just by asking. Twitter doesn’t have a signup screen, for example, but they seemed to be open about it when I asked, and I’m a relative nobody in the web world (not a friendfeed-size service, I mean). Ultimately, I ended up abandoning that particular project, and shut off receiving the feed. I wonder if I turned my openfire back on if I’d still get the feed again???
They cut everyone off of tat xmpp feed shortly after you abandoned yours. Only 4 companies have had access to the xmpp feeds over that past few months…
Thanks for the heads up – I wasn’t aware they’d cut people off. Seemed a bit strange that they just gave it to me in the first place. What’s stranger though is why they don’t just monetize access to that in the first place. Data is ‘open’ but hooking up to the is $x. Others could broker the data on a as needed basis (perhaps I only want access to a set of people, or terms, or whatever), and I’d pay less through those 2nd-level access people.
then what is the problem with twitturly? if twitter is opening the flood gates they must have it pointing to certain companies.
They opened the flood gates to gnip, which still doesn’t offer the full xmpp feed to other, just a subset of it. Twitturly will hopefully come back online soon, Biz Stone as well as some of their developers have said that we would gain access again, but they are just taking a while to do it… Hopefully sometime this weekend it happens…
Good Night!
That’s is indeed great news, hopefully Twitter will end up with fewer of those pesky whale pages (over capacity) – and yes, I would absolutely love to see some gtalk/gmail integration.
I wonder if the trend is heading toward outsourcing this part of the business. In this way a data provider can better deal with unpredictable load on its servers. It seems too expensive to maintain an “elastic” hosting service per each Web 2.0 service. I wrote about it in here. Gnip is leading the way. I think that they are onto something real.
Outsourcing Data Producer’s open API development and support – a new business opportunity?