Over the last few days a number of popular bloggers have complained, loudly, that it’s time to ditch Twitter and move to a decentralized version of the service that won’t go down every time usage spikes. Generally services like Twitter, once they reach a critical mass of users, can’t really be stopped because the network effect kicks in as a massive barrier to entry. But one aspect of Twitter – it’s openness – could also be its Achilles heel.
Scott Hanselman mused on how such a system might work yesterday. Dave Winer has also written extensively about this, although he’s more focused on simply backing up Twitter messages so that they are still available when the site goes down. He isn’t paying attention to the real benefit of Twitter – the spontaneous, asynchronous conversations that pop up between every changing groups of people. Instead, he just wants to make sure the data is secure.
Marc Canter also weighs in in a surprisingly lucid post. Infrastructure needs to be decentralized to be reliable, he says, pointing to DNS as an example.
Decentralizing Twitter isn’t about having backups of content if the service goes down. It’s about making sure that the service as a whole can’t go down, and allowing all those conversations to continue unabated no matter how popular the service gets.
Here’s How To Decentralize Twitter
Chris Saad, co-founder of DataPortability and founder of startup Faraday Media, thinks he knows one way Twitter can be decentralized effectively.
The key weakness in Twitter (and therefore opportunity for a new decentralized approach) is the fact that so much Twitter activity occurs off Twitter.com. Users are getting very used to using desktop clients (Twitterific, Alert Thingy, Twhirl, etc.), IM, SMS, and other interfaces to talk to Twitter. Those third party applications can be tuned to lock in to the new decentralized Twitter-like service instead or in addition to Twitter itself.
Users on the new system will post to a microblog. Third parties can create platforms for these blogs, and have them be certified as compliant with the microblogging standard – posts of 140 characters, no titles, etc. Users could also install compliant software on their own servers – much as they do with Wordpress.org today. There would certainly be an open source project around this shortly.
The hard part is putting these microblog posts together into a Twitter-like conversation where people subscribe to those writers they like, can respond via an “@[username]” mechanism, etc.
This can’t be done efficiently just via RSS because rapid and excessive polling would bring servers to a halt. Instead, Saad thinks wrapping RSS in XMPP, an open standards based instant messaging protocol that was originally created for Jabber and is now used in various applications including Google Talk, is the answer. XMPP allows for pushing of messages to subscribers, which removes the need for constant polling. For more of Saad’s thinking, see his site on their product SyncStream, and they’ve already written code that will do this based on their proposed standard called “GetPingd.” Twitter uses XMPP in their API already; third party applications like Google Talk integrate with Twitter via XMPP already.
If users begin to “twitter” using this new system, applications like Alert Thingy can simply add it to their functionality. Instead of using Alert Thingy to sign into just Twitter, you would also create an account at Alerty Thingy, too, which stores your subscription lists. This is analogous to what feed readers like Google Reader do today with RSS subscriptions.
XMPP already has a mechanism for tracking subscribers which performs the same funtion as the “followers” list in Twitter. Users can therefore have a list and count of followers back on their microblog.
Handling replies is a little more complicated (the @[username] feature on Twitter). One way to do this is to use the existing RSS infrastructure specifically with Google or Technorati Blog search to monitor for @replies and feed this to the user applicaitons. Results wouldn’t be limited to just microblogs, but I’m not sure that matters. It would also be simple to block spammy stuff by simply clicking on a button, the same way Twitter works today.
The net effect of this theoretical platform is to move the publishing to a completely decentralized network, and move the hard part of Twitter into the third party aggregators (which is already a competitive space). There would be no one central bottleneck that could fail.
Twitter (the company, the service, the site or the software) isn’t part of this new decentralized platform, of course, so they’ll oppose it. But can it happen? Absolutely, because it doesn’t require a hard reset to a new Twitter-free world. Existing Twitter clients could add support for GetPingd and the rest of the infrastructure and it would work seamlessly with the existing Twitter world. Anyone could create a website that duplicates what Twitter does today, but supporting the new decentralized framework. And a limitless number of microblogging applications could emerge to join in the fun.
And we’d never have to deal with outages again.
twitter is NOT going to like this post.
I think it’d be great to offer decentralized twitter, but who are the major players going to be? Who wants to foot the bill? Does everyone run their own server and hook up to the decentralized system or do we end up with 3 twitters instead of one?
Who holds all the tweets ever made so we have a history? Does every server get every tweet and store it away or does twitter still end up as the canonical storage platform?
How does track work? Do I have to jump from site to site to see everyone’s profile or can I still just go to twitter and check them out?
I feel that by moving away from a centralized twitter we gain reliability but lose ease of use. Will we go in this direction? Yes. Who will be the big winner? Whoever can answer all these questions and has the easiest to use product.
What about SMS updates? That’s one reason I love Twitter.
Nick – the idea is to turn the notion of “twittering” into the same system that handles “blogging” – no one owns it, no one is a bottleneck. A limitless number of companies compete using a set of agreed upon standards.
Nice article!
Metcalfe’s Law has kicked in big time. People are jumping on the twitter bandwagon at a rapid rate.
I just don’t see people moving off twitter despite the reliability issues. Once people use it and “get it” they are hooked on this newly evolving way of communication and listening.
The third party services built off of the twitter API is impressive and growing rapidly. Some are crap, some are really cool. Good ones like brightkite and twitpic are quite interesting.
Decentralizing is an interesting concept in theory and not without significant technological challenges of it’s own. The jury is out.
Cheers!
This is exactly why google or microsoft or yahoo or aol or facebook should buy Twitter now. It is going to take a massive effort to scale twitter yet there is an opportunity now for someone to cement it as a de facto standard. Would anyone mind if Google for instance created the decentralized twitter envisioned in this post? But do it starting with twitter vs copying it.
Forget about raising vc money for twitter. Go partner up now.
Sure, open twitter makes a lot of theoretical sense but still needs an owner to make it happen. Who could build it from scratch and commit to it? Mozilla?
The solution exists and has existed since the early 1990’s. It’s called IRC.
I’ve added a few more comments on my own blog – but most of my input made it into the post
http://chrissaa...ing-twitter-20/
Excellent read Mike! Your wrap up is making the entry I’ve worked on during the weekend a bit obsolete so I’ll have to re-write it, but this one is great.
Alex
I agree with Shawn – what about SMS and offering a route directly to and from mobiles? That can’t be readily decentralised with an obvious way to cover up for the costs of SMSing. And bear in mind that SMS (not using mobiles to access the web) is probably the way to reach (even more of) the masses.
The decentralized approach you advocate comes at a cost: increased complexity, increased development effort, and increased running costs. Accordingly, in response to Twitter’s unreliablity it is more likely centralized competitors emerge and one or two of them succeed. Given Twitter’s problems and potential, Twitter 2.0 will probably choose technologies that will allow it to be more fault tolerant in high traffic conditions.
I like Twitter! Most likely won’t be moving off it anytime soon especially with the desktop app Twhirl that makes Tweeting so much easier.
http://twitter..../Daniel_Richard
I haven’t faced any downtime by Twitter yet, most likely cos’ I’m one of those who signed up when Twhirl was released, and when I got an app from Facebook that allows status to be updated and sync between the two.
can we haz decentralized flickr, fireeagle and facebook just at the same time, please ? O_o
i don’t think recreating another twitter is the answer. problem could be in code or infrastructure. just throw money into it to scale up.
there are lots of twitter-clones: pownce, jaiku. people use them as well, though differently but twitter’s got a bit of community already.
twitter actually reminds me about irc but web2.0-ish and geared towards our 21st century mobile lifestyle.
guess it’s time for twitter to find a business model or be bought by a google or google-like entity that can feed it money.
Oh wow, we’re progressing by essentially remaking AOL Instant Messenger.
“One way to do this is to use the existing RSS infrastructure specifically with Google or Technorati Blog search to monitor for @replies and feed this to the user applicaitons.”
in a word, centralise
Centralisation DOES provide some benefits. For example how reliable are these server ? not from uptime, but from ownership. if I see a message today from x saying “@y suks …” I can reasonably reliably say that IS @x saying it.’ Depending on the network links between various servers under you scenario, we may not hear from someoen in a timely fashion ofn a particular conversation. is this because they don’t care, because they’re thinking, or because their twitter-server is currently stuffed (or owned)?
Perhaps implement Open-ID ? Even then we will have shelisrael.com BS all over the place again, if people don’t claim their names fast enough.
There still needs to be a central point that ‘authorises’ and verifies the twitter-servers. Like DNS, it can be buffered at various hierachial levels, but again this means someone / somebody has to be responsible for the secuirty of this data, AND the propagation of this data. Who pays ? users via subscription ? Server owners via subscription ?
Somebody mentioned this already, but what about the SMS part? That’s a pretty big part of the usefulness of Twitter for many is using SMS. At the moment Twitter covers the substantial cost of all the SMS notifications that people get, I don’t think that’s going to be an aspect that can be recreated so easily, basically as it doesn’t come free.
It’s definitely interesting times for Twitter though, I can’t really see how they can monetise the service. Blog posts like this just show how much twitter has become part of the infrastructure, it’s more of a protocol really, and as the majority of it;’s usage is via the API I don’t think ads are going to be the solution.
Very interested to hear Twitter’s response.
It’s much simpler than that – Simply go peer-to-peer (P2P). Liberating centralized networks since 1999
@Dimitris An SMS service would be easy to run on top of the architecture described in the post.
@Daniel Richard you would still use this imaginary new system via Twhirl or something very much like it.
@cocoy The Twitter ‘clones’ you mentioned are not clones – they are very different to twitter in many fundamental and important ways.
This sounds a great idea, by decentralizing the service it will allow it to operate if there is an outage in any part of the world, something that affects us in Asia nearly every day.
I really cannot see this happening with twitter though, I have been waiting for a while now to see what twitter rolls out to actually make them money, yeah they have received a lot of funding recently, but how are the money men going to rake back their cash on a decentralized system?
The most obvious way to bring in the dollars is through advertising and merchandise tie-ins; the merchandise I am sure they will make money on, but how are they going to rake in the advertising cash?
Or are they running a charity
In case anyone’s interested, last month I wrote a little about the inevitability of Twitter becoming decentralized, and what that means for Twitter’s business model (or lack thereof): http://benmyles...m/post/33195820.
I read blog posts and comments about twitter’s scalability problems for a while now. What i keep wondering is how come no one is looking at the traditional ways the software industry is scaling large scale data distribution issues. Look at stock exchanges, brokers and financial institutions: Data is very volatile and needs to be distributed to massive numbers of subscribers, while at the same time it needs to be captured no matter what the load or connectivity state of the application is. The answer is Message Oriented Middleware (MOM). Of course traditional MOM technologies are not designed for the internet era as they provide HTTP/client APIs by using servlet like technologies which break end-2-end security as well as guaranteed delivery at the web server layer. There are solutions though that are designed for the internet such as our own Nirvana product, providing a built in HTTP/S stack and comet/ajax client APIs. For more details take a look at http://www.my-channels.com
Many do use Twitter in an IRC manner (or Jabber for advancements), and except for the SMS part it already is there.
But what is sms really about? When I am on the road in Germany, I do use the mobile interface and not sms to communicate from and to Twitter, only when I have no data access it goes sms – and even then only occassionally.
The one I really do use is dm for direct messages and that is basically because I may not have all your phone numbers. It would be something I might be willing to pay for, while at the same time I’d want better control of who can send me those sms I pay for.
That set aside – why not go decentralized? It would finally allow for decent clients while at the same time allow for the idea to grow.
I explained two months ago how to build a decentralized Twitter with ejabberd Instant Messaging server.
The article is here:
http://www.proc...ication_server/
sorry I strongly disagree 50% of users are still using twitter.com according to rww research.
http://www.tech...ated-heres-how/
I commented this on the last post do you read your comments :p
I find it an interesting problem to solve as most hackers do, The @ just needs a system to point a url of a user. The bigger problem is conflicting usernames on different system.
1 final note I have is what are these problems people are having with twitter being up? I have not had any down time in a while and I use the .com for my client.
@chrissaad, yep. i know… they’re all used and act differently from twitter. people use twitter differently as well. as a point of clarification i use the term “clone” very loosely.
…twitteriffic has ads on its twitter feed. i don’t mind having those, especially if twitter’s getting its fair share.
…DNS… isn’t exactly decentralized. there are “root” servers see wiki for details: http://en.wikip...works_in_theory
what i think people want is to get twitter as a service to be as reliable as dns, not exactly whether or not twitter is decentralized. it would be interesting to know what twitter thinks.
opps wrong URL for 27
http://www.read...nitive_list.php
yes. i agree with the first commenter. ciao from romania.
I don’t know why you would want to wrap RSS which is already a wrapper. How about XMPP wrapping a set of microformats? The formatting bits can be filtered out when the messages get translated to connectors like SMS.
Open standards and Decentralization FTW. The “twittering” and “blogging” analogy mike gave makes it easy to understand. Same can be said for emailing. No one owns the ability to email.
One success of Twitter is SMS deliveries.. which could be expensive and not readily decentralizable (ie., not because of tech reasons)… might be when SMS becomes free… that said, a decentralized twitter without sms would still be very useful.
Great article and completely possible. All hard work (listing replies and updating followers) will be handle by google and/or technorati. And everyone offering services like sms, send as voicemail/email , will earn money form it, it will never be down as long as internet is not down
, every blog can post to twitter similar to generating rss or have it as widget, sideblog.
Only problem it needs to be done in 1 step hassle free process. Setting up an account and starting using it needs to be 1 click solution regardless of having host. Also it is better if all accounts are opened from one location to gain popularity to the service. otherwise it may die if everyone will refer to different service to signup. Unless term twitter is coined and added to international dictionary similar to term blog.
Does twitter have any patents relating to what they have developed to date to “protect” themselves from a decentralisation approach?
eg if they had a patent for the use of @xyz or the 140 character payload then at least they could keep themselves in the picture during the development of this infrastructure.
>> you would also create an account
Oh great – yet another account somewhere. Here was the perfect opportunity to talk OpenID – yet.
That’s a non-starter for me. The idea of services having to store their own tweets and then move them over later makes me envision those OWN services having their own scaling problems. It’s like solving a problem by simply spreading it out.
We can do all these or twitter can just upgrade their servers. Keeping it simple is very important in everything we do.
Chris touches on the future of user owned content. Liberated service agents servicing personal requests. In this case, I’d configure my agent to serve up my tweets and alert for others. Unfortunately, the decentralized model changes the profit paradigm as well. Twitter.com’s profit potential lies in creating this messaging backbone
http://www.tech-exposed.com
So what are the odds that this will happen? When could we expect change??
So ‘twittering/tweeting’ becomes like ‘blogging’, and anyone can run their own service. Services will compete for tweeters, and we’ll see a handful of platforms that handle a huge majority of traffic. This will become another form of centralization, and will have the same uptime problems we have now, just slightly less impactful. Yes, anyone can set up a blog, but a handful of service providers – blogger, wordpress, livejournal, etc – provide blog service for a large percentage of blogs out there.
twitter likes to be centralized – why else would the vc give them money?
Chris has just restated what I’ve been sayimg, calling for a bootstrap, the first step of which is to get the data stored off twitter.com.
nice linking to everything chris saad! i guess you get that as a house guest of casa arrington?
Michael, If you’re suggesting we move to a de-centralised twitter, then we need a de-centralised name for the initiative. Twitter is a brand mark of a commercial company.
Lots of wasted bandwidth if this thing has to communicate with millions of separate microblogs at the same time, rather then stemming from one place, no?
I think a decentralized Twitter (or Micro-blogging platform) could work very well if there’s a standard and if it’s easy enough to operate. Remember, not everyone is tech savvy and many people probably don’t want the trouble to operate their own “twitter” server.
By the way, I’m curious to see what Google will do with its Jaiku acquisition…
To clarify a few differences:
IRC, XMPP-based chat, SMS, microblogging, these are all, in essence, different ways to get one thing going: post a message so one or more users can read and may reply to it. But how they differ is in the intention you have when using the service.
In IRC and similar chat services, your intention is to have an immediate response, a ‘conversation’ with one or more other users.
In SMS, you can always expect a delay in response to your post, or have no response at all.
In microblogging (in essence, just like blogging), your intention is to post a message to the masses without any direct response (in other words, no conversation), but users can comment on your post (comment, @reply, etc).
Having that in mind:
Twitter was developed more as a microblogging service than anything else, even though it now has more of a conversation feel to it.
Chat services don’t usually store you conversations, generally speaking. They connect people and pass along messages. Whereas blogging and microblogging, all messages (posts) are stored on some server. For a decentralized microblogging service to work, all posts and replies would need to be semantically formated so they can be fetched with ease from any server and any similar product. Only then can the service be used by different products, but can still ‘plug into’ other similar products.
@coycoy – Please stop talking about something you know nothing about by backing it up with a wikipedia article.
I need Decentralized Twitter ( like WordPress)!
Interesting proposal, but some things don’t really work.
Replacing RSS with RSS wrapped in XMPP for example. Sounds like a great way to cut down when you have only a handful to share with, but this can get complicated. The advantage of RSS is that anyone can poll at any frequency provided they know the location and have permission (otherwise 403). With a push, if there are a ton of “subscribers”, you have issues:
- Delay – last on the list gets it after the first.
- Touch to cache – You can’t easily cache a push. You need to push for each user. This is really inefficient. RSS is cacheable because it’s a get. Both because you can leave a static file on a server, and because agents (think Google Reader) can make 1 GET for 1000’s of subscribers. If you push, that’s thousands of extra push’s. Just a giant waste.
I’d suggest WordPress is really on track to conquer this. There’s even feeds for comments already. The question is how to aggregate this into something useful. If you monitor 10 people, it’s no big deal, but if you follow hundreds, your going to need a few servers to manage all those feeds.
It’s AIM {seesmic_video:{”url_thumbnail”:{”value”:”http://t.seesmic.com/thumbnail/2v5QlQqrE9_th1.jpg”}”title”:{”value”:”It’s AIM ”}”videoUri”:{”value”:”http://www.seesmic.com/video/syo4J39HWE”}}}