The first Twitpocalypse was one of those events that you’re going to tell your children about one day. I remember where I was when it hit: On my way to Napa Valley with some friends as we heard sirens race by, likely signaling the end of the Twitter world as we knew it.
Okay, it didn’t end up being that bad. But it still was a pain in the ass for many third-party developers, especially the iPhone Twitter app developers, who had to wait in the App Store line like everyone else for their fixes to go through. And now it’s set to happen all over again!
A tweet today from the TwitterAPI account warned that the second Twitpocalypse was closer than people thought. Okay, most people probably didn’t realize that it could occur again, but it can, and it will, and it’s fast approaching.
The current estimate by the API team is that it will occur sometime in the next 60 days, probably at the end of September. They warn that it could happen sooner though.
So what does it mean, and why is it happening? Well, for those that don’t remember, the first Twitpocalypse occurred when the unique identifier for tweets hit 2,147,483,647 — the 32-bit signed integer limit. That number caused some third-party apps to start counting tweet identifiers as negative, screwing them up. This new Twitpocalpyse is similar, only it’s for the 32-bit unsigned integer value of 4,294,967,295.
In case you didn’t already realize it, the fact that this chasm between the two numbers was crossed so quickly once again shows that Twitter is growing very quickly. Though it’s not a 1 to 1 tweet-to-unique ID ratio, that the number will have doubled (an increase of over 2 billion) in just a few months is huge.
So what can you do to prepare yourself? Well the Twitter API team recommends developing your apps to use 64-bit integers, thus increasingly the number of tweets your app can recognize before it hits these integer walls.
Also, lock up the women and children.
Twitter ha to be one of the most incompetent, heavily venture funded companies. Constant blackouts (expected and unexpected) for maintainence, and apparently an inability to plan ahead. After the first one, a move to a 64 bit integer was obvious.
Yes, they handle a lot of traffic, but it’s not uncharted territory. Trading systems have been handling huge bulks of realtime data for decades. Perhaps what Twitter really needs is some expertise from the financial industry to help them build something both scalable and reliable.
You’re confused.
Twitter is not the one having a problem with these numbers.
It’s external Twitter API users (like desktop clients and websites). Those developers, not Twitter’s, often have not thought through their database design well enough and didn’t allocate enough space for the IDs.
If they had chosen to use uuids or similar instead of an integer, then API developers wouldn’t have any trouble and Twitpocalypse wouldn’t happen.
UUIDs for something this high volume doesn’t make financial sense. The CPU cost to generate them and the storage costs to save them are significant when you multiply by billions.
Twitter themselves do not have a problem, they’ve been using unsigned 64 bit integers for a long time, long before the first twitpocolypse. The Twitpocolypse is related to third party applications and how they store the ids.
APIs transmit the integer in XML or JSON as simple text, it’s up to the programmer using that API to store it how they see fit. The original problem is that some people were storing it as signed integers which ran out several months ago if you were going that way.
Still others, and arguably probably a lot more, stored them as unsigned 32bit integers (the default in a lot of databases), and that’s the limit we are going to hit in T-60 days and counting (or less, estimates are based on tweet volume which changes constantly).
If you go 64-bit unsigned (skipping 64-bit signed because there’s no point), you get up to
18,446,744,073,709,551,615 unique ids, which will hopefully stave off twitpocolypse 3 (or technically twitpocolypse 4 as those idiots who go 64-bit signed will run out after the statusid reaches 9,223,372,036,854,775,807) for quite some time.
If we run out of 64 bit unsigned integers in a short period of time, twitter is growing too fast
but we’ll probably end up going to some sort of alphanumeric ids.
For proof that they have been using unsigned 64 bit integers since at least the first twitpocolypse, see this tweet from @twitterapi:
http://twitter....atus/2048659057
And you know what?
Some people might even care. Or not.
Who knows?
“Also, lock up the women and children.” lol
Substitute “Jesus Christ” for “The Twitpocalypse” and you have yourself a pillar of Christian doctine in a tweet
Quick, someone calculate how long until the tweet ID reaches the maximum capacity of a 64-bit signed integer!
9,223,372,036,854,775,807 isn’t it?
357,913,941 years at the current rate of tweeting
Are you factoring in the rate of acceleration or just using the current rate?
It took from March, 2006 to June 12, 2009 to go from 1 to 2,147,483,647
From June 12, 2009 to today it went from that number to: 3,060,265,346
So while it took 3 years and 3 and a half months to get up to 2 billion, it only took 49 days to get 912,781,699 (very nearly a billion) more, increasing the number by 42.5%
So simply checking the current rate to see how long it will take to get to the 64-bit signed integer level is insufficient estimate.
Curve modeling shows us breaking the 64-bit limit sometime in March of the year 52,888.
i was not being serious.
your models should also incorporate expected saturation of growth and the plateauing of the growth curve as well as the finite size of the internet. If John Adams’ estimation about year 52,888 holds true, then i think we should all prepare. I ‘m aready building a treehouse
Just in time for my 50,906 and a half birthday.
I bet thet in your model, Twitter users grown like this, http://geekandp...iness-plan.html , don’t they?
Or Twitter is full of spam. You pick.
http://twitter....uses/3059678686
Twitter recommends you place a bandaid on your sucking chest wound.
why would anyone build anything around this pile of steaming ickyness.
what a fricking joke.
I love the smell of failwhale in the morning, smells like victory.
http://www.yout...h?v=bPXVGQnJm0w
very nice.
++$theHorror
i will embed
where’s the surfing scene?
http://en.wikip...rg/wiki/128-bit
I cringe at the thought of Tweet #: 340,282,366,920,938,463,463,374,607,431,768,211,455
I’d like to see the casting details here (32-bit int -> 64-bit int; Ev -> Marlon Brando; Robert Duvall -> Biz?)
I see what you did here.
What’s up with the word “Redux”? Seems like it’s being used a lot lately. I hate the word.
clearly you skipped apocalypse now redux.
This seemed blatantly obvious when it happened the first time that they would need to go to 64-bit. Maybe they just went to unsigned to buy a little bit more time. This shouldn’t be much of a surprise to any of the Twitter developers out there. Hopefully most of them thought ahead and made the change to 64-bit last time around.
Any developer who didn’t already do this based on the first “twitpocalypse” should not be a developer.
PS: Am I the only who can’t stand all these made up “twit” words?
Yes.
It all twepends on your twastes.
Too many twits and twats…
We have just updated the http://www.twitpocalypse.com website to track the Twitpocalypse Redux event. Enjoy
I would like Twitter to call out those apps that are using the ridiculous 32-bit integers. That way we can make fun of their developers.
That number caused some third-party apps to start counting tweet solarhotwatersystem.net identifiers as negative, screwing them up. This new Twitpocalpyse is similar,