If the money isn’t going to hiring REAL talent and clearly isn’t going to hardware architecture, it must be lining someone’s pockets.

Tradition: hot discussion about Twitter

I thought you complained about Federated Media all the time, but it seems Twitter is still the favorite.

Best to focus on ways that actually may help them. This to me does not do it. More finger pointing. Would welcome a post on Twitter that was constructive.

Firstly, this reporting is hearsay. Oh.. I know a guy who knows a guy that says that Arrington is the long lost brother of Blaine and that Mike used to sysadmin at Twitter. What BS.. really people! don’t fall for flame bait. duuh.

Its really funny about all the people that are posting here, criticizing twitter and really have no idea how to scale a site.

If you *do* know to scale a site (and you’ve done it before.. not just read a bunch of blogs) you obviously don’t truly understand the problem domain that twitter has to solve.

It’s not just obtain a bunch of stuff from the database, cache it, load-balance it and serve it up.

Any reasonable competent sysadmin can do this. The guys there are NOT stupid.. There’s plenty of resources on the internet on how to scale. The problem isn’t as simple and trivial as you think it is.

I was at lunch with a group of guys (at railsconf) and blaine joined us. I wanted to go get the ’scoop’ of what the whole problem was, but I thought, just leave the guy alone.. there’s too much anger focused on him personally and it must suck.

Remember you have a choice you use twitter or not use twitter.

If you think you can do better go ahead.. TRY.. I don’t think any of you whiners have the balls for that one.

A twitter fan.

I love the weekends. Bravo!

Before you launch again into a dissection of why Blaine’s done such a bad job (which is a bit rich since most of the recent problems seem to have happened immediately after he left), perhaps it would be worth looking back at Ev’s previous success - Blogger - which was also pretty much unusable for great tracts of time as a direct consequence of not being able to scale effectively. I’m not implying it’s his fault, just pointing - again - to the fact that a lot of things can impact on the ability of a service to stay up and much of it is going to be connected to the amount of financial and engineering resource you’re prepared to put onto a problem, as well as how effectively you hire and what product decisions you make. One option - for example - on the product side would be to stop new registrations as a way to maintain the service’s ability to function for existing users. That’s not an engineering decision, that’s a product decision and is one that is clearly not being made. You really really need to get away from thinking that how Twitter is architected, scales and is supported is all at the door of one man - particularly when the evidence would tend to suggest that while that man was at the company, the service was substantially less flaky.

… and humanity? That’s pretty funny.

I doubt any of these armchair architects can even configure a mysql instance from scratch.

Hey Michael I have a few questions too?

How many master MySQL servers should twitter be using?

What total number of ‘physical database machines’ would you find an acceptable number for powering Twitter?

How would you propose Twitter architect a fully redundant database architecture whilst maintaining an acceptable transaction throughput?

How long will it be until you are able to move on from Twitter and start doing some investigative journalism again?

You know - looking at your graph again at the top of the page, another way to read it would be that Blaine was the only thing holding that company together…

Sorry, I thought you said

…undo the damage Cook has caused to Twitter and TO HUMANITY?

not community.

Tweetards

I don’t use twitter as I usually do work when I sit at my computer, but I must say the comments are fun to read.

I picture in my head a room full of nerds (rich nerds, but still nerds) swinging their arms trying to hit each other as they call each other names.

Leave the fighting to the UFC and move on folks…it’s just a web service..and it doesn’t work.

Go find a new toy.

Anybody know if twitter blocking yahoo pipes from getting feeds? I have set up a feed of my friends using pipes and it has been dead for last day. When I run the pipe directly from pipes instead of my site where pipes is embedded I get the following error:

warning error fetching http://twitter.com/statuses/fr.....ience.atom (401 Unauthorized)

any ideas?

Tweetards, indeed. This has been the funniest post in a long time.

C Thomas Edwards :

“looking at your graph again at the top of the page, another way to read it would be that Blaine was the only thing holding that company together…”

YES. but the problem is, Blaine himself built it. He built a platform that required his constant attention to stay live. and a platform that no one else can keep live.

http://www.webomatica.com/word.....-it-crowd/

God that’s funny.

Great that the industry leaders are addressing the issue.. good for Twitter, great for the community. Do Web2.0 a favor!

see update in post above.

“Twitter continues to be annoyingly and constructively responsive to criticism”

hahaha classic.

@117
A striking resemblance to every ones favourite Twitter punching bag!

I can’t believe they even allude to the Mars rover on their blog.

In their response, they’ve indicated an alternative filesystem-based approach. This is in fact a good and viable solution (in the lack of zillion servers available like Google), though not a long term solution. In any case, they’ve a new batch of $15M, don’t they?..

Our new architecture will move our reliance to a simple, elegant filesystem-based approach, rather than a collection of database.

So wait, all of the problems are database related now? What happened to Ruby on Rails? Which one is it?

I am not a real geek. I like cool things and try to keep up on tech but just so everyone here catches this. There is this thing in the universe called critical mass. Facebook, Google, Microsoft, and now twitter have all caught this. I was talking to one of my friends that just graduated high school about a week ago and I asked him if he had heard of twitter. He had just started using it within the last 2 weeks. he said it was a great way to have his friends join him for lunch without having to setup something “official”. So long has twitter does not have a complete meltdown it will continue to explode because it is hitting the main stream. Oh, and Arrington relax.

So they basically said that everything in this post is true, they’re patching it up for the short term, and re-architecting for the long term. Wow, they really took the fun out of this..

Arrington you redeemed yourself with the above video. You have relaxed good job.

The pics twitter uses are so nice and reassuring, quite frankly I don’t really mind when they are down or overloaded. If twitter becomes super-efficient and smooth running I shall miss the enforced twitter breaks.

But then, whilst I’m a twitter addict - I’m not twitter dependent as Michael is - that’s the real problem. Michael really needs twitter to function.

Aleph (128) means that he’s a tweetard.

Was this post ghost-written by Ted Dziuba?

It smacks of his brand of “I read an O’Reilly book and am now going to insult you.”

I’m pretty over all of these twitter outage and scoble accusation blog posts all over the internet. It’s like high-tech passing notes in middle school. Play nice.

The proof that this blog post is garbage is simple.

Replace the word “Twitter” with “MySpace” or “Rapidshare” or even “MSN” in the original blog post.

Now what? now what?

You wanna know something about the “big hit” Twitter that Mike Arrington won’t dare to show you.

http://www.alexa.com/data/deta.....ize=Medium

Twitter can’t even hold water against a YouTube CLONE. Not youtube itself, a CLONE of youtube that specializes in pr0n. What does that tell you about this new mega-site of the year?

michael, you seem to be obsessed with twitter and the supposed shortcomings of its architecture. lots of websites, from startups to established companies, have scalability problems. what the big deal about this one? i don’t feel the urge to post every time a page is slow to load or i encounter a 503.

Mike - the backlash of technology is everywhere. The other day I had my GPS fool on me at the Lincoln tunnel, right in the middle, and put me on to the wrong lane and then had to recalculate. I don’t think I can complain that to Garmin. Having said that I understand that Twitter should have made the checks and balances in place to not make this news. But again, it is still the same “technology” that has its own limitations.

Why always so mad at Blaine Cook? I saw you calling him out on Twitter last night or the night before too. I must have missed something but I’m pretty sure that one person can’t be to blame for all this.

I feel kind of bad for Twitter, they didn’t really plan for it to get this big, and I’m sure they’re glad it did but you can’t blame them too much for being unprepared. They just got some big funding and they’re working on making it better, what more do you want?

When you launch a Web service, this should be planned in your risk assessment plan… Why so much amateurism? I am speechless.

@Sarah Austin
I can’t stand FriendFeed. Tried it… Makes me think of Plaxo and all these wannabees. Hopefully some senior engineer will come and work @Twitter and fix this. Would be regretful to loose the mass currently on Twitter.

@Amanda: I never knew Dane Cook was Twattr’s architect.

But seriously, the amount of grief Arrington, Scoble and other no-talent ass clowns are pouring on him is pathetic and offensive. Whether he failed in his position or not that gives nobody license to personally attack him because every time Twattr fails it makes them throw a tantrum.

That Arrington is a giant blubbering baby doesn’t give him the right to trash Cook personally and professionally.

Honestly Michael, listen to yourself! You really think it’s the sole responsibility of the architect to keep the whole company going even after he’s left? You don’t think the CTO / CEO / Product lead or whatever have *any responsibility*? And you’re prepared to sat these things with no knowledge of the resources at his disposal, with no knowledge of how he was managed, with no knowledge of how free he was to make decisions or act upon them? With evidence that the head of the bloody company has had trouble scaling projects before and with substantial evidence that the product worked better when the architect was around?! I mean - I’m not saying that he’s not responsible, just that from outside how the hell can you tell!? You can put the best people in bad situations and they won’t be able to do good work. You can hire geniuses and then ignore them. You can have great people and then give them nothing to work with. Again - you cannot know this situation as well as you pretend to and in the process you’re basically eviscerating a man who - as far as I can tell - is utterly unable to defend himself without basically burning all of his own colleagues. From what I hear Twitter themselves don’t feel negatively towards Blaine! Having said that though it’s probably fairly helpful for them if you absolve them publically of any and all repsonsibility in this area in public over and over again.

If you want to give Twitter a hard time, having accepted that they provide an extremely entertaining service, i would suggest you do so in the round. Criticise the company for not having been able to scale, criticise the company for not being able to monetize. Or put the responsibility where it belongs - at the top - and let them redistribute it as they see fit. Because really you’re not going to be able to assign blame. You really can have no idea what’s going on behind closed doors. And until you can say confidently that Blaine got all the support he needed, you’re basically just mounting an unwarranted and sustained ad hominem attack. Which is - I’m afraid - indefensible.

http://www.alexa.com/data/deta.....ize=Medium

I just noticed something, any website that clones youtube and hosts pr0n will automatically beat the crap out of Twitter.

Even a 1 man website can beat the entire twitter team in traffic and views.

Isn’t that something.

news.google.com/news?source=ig&hl=en&rlz=&q=california%20pornography%20tax&um=1&ie=UTF-8&sa=N&tab=wn

You guys may not have noticed this but California is passing a tough new pornography tax. That is going to push people to the internet for free content more than ever before.

Again, it seems pretty easy to beat twitter in a traffic matchup, with a script, ffmpeg, linux and a dream.

Arrington, remind me again how much you are paying to use this service?

@47. Supposedly there is no such thing as a stupid question. That has now been proven false with your question. What possible motivation does Twitter have to be “enterprise class”? Are you really trying to lump Twitter in with the likes of SAP, DB2, Oracle, SQL, Domino, Exchange, etc? All of which charge their customers thousands of dollars to justify being enterprise class applications? How much does Twitter charge their users? What SLA’s have they guaranteed their users?

Arrington, do you get equally upset when a radio or TV station goes off the air for a lengthy period of time? Another service YOU DO NOT PAY FOR!

@141: Last time his cable service went off for ten minutes Arrington shot the TV screen with his silver .22 handgun, after pulling it from his pink purse.

Erickson Smith #39

You nailed it. The problem is one of an exponential explosion of writes to the db that scales to the size of the community of listeners to a particular sender, which as I understand it is not limited by twitter accounts. The database setup they have is quite robust for normal high transaction applications. Unfortunately, those exponential write explosions is simultaneously twitters strength, it allows the quick reach out to the community of listeners) but it is also the weakness , all those simultaneous sends kill the db fast. What twitter needs is a way to slow down the propagation of new tweets *naturally* by setting up , distribution points architecturally.

I have a way to do this that I’ll be glad to share with twitter …for ehem…a fee. *grin*

“I have a way to do this that I’ll be glad to share with twitter …for ehem…a fee. *grin*”

Dude, totally get a clue. They’d turn to craigslist.org 80×10000power times before taking you up on an offer. We have teh best coders in the world here.
I’m so bored and drunk right now. Sorry for the sprinkle of reality.
I see startups like Twitter every single week in LA. I’ve toured the buildings, worn the id badges.

They’re focusing on this one for a reason, and I’m not sure why. I can tell you that they’re nothing special as far as this area goes.

if this is only de DB issue, SSD disks can help to feel the gap… but I doubt that this is the main issue.

If I may:

I am a non-tech-industry geek who enjoys staying ahead of the curve, on the bleeding edge, and generally keeping up with what’s going on in tech-land.

I discovered TechCrunch a few months ago and have enjoyed reading (most of) the stories covered here. It helps me stay informed.

But — I do not use Twitter. I find the occasional story about them interesting (’new release’, ‘raised money’, etc.) - kinda like everything else on here.

This obsessive level of coverage that the company’s been getting recently is absolutely mind-numbing, and frankly it makes me less want to read this site every day. I REALLY do not care about the low-level scalability problems they’re having or who screwed up what or who did what to whom in the bathroom with the candlestick.

So as a mild-mannered, average non-tech-industry “regular guy”, please hear me when I beg you to STOP MAKING EVERY THIRD POST ABOUT F’ING TWITTER!!!!!

That is all.

Noah - just skip the Twitter posts, read what interests you. Or bail. We’ll both be fine either way.

I’ll keep writing whatever the hell I feel like writing, you keep reading what interests you, wherever that is on the Internet, and things will work out.

I don’t know, I don’t know.
Comments are fan though at twitter posts.
But it’s obvious that twitter posts drive traffic here… whatever.

By the way I like question 3. For what it seems twitter’s architecture was
not meant to sustain such load, that is one thing no one can argue.

Yea. NOT HELPING!

Yeah, it’s the architecture not the implementation technology. It depends on the type of hardware too. There are single servers hosting 200′000 domains…