Hey Twitter I Have A Few Questions Too
by Michael Arrington on May 31, 2008

Lately Twitter has been cleaning house, raising money, doing interviews and actually talking to users. In a blog post last week they did a Q&A session, directly answering questions about Twitter’s architecture.

So I have a couple of questions, too, based on a couple of discussions I’ve had with people who say they’ve seen Twitter’s architecture.

  • Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
  • Do you really have a grand total of three physical database machines that are POWERING ALL OF TWITTER?
  • Is it true that the only way you can keep Twitter alive is to have somebody sit there and watch it constantly, and then manually switch databases over and re-build when one of the slaves fail?
  • Is that why most of your major outages can be traced to periods of time when former Chief Architect/server watcher Blaine Cook wasn’t there to sit and monitor the system?
  • Given the record-beating outages Twitter saw in May after Cook was dismissed, is anyone there capable of keeping Twitter live?
  • How long will it be until you are able to undo the damage Cook has caused to Twitter and the community?

Update: Twitter continues to be annoyingly and constructively responsive to criticism. They respond to this post here, saying “We’re working on a better architecture.” Kind of takes the air out of the balloon when you can’t get them riled up.

Trackback URL

Comments

Comments Pages: « 1 [2] 3 4 » Show All

Twitter, Twitter, Twitter. Seriously need change.

 

I wouldn’t be shocked if Twitter answered you. Like you said, the one thing they are doing well these days is communicating with their users.

 

Twitter doesn’t need to fix and improve their architecture at the moment. The community is so big its too late for that. Now they need to create a complete new framework that can handle major load.

 

Oh my F*ing god that is why the down times during middle of months I say in their down report looked damn familiar as a pattern..

This is DBA incompetence on the highest order if all conclusions hold and I am sure they will

BTW, that is why they cannot handle track its Database intensive..basically the SQL query to do track approaches infinity thus queries have to be more than cached they have to be systematically optimized programmatically to get away from approaching infinity..

 

Your questions are funny Mike,

It would be like asking you, Mike Arrington.

Is it true that TechCrunch and all the crunch sites are actually a GPL word press install on a single server with a custom html template?

Is it true that Techcruch can’t handle so much as bad javascript as Seesmic pulled it down the other day?

This line of questioning is mundane and irrelevant to most people

As a technical expert and software programmer, even I do not see the point of them.

Twitter, aka ANY SITE is a package or product. Either it works or it doesn’t. Why would you want to go in and act like their sys admin?

If it were any site but twitter, I gather you would say it sucks in other words and that would be the end of it.

It’s almost like you and Scoble are trying to mold this site into your next Facebook or something. Trying to steer it in whatever direction you want.
I think that’s a little disturbing. I don’t think you would do this type of amateur technical consulting for any other website. It actually makes me question the relationship you have with them as well as Scoble’s.

 

@Angry Fan:
Um, yes, ejabberd, and most other OSS XMPP servers support mysql, postgresql, as well as possibly the other commerial RDBMS. Are you sure you know anything about them?

 

@55: Chris, I too question Arrington’s relationship with Scoble. I think it’s amazing that two narcissists can actually have sex with each other. Doesn’t that go against their prime urge of exalting their own selves above all else?

Plus, that’s one porn you don’t want to see. Though maybe you want to get a twattr from them when they start doing the monster mash together.

 

Well done Michael.

A handful of emotive words and you have 50 comments and counting.

 

@56: Mr. Church, it may be your dyslexia, but I said IM applications of scale. Not some pissant OSS XMPP douchebaggery.

Lordy, you’ve got problems with reading comprehension.

 

Heh, seems by the time I had read the comments you picked up another seven ;)

 

Is that why most of your major outages can be traced to periods of time when former Chief Architect/server watcher Blaine Cook was there to sit and monitor the system?

Bing! I think you finally figured it out (or you’re getting good advice from a smart techie).

When did Twitter start getting unreliable? When Cook and Lee Mighdoll left. I don’t know which one was keeping it running or if both of them were, but the current situation has all the signs of people fumbling around trying to figure out which dial to turn when this or that happens.

This was a good post. The previous one was way off (and I said so).

Keep on truckin Mikey.

 

@Angry Fan:
I’m not sure how dyslexia would come into play? Maybe you could explain?

Many corporate and govenrment IT dept. are running customized versions of OSS XMPP servers. What are they using for DBs? What do you think google is using for a db? Multi-value dbs? I don’t claim to know, you do.

So, quit being a douchebag and tell us all how data storage is done on the “IM applications of scale” you seem to know everything about.

 

BTW, why don’t you write a post like this for your friend who runs Mahalo. You’re good at Emperor Has No Clothes pieces. We both know JC is running around buck nekkid. Why give him a free pass??

 

Michael,
I didn’t read the comments, but they should pony up and do some sort of co-sponsorship with Microsoft and throw the DBs onto MSSQL and run the servers on IIS. IronRuby is coming along nicely, although the RoR app might want to be upgraded to a more robust and tested framework/server OS, even if it isn’t IIS 7 and .Net

 

Does this mean CrunchBase can’t scale?

 

Ha ha, “damage to the community.”

You ever notice when the author of the post gets up on his high-horse that the posts follow suit. Everybody’s an expert! I offered pro-bono for bragging rights and they responded with NIH syndrome! RoR suxxxx0rz!!1

Hey, pile on everybody. Arrington, isn’t your training supposed to be in economics? Why not second guess their spending of the 15mil instead of blathering about IT? It makes you sound like Gillmor without the marijuana.

 

@Ericson Smith: you’ve said everything I was about to say.

No language “magically” recovers from database slave failures, so why would ruby?

I’m not sure what’s wrong with Michael Arrington recently, but his quality has defiantly gone down hill, every other post is about twitter.

 

I have a few questions too.

* Does your customer contract with Twitter state ‘zero downtime’? Oh wait…it’s a free service, no contract.

* What damage to Twitter and ‘the community’ are you talking about? They continue to grow rapidly and you continue to use it like a crack addict.

* Why do you continue to personally attack one of the guys who built it? Without his hard work it wouldn’t exist.

 
One guy's perspective - May 31st, 2008 at 6:27 pm PDT

Funny, but a bit harsh?

 
 

OMG - Michael Arrington talking tech - mentioning things like MySQL replication and stuff. Doesn’t everyone know he doesn’t know what the f*ck he’s talking about!!! He’s a goddamn lawyer and a pseudo writer hack. We’re at 14 minutes now. Time is almost up.

 

Arrington talks a good game, but I think that’s all he’s really capable of doing. I doubt he has even the first clue as to what any of this stuff means.

 

@57,

I am in no way implying that Mr. Arrington and Mr. Scoble are having gay sex together. I am simply saying that I question their ties to Twitter.

Former attorney turned journalist turned amateur system administrator and DBA.

Jack of all trades, or somebody who is a little too interested in Twitter? This is not really a story in my humble opinion.

 

@Dave Winer: Well, let’s not get carried away. Arrington has been pushing Twitter with its lack of business model for two years, Facebook shits bricks of gold in his eyes, Mahalo is the bestest search engine evarr, and Yahoo MUST SELL now (each post collecting some M$$$, I guess).

The source of his critique isn’t to be a serious tech blogger, it has more to do with his tendency to throw tantrums like an overgrown baby. You should see what he does when he runs out of cigars. He’s like that giant baby-thing in Spirited Away.

 

their problem are understandable. ruby, mysql are only suitable for content oriented sites. they just made a mistake of using it instead of something more solid.

 

Time to move on from the regular twitter bashing - you get what you pay for and at the end of the day its a free service.

 

ha, they don’t even have a dba position open :)

 

@Greg: Wrong, just because a service is free does not mean that they have any less of a responsibility to their users. They are building a business, make no mistake about it.

 

Hey, here’s to redefining the word “Tweetard”.

Tweetard: “A person who constantly bashes the Twitter service, yet can’t stop using it.”

Hey, i’ll give an iPod nano to the first person who can verifiably get this up in urbandictionary.com

Contact me through my URL above!!

 
 

Way to go armchair journalist. Feel good beating this dead horse? Instead of running your mouth at Twitter constantly, actually do something in technology. Have you ever dealt with the technical difficulties in something the scale of Twitter? Yeah, didn’t think so. Try it yourself, then you’ll have some respect for the people who fuel the service you spam daily for your personal marketing message. Tool.

 

What really impresses me about Michael is that he can keep up (and/or lead) in so many realms.

And that includes technical discussions.

It’s amazing.

Now (and not that he needs a defender in any way) but if Michael was a bit nicer to Scoble in print (sometimes he is nice) perhaps Robert would take him along on such interviews (and not that Michael doesn’t lead as well with his own interviews — and scoops - leading ability)

 

Michael, why go on a personal Jihad againt Cook? Also, do you evidence that he was dismissed????? (maybe you’ve already published that, I don’t know).

 

How long will it be until you are able to undo the damage Cook has caused to Twitter and the community?

I stopped reading your post too soon — that’s totally unfair.

 

@Ericson Smith - LOL - this one wins ;)

 

I’m pretty confident in saying that Arrington has never once set foot inside any type of MySQL database, let alone setup any type replication/failover situation. There are much larger sites running on much less hardware, you don’t have a clue as to what you are talking about.

 

I’ve just set up TechCrunch Sans Arrington, a piped feed containing TechCrunch’s articles minus Michael Arrington’s painful diatribes. If posts like this one make your brain hurt as much as mine, be my guest and subscribe.

 
 

read comment 28 by bob wyman - and that’s all you need to read.

oh, look at the # of comments here - mission accomplished for the author.

 

@Ericson Smith: you win. “Tweetard” is more effective then my favorite replacement, twats twatting messages.

 

Only in SF would the overwhelming response to an article rightfully calling bullshit on a PR push to deflect attention from fundamental incompetency be “awww. Why you got to be so mean?”

Honest and unmerciful Arrington. Keep putting the feet to the fire.

 
Please make it stop - May 31st, 2008 at 7:50 pm PDT

Michael, like many, I am a TechCrunch addict but over the last few weeks I’ve become twittered out and I DON’T EVEN USE IT. I have never seen ANY company get as much (obsessive) coverage. You’d think Twitter was the most important thing to come along since Jesus himself.

 

This is the funniest TC post I have ever read.

 

You know what’s so funny Michael, you know sheeit about architecture let alone databases, well may be from a 1 million feet.

PUT YOUR MONEY WHERE YOUR MOUTH IS and contribute to Twitters CapX. Because right now you sound like a little girlie man.

BTW, question for you. Since your cred is worth zero amongst tech’s, who do you think will work with let along for you now??? Find the problem and fix it. Don’t blame people.

You’re probably just ticked because Scoble out scooped your egotistical self ;-)

There you got one more page view - go get a doggy snack

 

WOW!! I think we all totally forgot that you have sooo much IT experience and background and that you weren’t a lawyer at all and have tons of technical training. Right?

Dude, a little tip since you clearly can’t see beyond the 5 feet surounding your desk but NO ONE USES TWITTER OUTSIDE OF THE UBER GEEK/TECH COMMUNITY!! IT will never get mainstream adoption because…you know..we have these things.. called CELL PHONES, yeah it’s cool, they let you talk to people WITHOUT the Internet!! and they also have tese things, you might have heard of them, called TEXT MESSAGES that allow you to send short messages to a group of people with 160 characters! W.o.w.

You must be so smug, just sitting there in your chair looking like shit (we’ve seen your seesmic videos) and bashing other people’s hard work. I guess it’s soo easy to do when you don’t have the balls to try yourself huh? Oh, don’t get me wrong you may invest in various companies and call yourself a “founder” but you surely don’t have the pleasure of running any of them and dealing with their day to day issues huh?

And your job board company was a huge success, wasn’t it? Who burns thru MILLIONS with no significant revenue? I mean come on, no one is that stuipd. Looks like you should stick to “reporting” and leave the real jobs to the people who know what their doing.

After all, those who can’t do “teach” right? And your out to teach us all a big lesson about what everyone is doing wrong and how they suck, etc, etc, etc. Douch bag. You couldn’t even cut it as your own CEO, how lame.

So I tell you what, how about you do something significant, like run a startup that tries to develop a new business or technology from the ground up and then you get the right to tell other “owners” if they are doing a good job or not. Your so stupid you would have panned Google back in 1998 or Yahoo back in 1994 because it wasn’t XYZ enough.

Go screw yourself, you are NOT relevent to the world at large and you are NOT a journalist because you only write about companies that your VC cohorts tell you to write about. Way to go on independent thinking there! But I guess if you were a worth while person you wouldn’t be 38 and single huh?

Someone seriously needs to knock you down a peg, like a stupid HS girl who thinks she’s all that…the bigger they are the harder they fall.

 

Manny, Twitter doesn’t need Mike’s money - watch the video. They have the money. They need more experience and more help, along with a new architecture, which could take them months to develop.

 
How to help Twitter - May 31st, 2008 at 8:19 pm PDT

OK, your a great developer and have experience that can help twitter scale and you want to help.

The secret is to shut up! The founding developers know they are over their heads and are hanging on to their jobs with their teeth. Do you think they want someone to come in who can clean up the mess and make them look as inept as they are?

Tweak your resume so you sound like a mid level developer who can help them solve the problem and make them look like the savers of the system!

Once your in then start fixing the system (remember to document your solutions) and remember to send firendly notes about what you found and one or two recommended fixes to the project lead and your manager.

I’ve worked at many companies that the managers wanted someone who could solve the problems but the interviewing developers were deathly afraid of someone who had more experience than they did.

 

Blaine Cook watching twitter server every second. When it failed, he will manually switch the cable from one server to another.

Wow, it is the future of server computing. The new approach in solving scability. ;)

 

I’m giving these guys a break. Sure there’s plenty of work to do and in this business they can ill afford to have the downtime and negative press they’ve experienced lately, but I’m never quick to judge. Unless you’re on the ground of a company, there’s no way of truly understanding the challenges within. Business culture, management, and investor relations sometime bleed into the tech requirements and resources.

We have NO IDEA what was being asked of these guys or what kind of pressures they’ve been under in the past year or so. I know from experience that when the term “monetization” begins to enter the discussion, tech is usually an afterthought…at least on the minds of the business folk. They do need to get their act together, but I sympathize greatly.

 

@95: Right on. A storm in a teacup. Or a tornado in a bucket of Arrington’s urine.

 

Comments Pages: « 1 [2] 3 4 » Show All

Leave Comment

« Back to text comment

Commenting Options

Enter your personal information to the left, or sign in with your Facebook account by clicking the button below.

Alternatively, you can create an avatar that will appear whenever you leave a comment on a Gravatar-enabled blog.