It’s sad. Every time I become kinda hooked to twitter and twitter regularly via IM, the gtalk twitter bot goes down for weeks and my twitter activity goes down to zero.

yes, twitter need to hire back the former chief architect Blaine Cook so he can watch the server and manually turn it on when the server goes down. It is 24 hours job and he must not blink when watching it. he he he

“The smartest thing Twitter could have done would be to hire former Chief Architect Blaine Cook back as a consultant to keep an eye on things for the day (he seems to be the only person that can keep his crazy architecture actually live).

Nice compliment (and backdoor apology?) aimed at someone who could use it after you hammered him in previous posts. Right thing to do, MA, and well done.

@mike - where’ the logic in your argument? It’s known the architecture behind this thing is borked and wasn’t Blaine the brain’s behind it? Your own referencing post says he failed at the scaling job.

How the heck does bringing him back help? And with due deference to your ‘experts’ we can all be armchair CTOs when we’re not in there with an architectural reference map in front of us.

I never have problems with Twitter, maybe your Internet connection is bad.

http://twitter.com/_p_j

I agree. In this case, under-promise and over-deliver would have been a better approach. Now, the BEST they can do is meet expectations.

if twitter goes down,
“at least they tried”

I’ll be busy trying to break Plurk.com

“We’ve moved much of the load off our database by utilizing more memcache”

I bet they just changed the key index memory size in my.ini in 5 secs with vim.
I also bet they just found out about it.
Adding 1Us in a SQL cluster is cake. It takes about 5 minutes.

What this reminds me of is people with absolutely no DBA training winging it trying to run a large scale website.

People go to college for this ya know. Others simply learn it.

People that do this on the fly are playing with disaster.
We are we all hand holding these developers?

They clearly aren’t ready to run this website. Why is it so important that Twitter succeed no matter what?
What if they fail. Why not just pimp them to death on round 2 of “try a venture”

This is BS.

mathew, it’s a backdoor trashing, not compliment.

I really hate seeing twitter having so much problems. They have a brilliant product and at this moment all eyes are turned to them.

It would probably be a good idea to get some funding and use huge part of to seriously optimize twitter code. The rest of it should be used for people/hardware to control their downtime and minimize it till they finish optimization or complete code rewrite.

Even a for a great product like twitter surviving that much downtime isn’t easy. And it definitely won’t last long if things don’t change soon.

Proof that Twitter is a tiny niche web service that no one outside of Silicon Valley cares about or even knows about: they “are expecting approximately ten times (their) normal daily traffic” due to the announcement of the second generation of a cell phone.

Seriously, why does this company keep getting covered?

Sorry about the windowz reference by the way. I’ve been through a lot this week. I of course meant my.cnf. The point is that these guys have way enough money to visit craigslist and place an ad for a DBA. This is BS, and furthermore, if they want to learn DBA’ing whilst running a service, don’t have your friends pimp it like the next Google. Get a clue Twitter.

Indeed, as the “anonymous source” says, why didn’t they do this earlier?

Twitter may become the next Friendster. Remember when?

Friendster was great, until it hit usage that the system could not cope with.

Then someone else came along.

Twitter is not rocket science. Someone else will come along if they don’t get the systems in order.

Shaun Dakin
StopPoliticalCalls.org

any good customer service always under promises and over delivers!

It seems like a pretty safe gamble; who’s really going to be twittering and watching Twitter for coverage? The 3 best sources (MacRumors, Gizmodo, and Engadget) do a much better job as liveblogs — including pictures in the stream. So why not claim a big event is a big threat (but they’re prepared) when in fact few will be using Twitter as a primary platform for the event’s coverage.

“Then someone else came along.”

http://www.freemyspace.com/history.htm

“Friendster was a very basic website so copying its core features was not a big technical hurdle.” - Brad Greenspan.

I have a brilliant idea. Why not do a twitter for BANDS, and get the support of the entertainment industry in LA???

Who’s game? We can use my office in Long Beach?
I’d drop what I am doing except for the SN script I am going to sell on hotscripts for equity, like a good amount of equity.

Nobody likes Twitter because like Friendster, it’s not cool because there’s no music or movies. Why not do twitter based on real bands and sell ring tones?

I’ll invest 10k right off the bat if there are interested parties and other investors.

TC comment on: Twitter Tempts Fate 

@Matthew/#3 - I was actually thinking that was another backdoor diss — note the name-calling of “his crazy architecture”.

Looks like I kept this window open a little too long before posting my comment (#21). Arrington admits it was a “backdoor trashing” - comment #10.

This tireless ranting about Twitter’s uptime is getting old. Additionally, Mike Arrington’s repeated attacks against Blaine Cook are unwarranted and, according to Twitter, incorrect. Like they say, its a team effort. I wonder how many startups out there would be able to handle the scaling. Mike is an economics major, not a hacker. He doesn’t have an intuitive feel for computers, programming or being a competent sysadmin. He’s standing here on his soapbox preaching about things he doesn’t know anything about. I don’t care how many startups you’ve worked for Mr. Arrington. If you couldn’t set up the slave by yourself in ten minutes then you need to let it go. Anything less is tantamount to you bullshitting us for ad impressions. And I am constantly surprised that you manage to fit that ego inside of your skull. A bit of humility would go a long ways to reducing the number of people who repeatedly come to tell you how much of a jerk you are on your own blog. Try being nice for a change.

If twitter keeps failing so often, they should seriously consider reengineering it from the ground up. The service itself is very simple, just a lot of database C-R-U-D.

@Chris

I’ll code it in Symfony

I don’t think it’ll matter in the end. If Twitter stays up, they’re heroes and everyone loves them. If Twitter crashes and burns, it’s the same situation as today, and everyone loves them.

I’m making a site that is above and beyond Twitter in every way. It is more exclusive and I have already beat them hands down in amount of uptime. I have not yet had the site up even ONCE!!

Try to beat THAT Twitter!!!

(Maybe I better not say that too loud since they are close to beating my record.)

Well, I for one am excited that some online publishers are using dedicated SMS to RSS services like Pingie.com to deliver information. Twitter has become an unreliable tool to deliver instant information. The time has come for dedicated services that specialize in a specific task to take over where Twitter has left off. Leave the Twittering to socialization, not news delivery.

Since when is adding some database nodes and tweaking your queries news!

Wow, Chris. Talk about armchair commentary. You’re basing you assumptions on an executive outline of what was done. You have no idea how their architecture is laid out. Your claims are moot until you’ve got some facts to back it up.

And if you think it’s so easy, how about you deal with a service that was getting 10,000 requests per second last time I heard (and that was over a year ago)? They’ve been trying to do two things at once: keep the current Twitter running and build an entirely new version from scratch. Given their limited resources, that’s a rather sizable task.

So, rather than claiming we’re all experts, let’s let them get back to fixing it and not have them waste time dealing with our bitching. The easiest way to make your voice heard is to simply stop using the service.

@25, Symfony works for me.
@27, Do you have the entertainment industry behind you?

That was the key to Greenspan quickly overtaking Friendster.

http://www.freemyspace.com/history.htm
“As eUniverse got well known in Los Angeles, and with Sony as one of my key investors, I was able to get better and better guys to come work for me.”

What about a 2-peat?

As a business owner in LA, I’ve met people from all over the technology side of the entertainment industry so far. I can’t mention them here or risk getting sued, but I am on good terms with most of them.

The problem is that I’m acquainted with the actual execs in the music industry. I could work on that though.

Read about what Brad did. If that sounds good to you, starting with the Twitter feature set and using bands and their fans to get signups, then get a hold of me.
A few of us together and 7-10 devs could probably pull a MySpace 2peat in the Twitter space.

I am rolling up a script to sell on hotscripts, but aside from that, I would be willing to drop our jobs site for this, if there are more investors.

@31 meant to say not acquainted with music execs. Again I could try to fix that.

This repeated negative posts regarding Twitter fascinates this new reader of techcrunch. Your coverage is remarkable for at least two reasons: first, it’s evidence that you’ve found software so compelling you’ll complain about it when it fails rather than just ditch it, and second that concepts of software-scalability and architecture are rising in importance and almost becoming mainstream.

This is an age of ubiquitous product concepts, and they are mostly tried and discarded without much care or worry. Nothing hurts an inventor more than apathy. Frankly, most web products are rather hum-drum, either derivative or not terribly useful, and if they faced outages most people wouldn’t even bother to complain - they would just vote with their feet. It seems to me that loud criticism shows more caring than silence, and you obviously care very much about Twitter!

Your posts also presage an age where individual programmers are being recognized (or blamed) for their contributions to important pieces of software. Too long, I think, have software products in particular been produced and consumed in an impersonal way. It’s unthinkable that 10 years ago any particular hacker would be recognized as the force behind any particular website, for good or ill. Now that’s changing, and I think it’s good. Some programmers may enjoy being anonymous, but I think many more do not, and if consumers would take the time to figure out who the people are behind the products they love, it can only improve the connection between producer and consumer, and so improve the product.

I would caution, though, that repeated public rebuke in a well-traveled blog are in fact dangerous to the poster, as such can quickly turn from constructive feedback, good for all, to mere peevishness, good for none. I hope that you can maintain that fickle balance in favor of the former end, Mr. Arrington, as I perceive you are dangerously close to veering toward the latter.

Now, Twitter is a little too forthcoming. It’s great to be honest and open, but sometimes it becomes TMI.

By the way, this is off-topic, but wasn’t the number of subscribers as reported by FeedBurner above somewhere around 850k a couple of weeks ago? It’s suddenly 766k right now. What happened? It was trending upward.

Enough already with twitter uptime. You have mistaken them for a piece of mission critical infrastructure. That’s on you. Right now they are a startup that has digested, what $5M in capital and has been in business for two years. You don’t pay them enough (oh, wait… anything) to run better.

Right now, at their stage, they get to be good at two of the following three things, whether they consciously recognize it or not:
*Grow
*Innovate
*Stabilize

Maybe they should have gone for stable, but then you might never have even noticed them, and they would have ended up in the dust bin. Yeah, they need to clean up, but stop whipping them like your life depends on it. It’s going to take time and that whole, new slug of capital, but they can probably get it straight.

I predict Twitter down by 10:37 am PT

@chris
“‘We’ve moved much of the load off our database by utilizing more memcache”
I bet they just changed the key index memory size in my.ini in 5 secs with vim.”

Memcache != MySQL.
Also, Blaine seems like an emacs man to me. I think it’s the hair.

“Adding 1Us in a SQL cluster is cake. It takes about 5 minutes.”

AFAIK, they’re using a master->slave replication solution, which is a little more difficult to just add nodes to, especially if you have no load balancing architecture in place.

why not use an alternative and stop complaining, I like http://www.noisewater.net

Thanks for the fascinating post, TwitterCrunch!

@19 : interesting, i’m working on a microblogging technology based on xmpp/pubsub, can we discuss your idea further ?

http://www.usshortcodes.com/csc_obtain_a_csc.html
FYI, the short code is 1k per month, and bulk SMS come out to about 3-5 cents a piece. Not sure how to implement sending ringtones and media yet, but that could be worked out in a developer meeting in LA.

If you’re interested in Twitter features for bands and film makers to instantly share media instead of just text, please contact me. I’ll match any reasonable investment, but would not be willing to draft a plan-vc for it. I would just want equity and to work on the code only.

We could whip up the DB schema and the skeleton with cake or symfony within a month with a decent sized team.

@42, sure. I will be back in Los Angeles this coming Saturday.
You can contact me at twitspaceproject@gmail.com in the meantime.

I can see VC jumping at this and I still have my contact at Mangrove from February.

Sorry, @43 was for @41

Looks like they’ve ceded WWDC tracking to Summize — great way to show you can withstand the load.

Odds are right at 50/50 of a crash LOL:

http://www.hubdub.com/m8594/Wi.....une_9th_08

With all efforts it shouldn’t go down.

If I was Apple, I would put my own money for Twitter extra servers for a week\month, so the massive ad campaign by users wouldn’t crash. 3 more servers would take about just $400 but would add thousands free user ads of their product.

@41, I am very familiar with jabber, but was not familiar with pubsub. Thanks for the heads up. I have not yet gone over the spec, but I have read the overview and it looks very appropriate.
xmpp.org/extensions/xep-0060.html#intro

I also went to http://sourceforge.net/projects/mod-pubsub
To check out the java server implementation. I think we could do the entire server in Java, and the UI with either my own or another framework pretty quickly. I’m actually very good doing RPC with soap and I programmed a jabber client way back so this spec shouldn’t be too hard to mod from the sourceforge example code.

If you contact me at that temporary twitspaceproject@gmail we could go over an initial plan and see about resources. The key is really the involvements of bands. IE, getting a marketer and setting up demos at concerts in the LA area. That was Greenspan’s deadly weapon. You have to throw parties ect… in densely populated areas like LA and get band involvement or there is nothing. It’s really not even about the tech.
The tech has to be good though. I think pubsub may do it if the current gen cells can subscribe. I think we could do it.

@41
1k is for a vanity shortcode, which aren’t quite ready for mass adoption and acceptance.

3-5 cents per SMS is utterly outrageous, shop around for your aggregator, you’re off by an order of magnitude.

if you don’t like a service don’t use it.
Its obvious a lot of people like/need twitter because they complain about it when its down