On the same day that new Comscore numbers came out indicating new traffic highs for YouTube, the site went down for six hours this morning in the first unplanned outage since launching in February of 2005. Though users were told that new features were in the works, press inquiries have confirmed that it was actually a database failure that took the site down. Periodic planned downtimes at night, US time, are common but today marked the first major service failure for the site.
The biggest question on the table has always been about YouTube’s business model; today’s outage makes you wonder whether the company will be able to scale. YouTube announced that it was serving more than 100 million videos per day as of last month.
Update: YouTube’s problems today are now all over the web, getting loads of mainstream media coverage. Comscore is getting huge press today too, since the traffic report came out on the same as the failure. Freak PR windfall for both companies, unless YouTube continues to have service problems that is.
I really do wonder what their business model is. I don’t see many ads on the site and I don’t see premium accounts. There is no way the ads alone can pay for that bandwidth bill.
Any idea as to how many database servers went down? Just the main node or something?
Cody Mays
http://www.threadbound.com
Last I checked, they were using MySQL for the DB.
I don’t see their business model neither. No ads, no premium accounts as cody has said. Maybe, they can sell some data, or estadistics about the people who use it. But I think You Tube is a bottomless well. (is correct that in english? has it sense?)
Business model? Today (or maybe yesterday) there was a Slashdot post (http://slashdot...15/148257.shtml) about YouTubes competitors. Some of them have revenue sharing … and thats the way to do it. Where is my incentive to use YouTube over other services? As I see it now, there are barely any Ads on YouTube … maybe that is the incentive, but eventually they are going to change this somehow, becuase they cant keep up with growing storage and bandwidth needs if they dont. Once that happens.. then the only incentive that I can see is gone. Bring in the next wave of video sharing services who share their revenue with me! I think that is a nice incentive.
“. . .wonder whether the company will be able to scale”
Do you mean economically? If they are serving 100 million video’s in a day, I think they’ve proven they can scale technically.
Do you think they lack funding to keep up with their demands?
I would have thought PornoTube would have gone down first… sorry had to be said.
What do you mean Scale?? Do you have any clue what it takes to server 100 million videos per day? I think they have proven they can manage their infrastructure and scalability.
These types of outages like they had today are simply some growing pains.
Fair enough. We’ll see.
Growing pains…. I’m impressed they haven’t crashed and burned already but then again they have the $ to pay for good consultants to help with DB issues.
Kevin
How? How? How? I mean come on YouTube. Biggest video site on the net and don’t have a disaster recovery plan? Horrible.
Sorry, but I’ve seen Youtube down a couple of times before with the same error message. So no, it’s not the “first time.”
when?
100 million videos per day is astounding.
Who’s saying it was a db issue? Could have been a Limelight problem, no?
Er, giants like Amazon and EBay have had similar outages, even when older and richer than YouTube. Even Google search had a brief outage May last year. Anyone reading bad portents into this needs a chew toy, not a keyboard.
we should not care business model of youtube. I think he will get great success. BTW, the video sharing needs creative. http://www.biku.com and http://www.jumpcut.com both have unique online video editing technoly to promote the original video
It’s irrelevant what the server was hosting. You have hardware pain, you’re supposed to have another server around to fail over to. That’s how you prevent outages from lasting a long time.
Brightcove:
CREATIVE SPAMMERRRR
Paul Tuckfield to the rescue!
Are you really going to question the future of YouTube based on one database failure? As someone mentioned earlier, even Google has unplanned outages from time to time. It just happens, it doesn’t mean they can’t scale to meet demand. Yes, their business model is quite hazy, but nevertheless, creating speculation about this is quite harsh in my view. Plus, it hasn’t had any unplanned downtime since it launched, that’s hardly bad going now is it, really?
Not bad, except for 6 hours in the middle of the busiest day online of the week. Goog has had outages, but I don’t remember the one that lasted 6 hours! Lol , yeah Ed I agree though that things will probably be ok.
I don’t understand why I keep coming to this blog. I’m hoping more of these come out and create some kind of brand name for themselves with sane reporting. Do you realize what it takes to support 100+ million video views a day…and that too for a company of size 30-40 emps? I’m surprised they didn’t crash before. If they do use mysql then what I can say is that it is NOT the most reliable db out there and is not intended to support such crazy traffic. You really have to get a lot out of the db for this type of traffic. Anyway, just my thoughts.
Mr. Donaldson et al – yes, point taken that 100m videos demonstrates scalability but jesus christ does 6 hours of downtime at peak hours not also demonstrate a hint of a problem? Likewise, as note worthy events in the Web 2.0 startup world go, I’d say this makes the mark.
@Sam Donaldson, who said, “If they do use mysql then what I can say is that it is NOT the most reliable db out there and is not intended to support such crazy traffic”, I would only say that you should check your facts, mate. MySQL is running most of the top Web 2.0 sites on the Internet, as well as a large majority of the highest-hit sites in the world, including Wikipedia, Slashdot, LiveJournal, Feedburner, mixi.jp (MySpace of Japan), Technorati, Flickr, Craigslist, Yahoo! Finance, Google Adsense, del.icio.us, digg, and the list goes on and on.
Gimme a break.
Raises the question as to whether YouTube can scale? What? Huh? Are you serious on this? It’s one of the 20 most frequently visited sites in the U.S. serving more than 100 million videos a day…not search results, but videos. Not scale? What? Huh?
Call me a bloody fool…but that does seem like scale to me.
hum…
1) the company is barely 18 months old. Growing problems are natural. And they offer their service for free, hence their SLA would allow them every now and then a little outage. I think they’ve done tremendously well with their limited cash, unmber of employees, and time to build all of it so quick.
2) now that they have the audience, how difficult would it be for youtube to start doing revshare ? or splash ads a little more ? and in addition, they could go into a myspace-google kind of deal in a snap to monetize the traffic.
100m videos served a day… that’s 3b a month !
Search for ‘ebay outage’ and ‘amazon outage’ and you’ll see longer outages from these sites even in years when they were (1) public companies; (2) with hundreds of employees; and (3) profitable.
Seriously, lots of TechCrunch’s coverage has the perspective and attention span of a hyper puppy. Good for spawning long discussion threads. But not very credible.
Does anyone know how many servers they are running to manage that kind of load?
I’m pretty impressed that they have been able to scale thus far without a hitch and 6 hrs of downtime is really nothing considering their growth.
I’m not a data admin, but I know there is so much backend work in scaling hardware/database efficiently and they’ve done a great job thus far.
Outages will happen, but YouTube really doesn’t have a viable business model. It’s far to expensive to run and I don’t think ads (banner or video) will cover the operational costs.
An abomination.
100 million videos a day = 400+ terabytes (assuming 5 MB/video) a day!
OMG.
“Where is my incentive to use YouTube over other services?”
Superior ease-of-use and quality of playback than competitors.
“MySQL is running most of the top Web 2.0 sites on the Internet, as well as a large majority of the highest-hit sites in the world, including Wikipedia, Slashdot, LiveJournal, Feedburner, mixi.jp (MySpace of Japan), Technorati, Flickr, Craigslist, Yahoo! Finance, Google Adsense, del.icio.us, digg, and the list goes on and on.
Gimme a break.”
Have you witnessed the trouble the Wikimedia sysadmins go through to maintain their MySQL servers? I have, and to say they’re pushing the envelope is an understatement. Lots and lots of pain.
All the other sites you list are tiny compared to the big boys. The *very* largest sites on the net DON’T run MySQL; indeed there are superior databases out there.
You don’t see YouTube’s business model? You idiot’s just don’t “get it.”
YouTube Explained:
http://www.revv...ideo/41613/3331
The Guy Who Will Save YouTube:
http://www.revv...ideo/39028/3331
Marshal,
“press inquiries have confirmed that it was actually a database failure that took the site down”
Press enquiries have confirmed “a database-related issue”. I suggest that you ask YouTube what they currently think the cause was. The difference is just where the words imply that the failure happened. A broken network cable, for example, would be database-related but not database server software.
Dan100, forgive the directness here: you’re talking rubbish.
Yes, I do know how much pain is and isn’t involved at Wikipedia and _why_ it’s pain. MySQL is rarely the _cause_. Power outages, high load, lots of servers so some fail, not as much redundancy as would be nice (funding limited) etc. are the usual reason. MySQL itself basically just motors on delivering results. Everything is painful once you get large enough – there are so many things to tune and monitor and break that everything requires regular attention and tuning and maintaining and hassle. You’re probably not allowing for this “big is painful for everything” factor.
As for the very largest: how about 250 billion transaction a quarter for Overture? Or Google Adsense and AdWords? Or Yahoo? Sorry, but the *very* largest sites on the net are using MySQL successfully. It’s so routine it might even be a good guideline to how to take over the world: “start with MySQL and the rest of the LAMP stack”. You can do it in other ways but this is the best beaten track to the top.
First DBA, Wikipedia.
Support Engineer, MySQL AB
You tybe is down right now, i didn´t thoght they were prepare for this, as i know to may users can´t take down a server, i will apear slow but do not stop serving. I don´t know, it seams that they´re having problems attendig the request or some haking problem. Anyway this start bothering me, i need people to see my videos…
sorry about my english…
muy buena la pajina ajaj
Every Website has issues, as they are all dependent on technology to run properly. As for them not having the revenue to scale their business model, that is a load of crap. I make anywhere between $1-$10 per 1000 PageViews on Adsense. So with YouTube’s 100,000,000 pageviews daily, that means they are making $100,000 per day roughly. they can afford the bandwidth plus pimp rides for the staff.
Youtube is down? Oh shit i didn’t notice ._. No seriously…. It works fine for me ._.
good