Speculation was rampant the last few weeks that Google had to rely on a third party content delivery network to make the YouTube Live live concert stream properly at scale. Despite the fact that Google has it’s own quite impressive CDN, streaming live video (as opposed to progressive downloads, which YouTube has historically relied on) is hard stuff. And expensive – you have to license Adobe’s Flash Media Server, or a competitor like Wowza, and pay at least a couple of cents per gigabyte transferred on top of normal costs.
We’d heard rumors that Google had partnered with one of the big three live streaming services – Mogulus, Ustream or Justin.TV. And in fact Google has met with all of those startups to discuss partnerships or an outright acquisition.
But instead of working with them, or building their own streaming media CDN, they chose to work with Akamai. Google won’t confirm this, but it’s fairly trivial to detect (see screen shot below). Why did they go with Akamai instead of partnering? One key factor may be that Mogulus, Ustream and Justin.tv haven’t streamed live events with much more than 100,000 simultaneous viewers (correction: one person associated with Justin.tv emails to say they’ve hit “well over 400,000″), so tonight’s concert would have been an experiment in scalability for them.
It appears based on public Akamai data that about 700,000 people were watching the YouTube concert at its peak. There’s more information on the Mogulus blog, but basically Akamai was serving about 150,000 live streams across its networks right before the event started, and 863,000 at the peak of the concert.
All this expensive CDN infrastructure really isn’t necessary to handle live video streams effectively. P2P software can handle it effectively and far cheaper since the users are serving most of the video to others. That requires getting software directly onto user’s computers, however, something Joost tried and has mostly abandoned. Eventually this stuff will likely just be built into the browser directly.
One thing tonight proved – Akamai may be expensive, but it sure does work. I wonder if Google will let them issue a press release about it.
I was surprised that they pulled it off so well. There was only one point when the feed froze.
The show itself was an epic fail.
You’re an epic fail for misusing memes and just parroting stuff you’ve heard on the interwebs.
I thought the show was awesome, personally. Not everything was a hit, but considering the variety it was very entertaining.
YouTube (Google) couldn’t afford to flop on their premiere so they made the right choice by selecting a proven service provider. In the future I expect them to have their own internal delivery mechanism. Overall, nicely done!
Epic fail?
I watched it for a bit and it looked pretty representative of YouTube–a lot of diversity you wouldn’t find from MSM.
Streaming YouTube is a joke. FAME IS DEAD.
I’m going off topic here, but have you seen Google’s timeline results? They are getting pretty good. Here’s an example.
http://www.goog...11&ct=title
I had some hiccups with the feed, but compared to other live streams on the web, wow really well done! Overall, I like it and Ilove the fact that YouTube is trying something new.
I don’t think the future is for P2P to be built into the browser. I think the best solution is for IP multicasting to be actually implemented by the router manufactures and backbone providers.
Actually the future is nearly here as Flash Player 10 appears to include re-broadcasting and swarming technology, which is disabled by default. It was demoed at Max 2008 event just this Tuesday (hmm .. interesting timing actually) and it is based on Amicima stuff they bought some time ago.
http://www.flic...@N06/3044147140
CNN.com’s live video is using a plugin within Flash that does P2P transfer mixed with CDN. I honestly forget the name off the top of my head, but the install was relatively seamless thanks to me having Flash already installed. In fact, I didn’t even know you could do plugins within Flash. In any case, there are companies working on this and CNN appears to believe in at least one of them for the future of it’s streaming.
Michael, you wrote that “all this expensive CDN infrastructure really isn’t necessary to handle live video streams effectively. P2P software can handle it effectively and far cheaper since the users are serving most of the video to others.”
I’m not sure that’s the case.
Bittorrent-style P2P works efficiently because chunks are designed to arrive out of sequence so as not to create a bottleneck on the necessarily smaller number of hosts with chunks to serve. Whereas live streaming requires chunks to arrive in order and with little delay.
How does P2P help here?
I get where he was heading with the Joost comment, but I tend to agree with you DeWitt. P2P hasn’t proven an effective platform for live distribution, nor will it. I think the next best hope would be some sort of national IP Multicast implementation (hopefully with IPv6) which would offload the need for a CDN and operate a router as a mini CDN.
His sentence should have read:
“You wouldn’t need all this fancy CDN infrastructure, if P2P were well I suited to delivering data in order, which it of course isn’t.
But in web 3.0, browsers will be servers too: when three people visit a site, their machines will start serving the data to each other! This will make the companies profitable enough to revive the VC industry. It a great idea for the community-of-three to serve other people’s data, especially when it’s low enough value that they can’t afford to do it themselves.
For example, if three people got together, we could serve all the Seesmic 3.0 data directly from our touch screen TechTablets. The Three’s Company Cluster is what I want, now. “
lousy stage production
Lousy comment production.
If that number of 700,000 viewers is correct. Then here are my calculations, Google was using something like 540gbit/s (if 800kbit/s per stream (for High quality mode)) thus used probably about 250TB of bandwidth for this 2 hour event. That is if the figure of 700 thousand viewers is correct, and I’m timing that by 1 hour even though the whole event was 2 hours, for average. If bandwidth costs $0.10 per GB, then cost of bandwidth might have been just something like $25,000. That wouldn’t be very expensive at all for bandwidth would it..
Google clearly could cover the bandwidth costs quite easilly without having to show too many ads in such a show wouldn’t they. I’d say the biggest cost to producing this type of show is probably paying the artists a fair amount of money for their awesome performances, and paying the very professional multi-camera production crew..
I’d say, other then thinking about soon launching Youtube Live service for all producers to broadcast live, Youtube could create an online worldwide channel that would regularly send cool live programing to compete with the crappy regular TV programs on the old fashioned TV networks (those that don’t want White Spaces to happen).
I thought, although the artists sometimes had too little time on stage, sometimes just a few seconds, I thought this show was awesomely funky, Mike Relm the video DJ was just fantastic, the human beatbox was amazing, breakdancing was fenomenal, the gay hosts were hillarious and all the Youtube celebs were very very pro in the execution of their comedy, I’d say this is much better then Saturday Night live or any such kind of old fashioned TV comedy.
Google as a producer, not only Seth Mcfarlane 2 minute clips, also a bunch of live and on-demand shows all around the world. In prime-time, would be cool.
Now they also need to launch a P2P streaming client software.
And I bet Google got a sweet deal….
At $.10/GB it would be more like $250k but still not a big deal for Google.
250000GB x $0.10 = $25,000…
The real failure here is that when sending a single stream of data to many recipients, most of these systems are still relying on unicast transmission to send data. Had multicast been properly deployed into the Internet, we would have efficient ways of transmitting large video streams to many people.
Multicast deployment is very ad-hoc these days, with few ISPs properly supporting it.
In a sense, what akamai does is like multicast, by sending blocks of data to edge servers and then sending that data to clients out on the edge. The fan out is handled globally instead of from youtube’s servers directly to the client.
As a publicity stunt, this has worked for Google and You Tube, but as a demonstration of new technologies, I don’t see it being anything more than what was happening ten years ag.
It’s unlikely that they’d get $.10/GB — that’s really low. It would probably be closer to $.20 or $.25.
Live streaming costs are typically higher than straight CDN costs.
Just thinking out loud, but why would Akamai need a press release when they now have this article and front page Techmeme?
Shhh. Akamai paid Arrington and Rivera. Don’t tell anyone!
they are just amazing. so well done. The tv is irrelevant to my 6 year old – he only likes u tube – i bet its the best 1.65 ever spent –
no chance on the press release – i think 23 and me might get one as a baby gift!
You reproduced?
Only time will tell, they did this for a reason, to improve the service.
Someone from Justin.tv says “well over 400,000″ and you just report it? So they had some event that was 50% the viewers of one of the most heavily promoted events on one of the biggest sites on the internet? I’d love to hear what it is and see some proof. I call bullshit.
On http://blog.jus...new-record.html they claim 100k viewers across the entire site (not just one stream). That was in August, so we can assume it may have hit higher since, even so, it’s not for a single stream. Streaming copyrighted movies and NBA/NFL/NHL sporting events is popular, but not as big as a heavily promoted YouTube event.
Reporting on this kind of technical matter should be left to technical people. Saying that P2P is capable of handling massive one to many live streams? Comparing Joost with this? Someone knows enough to be dangerous.
Akamai owns 15% of web infrastructure. It has proxy servers in major hubs and caches the web to speed up internet traffic. My friend made millions off the company’s stock back in 1999 before the tank. It came back from 0.57 after 9/11 and is now back to riding between $15 and $50 a share.
let me guess you came up with their name
Nicely done. Great for Google & Akamai.
Impressive for sure, although the wife asked “can we watch that on tv somewhere” /snicker
Yeah, I really wished I could watch it on my TV. Sitting here in London, it felt like Times Square invaded San Francisco and a ball was going to drop somewhere!
The whole concept was really cool and I’m looking forward to how this changes the landscape of old media even further.
@sean:
Haha… you know I have a large monitor but I still prefer to sit in front of my 27″ TV to watch movies… so even though there are a lot of content on Hulu.. I seldom watch it. Maybe its because the couch is so much more conducive for activity than my crappy office chair. LOL.
We watch a lot of youtube videos via Apple TV but it wont support streaming video like tonight. Boxee looks interesting too.
You all really don’t have an old XP PC hooked up to your living room digital TV? Aren’t you all supposed to be early tech adopters? I’ve got old Intel P4’s hooked up to most of my friends’ living room TVs, and they’re practically tech illiterate. I can’t believe people say this crap on this blog.
AppleTV? Really? What a total waste. LMFAO.
“one person associated with Justin.tv emails to say they’ve hit “well over 400,000″”
That sounds like BS to me. They don’t even have that many monthly uniques.
How the hell would you know?
Michael, I think this time you missed the point. Youtube did not develop anything interesting this time from a technical point of view. They have made a viewer with a third party firm and pushed the stream directly to Akamai. Anybody can do this (Ustream for example uses Akamai too for big streams, so it could serve that viewer number smoothly). But this model does not work with a lot of small broadcasters. So technically Youtube currently is far behind from the old players in this area.
Ah…I see. Dave works for Ustream….
Too bad YouTube Live Tokyo failed to go live.
so THAT’s what happened to tokyo live. I was waiting the whole time and it never materialised. shame. google pr should say something about this. comments reveal that plenty of people were waiting on youtube.com/tokyolive.
strangely enough these comments have been deleted. Google, come forth and protect your name.
What a thread! Guys, here are my comments after having spent the past 3 years working on live streaming and P2P.
1. The cost of bandwidth is not the issue. The infrastructure is. Akamai must have allocated 80% of their whole CDN for the You Tube concert. This is clearly a one-off as it would be very demanding for Akamai to be able to do this on a regular basis. The hardware and infrastructure cost would be enormous and would overshadow the bandwith costs by a long way.
2. While 700,000 is a very impressive number for online live, compared to TV audiences it is still very far off. This point to say that should people start really want to watch live (events or TV) online you would need tens of Akamais out there.
3. Multicast it’s by far the best solution but it’s highly unlikely it will ever happen (speaking to the telco boys) for cost and security issues.
4. P2P for live is hard but possible and could help a lot. We have been working on it for a long time now and we are rolling it out with very good results. You need a different breed of P2P as the Bit Torrent type (as mentioned in one of the comments) is not designed for synchronous situations. Live P2P could give you a boost on the existing infrastracture – in particular if combined with a CDN – and the key variable is the available upstream. Example: if the average upstream of the people watching is 500kbps and the stream is 800kpbs you would have to supplement only 300kpbs from the CDN. Clearly this would be very helpful as with the same infrastructure you could serve an audience more than double. Check out Livestation for mrore on this topic (www.livestation,com)
I agree that only streaming P2P (or if Pando get its way P4P) has the potential to serve audiences at the scale of broadcast TV. It’s not just a matter of bandwidth. Advertisers don’t value 700,000 people watching 700 clips a thousand times; they don’t know how to place ads next to that much fragmented programming. OTOH give them 1.4m. eyeballs watching the same stuff at the same time and advertisers can do business. And to do that you’ll need P2P streaming supporting dozens of programs like we saw on YT.
that’s assuming the stream holds at a constant bit rate which P2P never does and max speeds are usually well below what is promised by the telco.
a good video stream need to be at least 1,000 kbps to be watchable especially if it’s real time and something as visually complex as a live concert (probably need more than 3000kbps for a decent picture).
as for complex infrastructure not being “necessary”, once again Mr Arrington you display complete ignorance on subjects about which you should not be writing.
Agree. But that is typical of TC posts.
P2P for live is good, except you will have a 2-3 minute delay. Is it live then?
It would be 2-3 minutes if you used a bit-torrent type approach. Next generation P2P for live are designed in a different way and can achieve near realtime performance. If you miss school days, check out the Splitstream papers bu Microsoft. Complex but good reading.
They have been promising this for years. I highly doubt it.
Even 30 seconds is too long, especially for interactivity (like Tom Green show).
P2P has a long way to go before it can match the sites like Ustream, etc in latency.
MTV > You Tube live. You Tube should power the people not compete against MTV.
Doesn’t justin.tv use Akami as well?
[geek@australia ~]$ traceroute www-cdn.justin.tv
1 mel.intervolve.com.au (122.100.4.253) 0.551 ms
2 HSRP3-GW.VIC.ESC.NET.AU (210.56.80.60) 2.243 ms
3 AS4802.melbourne.pipenetworks.com (218.100.13.12) 1.815 ms
4 gi0-2.mel-pipe-hsa1.ii.net (203.215.6.9) 3.212 ms
5 203-206-138-137.deploy.akamaitechnologies.net (203.206.138.137) [open] 2.800 ms
[geek@australia ~]$ _
[ross@sparta ~]$ nslookup www-cdn.justin.tv
Server: 173.45.224.4
Address: 173.45.224.4#53
Non-authoritative answer:
www-cdn.justin.tv canonical name = http://www.just...v.edgesuite.net.
http://www.just...v.edgesuite.net canonical name = a31.g.akamai.net.
Name: a31.g.akamai.net
Address: 205.234.225.107
Name: a31.g.akamai.net
Address: 205.234.225.128
[ross@sparta ~]$
exactly! Thanks for pasting this in here.
Wow, no way, one of the busiest and most traffic-intensive websites in the world uses the same CDN as all their other services to distribute content.
Why is this even news?
The technology was on point but i felt the show itself was sketchy, parse and downright “NOT” entertaining. Youtube needs to step its game up!
Oh btw
Check out http://www.jobstaxi.com
New Jobs. HUGE Inc. Starbucks. Facebook. Puma North America Inc. Ignition Entertaiment.
Wowza is a lot cheaper than Adobe FMS and charges a simple per-server fee. The comment “And [Flash streaming is] expensive – you have to license Adobe’s Flash Media Server, or a competitor like Wowza , and pay at least a couple of cents per gigabyte transferred on top of normal costs.” is only correct as it applies to CDNs using FMS. Adobe’s “Flash tax” is either passed on to customers or the CDN takes a margin hit. Wowza server licensing doesn’t tax Flash services with per-GB fees.
Why would it be a bad thing for GOOG to use AKAM services for something like this? Seems like a great example of buying instead of building. Reward companies who have the sense to work like this.
(Although it would have been cooler if they could deliver a ‘live experience’ via progressive download… wouldn’t ADOBE love to see that happen?)
CG
if youtube offers a live service, would this mean death to the other live services?
Michael,
Your comment that “P2P software can handle [live broadcasts] effectively and far cheaper…” is both accurate and timely. Abacast’s Peer-Assisted Live delivery is doing exactly these types of live broadcasts today at scale and enabling higher bit rates for consumers and with a higher QoS than utilizing unicast delivery exclusively. We recently did a live broadcast of the Beijing Olympics for SBS (http://www.abac...9032008_sbs.php) using our peer-assisted technology licensed by FutureStream Networks. We served higher bit rate content than the YouTube broadcast (1.5Mbps), with very high audience satisfaction and resulting in significant server infrastructure savings and over 13Gbps of bandwidth savings.
While both unicast delivery and peer-to-peer have advantages, their sum is greater than their individual value, something Abacast has come to realize as the company has evolved.
-Jim Kott
Abacast, Inc.
So with AKAM stock at 12 and change, looks like Google can just buy them and Youtube can go live 24/7.