Google Research prototypes ambient audio contextual content
by Marshall Kirkpatrick on June 8, 2006

A team from Google Research has developed a prototype system that uses a home computer’s internal microphone to listen to the ambient audio in a room, determine what is being watched on TV and offer web-based supplemental information, services and shopping contextual to each program being watched. It’s strange, but it sounds like it works and people might really like it. There’s no indication yet whether or when this could be available as a service.

Google Research team members Michele Covell and Shumeet Baluja along with Michael Fink of the Hebrew University of Jerusalem’s Center for Neural Computation were given the best paper award for their report on the system at the the Euro ITV (interactive television) conference last week. (“Social- and Interactive-Television Applications Based on Real-Time Ambient-Audio Identification” 10 pg PDF, see also the Google Research blog post on the paper.)

The system compresses the captured audio into irreversible (emphasis theirs) summary statistics which are then compared to a database of mass media statistics and used to determine what the browser should display. Possible service offerings discussed in the paper fall into four categories:

  • Personalized information layers Here’s what Tom Cruise is wearing in the show you are watching and here’s where you can buy the same clothes in your zip code.
  • Ad hoc social peer communities If you would like to chat about this show, ten of your college friends are watching it right now as well.
  • Real-time popularity ratings Nielsen requires hardware and the results aren’t available in real-time. You might want to know if there is a spike in viewers watching the show on channel 9 right now. Advertisers might want to know that too.
  • TV- based bookmarks Click to save a show or clip into your video library and there will be more than just a few shows available for watching later.

The system requires no dedicated television-connected hardware, protects users’ privacy and is technically feasible, the researchers report. Experiments with a laptop placed in the lap of a person ten feet from a television and engaged in loud conversation with some one next to them were successful in providing matching online content - when channel surfing was taken into account.

Lest you fear that all broadcast TV is a huge data-set, the report says that ff the database of summaries holds only 32-bit descriptors of 5 second clips, then up to one year of broadcast information could be held in less than 1 GB. The researchers report that this is made much more feasible by re-runs.

Privacy concerns were addressed in the prototype by compressing captured audio on the user’s computer before transmitting summary data to the database for comparison and by offering a mute button in the program. Given Google’s recent ethical issues, these privacy measures may not be enough to assuage some people.

Responses

Trackback URL

Comments

Comments Pages: [1] 2 » Show All

Is google crazy?!? I can’t even imagine what this current project would make the privacy advocates say. Letting it hear us over a mike, and then making ads? This is giving me the creeps, and I have no intention in allowing 1984 style loss of privacy coming any quicker that it already is. I can see this information being used for good, but in the hands of the wrong people. Oh my god! What if the government is successful in subpoening google for information? So, now apart from phone calls, they literally tap into our living rooms, and hear our conversations.. Absolutely not!

 

This sounds interesting and cool but good point Joy, that would be scary.

 

The NSA will just love this.

You don’t even need to sneak and an plant bugs in order to wiretap someone now.

 

Great!!! Another way to get more, better spam!!! More ads, consumers rejoice?

 

I agree with Joy. Aside from the other obvious issues with this idea, users are increasingly aware of the need to preserve and protect (what’s left of) their privacy online and unless the benefit overwhelmingly outweighs the cost are unlikely to have it wrested from them in the interest of “supplemental content” of dubious value (really — who cares where Tom Cruise bought his jaunty top? And the social thing — people have already found ways to self-organize around shared-viewing experiences that seem to work well…) I guess I just don’t see how organizing the world’s information and making it accessible includes spying on the world’s users and blatantly disregarding the basic right to privacy in one’s own living room to (and let’s not kid ourselves here) sell ads.

 

From technical stand point great idea and out of the box thinking. But from the consumer stand point, if I love my DVR, why the hack do I want google to serve me add based on what I am watching. For F*** sake T.V is for the entertainment and not for Google to listen to what I watch.

 

Why stop with TV? Google should train their system to listen in on everything that goes on at home and serve ads as necessary. V*agra! Ritalin! Divorce lawyer! Metamucil!

 

Serdar, I think you win the funniest comment so far award. Tough competition though.

 

This is really scary. Google is set to invade its users privacy now more than ever before. F*** that!

 

Like it or not this type of internet technology is only the beginning. Good for Google to be the first to wet their feet. And for those that think i’m crazy. Think about it this way. Ten years ago the internet was a 2400 modem. Not to mention text based only internet. Irc became chat. Text based pages became web 2.0. I could go on and on. You get my point. Hold on for major life changes.

 

To be fair, the research paper says that conversations will NOT be listened to, they will be filtered out. The audio data will be compressed to represent the TV shows while it’s your computer before it’s transmitted to Google. They say it several times throughout the paper.

 

Thought it was a post from The Onion.

“but it sounds like it works and people might really like it”

Breathtaking naivete. How old are you? 17?

 

Steve, I think there is ample evidence that many people would welcome a system like this and would accept assurances that they are not being spied on.

 

Just disconnect the microphone or eat gobs of beans and fart to mask the sounds if leaving your mic live. Of course you risk getting ads about these:
http://health-beauty.search.eb.....acatZ26395

 

Sounds like they want to dip their toe into TV ads. They’ve got video ad capability added to Adwords and it sounds like this would be excellent to integrate into that. I doubt this will make it to even open beta.

What they could do with it is go to ABC or NBC and propose contextual advertising for TV shows (which I’m guessing they already have done or have plans to.)

Just having technology that would let them by-pass the TV stations probably gives them some leverage. From the little I know bout the media industry, they’ll need every leverage they can get.

 

What a great media supplement. Heck im in front of the tv right now and on my laptop. I bet this type of interaction will be hot with the consumer.

 

“I have seen the future… and it sucks”.

Could someone please draw a better future for us, humans? I mean… consumers?

 

So if you’re wheezing will you get a lot of porn ads?

Beating your spouse? Ads for guns!

I’ll reserve judgement until I can test this out… until then, typical overreactions being seen.

 

For people watching their programs on a media center PC this roundabout method can be replicated with a widget for sideshow.

I think Bill Gates Keynote did show something similar at Las Vegas (whatever show that was).
They demoed something similar for HD-DVDs, too.

 

Google will push a lot of its ads into that if it becomes available as a service! Makes ‘ad’ sense.

 

Google is getting CREEPY. From holding alll your data on GDRIVE to recording your emails and chats. I’m starting to get wary. Now they’ll have all my conversations and are going to keep records of everything I watch on TV? I forget, they also have records of everything I’ve searched for and all my research notes. And let’s not forget all my blogposts and calendar items and address book, so they know where I’ll be when and who I’m meeting and why. Are we sure we can trust an entity with so much information to stick to its motto of doing no evil? It’s been proven time and again that people and entities with power misuse it. Oh and they also know exactly who I’m meeting and chatting with on Gtalk and Orkut. They have my picture on Orkut, every personal detail of mine and what I think is a perfect date. Oh, now I know why Ms. Jr. Page was able to seduce me the other day! She knew I liked walks on the beach and tequila!

 

Wow.. that’s almost content theft on a massive scale.. So Channel4 (UK) spends a few million quid licensing Lost so they can make a packet on advertising, and the big G displays advertising all the way through it for free… I think I’ll go into my local hypermarket and stick stickers over all the beer cans for a site I own…

Hats off for innovation though!

PS - wouldn’t it have been much simpler to have a web page that said ‘what channel are you watching’.. that’s 1 button click… hell, maybe we should do that instead, then scrape G through the API and display TextAds on the content.

 
Marshall Kirkpatrick - June 8th, 2006 at 10:50 pm PDT

caw - I think you’re missing the point - people have to be able to passively enter into the active phase of TV watching :) Clicking your channel number is too much effort to ask from someone in order for them to enter a live chat they are honestly interested in. (For our international readers visiting right now- in case it’s not clear, I’m totally being sarcastic.)

 

Dang, you’re right.

 

Well, so the Google Labs reinvented, what “Radiocontrol” already does for years…:
http://www.radiocontrol.ch/english/home.html

 

Clever stuff … but ultimately doomed technology. With the merging of TV and internet (and radio) technologies it will all come through the same connection anyway eventually, so no need to ‘listen in’ - the data will already be logged.

 

Between this and Google’s FireFox browser sync tool (http://www.google.com/tools/firefox/browsersync/index.html) storing info on Google’s servers, I wonder if anyone is going to discuss privacy issues today? ;-)

 

This research ties in nicely to Cringely’s prediction that google is poised to take up video distribution (with included tv advertising model):

http://www.pbs.org/cringely/pu.....60105.html

 

But this is like sticking a web cam so that Google knows what site we are visiting! I mean, why a mic? With IPTV, it will not be an issue to get to know what each home is viewing. Is that the expiration date of this product?

 

a new version of spy. what coming next? Google-Webcam? We Set you ONLINE ?!

Google = Evil => more and more

 

seekXL wrote: “Google = Evil => more and more”

Hey seekXL, did you notice that the .pdf paper ( click link in post ) is 666KB?

 

I think some of you are being complete jackasses. How many comments do we need about Google serving ads based on our conversations and the sounds of our actions, after it has been established that these noises are to be filtered out, for the point of the technology: to serve ads contextual to what you’re watching on TV.

What’s with all the bitching about Google knowing everything about you? Don’t use their products then! All I’m seeing here is the inevitable. And it’s cool.

 

Why cant google start an online ad making agency.Needless to say hottest in town. They can claim that they know the customer than any one because of tons of datas and clicks they can claim.

 

When a company has an insane stock valuation, insane proposals start becoming accepted as reality. I cannot imagine any scenario where I’d give a company access like this (especially Google Inc.).

 

Awesome…bring it on. I’d be glad to be aguinea pig for this if it meant that audio (and the video) could be searchable items on google.

This has GOT to be the next big thing. With podcasting growing so much, imagine being able to search audio and download sections only relevant to what you want.

Despite the cliche about Google-Love this is a massive step in the right direction for me.

 
 

Googl - жжёт))
ток я по английски не рублю нехуя=)

 

I really hope anything like this *ever* gets approved. Privacy out the window!

 

I mean *NEVER* — sorry!! :)

 

I wanted it to provide ambient music based on my environment!

That would not be evil.

I say “Rats”.

 

Wow, fuck that, Google is invading our fucking privacy with that shit

 

I recall radioshack doing a similar form of home privacy invasion about 5 years ago with the cuecat bar code reader. Among the various similar give aways they offered a device that attached to your tv directly (i suppose ambient is enough to differentiate this as a new idea in the US’s defunked patenting system) and intercepted the audio linking you to the particular product on television. Unfortunately this was a little before widespread broadband integration and as you know never caught on. But thanks google, your love of feeding the ad machines whille playing us like pawns fed by cool techy services is what keeps the internet fresh (and your pockets loaded).

 

#12 - the first thing I thought of when I saw this post was the Onion article with this headline:

“Tivo gets it - you like porn”

 

The number and weighting of comments about this post says it all. This is a deeply flawed concept and going to kick up a shitstorm.

 

Is anyone else sick of being survey and marcketed to ALL the time?

 

Just a few days ago, Microsoft was awarded a patent on a system for monitoring “television programming” and inserting ads in content on computers as a result of that monitoring. i.e. The ads you see on web pages will be sensitive to what you watch on TV… see: http://patft1.uspto.gov/netacg.....er=7051351

The Microsoft patent covers the case where “the client system has access to television programming viewed by a user of the information retrieval system.” The claims are not specific about the means by which “access to television programming” is achieved. The description clearly anticipates some kind of physical connection to the television device, however, the claims would appear to be written broadly enough so that picking up “ambient noise” might constitute “access to television programming.” The Google approach, being more general than the Microsoft patent, might still have a problem if it is used to do what the more constrained Microsoft system claims.

bob wyman

 

Thanks for pointing that out Bob, that’s really interesting. I wonder how people would have reacted if this was a story about MS instead of Google.

 

I’m just going to have to start talking nonsense whenever I’m surfing the web now.

 

Thanks Marshall (Comment 47). If you’re interested in this type of patents, you can find lots more at http://www.uspto.gov. Here are a few more recent Microsoft patents concerning various means of implementing advertising spyware:

Tracking usage behavior in computer systems:
http://patft1.uspto.gov/netacg.....er=7039699

Transmission of information during ad click-through: http://patft1.uspto.gov/netacg.....er=7058592

bob wyman

 

http://google-watch.org/
google is not to be trusted.
google is run by NSA Spooks

 

Comments Pages: [1] 2 » Show All

Leave Comment

« Back to text comment

Commenting Options

Enter your personal information to the left, or sign in with your Facebook account by clicking the button below.

Alternatively, you can create an avatar that will appear whenever you leave a comment on a Gravatar-enabled blog.