Google Research prototypes ambient audio contextual content
Marshall Kirkpatrick
150 comments »
A team from Google Research has developed a prototype system that uses a home computer’s internal microphone to listen to the ambient audio in a room, determine what is being watched on TV and offer web-based supplemental information, services and shopping contextual to each program being watched. It’s strange, but it sounds like it works and people might really like it. There’s no indication yet whether or when this could be available as a service.
Google Research team members Michele Covell and Shumeet Baluja along with Michael Fink of the Hebrew University of Jerusalem’s Center for Neural Computation were given the best paper award for their report on the system at the the Euro ITV (interactive television) conference last week. (“Social- and Interactive-Television Applications Based on Real-Time Ambient-Audio Identification” 10 pg PDF, see also the Google Research blog post on the paper.)
The system compresses the captured audio into irreversible (emphasis theirs) summary statistics which are then compared to a database of mass media statistics and used to determine what the browser should display. Possible service offerings discussed in the paper fall into four categories:
- Personalized information layers Here’s what Tom Cruise is wearing in the show you are watching and here’s where you can buy the same clothes in your zip code.
- Ad hoc social peer communities If you would like to chat about this show, ten of your college friends are watching it right now as well.
- Real-time popularity ratings Nielsen requires hardware and the results aren’t available in real-time. You might want to know if there is a spike in viewers watching the show on channel 9 right now. Advertisers might want to know that too.
- TV- based bookmarks Click to save a show or clip into your video library and there will be more than just a few shows available for watching later.
The system requires no dedicated television-connected hardware, protects users’ privacy and is technically feasible, the researchers report. Experiments with a laptop placed in the lap of a person ten feet from a television and engaged in loud conversation with some one next to them were successful in providing matching online content - when channel surfing was taken into account.
Lest you fear that all broadcast TV is a huge data-set, the report says that ff the database of summaries holds only 32-bit descriptors of 5 second clips, then up to one year of broadcast information could be held in less than 1 GB. The researchers report that this is made much more feasible by re-runs.
Privacy concerns were addressed in the prototype by compressing captured audio on the user’s computer before transmitting summary data to the database for comparison and by offering a mute button in the program. Given Google’s recent ethical issues, these privacy measures may not be enough to assuage some people.


June 8th, 2006 at 4:32 pm
June 8th, 2006 at 5:31 pm
June 8th, 2006 at 5:58 pm
June 8th, 2006 at 8:48 pm
June 8th, 2006 at 8:59 pm
June 8th, 2006 at 9:17 pm
June 8th, 2006 at 9:44 pm
June 8th, 2006 at 10:13 pm
June 8th, 2006 at 10:17 pm
June 8th, 2006 at 10:25 pm
June 8th, 2006 at 10:51 pm
June 8th, 2006 at 11:23 pm
June 8th, 2006 at 11:35 pm
June 9th, 2006 at 1:29 am
June 9th, 2006 at 2:48 am
June 9th, 2006 at 4:30 am
June 9th, 2006 at 5:59 am
June 9th, 2006 at 6:20 am
June 9th, 2006 at 6:32 am
June 9th, 2006 at 6:43 am
June 9th, 2006 at 7:05 am
June 9th, 2006 at 7:13 am
June 9th, 2006 at 8:00 am
June 9th, 2006 at 11:52 am
June 9th, 2006 at 11:55 am
June 9th, 2006 at 11:55 am
June 9th, 2006 at 2:19 pm
June 9th, 2006 at 3:43 pm
June 9th, 2006 at 4:30 pm
June 9th, 2006 at 6:14 pm
June 9th, 2006 at 6:17 pm
June 9th, 2006 at 6:20 pm
June 9th, 2006 at 6:41 pm
June 9th, 2006 at 6:48 pm
June 9th, 2006 at 8:16 pm
June 9th, 2006 at 8:54 pm
June 9th, 2006 at 9:38 pm
June 10th, 2006 at 4:42 am
June 10th, 2006 at 6:50 am
June 10th, 2006 at 3:56 pm
June 10th, 2006 at 6:19 pm
June 11th, 2006 at 8:12 pm
June 12th, 2006 at 6:35 am
June 13th, 2006 at 5:05 am
June 15th, 2006 at 1:05 am
June 15th, 2006 at 2:25 am
June 21st, 2006 at 1:00 am
July 2nd, 2006 at 8:17 am
July 14th, 2006 at 7:24 am
July 20th, 2006 at 7:34 pm
July 30th, 2006 at 8:03 am
September 5th, 2006 at 7:55 am
September 8th, 2006 at 8:51 am
September 11th, 2006 at 7:07 pm
September 18th, 2006 at 8:19 am
October 26th, 2006 at 6:55 pm
October 27th, 2006 at 3:16 pm
October 28th, 2006 at 4:57 pm
December 27th, 2006 at 8:21 pm
January 31st, 2007 at 4:06 am
April 8th, 2007 at 11:54 pm
April 17th, 2007 at 10:49 am
August 26th, 2007 at 8:51 pm
August 31st, 2007 at 7:22 am
September 5th, 2007 at 8:56 pm
November 18th, 2007 at 2:48 am
December 2nd, 2007 at 3:56 am
December 23rd, 2007 at 5:24 am