« Previous post
Next post »

April 27, 2008

Google Experiments With Next Generation Image Search

Michael Arrington

64 comments »

Two Google scientists presented a paper (pdf embedded below) at the World Wide Web Conference in Beijing last week that outlines their vision for the future of image search.

Notably, the new image search technology doesn’t just index text associated with an image in determining what’s in it. Google is now talking about using computers to analyze the stuff in photos, and using that to associate it in a ranked way with keyword queries. In effect, they’re talking about something similar to PageRank for images (but without the linking behavior).

Today when we talk about search all we really mean is text search. That’s sort of like only being able to see in one color. And when we search for image, video and audio content, the only data that search engines use to do those searches is the text that is associated with those files. That’s like trying to describe the color green when you can only see in red.

To date Google and others have spent a significant amount of effort on making the metadata around rich content better. One example of this is Google’s Image Labeler game that uses human labor to properly tag images. Innovative, yes. But it’s still trying to “describe green” when you you can’t actually see it.

Once computers are able to analyze rich content as easily as they can analyze text, a whole new dimension to search will emerge. Humans will no longer be needed to do the heavy lifting in describing what is included in rich content, and that means that content will no longer be invisible on the Internet.

PageRank For Image Search

Googlers Yushi Jing and Shumeet Baluja argue that Google is now ready to see beyond text. In their paper they talk about their efforts to apply state of the art image recognition software to figure out what stuff is in an image. “Commercial search-engines often solely rely on the text clues of the pages in which images are embedded to rank images, and often entirely ignore the content of the images themselves as a ranking signal,” they say. Their experiments in actually digging into the images themselves “show significant improvement, in terms of user satisfaction and relevancy, in comparison to the most recent Google Image Search results.”

Google is looking at the visual characteristics of popular images and then determining rank based on similarities between images. In the figure below, the largest two images contain the highest rank.


PageRank for Product Image Search - Get more documents

  • Sphere It

This entry was posted on Sunday, April 27th, 2008 at 11:28 pm and is filed under Company & Product Profiles. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Trackbacks/Pings (Trackback URL)

  1. Google PageRank for Product Image Search | My Blog Posts

    April 28th, 2008 at 12:09 am

  2. The Escape Blog : Blog Archive : Next generation image search

    April 28th, 2008 at 12:23 am

  3. Dekut.com

    April 28th, 2008 at 12:53 am

  4. Google’s Vision of the Future of Image Search

    April 28th, 2008 at 1:46 am

  5. e-Bouw.org » Blog Archive » Google PageRank for images

    April 28th, 2008 at 2:33 am

  6. Google experiments with searchable images | Domain Name News | Domain News | Expired Domains

    April 28th, 2008 at 2:45 am

  7. TechCrunch Japanese アーカイブ » Google、次世代画像検索を実験中―画像にページランク導入へ

    April 28th, 2008 at 2:56 am

  8. Commentary on Google Image PageRank Research Paper

    April 28th, 2008 at 6:53 am

  9. Pagerank para imagens | wOOt

    April 28th, 2008 at 10:07 am

  10. New Ranking Methodology for Google Image Search : Natural Search Blog

    April 28th, 2008 at 12:13 pm

  11. April 28, 2008 | next media update

    April 28th, 2008 at 12:30 pm

  12. PageRank Para Pesquisas de Imagem | Web 2.0 Brasil

    April 28th, 2008 at 1:44 pm

  13. Google Experiments With Next Generation Image Search

    April 28th, 2008 at 1:45 pm

  14. Blern

    April 28th, 2008 at 2:24 pm

  15. Pagerank para imágenes « El rincón seo de Raquel

    April 28th, 2008 at 3:10 pm

  16. Do You See What I See? | Tropophilia

    April 28th, 2008 at 9:15 pm

  17. the new shelton wet/dry

    April 29th, 2008 at 12:21 am

  18. Google rinnova Image Search | FDS

    April 29th, 2008 at 6:15 am

  19. reemst.com » Blog Archive » Google experiments with VisualRank

    April 29th, 2008 at 7:27 am

  20. ::::::::Zogntai::::::::

    May 7th, 2008 at 2:06 am

  21. Google Reformula Busca por Imagens « Our Digital Life

    May 9th, 2008 at 7:31 am

Comments

RSS feed for comments on this post.

  1. Yakov

    April 27th, 2008 at 11:33 pm

    It sounds like a breakthrough, though, image search is not mainstream

  2. SearcH◆ EngineS WEB

    April 27th, 2008 at 11:33 pm

    The next evolution of this technology will probably be implemented in Videos.

    To some degree this technology is in an embryonic stage already with the face and torso search options in Google images and their ability to help track child porn in videos

    Perhaps by the end of the next decade - this intelligent image search will be very common

  3. MyMesh.com

    April 27th, 2008 at 11:34 pm

    Web 4.0 in preparation? 8-)

  4. Michael Arrington

    April 27th, 2008 at 11:34 pm

    i just want computers to be able to understand words in audio streams, and index that. it would be such a huge step forward.

  5. Vishal

    April 27th, 2008 at 11:39 pm

    “calling the much needed semantic web”
    *sigh*

  6. Munim

    April 27th, 2008 at 11:44 pm

    Understanding words in audio streams? so it will be like one of those voice recognition/dictation software installed on the crawler to index words!
    technically I think it can be implemented… but these software aren’t very accurate as of now.
    why hasn’t anybody done this yet?

  7. Elias Bizannes

    April 27th, 2008 at 11:54 pm

    Sounds useful.

    Combine it with facebook, and a picture of your favourite body type, and you’ve got a new dating service.

  8. Sam Daams

    April 28th, 2008 at 12:08 am

    Are you just trying to score rankings for unrelated terms Michael? WTF does PageRank, a measure of inbound links to a page, have to do with image analysis?

  9. Emre Sokullu

    April 28th, 2008 at 12:14 am

  10. Robotics

    April 28th, 2008 at 12:23 am

    @Sam Daams, take a moment to read the paper linked in the article and you will realize that “PageRank for Product Image Search” is the paper’s title. And if you read the actual paper, you will see how the authors use the similarity of image features as links between images and hence extent the PR mathematical framework to image search.

  11. sean

    April 28th, 2008 at 12:34 am

    is it possible to know what’s in the picture?

  12. V

    April 28th, 2008 at 12:37 am

    Anyway to get that pdf without having to register on a random website? You know, like a link to the pdf? It used to be possible on the good old web 1.0…

  13. V

    April 28th, 2008 at 12:38 am

    Sorry, I got it from the original conference site (1st link in the article)

  14. Ethan Stock

    April 28th, 2008 at 12:49 am

    Mike,

    I’m frustrated that this is a recycled press release, not journalism. Markoff in the Times managed to at least get a perspective quote from Munjal at Riya/Like. This post is just a reprint of what Google pushed out to the world, and I can get that direct from Techmeme. I understand the pressures you are under to publish fast in the nerve-twitch world of blogging, but you have access to damn near every CEO in Silicon Valley, and most of them are happy to take your call. Either they, or their CTOs, could give your piece some perspective that no guy in New York could ever match. I would love to see you and TechCrunch as the consistent source of that kind of perspective. Over time, I tend focus on the news sources that provide a consistent, deep, unique perspective. Blodget, for instance, is doing a dynamite job of analyzing the public-market/technology intersection - his personal sweet sport, and one well done from New York. TechCrunch should be crushing the peculiar dynamics of private company/technology — which means personal access to, and insight from, VCs, uber-geeks, and management types, most of whom are one VentureSource search and one IM or email away from you at this time of night, or any other. I would love to see you maximize that strength and access, for all our benefit.

    — Ethan

  15. Abi Bakar

    April 28th, 2008 at 1:03 am

    Google does know how to give well service to its users

  16. david saintloth

    April 28th, 2008 at 1:10 am

    A Microsoft technology for recognizing objects in photographs was demoed some time last year , quite impressive. It allowed the reconstruction of 3D data for items in separate photographic images, I forget the name of the technology but Microsoft had it previewed on their silverlight website. The news was trumpeted on digital photography websites like dpreview.com as a very interesting technology for exploring the 3D map of photographs taken by different people of the same landmark or structure.

  17. aaron

    April 28th, 2008 at 1:11 am

    Visual search engine coming to iPhone

    http://gizmodo.com/381352/visu.....ne-in-june

  18. Sumanth

    April 28th, 2008 at 1:14 am

    David,

    This is probably what you are talking about

    http://phototour.cs.washington.edu/

  19. bryan

    April 28th, 2008 at 1:20 am

    Envisional (UK based? I think) have had a pretty sophisticated image matching technology for years. Always surprised that it doesn’t pick up much comment but i don’t think they promote it much. When I worked for [insert name of very large bank], it used to pick up our logo on phishing sites, fraud sites, all kinds of stuff. No idea how it worked but it was v.clever and v.useful.

  20. Sam Daams

    April 28th, 2008 at 1:22 am

    @10, I saw that’s what the paper is called, but if you write something like this paragraph:

    “Notably, the new image search technology doesn’t just index text associated with an image in determining what’s in it. Google is now talking about using computers to analyze the stuff in photos, and using that to associate it in a ranked way with keyword queries. In effect, they’re talking about PageRank for images.”

    then you really have not done your research on PageRank, which is strange for Arrington since this concept has existed for more than a few years now. Just because you read ‘ranked’, does not mean it has the first thing to do with PageRank (in face G now says ranking has very little to do with PageRank, but I digress). That entire paragraph was about analyzing an image, figuring out what’s in it and then how to return it for keyword queries. The conclusion that ‘in effect’ they are talking about PageRank is incorrect and misleading based on that paragraph.

    Perhaps it just comes down to what @14 writes. The whole article just reads like a hack job of trying to put out content as fast as possible without taking the time to properly research the story.

  21. Sathyan

    April 28th, 2008 at 1:32 am

    Sounds great.
    An insight to how search would be in the future. its amazing.

  22. Robotics

    April 28th, 2008 at 1:32 am

    @Sam, I simply quote from the paper’s abstract, “Through an iterative procedure based on the PageRank computation, a numerical weight is assigned to each image; this measures its relative importance to the other images being considered.”

  23. Jason Jenkins

    April 28th, 2008 at 1:47 am

    fine artists will no longer starve!!!

  24. 113.com

    April 28th, 2008 at 2:18 am

    Interesting research, good for the web… :-o

  25. Steve Dukes

    April 28th, 2008 at 2:40 am

    Very interesting. PIXSTA (www.pixsta.com) is already enjoying success at monetising ‘image-to-image’ search via its AdImages network and Visual Product Search technology.

  26. Alexei Yavlinsky

    April 28th, 2008 at 3:26 am

    Very good article indeed. Great to see that big image search players are waking up to content-based image search! However this is system is not entirely novel. A smaller scale prototype of this type of system has existed for a few years now. It is called Behold and was developed by Alexei Yavlinsky during his PhD studies. You can find it at http://www.behold.cc and a comprehensive about page about its functionality is here: http://www.beholdsearch.com/about/

  27. Fabian

    April 28th, 2008 at 3:54 am

    I have been working for the German Research Center for Artificial Intelligence on project which also focuses on searching similar images. The project is called Flexible Image Retrieval Engine (FIRE). You can have a look at a demo here:
    http://demo.iupr.org/cgi-bin/fire-parts.cgi

  28. Alex Linhares

    April 28th, 2008 at 4:04 am

    This is years off. Human-level image recognition? Human-level speech recognition? Years, years off. A lot of work in cognitive science has to be done first.

    Google should try to solve Bongard problems in order to really grasp the issue of meaning extraction. Until then, nothing will happen here except demos that fail spectacularly (like google image search does).

  29. Anita CM

    April 28th, 2008 at 5:09 am

    It seems the day of artificial intelligence taking a lead over human intelligence is not very far off…

  30. Sahar Sarid

    April 28th, 2008 at 6:00 am

    “PageRank for images (but without the linking behavior).”

    That is a funny statement as pagerank is all about linking behavior.

    Sahar

  31. Peter

    April 28th, 2008 at 6:03 am

    I am tired of reading reports like that. They either discuss academic papers that go nowhere or some lame demos that can only retrieve images based on color similarity.

  32. Utkarsh Sinha

    April 28th, 2008 at 6:23 am

    @M.Arrington,@Munim
    Hi, we might fulfill your wish :) [extreme stealth]
    Hint: How about understanding video contents?

  33. Ontario Emperor

    April 28th, 2008 at 8:22 am

    Commercial entities have been pursuing image recognition for some time; Neven Vision (see #9) was an independent concern for a while before its acquisition by Google.

    It’s clear that our search processes could be much, much better, but continuing research in the field, coupled with some initial attempts at betaware if not actual commercial products, brings us closer to the time when we can truly search everything, and not just text.

  34. Spambot

    April 28th, 2008 at 8:53 am

    @MA. I think it is possible. Will require a very powerful audio recognition tool.

  35. Jai

    April 28th, 2008 at 9:33 am

    I think there are already amazing search engines out there. If google can replicate the features of some of them, its job is done.

  36. Peter Rothman

    April 28th, 2008 at 10:53 am

    Very interesting paper. This approach is quite similar to work we are doing with facial images.

  37. david saintloth

    April 28th, 2008 at 2:00 pm

    Re. 18 by Sumanth

    Sumanth, that is the very technology photosynth , thanks for finding the link!

  38. CRS

    April 28th, 2008 at 2:23 pm

    It’s a good paper!

    Dealing with generic images is a very challenging research problem. I am glad to see that Google is finally making the move. To give a bit of history to this forum - IBM’s QBIC system was one of the pioneer systems back in early 1990 that can do query by image contents. Since then, more than thousands of research papers were published mainly on computer vision, image processing, information retrievals, machine learning, and some database indexing.

    To my observation, there are some approaches that have some degree of success, but none can claim a true victory. Over the past decade, we do see many “specialized” image/media search engines that can search large-scale image/media databases in seconds with high accuracy. These media range from medical radiology images and satellite images to genomic sequences and protein 3D structure.

    Two examples of powerful search could be “given a new patient’s medical image, find cases with similar pathology visually and their follow up treatments.” “given an unknown protein 3D structure, find structures that are locally similar to the unknown structure so that we can predict potential functions of the new structure.”

  39. michael mandel

    April 28th, 2008 at 3:17 pm

    As part of the PhD research, I’ve been working on very related problems in content-based music search. I just put a demo up of some of the searches we can do. In particular, it uses data from a human computation game to train “autotaggers”, i.e. models of what tags sound like. It takes 20-30 good examples of a tag to train an autotagger, and we only have about 50 so far, but as more people play the game and more tags cross that threshold, we’ll be able to train more autotaggers.

  40. Harry Wang

    April 29th, 2008 at 8:02 am

    Let’s see it implemented first. Have not really seen great automated image detection/search anywhere sans Google Image Search which is image tagging dependent for the most part.

    Harry “wants his image search accurate” Wang

  41. Miles (SEM iCluck)

    April 29th, 2008 at 1:00 pm

    Google just has to replicate the other image searches - I do love the idea of relevance decided based upon their algorithm and potential patent though - Imagine if they give that technology to the public what type of mashups will come about and what type of communities will come into fruition =o)

  42. Alexander Straub

    May 6th, 2008 at 1:46 pm

    Great article from Mike about PIXSTA - PICTURE YOUR SEARCH company on TechCrunch UK
    http://uk.techcrunch.com/2008/.....out-image/

    Clearly google is not the first company to make an inroad into the field. images.google.xx has still room for improvements and several start-ups have taken the lead as described in the comments above.

    Maybe time for image to image search is emerging just NOW - I guess search is sill 10 blue underlined lines and could move from the DOS like experience within the next 10 years.

  43. Alexander Straub

    May 6th, 2008 at 2:16 pm

Leave a Reply

Continue the conversation in TechCrunch Forums

« Previous post
Next post »