Comments on: Yahoo Search Wants to Be More Like Google, Embraces Hadoop

By: 14153f82220c

14153f82220c — Sat, 10 May 2008 06:53:35 +0000

14153f82220c…

14153f82220cdd728696…

By: Ryan Ward

Ryan Ward — Fri, 29 Feb 2008 01:25:35 +0000

Reading through the shear size and amount of data is mindboggling.

By: mapreduced

mapreduced — Thu, 21 Feb 2008 07:23:55 +0000

Thanks for this post and the clarification. Yahoo itself played a little bit with the words map and reduce so ordinary people understand and remind it. Good to see that Yahoo made some decissions in the past that now add value to the web. Combine this with some tough Microsoft deal makers and distribution opportunities and the other Company will get some serious competition the first time. Yahoo, you made my day ;

By: Anonymous

Anonymous — Thu, 21 Feb 2008 05:56:42 +0000

You guys are the lamest of all Google fanboys. Yahoo! follows google’s footsteps? Yahoo! has been one of the largest proponents and developers on Hadoop.

Google has nothing to do with Hadoop. They do closed source software. They should be marching in Yahoo!’s footsteps … but, wait. Why would google do anything in open source? They are a black box *non* evil empire.

Happy 1984 to all the fanboys.

By: chris smith

chris smith — Thu, 21 Feb 2008 02:23:04 +0000

Interesting article, looks like you missed another similar story at the begining of FEB when Hypertable.org launched…..Seeing that they have had nearly 300 downloads and are talking with some very hip Valley companies…. why would you not of seen or covered this…(http://onotech.blogspot.com/)

Fred could not possibly be right could he….

By: Markus Thomson

Markus Thomson — Wed, 20 Feb 2008 23:07:31 +0000

Haha, today’s XKCD comic is just so perfect for this thread:

http://xkcd.com/386/

awesome.

By: Erick Schonfeld

Erick Schonfeld — Wed, 20 Feb 2008 22:58:46 +0000

Thank you Qian Wang for setting me straight. I’ve updated the post with a clarification.

By: Eric Baldeschwieler

Eric Baldeschwieler — Wed, 20 Feb 2008 22:48:34 +0000

Hi Folks, thanks for the write-up. A couple of comments:

Comparing a single job to all jobs run at google is not too informative. Google clearly has a larger plant than we do and has undoubtedly run much larger jobs than ours, but we have been producing a comparably sized web search index for many years and the point of our announcement was that Hadoop can now support jobs of the scale needed to build Yahoo or Google scale services. Doing some math on the Google numbers you quoted…

Their average job produces 14,000TB/2,217,000 jobs -> 0.0063TB / job compared to our 300TB job. So we can conclude that we have run a job 47,000 times larger than their average job! Again, the point is that we can do full web work on Hadoop, not to compare our plant to theirs.

Also our team is very proud of our investment in the Hadoop platform. It is our contributions to the Hadoop project that have made it possible to do full scale web search work on it. We’ve been working towards this milestone for several years.

http://developer.yahoo.net/blo.....adoop.html
http://radar.oreilly.com/archi.....-on-h.html

By: ONLINE SERVICES/INTERACTIVE MEDIA « Daily Marauder

ONLINE SERVICES/INTERACTIVE MEDIA « Daily Marauder — Wed, 20 Feb 2008 22:17:57 +0000

[…] and “reduces” them to a map of the Web so that ranking algorithms can be run against them. (http://www.techcrunch.com/2008/02/20/yahoo-search-wants-to-be-more-like-google-embraces-hadoop […]

By: Colin Brumelle

Colin Brumelle — Wed, 20 Feb 2008 21:02:57 +0000

For those interested, there is a well written explanation (in classic Joel style) of how “Map” and “Reduce” work here:
http://www.joelonsoftware.com/.....08/01.html

By: Dyde

Dyde — Wed, 20 Feb 2008 20:58:54 +0000

Yahoo can do one thing to gain on Google, get rid of that slow page design on the search pages. Get rid of Javascript, gazillion cute images. Show only 7 results and 3 more links on the bottom. That will make it load faster than Google. And they should not index subdomains and biz and info sites. Most of these are spam. But of course, I’m probably not the first to suggest these improvements. I’m sure their engineers do suggest similar changes but these suggestions die in huge bureaucracy Yahoo has become.

By: TechBeta » Blog Archive » Yahoo Search Wants to Be More Like Google, Embraces Hadoop

Wed, 20 Feb 2008 20:22:43 +0000

[…] more: techcrunch.com addthis_url = […]

By: till

till — Wed, 20 Feb 2008 20:21:09 +0000

@Metabass: And apparently it takes even less to comment on here.

@Erick: Still an interesting post. Thanks. I don’t get the fuss.

By: Qian Wang

Qian Wang — Wed, 20 Feb 2008 20:11:47 +0000

Erick, the confusion comes from your use of the words “map” and “reduce” in a different way than what they mean in MapReduce. In MapReduce, “map” refers to mapping a function over a set of data. This is an operation that can be quite easily parallelized, which is why Google is able to bring such a large array of processors to bear. The “reduce” part takes all the results of the mapping step and recombines them into the end result (or set of results).

Your paraphrasing would be fine if you just didn’t use the words “map” and “reduce”.

By: some guy

some guy — Wed, 20 Feb 2008 19:53:22 +0000

the map step breaks the problem down into many small chunks and sends them to individual boxes for computation.

each box then does computation on it’s small chunk of data

the reduce step take the result from all the small chunks and combines them back into one big solution.

that’s the rough explanation of what’s going on.

By: YDRIVE

YDRIVE — Wed, 20 Feb 2008 19:52:09 +0000

@19 -

1) http://feedblog.org/2008/01/06.....-clusters/
(and click on the pdf file)

2) http://wiki.apache.org/lucene-.....pMapReduce

By: Metabass

Metabass — Wed, 20 Feb 2008 19:45:13 +0000

TechCrunch… It takes all the technologies on the Web 2.0 found by Erick and “crunches” them down into inaccurate blog posts.

By: Mike

Mike — Wed, 20 Feb 2008 19:43:33 +0000

In other news… Yahoo’s YUI 2.5 is released.

http://yuiblog.com/blog/2008/0.....-released/

By: Erick Schonfeld

Erick Schonfeld — Wed, 20 Feb 2008 19:37:13 +0000

@3 and @6, I clearly state in the post that Hadoop “also works for large-scale computing problems beyond search”

Also the “It” in that sentence you both cite refers to Hadoop, not MapReduce, and is a paraphrase of Yahoo’s own description of what Hadoop does:

http://developer.yahoo.com/blo.....adoop.html

“The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.

“The Webmap build starts with every Web page crawled by Yahoo! and produces a database of all known Web pages and sites on the internet and a vast array of data about every page and site. This derived data feeds the Machine Learned Ranking algorithms at the heart of Yahoo! Search.”

I read that to mean that Hadoop creates a map of the Web and reduces (i.e. compresses) it into a manageable set of data that can be placed into a database so that the ranking algorithms can do their work.

If this is wrong, someone please explain (and, no, that does not mean linking to Wikipedia). Explain it in English.

By: Owen O'Malley

Owen O'Malley — Wed, 20 Feb 2008 19:27:10 +0000

It really runs the other way. Yahoo needed to update their infrastructure and decided to invest in an open source infrastructure rather than a proprietary one. Yahoo now has the advantage that undergraduates at schools like UCB and UW are being taught to program using Hadoop.

The Yahoo webmap is one Hadoop application, but it is far from the only one at Yahoo. One of the side benefits of having a good distributed computing infrastructure is that it becomes much easier to write ad-hoc programs that run on a large cluster of machines. There are more than 40,000 cores running Hadoop at Yahoo.

Hadoop was the name of Doug Cutting’s *son’s* stuffed elephant.

By: YDRIVE

YDRIVE — Wed, 20 Feb 2008 19:11:39 +0000

Yahoo! does still have lots of bright engineers.. only a few noisy guys at the top make themselves look stupid.

By: WebSide Ventures

WebSide Ventures — Wed, 20 Feb 2008 19:04:50 +0000

Hey, I totally bought the MapReduce quote…looks like you can’t slip much past this crowd though.

By: Nick

Nick — Wed, 20 Feb 2008 18:58:14 +0000

Well, this is not like an impulse idea from Yahoo!. It was obvious this day would come already when they hired Doug Cutting over a year ago. Yahoo!, with Doug leading the way, have since been the single most proponents and developers of Hadoop.

By: Owen

Owen — Wed, 20 Feb 2008 18:54:52 +0000

Interesting that it is more efficient - should increase overall search throughput - shouldn’t affect quality of results though. Not that that matters much. As far as I (a sample of one) can tell, Google’s constant tinkering with search algorithms has led to worse and worse results over the past two years. Yahoo’s results have been better for well over a year now. It may not be Google’s fault entirely. There is so much attempted manipulation of its results that playing defense against that may be hurting the overall quality.

By: RJacobsen

RJacobsen — Wed, 20 Feb 2008 18:43:50 +0000

This is great news.. as just this morning I was wondering how come Google is referring 10 times the amount of traffic to my site than Yahoo is. I hope that this new algorithm will help my targeted audience to be able to locate my site easier, when entering certain keywords pertaining to my site.