Comments on: Yahoo Search Wants to Be More Like Google, Embraces Hadoop

By: Yahoo Open Sources Traffic Server

Yahoo Open Sources Traffic Server — Mon, 02 Nov 2009 23:31:46 +0000

[...] the open-source software. Hadoop now provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services. CrunchBase Information Yahoo! Information [...]

By: Yahoo Releases Internal Hadoop Source Code « Castilho1024’s Blog

Yahoo Releases Internal Hadoop Source Code « Castilho1024’s Blog — Thu, 11 Jun 2009 20:27:25 +0000

[...] the open-source software. Hadoop now provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services. Hadoop runs on more than 25,000 servers and [...]

By: Yahoo! Releases Its Own Hadoop Distribution | CloudAve

Yahoo! Releases Its Own Hadoop Distribution | CloudAve — Wed, 10 Jun 2009 23:46:17 +0000

[...] includes more than 25,000 servers and provides the framework for many Yahoo properties including Yahoo Search, Yahoo Mail, and several content and ad services. Yahoo says its opening up the source code to [...]

By: The Far Edge » Blog Archive » Yahoo Releases Internal Hadoop Source Code

Wed, 10 Jun 2009 22:15:48 +0000

By: Yahoo Releases Internal Hadoop Source Code | Blog YODspica Ltd

Yahoo Releases Internal Hadoop Source Code | Blog YODspica Ltd — Wed, 10 Jun 2009 22:09:27 +0000

By: Yahoo Releases Internal Hadoop Source Code

Yahoo Releases Internal Hadoop Source Code — Wed, 10 Jun 2009 20:22:38 +0000

By: With Hadoop, Amazon Adds A Web-Scale Data Processing Engine To Its Cloud Computer « Programming and more

Fri, 03 Apr 2009 06:22:28 +0000

[...] is using Hadoop, which is the open-source version of MapReduce. Yahoo also started using Hadoop last year. While Google and Yahoo use this technique for searching the Web, it can be used for any [...]

By: Hadoop, Amazon Adds A Web-Scale File System To Its Cloud Computer | Blog YODspica Ltd

Thu, 02 Apr 2009 16:28:21 +0000

By: With Hadoop, Amazon Adds A Web-Scale File System To Its Cloud Computer

With Hadoop, Amazon Adds A Web-Scale File System To Its Cloud Computer — Thu, 02 Apr 2009 15:42:33 +0000

By: Parascale Promises Data Center Heaven: Private Cloud Storage At About $1 A Gig | aboutCREATION

Mon, 08 Sep 2008 06:17:25 +0000

[...] its approach is closer to Google’s MapReduce file system or Hadoop, the open-source version of that. Krishnan doesn’t rally see these as competition. [...]

By: Parascale Promises Data Center Heaven: Private Cloud Storage At About $1 A Gig

Parascale Promises Data Center Heaven: Private Cloud Storage At About $1 A Gig — Mon, 08 Sep 2008 05:55:55 +0000

[...] its approach is closer to Google’s MapReduce file system or Hadoop, the open-source version of that. Krishnan doesn’t rally see these as competition. He says: [...]

By: Update: ImageShack CEO Hints At His Grander Ambitions

Update: ImageShack CEO Hints At His Grander Ambitions — Tue, 20 May 2008 23:05:23 +0000

[...] or to syndicate that targetting data to ad networks). Like Google and Yahoo, he is deploying the open-source Hadoop software to create a massive distributed supercomputer, but he is using it to analyze all the data he is [...]

By: Ryan Ward

Ryan Ward — Fri, 29 Feb 2008 01:25:35 +0000

Reading through the shear size and amount of data is mindboggling.

By: mapreduced

mapreduced — Thu, 21 Feb 2008 07:23:55 +0000

Thanks for this post and the clarification. Yahoo itself played a little bit with the words map and reduce so ordinary people understand and remind it. Good to see that Yahoo made some decissions in the past that now add value to the web. Combine this with some tough Microsoft deal makers and distribution opportunities and the other Company will get some serious competition the first time. Yahoo, you made my day ;

By: Anonymous

Anonymous — Thu, 21 Feb 2008 05:56:42 +0000

You guys are the lamest of all Google fanboys. Yahoo! follows google’s footsteps? Yahoo! has been one of the largest proponents and developers on Hadoop.

Google has nothing to do with Hadoop. They do closed source software. They should be marching in Yahoo!’s footsteps … but, wait. Why would google do anything in open source? They are a black box *non* evil empire.

Happy 1984 to all the fanboys.

By: chris smith

chris smith — Thu, 21 Feb 2008 02:23:04 +0000

Interesting article, looks like you missed another similar story at the begining of FEB when Hypertable.org launched…..Seeing that they have had nearly 300 downloads and are talking with some very hip Valley companies…. why would you not of seen or covered this…(http://onotech.blogspot.com/)

Fred could not possibly be right could he….

By: Markus Thomson

Markus Thomson — Wed, 20 Feb 2008 23:07:31 +0000

Haha, today’s XKCD comic is just so perfect for this thread:

http://xkcd.com/386/

awesome.

By: Erick Schonfeld

Erick Schonfeld — Wed, 20 Feb 2008 22:58:46 +0000

Thank you Qian Wang for setting me straight. I’ve updated the post with a clarification.

By: Eric Baldeschwieler

Eric Baldeschwieler — Wed, 20 Feb 2008 22:48:34 +0000

Hi Folks, thanks for the write-up. A couple of comments:

Comparing a single job to all jobs run at google is not too informative. Google clearly has a larger plant than we do and has undoubtedly run much larger jobs than ours, but we have been producing a comparably sized web search index for many years and the point of our announcement was that Hadoop can now support jobs of the scale needed to build Yahoo or Google scale services. Doing some math on the Google numbers you quoted…

Their average job produces 14,000TB/2,217,000 jobs -> 0.0063TB / job compared to our 300TB job. So we can conclude that we have run a job 47,000 times larger than their average job! Again, the point is that we can do full web work on Hadoop, not to compare our plant to theirs.

Also our team is very proud of our investment in the Hadoop platform. It is our contributions to the Hadoop project that have made it possible to do full scale web search work on it. We’ve been working towards this milestone for several years.

http://develope...hoo-hadoop.html
http://radar.or...s-bet-on-h.html

By: ONLINE SERVICES/INTERACTIVE MEDIA « Daily Marauder

ONLINE SERVICES/INTERACTIVE MEDIA « Daily Marauder — Wed, 20 Feb 2008 22:17:57 +0000

[...] and “reduces” them to a map of the Web so that ranking algorithms can be run against them. (http://www.tech...embraces-hadoop [...]

By: Colin Brumelle

Colin Brumelle — Wed, 20 Feb 2008 21:02:57 +0000

For those interested, there is a well written explanation (in classic Joel style) of how “Map” and “Reduce” work here:
http://www.joel...2006/08/01.html

By: Dyde

Dyde — Wed, 20 Feb 2008 20:58:54 +0000

Yahoo can do one thing to gain on Google, get rid of that slow page design on the search pages. Get rid of Javascript, gazillion cute images. Show only 7 results and 3 more links on the bottom. That will make it load faster than Google. And they should not index subdomains and biz and info sites. Most of these are spam. But of course, I’m probably not the first to suggest these improvements. I’m sure their engineers do suggest similar changes but these suggestions die in huge bureaucracy Yahoo has become.

By: TechBeta » Blog Archive » Yahoo Search Wants to Be More Like Google, Embraces Hadoop

Wed, 20 Feb 2008 20:22:43 +0000

[...] more: techcrunch.com addthis_url = [...]

By: till

till — Wed, 20 Feb 2008 20:21:09 +0000

@Metabass: And apparently it takes even less to comment on here.

@Erick: Still an interesting post. Thanks. I don’t get the fuss.

By: Qian Wang

Qian Wang — Wed, 20 Feb 2008 20:11:47 +0000

Erick, the confusion comes from your use of the words “map” and “reduce” in a different way than what they mean in MapReduce. In MapReduce, “map” refers to mapping a function over a set of data. This is an operation that can be quite easily parallelized, which is why Google is able to bring such a large array of processors to bear. The “reduce” part takes all the results of the mapping step and recombines them into the end result (or set of results).

Your paraphrasing would be fine if you just didn’t use the words “map” and “reduce”.