Comments on: Google Processing 20,000 Terabytes A Day, And Growing http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/ Startup and Technology News Mon, 13 Oct 2008 23:24:51 +0000 http://wordpress.org/?v=2.5.1 By: Running Appcelerator on the Google App Engine http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2158328 Running Appcelerator on the Google App Engine Thu, 10 Apr 2008 01:40:10 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2158328 [...] about Google’s App Engine. How can you help it? If the opportunity to run your apps on a 20,000 terabyte per day cloud computer isn’t enough, certainly the ability to run Python on that cloud must [...] [...] about Google’s App Engine. How can you help it? If the opportunity to run your apps on a 20,000 terabyte per day cloud computer isn’t enough, certainly the ability to run Python on that cloud must [...]

]]> By: Yahoo Search Wants to Be More Like Google, Embraces Hadoop http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2001073 Yahoo Search Wants to Be More Like Google, Embraces Hadoop Wed, 20 Feb 2008 22:57:38 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2001073 [...] this to some data from Google on its MapReduce computing infrastructure (it is not quite apples to apples, but Google was [...] [...] this to some data from Google on its MapReduce computing infrastructure (it is not quite apples to apples, but Google was [...]

]]> By: Yahoo isi modifica infrastructura motorului de cautare • Arena IT - Publicație online de IT&C http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2000892 Yahoo isi modifica infrastructura motorului de cautare • Arena IT - Publicație online de IT&C Wed, 20 Feb 2008 19:14:06 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-2000892 [...] this to some data from Google on its MapReduce computing infrastructure (it is not quite apples to apples, but Google was [...] [...] this to some data from Google on its MapReduce computing infrastructure (it is not quite apples to apples, but Google was [...]

]]> By: roddy.ie» Blog Archive » Scale of the googleplex http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1997998 roddy.ie» Blog Archive » Scale of the googleplex Mon, 18 Feb 2008 20:59:06 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1997998 [...] not heard any speculation about the size of Google’s server farm for a while, but this TechCrunch post makes it easy to have a guess: in September 2007, they used 11,081 Machine-Years, or had [...] [...] not heard any speculation about the size of Google’s server farm for a while, but this TechCrunch post makes it easy to have a guess: in September 2007, they used 11,081 Machine-Years, or had [...]

]]> By: marks.dk – links for 2008-01-13 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1917621 marks.dk – links for 2008-01-13 Sun, 13 Jan 2008 01:30:54 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1917621 [...] Google processes 20 petabytes of data each and every day! (tags: google search data technology scalability petabytes) [...] [...] Google processes 20 petabytes of data each and every day! (tags: google search data technology scalability petabytes) [...]

]]> By: SmoothSpan Blog http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1915891 SmoothSpan Blog Fri, 11 Jan 2008 22:46:52 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1915891 <strong>The Big Switch Requires A Big Shift In Thinking And Strategy...</strong> The Big Switch is Nick Carr’s term for the emergence of Utility Computing in the cloud over software installed in individual data centers or machines.  It’s also the title of a new book he has out on the topic.  There’s a lot for ev... The Big Switch Requires A Big Shift In Thinking And Strategy…

The Big Switch is Nick Carr’s term for the emergence of Utility Computing in the cloud over software installed in individual data centers or machines.  It’s also the title of a new book he has out on the topic.  There’s a lot for ev…

]]> By: Matt Mahoney http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1913657 Matt Mahoney Thu, 10 Jan 2008 20:16:13 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1913657 Actually 11,081 machine years / (2217 jobs x 395 sec = .0278 years) implies 399,000 machines. Since this doubles about every 6 months I guess they are up to about 600K machines by now. Actually 11,081 machine years / (2217 jobs x 395 sec = .0278 years) implies 399,000 machines. Since this doubles about every 6 months I guess they are up to about 600K machines by now.

]]> By: Reader http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912775 Reader Thu, 10 Jan 2008 10:08:50 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912775 From these numbers you can deduce that they have over 11 000 machines used for these job (exact number depending on percentage of comoputing power used and machine downtime). From these numbers you can deduce that they have over 11 000 machines used for these job (exact number depending on percentage of comoputing power used and machine downtime).

]]> By: Nicholas Bedworth, CTO, DigitalDirect Development Corporation http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912584 Nicholas Bedworth, CTO, DigitalDirect Development Corporation Thu, 10 Jan 2008 06:28:08 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912584 Hadoop is also being used by Barney Pell at Powerset... Anyone have an idea what the Google datacenter (or perhaps a dataplex) costs to operate on an annual basis? Hadoop is also being used by Barney Pell at Powerset…

Anyone have an idea what the Google datacenter (or perhaps a dataplex) costs to operate on an annual basis?

]]> By: David Litsky http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912492 David Litsky Thu, 10 Jan 2008 05:07:13 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912492 @2 Shawn “The average MapReduce job runs across a $1 million hardware cluster, not including bandwidth fees, datacenter costs, or staffing.” Is this the equivalent of saying a car costs $20,000 to own, but excludes the cost of gas, maintenance, and a chauffeur? @2 Shawn
“The average MapReduce job runs across a $1 million hardware cluster, not including bandwidth fees, datacenter costs, or staffing.”
Is this the equivalent of saying a car costs $20,000 to own, but excludes the cost of gas, maintenance, and a chauffeur?

]]> By: Boring Market http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912479 Boring Market Thu, 10 Jan 2008 04:59:49 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912479 I think the increase of daily use of Google products has some effect on it. Sure everyone uses Google to well Google, but think about when people use Adsense or Gmail. Don't they then use Google has their primary search engine? I know I do, I switched from Yahoo! to Google after my increase of use in Google products. I think the increase of daily use of Google products has some effect on it. Sure everyone uses Google to well Google, but think about when people use Adsense or Gmail. Don’t they then use Google has their primary search engine? I know I do, I switched from Yahoo! to Google after my increase of use in Google products.

]]> By: Venkat http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912270 Venkat Thu, 10 Jan 2008 01:45:05 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912270 is 20 petabytes/day compressed or uncompressed size? is 20 petabytes/day compressed or uncompressed size?

]]> By: Eric Baldeschwieler http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912068 Eric Baldeschwieler Wed, 09 Jan 2008 23:45:44 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1912068 > Eric, can you please explain when’s the best scenario to use map-reduce? A couple of links: http://en.wikipedia.org/wiki/MapReduce#Uses http://open.blogs.nytimes.com/tag/hadoop/ > Eric, can you please explain when’s the best scenario to use map-reduce?

A couple of links:

http://en.wikipedia.org/wiki/MapReduce#Uses
http://open.blogs.nytimes.com/tag/hadoop/

]]> By: ettore http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911963 ettore Wed, 09 Jan 2008 22:43:38 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911963 To put things into perspective, it would be very interesting to have comparable measures from Yahoo and other Google competitors. To put things into perspective, it would be very interesting to have comparable measures from Yahoo and other Google competitors.

]]> By: LiquidSilver http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911923 LiquidSilver Wed, 09 Jan 2008 22:10:52 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911923 Eric, can you please explain when's the best scenario to use map-reduce? thanks Eric, can you please explain when’s the best scenario to use map-reduce? thanks

]]> By: Eric Baldeschwieler http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911754 Eric Baldeschwieler Wed, 09 Jan 2008 20:32:38 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911754 The same techniques are in used by many other organizations on large volumes of data. My team at Yahoo! has invested in making the open source Apache Hadoop map-reduce framework run at very large scale (http://lucene.apache.org/hadoop/). This is used by Yahoo! and other organizations with similar data analysis needs. No one is doing as much map-reduce computation as Google, to the best of my knowledge, but others are to running individual jobs at the same scale on similar data sets. The same techniques are in used by many other organizations on large volumes of data.

My team at Yahoo! has invested in making the open source Apache Hadoop map-reduce framework run at very large scale (http://lucene.apache.org/hadoop/).

This is used by Yahoo! and other organizations with similar data analysis needs. No one is doing as much map-reduce computation as Google, to the best of my knowledge, but others are to running individual jobs at the same scale on similar data sets.

]]> By: debloggers.de http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911692 debloggers.de Wed, 09 Jan 2008 20:11:31 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911692 <strong>Google macht 20.000 Terabytes Traffic am Tag...</strong> Google mit all seinen Diensten, also Webseiten und den Suchcrawlern soll am Tag 20.000 Terabytes, dass sind 20 Petabytes, an Traffic verursachen. Echt häftig. Um das mal zu vergleichen Youtube soll in einem Monat 27 Petabytes machen. Ob die nun ... Google macht 20.000 Terabytes Traffic am Tag…

Google mit all seinen Diensten, also Webseiten und den Suchcrawlern soll am Tag 20.000 Terabytes, dass sind 20 Petabytes, an Traffic verursachen. Echt häftig. Um das mal zu vergleichen Youtube soll in einem Monat 27 Petabytes machen. Ob die nun …

]]> By: Outdoor Directory http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911642 Outdoor Directory Wed, 09 Jan 2008 20:00:18 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911642 Thanks Technicle. Thanks Technicle.

]]> By: Technicle http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911605 Technicle Wed, 09 Jan 2008 19:50:37 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911605 > What’s a comparison the average person could understand? a) 20PB or 20,000TB == 40,000 disk drives worth of data, assuming 500GB/drive (latest max-capacity per drive is 1TB/drive, ie., 1000GB/drive) or another perspective - b) 1,500 disk drives working perfectly in parallel over 24 hours to turn up that much of data, assuming 1.5Gbits/sec SATA drives all working [IMPOSSIBLY] perfectly to their theoretical design limits, ie., some 150MBytes/sec (practically, probably 30%-perfect efficiency, or even less, thus, at least talking about 5,000 drives working in parallel for 3600x24 seconds -- just the data i/o part.) Very roughly, of course. > What’s a comparison the average person could understand?

a) 20PB or 20,000TB == 40,000 disk drives worth of data, assuming 500GB/drive (latest max-capacity per drive is 1TB/drive, ie., 1000GB/drive)

or another perspective -

b) 1,500 disk drives working perfectly in parallel over 24 hours to turn up that much of data, assuming 1.5Gbits/sec SATA drives all working [IMPOSSIBLY] perfectly to their theoretical design limits, ie., some 150MBytes/sec (practically, probably 30%-perfect efficiency, or even less, thus, at least talking about 5,000 drives working in parallel for 3600×24 seconds — just the data i/o part.)

Very roughly, of course.

]]> By: Technicle http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911523 Technicle Wed, 09 Jan 2008 19:22:34 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911523 Kevin - sorry.. I just "Copy link location" while mouseover'ing the link... would've pointed it to your blog article but there was no ref of it in http://www.niallkennedy.com/blog/2008/01/google-mapreduce-stats.html#comment94130 where i learned of the link... my apology! ;-) Kevin - sorry.. I just “Copy link location” while mouseover’ing the link… would’ve pointed it to your blog article but there was no ref of it in http://www.niallkennedy.com/bl.....mment94130 where i learned of the link…
my apology! ;-)

]]> By: Mike http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911495 Mike Wed, 09 Jan 2008 19:13:46 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911495 The huge costs can't be an advantage over yahoo or ms since MS could afford to have that much power should they want . The reason Goog is still better than yahoo/ms thus only marginaly is because good still delivers more accurate . Somebody needs to come up a new search engine. Goog is the best technology can offer yet we have all the ebay spam and shopping spam coming up with every search. --------- www.xenbet.com The huge costs can’t be an advantage over yahoo or ms since MS could afford to have that much power should they want .

The reason Goog is still better than yahoo/ms thus only marginaly is because good still delivers more accurate .

Somebody needs to come up a new search engine. Goog is the best technology can offer yet we have all the ebay spam and shopping spam coming up with every search.

———
http://www.xenbet.com

]]> By: Kelli http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911468 Kelli Wed, 09 Jan 2008 19:06:54 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911468 Love them or hate them, Google is the most impressive computer/tech company. Though I would be really interested to see how they secure their system. I wonder what the private thoughts are from Yahoo and Microsoft when they see this. Does anyone know how this would compare to say a government agency system like monitoring weather etc? Love them or hate them, Google is the most impressive computer/tech company. Though I would be really interested to see how they secure their system. I wonder what the private thoughts are from Yahoo and Microsoft when they see this. Does anyone know how this would compare to say a government agency system like monitoring weather etc?

]]> By: Outdoor Directory http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911441 Outdoor Directory Wed, 09 Jan 2008 18:58:41 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911441 I know these numbers are huge, but can someone put these numbers into context? What's a comparison the average person could understand? I know these numbers are huge, but can someone put these numbers into context? What’s a comparison the average person could understand?

]]> By: Kevin Burton http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911435 Kevin Burton Wed, 09 Jan 2008 18:55:21 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911435 By 'real link' I mean the real link to the whitepaper: http://feedblog.org/2008/01/06/mapreduce-simplified-data-processing-on-large-clusters/ It's pretty good. Mostly a refresher. Niall pried out some good info though. Onward! By ‘real link’ I mean the real link to the whitepaper:

http://feedblog.org/2008/01/06.....-clusters/

It’s pretty good. Mostly a refresher.

Niall pried out some good info though.

Onward!

]]> By: Kevin Burton http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911432 Kevin Burton Wed, 09 Jan 2008 18:54:26 +0000 http://www.techcrunch.com/2008/01/09/google-processing-20000-terabytes-a-day-and-growing/#comment-1911432 Your whitepaper is broken... Here's the real link. http://feedblog.org/2008/01/06/mapreduce-simplified-data-processing-on-large-clusters/ Technicle, please don't deep link :) ... I can't view stats on that page :) Kevin Your whitepaper is broken…

Here’s the real link.

http://feedblog.org/2008/01/06.....-clusters/

Technicle, please don’t deep link :) … I can’t view stats on that page :)

Kevin

]]>