« Previous post
Next post »

July 27, 2007

Wikia Acquires Distributed Web Crawler Grub

Duncan Riley

27 comments »

grub.jpgWikia has acquired the distributed web crawling engine Grub from Looksmart.

Grub utlizes spare computing cycles on users’ computers to index the web, a similar concept to that used by SETI@home in the search for extraterrestrial life.

Grub originally started in 2000 and was purchased in 2003 by LookSmart. Grub was shut in 2005 following some minor controversy over the closed source code and the failure to comply with Robots.txt blocking requests.

Wikia will open source the Grub code “as soon as possible” and use the data from Grub to feed the Search Wikia Project.

The price of the acquisition was not disclosed.

(via R/WW)

  • Sphere It

This entry was posted on Friday, July 27th, 2007 at 12:51 pm and is filed under Company & Product Profiles. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Trackbacks/Pings (Trackback URL)

  1. Dan's Diary

    July 28th, 2007 at 6:20 am

  2. 破百的老兵

    July 29th, 2007 at 6:59 am

  3. Wikipedia Grub

    July 29th, 2007 at 12:51 pm

  4. share.websitemagazine.com

    July 30th, 2007 at 6:39 am

  5. El blog de Javier Mendoza

    July 31st, 2007 at 8:18 pm

  6. Wishful Thinking » Get Ready For Wikia Search

    December 17th, 2007 at 3:57 pm

  7. Le Blogueur » Blog Archive » Première capture d’écran du futur moteur de recherche Wikia

    January 2nd, 2008 at 3:36 pm

  8. Wikia Search Is A Complete Letdown.

    January 6th, 2008 at 10:24 pm

  9. Le Blogueur » Blog Archive » La déception Wikia

    January 19th, 2008 at 9:02 am

  10. Wikia Search in versiune alfa

    February 15th, 2008 at 12:23 am

  11. Get Ready For Wikia Search; First Screen Shots Shown In South Africa

    March 12th, 2008 at 11:43 am

  12. Complete List of 2007 Web / Technology Acquisitions » The StartUp Blog at PartnerUp

    May 21st, 2008 at 8:17 am

Comments

RSS feed for comments on this post.

  1. James

    July 27th, 2007 at 12:58 pm

    I have a feeling this project will do much better now under its new ownership. A distributed Web crawler is a very neat system, especially one that works similar to *@home systems. Wikia’s “Distributed Lab” is going to house projects focused on distributed computing, crawling, and indexing (more interesting is the Semantic Lab).

  2. Vijay Teach Me

    July 27th, 2007 at 1:05 pm

    crawling and indexing and using my resources for the betterment of my kind, I will give it a shot.

    James what is a Semantic Lab.

    Vijay

  3. wolf

    July 27th, 2007 at 1:14 pm

    http://www.faroo.com goes even a step further: distributed crawling and distributed search.

  4. What's Hot Today.com

    July 27th, 2007 at 1:15 pm

    I wonder what Grub.com sold for?

  5. MikeM

    July 27th, 2007 at 2:14 pm

    My guess is they gave it away for near nothing. After all Wikia has leased LookSmart’s AdCenter platform and some search tech.
    Good things ahead for Wikia I feel. LookSmart’s stock in the industry is also rising with partnerships with the likes of NYTimes, Reed, Facebook, Wikia, ASK and Active Athlete.
    I can’t wait to see the wikia Search fully deployed.

  6. Nick Harris

    July 27th, 2007 at 2:22 pm

    I can see a really interesting revenue model forming here:
    Since distributed indexing will cut down costs, they could actually share some ad revenue with their users based on how much processing power they contribute. I could see that getting a lot of people hooked to the system.

  7. rod / techfold.com

    July 27th, 2007 at 2:23 pm

    On the surface a distributed crawler seems like a great idea. That being said, though, if the algorithm used to index crawled data isn’t any good, all the processor cycles in the world will get Wikia nowhere.

    BTW, thanks for making me feel dirty for using the word “algorithm,” Ask.

  8. DennisK

    July 27th, 2007 at 2:55 pm

    Truthfully I don’t see the use in this acquisition. While distributed crawling using the community is useful, search is a hardward intensive business because of the serving of search results, not the crawling of the web. Having a distributed crawler does little to impact the overall effectiveness of a search system or cut down on hardware costs.

  9. Vijay Chakravarthy

    July 27th, 2007 at 5:00 pm

    I dont understand how they plan to use this..

    Using distributed crawling will make it even easier for people to generate spam links as search results for everything under the sun. First thing many people will write is something that returns their set of pages for every single query.

  10. David Mackey

    July 27th, 2007 at 9:13 pm

    I like this acquisition b/c I’ve always thought Grub was an interesting idea. What in the world is Looksmart up to these days? It seems they closed down most of their interesting services.

  11. housetier

    July 28th, 2007 at 1:20 am

    You can download grub here: http://www.gnu.org/software/grub/

    ;)

  12. Eric

    July 28th, 2007 at 11:59 am

    Dang - the Grub website has been down most of the day today - overwhelmed by potential volunteers?

  13. Jean Witter

    July 29th, 2007 at 7:00 pm

    Exactly

  14. Martin Belam

    July 30th, 2007 at 8:35 am

    >> Grub was shut in 2005 following some minor controversy over the closed source code and the failure to comply with Robots.txt blocking requests.

    It wasn’t a minor controversy if Grub got stuck on your site, disobeyed robots.txt and then crawled it to death ;-)

  15. Kord Campbell

    August 1st, 2007 at 1:20 pm

    Wikia’s plans for Grub go beyond what we were working on at the time LookSmart acquired us. Crawling in a distributed fashion isn’t that cool in and of itself - it’s what you can do with the data in a distributed fashion that makes things really exciting.

    The best place to start getting involved with Grub, and the other pieces of the project is http://search.wikia.com/. The guy in charge is Jer Miller, of Jabber fame. We are looking for a few people to start working on porting the client back to Linux and starting a few projects that can reuse the crawl data - which will be open to everyone.

« Previous post
Next post »