A reader noticed that stealth search engine Powerset is using Amazon’s Mechanical Turk service to gauge user reactions to search results.
See the screen shot (click for larger view) - users are shown a query and a number of results and are asked to evaluate the relevancy of each result from five choices. In this case, the query is “revealing bikinis.” Users are asked to evaluate four sets of results within ten minutes, and are paid $0.02 for the effort.
The current batch of Powerset projects have run their course, and there are currently no other projects available on Mechanical Turk.
I spoke with Powerset CEO Barney Pell this evening who confirmed that they are using Mechanical Turk to get human feedback on search results. He says the results are not all Powerset generated - rather, they show results from Powerset, Google and others to see which users prefer for a given query. He also says this is an ongoing project, and new ones will be added soon.
Pell also said that Powerset plans to use Mechanical Turk over the long haul, even after launch. They’ll put actual user queries into Mechanical Turk in real time, add Powerset and competitor results and see which results people find more relevant. If results suggest Powerset isn’t more relevant, they’ll adjust their engine.
Powerset also uses the EC2 computing service, another web service offered by Amazon. They recently released some of their internal growth models that allow people to compare the relative costs of EC2 to building out a real data center.
I don’t think Powerset is able to understand this toughest question:
1. I want the truth. Give me real reason why JFK was Assassinated?
2. How did Rosie O’Donnell get so fat. What food did Rosie O’Donnell eat?
I hated when natural language search engine appear same format:
“Good Question. I’m glad you ask… Maybe, this links suits you.”
Oh Fuck!!!
Oh I didn’t realise this was unknown - I did some HITs for Powerset in September on MTurk
wow, looks like these guys are doing everything except launching an actual product, good for them…
Another good reason for me to login to my MT account. If only they were going to open up more tasks.
what’s a hidden agenda behind this initiative? To get users involved into making an index more relevant?
i read somewhere that it build natural language search engine that reads and understands every sentence on the Web, is this module is purely based on Latent semantic indexing?
@Yakov: Hidden agenda? They pay people to make their algorithm better. There’s nothing hidden about it.
Most overhyped company of 2007 - when will their product launch? It’s already a month past tc40.
it has been sooo long they ever launched?
It’s pretty hard to be a threat to Google when there is no product to launch. These guys lack leadership among other things.
I signed up, let’s see how this shakes out.
http://www.whatshottoday.com
I will continue to make the same argruments against Powerset: I’ll pay some, any attention to them *after* they actually launch a product. Right now they have *zero* credibility in the marketplace. To me this smells like a project that may never really get off the ground. They keep launching BS services but seem far away from launching a real enduser search engine.
Keep these guys very close to the deadpool — remember this post when it happens.
Great post michael!!
I have always thought that mturk is a very good idea. It is using humans to simulate Artificial Intelligence, and that is like turning the trend of using computers to simulate human actions on its head :). I also wrote few comments on my site about what an engaging idea mturk is!!!
How much confidence can one have in a search engine using people from third world countries being paid 2 cents to learn English?
I would not be surprised if Google supplements its highly searched results using humans, but I am pretty sure they are of a much higher quality that you would get though Turk
Mike is the “revealing bikinis” = “erotic humiliation” just a pick to reflect your skepticism about Powerset and how it’ll process natural language? If that is a typical result Google has nothing to worry about.
Powerset is empty hype.
Hey Mike, wasn’t Powerset suppose to launch at TC40? Wasn’t that the requirement? What happened? Did they lie to you guys to get in? Why are you still covering this dead horse if they did lie to you?
Steve
Tech Crunch,
I’m all for you guys making money on Ads, but the new ad you’ve placed in the top right corner of the site should be removed.
2 cents for 10 minutes, so their workers are getting paid 12 cents an hour? yeah, you’ll get some high quality testers at that rate!
Powerset is really EVIL. What’s taking them soooooo long?
My guess is….
1. Maybe Powerset development team or engineers accidently press wrong button and ease half of powerset source code.
2. Maybe Google won’t allow powerset to grab anything Google search results.
3. Maybe Google Adsense banned powerset.
4. Maybe Powerset party guys want to break ten commendments. They want commit adultry, having sex with swingers or hippie wives, granny swingers, drink, stealing, murdering Google’s icon, etc. I wonder why they are soo many parties around powerset. FLICKR.COM explains all.
http://www.flickr.com/search/?q=powerset
There’s picture of powerset party, three dollars, stealing Google image, parties, employees, parties…
I think Google motto said right “Don’t be Evil”. I don’t think Google like this image either.
http://www.flickr.com/photos/b.....091668561/
http://web.archive.org/web/200.....erset.com/
That’s cool and everything, but I’m pretty sure bigger companies, cough google, can actually hire full time usability testers.
And WTF is up with “erotic humiliation” as a search example????
At any rate, Our new search engine changes the way people search, not the accuracy of the results. You can read this book to narrow that down with a recursive algorithm.
http://www.amazon.com/Googles-.....0691122024
Why doesn’t powerset contact me and see what we did to improve search? Maybe we could put our technologies together?
I saw Google Ads right next techcrunch, it says — “Server not found” I’m serious. I see is embrassessing iframe. I think Google likely to attack Techcrunch & Powerset.
Techcrunch sponsoring Powerset?
Atleast Powerset it trying
revealing bikinis???
erotic humiliation???
I think Powerset just broke 10th commandments.
Maybe Powerset party guys commit Adultry, having sex with swingers or wives, stealing Google image, etc…
Go to Flickr type powerset. See how many guys partying….
I don’t think anyone may waste 10 minutes for 0.02$.
Powerset couldn’t built its own search engine rather than putting metabase search. In order to get into IPO. They need write real tarabyte search crawler.
Powerset don’t have real time search engine.
I think the powerset coverage is REALLY good here on TechCrunch. I would like to see more in depth technical reporting about powerset. Perhaps even a video at their headquarters.
http://www.flickr.com/photos/1.....792313120/
Look all the gray hair. I don’t think they stand a chance at playing a fair match against Google. If they were college kids with stained clothes and unkept hair, I would seriously be worrying right now if I were google. That doesn’t seem to be the case.
The bigger question is why a search for “revealing bikinis” returned a Wikipedia entry on “List of Arctic Explorers”
Semantic technology wins again!
$0.12 an hour? AWESOME, where do I sign up? no more flipping burgers for me
MicrosoftTurk is better.
http://fakesteveballmer.blogspot.com
>MicrosoftTurk is better.
TurkishTurk is the best though!
We’ve been hearing/reading about Powerset for more than a year and still no product. If we don’t see a fully functional product by Q1 2008, this company is a failure — write this down everyone — a failure. It doesn’t take more than 18 months to build *any* Internet-based product. My hypothesis is that they are all well intentioned but in reality their thesis that semantic search can work better will be proven wrong. They may never release a fully baked product because their own internal tests prove this to be true. Watch this deadpool company!
Hey Sam: I’ve done quite a few of the Powerset HITs. The last time I checked, I lived in the United States, I was quite fluent in English, and I can tell whether something is relevant or not.
Speaking of English, you forfeit your right to comment on other people’s use of the language, “third world” residents or not, when you screw up the language yourself. Twice.