Regular search engines such as Google and Yahoo use statistics to make sense of the Web. They count links, keywords, and other items on a page to determine its rank in search results. Semantic search engines try to actually understand the meaning of the words found on the Web and other documents to bring back the most relevant results to a query. Microsoft bought Powerset for $100 million to gain semantic search expertise, but so far all it can search is Wikipedia.. Hakia, Textwise, and other startups are also working on semantic search. Now comes NetBase, which brings a slightly different approach that its says can scale to the entire Web.

NetBase has been around for a while. Originally called Accelovation, it has raised $9 million in two rounds of venture funding over the past four years, has 30 employees, and counts among its current customers P&G, Caterpillar, 3M, BP, Kraft, BASF, and Goodyear. It is now changing its name and offering its core semantic indexing technology as a platform for other companies to build their own products. Already, scientific publisher Elsevier uses NetBase to power its Illumin8 research tool for searching scientific articles, patents, and Websites.

NetBase takes a sophisticated linguistic approach, actually diagramming sentences to determine the relationship between words and phrases. It does particularly well with causal relationships, allowing it to tease out cause and effect from raw text. For instance, in the sentence, “The calcium, potassium and magnesium found in yogurt can help reduce your risk for hypertension often resulting from stress, obesity, and other factors” NetBase can identify that “stress” and “obesity” are causes of hypertension and that “calcium,” “potassium,” “magnesium,” and “yogurt” can be used to counter hypertension.

The company has already indexed about 8 billion Web pages and processes 100 billion sentences a month through its semantic parsing. Once it identifies causes, effects, and other relationships, it can serve them up in search results along with top-ranked links. For instance, a health-related search could turn up a guide that includes related symptoms, causes, drugs, and treatments. The technology also lends itself to Q&A types of searches. You could ask, “What companies are developing semantic search technologies?” and it will return a list of companies along with the snippets of mention that company and semantic search.

I’ve tried a few demo searches set up to do various things such as provide the pros and cons of a product, the companies in a particular market, or causes and effects of a medical problem. The results were impressive. On the whole, I’d say they were at least 70 percent relevant, compared to the much larger proportion of irrelevant links I get when I do a Google search. But it was slow. NetBase took 5 seconds or more to return results, something it says won’t be as big an issue in a production versions of its technology.

NetBase is not building its own search engine, although it plans to create a health-related search engine around PubMed content as a proof of concept Instead, it is targeting large publishers and companies that want to create their won vertical search tools, which combine data on the Web with their own databases of content. This is definitely an enterprise play. Licensing starts at about $100,000 and goes up from there.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Instead we have more Live Search results with dedicated answers (not sure why Powerset was needed for this), and better Wikipedia results (“Since Wikipedia articles show up in a large percentage of Live Search queries, it’s important that the captions are top notch.”).

The Powerset team is saying they have “plans for deeper integration in the future.” Let’s sure hope so, because better Wikipedia search results is not going to drive me to Live Search as my primary search engine.

Powerset promised to change the way we searched the web. Let’s see some of that.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Microsoft intends to use Powerset’s natural language search technology as a major differentiating factor v. no. 1 search player Google (see our recent coverage of Live Search Cashback, a another Microsoft search effort aimed at getting more market share).

TechCrunchIT goes into detail on how effective Powerset may be as a weapon. But a few things are clear – the resource limitations (cash and computing resources) that slowed Powerset’s development are now history. The relevance problem is less important since Microsoft core search relevance is quite good. And users really seem to like the beta launch of Powerset even with the limited dataset.

Naam says 5% of searches contain elements of natural language that keyword based search algorithms don’t handle well, and there’s an assumption that as better results are returned, more people may start to simply type a normal sentence instead of a couple of keywords. Microsoft will integrate at least parts of Powerset technology into Microsoft Live Search by the end of the year, Naam says. I expect we’ll be hearing a lot more about natural language search coming out of Microsoft shortly.

The full interview transcript is below, and you can listen to the MP3 over at TalkCrunch.

Michael Arrington: Hello this is Mike Arrington with Techcrunch. I have on the line today Barney Pell the co-founder and CEO of Powerset which was acquired, or actually announced that it was going to be acquired by Microsoft, earlier today the ancones was made. And from Microsoft I have Ramez Naam on the phone as well. He’s the group program manager for Microsoft Live search. Welcome Guys.

Ramez Naam: Thank you.

Barney Pell: Thank you.

MA: So just to be clear, what exactly was announced today? You announced that you’ve signed a deal, but not closed it yet.

RN: That’s right, we’ve signed the deal but the transaction has not happened but we’ve agreed on all terms for Powerset to become a part of the Live search team at Microsoft.

MA. Is there anything that can happen can stop the closing at this point?

RN: It would be very, very unexpected for anything to stop the closing at this point.

MA: Okay, and how long do you think it would take to close the deal, or more importantly really integrating the teams and move the product forward using Microsoft’s resources.

RN: Well, closing the deal, it’ll take a typical amount of time and not too terribly long, as far as integrating the teams, I think we’ll start on that immediately. And this is both a short term and long term task: Short term we think that Powerset has an amazing team, great people like Tim Converse, Chad Walters, Lorenzo Thione, Scott Prevost, Barney and they’ll have a big impact on Live Search before the end of the year your going to see significant changes and then long term as Barney likes to say, this is a 20 year vision, really the understanding what the pages are about, what queries mean. This is the front line of artificial intelligence, computer science and we’re going to be working on this for quite a long time to come.

MA: Ramez, did you say that your going to be integrating the teams right away?

RN: Yea, essentially yes.

MA: Ok, so effectively the deals close, that means that the deals closed, from an outsiders perspective, you guys are one company and you’re moving forward right now.

RN: We’re certainly laying our plans right now and talk about what we’re going to start doing. You won’t see any impact until the deal actually closes, but we have a a lot of ideas and a lot of conversations.
MA: Barney I’d like to, before we jump in too much more, I’d like for you to give a little bit of a background of Powerset, just a couple of minutes on when you first had the idea, the early days when you founded the company and just a couple minutes on that, and the basic ideas you had that resulted in you founding Powerset.

BP: Ok great. 3 years ago I was an Entrepreneur in residence at Mayfield, a venture capital firm. And I was looking at what was going to be the future of search, projecting forward in an open ended, visionary way and looking at what were the major trends. I had a previous background in a lot of artificial intelligence, and taking advanced AI technology from research labs, and getting them into the world. Either through Mission Critical, or mission operations at NASA, spacecrafts when I was at NASA, or internet search related technologies among other things.

I could see that there was going to be a huge amount of computing power becoming available over time, and that a lot of the work in AI and in particular natural language, was sort of nearing the time where it was going to be commercially ready, and these two trends would be converging just as search was becoming the center of our interactions with computers and tapping into all the information that is out there on the internet. So, I could really see that there were a set of trends that were going to converge it looked like the center of a perfect storm, I then having seen this vision, went out to evaluate how good was this natural language technology across the different groups, the research organizations, identified the key requirements that would make it work at scale, what kind of properties would a natural language search have to have to work at scale, how would the economics work now, and was now really the right time. And in my assessment of the different technologies out there, I found that the technology at PARC, after 30 years of development, had come to a point where it was actually ready to be taken out, and to be commercialized. And in principal ought to be able to work at scale. I began negotiating with PARC, with Ron Kaplan, who was leading the natural language group there for 30 years, and also with his colleague Danny. These were sort of the fundamental guys in computational linguistics over all this time.

And in parallel I found that there was another group that was related, working with the same PARC technology, already looking to apply it to search, in a research basis, and these were the folks at Fuji-Xerox Palo Alto labs including, Lorenzo Thione, and two of our other key people inside Powerset. And they had a similar vision on a research side, and they were already looking at the PARC technology and they were saying that this should be able to work. So, we actually had a shared vision, a shared recognition across all of us, that this could be possible. And Lorenzo would up saying right away, “let’s go do this, you know I want to join you”, and became a co-founder.

We then spent a long time negotiating with PARC to set up the right kind of teaming and partnership relationship, and to ultimately develop a very powerful license for this technology that would work for everyone. And during that time, we wound up, we were stealth for a while, and we would up hiring and building a great team, raising several rounds of funding, and basically building our product. And we had to build a lot of challenging infrastructure, take this natural language technology from the research labs, and really making it work on a large scale. Bringing together a world class search team, with together people like Chad Walters and Tim Converse coming on board, bringing their expertise and figure out how to make this stuff work on scale with natural language and the best of search, and then build up a product and user experience team that would be able to make this work in a way users would understand and be able to see the differentiation and like it.

Ultimately, after two years after we hired the first employees, we launched our product a couple months ago, demonstrating this capability on Wikipedia, and the response, you know we talked about this mike, the response that we had is that people generally really like the system and they just want to have it on the whole web. So I guess that’s a basic tour through the history of the company it’s only been 2 years since we hired any employees at all, and now the company is 63 people.

MA: How many of those 63 are search engineers and scientists?

BP: Most of them.

MA: Okay, how much money have you raised?

BP: We reported our series A round which we raised 12.5 million and that was including our angel bridge round. We actually didn’t report anything after that; clearly we did raise some more money, but we you know, we didn’t report anything.

MA: And what was the acquisition price?

BP: We’re not discussing that?

MA: Really, how about a ballpark? Everyone said 100 million is that where it ended up?

BP: We’re not even discussing ballparks, we’re not talking about it all together.

MA: That’s a great background of the company, but let’s talk a little bit about what you actually do that’s different in terms of thinking about Yahoo! Or Google search or search that Microsoft has today that’s keyword based and I’d love to go back to a post you did probably over a year ago now, maybe a year and a half ago, I think on your personal blog, where you talked about Powerset for the first time, and what you were trying to accomplish. From a non technical standpoint, what’s your vision for helping users search?

RN: One way to think about it is today’s systems that are out there, they don’t really understand language, so they don’t understand what a user is really saying, and the intent that’s behind the users query and they also don’t understand the documents that they are reading that they are ultimately trying to let the users find and by the way, they don’t understand the ads. So they don’t really understand anything and their based largely on statistical properties. Does this particular stream of characters appear with the right frequency in the right locations on certain pages? Does it all match? And it kind of does it pretty good job for being such a basic approach.

Now if you think about, what could you do if you had a system that could understand language? What if I could read? What if it’s already read everything in the document collection you’re interested in? Whether that’s a smallish collection like Wikipedia, or whether that’s potentially the whole Web? How could that actually help you? Well it could help in many ways. One, is you could just use more natural queries, just stating your intent as you actually mean it. Where that’s a full sentence or a question, or just a little bit of a linguistic phrase, or just some persons name. But it could understand that better and it could figure out what you want to do with this and how can I help. And then on the content side, if it could really read, then it could do a much better job matching the meaning of your query to the actual meaning that’s there in the documents. Moreover, it could present for you the results, you often have a challenge when you’re looking at search results of you see a little bit of a snippet kind of two lines worth of characters and you have to figure out from that, is that what I actually wanted? Because the system we have today don’t actually understand the queries and they actually don’t understand the documents, all they can really show you is where the keywords you asked are matched approximately in the right regions. But if they actually could understand both the documents and your query then they could present results, first of all, better two lines, or potentially a whole new kind of presentation.

MA: Just to cut in for one second the way you have described this before I have heard you talk about this is Google and other search engines look for key word batches and then present results ranked according to some sort of algorithm that determines how important a page is. You’ve said before that what Powerset does is it pre-reads the content. It uses artificial intelligence to actually try to understand what sentences mean and in the live search blog post today, the Microsoft announcement effectively of the deal they talked about a couple of examples that you know, a shrub and a tree are similar concepts that was one example, or that the word cancer could mean a disease or a horoscope. How does… Ramez maybe you want to jump in here to. How does that actually happen and what… a computer receives a sentence, your server sees a sentence, how does it actually start to parse that, again as non-technical as you can describe it.

BP: Okay, I’ll take that and Ramez, you can jump in on your examples.

RN: Great

BP: I guess one way to think about it is like when you are learning how to diagram sentences in elementary school. You draw these trees of a sentence and find here is the noun phrase and a noun phrase has a determiner like “the” and then it has a noun like “dog” and here is a verb phrase, and it might have a verb like “barks” and then what does it mean for the that word, bark is a verb and it has a “S” at the end and the way that it works, which we call morphology, that’s the present tense of that verb. And then the whole sentence is composed of those pieces, and so the meaning is built out of those. So you draw these diagrams when you are learning how to do it. And the kind of knowledge that’s in a natural language processing system like Powerset is using is sort of like that. Its basically extracting out both the surface structure, that kind of a tree structure of a sentence, and then its converting that into a series of different representation, ultimately into one which expressing that thing in fact. So it will basically say that there is a kind of activity here and it is a barking activity and the thing that is doing that activity, the subject of that activity, is a dog. Ok. So it is going from that sort of a surface structure of the language that you are seeing and converting it into a semantic factor representation. In addition, it is then able to draw on the individual meaning and relationships between words so if you saw that the sentence said “The poodle barks.” Then the system knows, if it can draw upon other knowledge about the relationship between words, as Powerset does, that poodles are a kind of dog. So if you as the user were able to say, “I want dogs barking” then it can actually then match the concept of dog to the concept of poodle and it is matching barking to barking and it is then doing this sort of semantic match for you which uses words you are not even using in your query and matching those against the document.

RN: I think everything that Barney said was right on. I think you see search engines including Live Search and also Google and Yahoo are starting to do more work on this matching not exactly what the user entered but it is usually limited to very simple things. So now all of us do some expansion of abbreviations or expansion of acronyms. If you type “NYC” in a search engine these days, in the last couple years, it understands that it means the same thing as New York. These are very very simple rules based things, and no one understands that bark has one meaning if it about a tree and a different meaning if it is about a dog. Or an example that someone gave the other day was the question of “was so and so framed.” And framed could mean a framed picture or it could mean set-up for criminal activity that did not occur, and so on. And you have to actually understand something of it is a person’s name then it applies to one sense of the word framed if it is not then it doesn’t. So one of the things that Powerset brings that is unique is the ability to apply their search technology to the query to the user’s search in ways that are beyond just the simple pluralization or adding an “-ing” is that Powerset also looks at the document, it looks at the words that are on a web page and this is actually very important. If you look at just the users query, what you have available to you to figure what they are talking about are three words four words five words, maybe even less. That can give you certain hints. If you look at a web page that has hundreds or thousands of words on it you have a lot more information you can use if you understand it linguistically to tell what its about, what kind of quieries it should match and what kind of quieries it shouldn’t match. And Powerset is fairly unique in applying this technology in the index on a fairly large scale already and with Microsoft’s investment and long term commitment we can scale this out even further, an apply it even more of the web, not just the wikipedia content they have thus far.

MA: Ramez how much work has Microsoft done in this area before today? Is it something that has been simmering, that you guys have been interested in, do you have a number of people on staff that are experts in this area, that have built technology around this? it would be interesting to know what you have done to date in this area.

RN: Well Microsoft has some leading people in natural language processing. We have applied the idea in machine translating, translating from one language to another, and in other areas of natural lanaguage, even things like the grammar checker in Microsoft Word comes out of our natural language work in some ways, and that is very exciting. The thing about the Powerset team is that it is purely additive, like the people inside of Microsoft research I have talked to about this are extremely excited. They see the Powerset team in San Francisco as great collaborators and see this as a great chance to exchange data, ideas, tools, and so on. All of this is going to help us directly. Also this is the first time we have had a focused team working just on natural language applied to search specificially, and not a broader area. With this kind of focused effort and the great technology that the Powerset team has built we’ll be able to make really rapid progress.

MA: Where are your search engineers today? Are they in Washington, or in your Mountain View office?

RN: The bulk of our team is in Redmond, and we have a small team that is in Mountain View, as well.

MA: For now is Powerset staying in their San Francisco offices?

RN: Powerset is absolutely staying in San Francisco. They have a fantastic office. I plan on staying down there a couple days a week myself. It is a fantastic location, and we want to grow the team so we are looking for more and more qualified search engineers and more and more computational linguists to join the team at Powerset, and keep scaling up.

MA: Barney, how many of you’re current employees, how many of your employees previously worked at Microsoft? Did anybody get hired back after leaving Microsoft?

BP: Actually I have not counted. I think we have a few Microsoft people, but it is not a high proportion.

MA: One of the things that has obviously hindered Powerset is that you need to index the entire web in a different way than search engines index them today because as you say your reading web pages instead of just noting key words and publicly you have said that you are not prepared to do that yet because it costs money and you wanted to prove it out with the beta product that looks at Wikipedia first. Beyond the fact that it is more expensive to index the web that way, that’s obviously, expense is not as much of an issue now that you are part of Microsoft, how long will it take. If you turned on the gas now full blast and wanted to launch a full version of Powerset that indexed the web, what is the fastest we could expect to see it.

BP: Umm, we are just getting together as a team to look technical integration and look at the best ways for our teams to work together and how we are going to combine and really leverage the resources that Microsoft has, so it is early a little early to say how long it is going to take before you see it. What I can say…

MA: Barney you have become media trained.

(Laughter)

MA: (Mocking) We are Microsoft. We cannot comment on future product releases. You gave me thirty seconds of nothing.

BP: No, No I prefaced it (laughter). I am not finished yet. With all that said about what I can or can’t actually say, what I think I can say is that Powerset has already been doing some experiments processing web pages. Arbitrary, random web pages using our technology, and those results are looking pretty good. It is already a pretty parallel system, so to some extent the basic experience you see right now could be replicated just by running the larger set of content that Microsoft already has using our technology running on the machines that Microsoft already has. Now that doesn’t mean that you would get the full search experience because there’s all the rest of the features that Microsoft has developed that we would want to integrate together to give a really coherent and good search experience. But some of the things you see already like the facts that Powerset extracts from the documents, to building profiles automatically of any kind of concepts that you have and the ability to show the pages with their automatically generated summaries. A lot of those features could really be done, at least to some level of quality today just by running it on a Microsoft infrastructure with resources that exist today. So we are going to have to figure out on what order are we developing what, but we feel that fundamentally the challenges of getting this up to web scale, the main barriers that were in our way, with Microsoft are now removed.

RN: A.) Barney is really showing his media training here, I am really impressed. His answer is also spot on, and something to bear in mind is, at this point, it has been primarily senior people across the teams that have been talking. And we really do have a very bottoms up culture inside of search. I think Powerset does as well. So we are going to connect more and more engineering teams now that we have announced this and we can start working on detailed plans. What we have super high confidence in is that this is a great fit, with great people. The cultures are actually very similar, and this is right on strategy with what we see as the big barriers to customers getting high quality results. You are going to see some short term stuff. We are going to get some stuff out there that is available to you on the live search site before the end of this year for sure. And then we are going to, as Barney was saying, take the current technology and start to scale it out out out. And will we go straight from wikipedia to the entire web? Will we have some interim stuff? I am not sure yet. But we will start scaling it up, and getting more and more benefit for customers over time.

MA: So do you think that you will launch this technology on live search, or will you launch something on Powerset, and sort of keep the brands separate for a while? Or are you ditching the Powerset brand? Have you thought about that yet?

RN: We are going to keep Powerset alive, we think it is a fantastic technology showcase, and we will probably always have some things that are really interesting to play and show people, but that aren’t quite ready yet to be exposed to all of our customers. But what really is the payoff for us is integrating the Powerset technology deep within live search, and really making that product the one that really shines, in addition to Powerset. We want to take Powerset’s technology and really broaden it out and impact tens of millions of people, if not hundreds of millions of people with the benefits of what Powerset brings.

MA: In December of last year, Peter Norwig, head of research at Google, was interviewed, and he said some things about natural language search that were interesting, and I’ll link to this when we post the podcast, but he said that, I will quote him. I would love to get your guys’ reaction out of this on just a product and science level. “We don’t think it’s a big advance to be able to pose something as a question as opposed to keywords. Typing what is the capital of France won’t get you better results than capital of France.” To me that doesn’t really respond at all to what Powerset is promising to do, and what it is already doing with wikipedia. But then he went on to talk about the limiting value, in his opinion, of natural language search. He said, “We think that what’s important about natural language search is the mapping of words to concepts that users are looking for.” He gives some examples: New York is different from York, but Vegas is the same as Las Vegas, and Jersey may or may not be the same as New Jersey. That is a natural language aspect that we are focusing on. Most of what we do is at the word and phrase level. We are not concentrating on the sentence. We think its important to get the right results rather than change the interface. What is your response to that?

RN: I think what Peter Norwig is saying has some degree of accuracy and that he is also ignoring some things. So, just for normal queries, queries that are not phrased as questions, there is a lot of linguistic structure. If someone types in a query that is “2 bedroom apartments, under 1000 dollars, within a mile of Portero Hill.” That query is loaded with linguistic content. And that’s a realistic query. That is the type of thing that customers actually want to find on the web. Today there is a sort of helplessness, where customers know that certain queries are too complicated, and they wont even issure them to a search engine. They will go to some deep vertical search engine where they can enter different data into different boxes. What is the capital of France vs. capital of France; that is not really an area that is that interesting. But some of these more complex queries really are. For example, shrub vs. tree. If I do a search for decorative shrubs for my yard, and the ideal web page has small decorative trees for my garden, it really should have matched that page and brought it up as a good result. But today Google won’t do it, Yahoo won’t do it, and Live won’t do it. So even in these normal queries there is a lot of value in the linguistics.

BP: That’s right. So in addition, Powerset just launched the product, and I think that some of the features are really well called out in the iPhone product that we just launched. It’s just another version of our web site, but designed to be used on an iphone. And Mike you have sort of blogged about it. I have been using Powerset on a mobile device ever since we launched, and it’s kind of funny because you have a very limited real estate, and you know what you want in your head but you know it is going to take a long time for the pages to come up. I see a movie, Iron Man, and I wanna know, what other movies did Jeff Bridges star in? How do you want to ask that question? How do you want to get the information? You want to say what movies has Jeff Bridges starred in? Who was that blonde reporter in Iron Man? How are you supposed to ask that? All these things that we think in our head in language, and then we have to figure out how to translate it. It doesn’t mean that you should have to do more typing to get back worse results, but it means that you should be able to do anything in the most natural method possible. We are humans, language is our unique human endowment, yet we have not been able to take advantage of that when interacting with machines.

MA: Wait, wait, wait. So I have been an internet user for 13 years now roughly, and I know better than to type a sentence into a search bar. What I would do is…

BP: That’s the learned helplessness.

MA: Yes, but what I would to is type in iron man, and look up the name of the blonde reporter from there. I have learned to do that because I have been using the internet for so long. Do you think that anyone still searches that way anymore with long sentences? It seems we tried in the early days and realized it didn’t work. So, does anyone even bother searching that way? And a follow up question would be, Barney, with regards what you are seeing in the Wikipedia engine, are you seeing longer queries sort of slowly developing as people learn to speak to a search engine?

BP: Let me answer the question of does anybody actually search this way. The answer is yes, people do this. It isn’t the most common mode, but we do see that probably 5% of queries are natural language queries. These are not all queries that are phrased in complete sentences, but they are queries where the customer has issued something that has some sort of linguistic structure. Almost any query with a preposition: X and Y, A near B, attribute A of Y, etc. Those things are loaded with linguistic structure.

BP: So there’s a couple pieces. One was does anybody do things? I think we all have the experience – if you just get your most basic expression query and your system comes back with a result that’s good enough you’re done and you’re happy. Well, what is it that happens when you don’t get back the result the first time? You have that moment of frustration and you know you’re in for a project. What happens is that moment of prayer, where you’ve basically tried a few different versions and you’re just frustrated and how do you express your query? You express your prayer and you say just let me say what I want and I know I’m not going to get results, but darn it, I’m just going to poke.

MA: I think that’s why so often Yahoo Answers pops up, because they have those questions that are literally a quotation of the question. Somebody else has asked and answered it, but that may not be the best resource for the answer, but it’s the best place to the search engine can find to send me to.

RN: I have a list of some natural language queries in front of me. Can we just show you some queries that our customers have actually sent to us and are random examples. The first person to see the dark side of the moon. How to get a credit card in Malaysia. Enabling system restore in group policy on domain controller. Timeline of Nvidia. How to measure for draperies. What is the difference between Mrs. and women’s sizes? Does my baby have acid reflux? I could just go on and on and I. These fit in the category that we’ve labeled that match about five percent of queries and they’re really just cases where the customer can’t think of a simpler way to express it.

BP: Now I’m going to elaborate Mike on you’re second part of you’re question, which was Powerset launched and have we seen that users are actually doing anything regarding natural language and if the queries look at all different.

MA: Yes.

BP: And the answer is absolutely yes. Our users have had absolutely no problem at all in throwing longer, more interesting, more complex at the system. You know, it’s just a flood of them and so when we watched the initial queries come in at launch, it was kind of a fun moment for us because it was some sense of initial reputation. There was no issue about could users use English or use ways of expressing themselves in all of their daily lives, could they actually manage to do that with a search engine if given the chance. Absolutely, if users are given the chance, the users do and users will. I want to go back about another point though. We don’t want to harbor all on the query side and expression of intent, because all of these billions of documents you’ll look at are all loaded with language. So the ability to read them in advance and extract the key information and then use that, even if you just did a small little simple query by automatically generating a profile. As for example Henry VIII I think you’ve blogged about Mike. Or, when you’re reading an article, giving you the summaries of the article.

MA: You return answers, not web pages sometimes and that’s amazing.

BP: We return answers. We actually synthesize, so if you were to say, “What did Tom Cruise star in,” you actually get not just the movies, but the cover art for the different movies. It synthesizes multiple pieces of information to give you a whole different kind of presentation. Or, if you were just to say, “Bill Gates” you’d be given an automatically generated profile of Bill Gates, pulled across many, many articles. It’s no longer just about 10 links, although we can certainly do more relevant job (and will) of the blue links, and a better job of presenting those links. With the language understanding systems which we now have, we can go way beyond that and open up a whole new door in user experience until you think, “oh god, that’s how I used to search, now I want this whole new different kind of thing.” And now the question is, which are users are asking, is how do I get this on the whole web and with this partnership we’re now going to deliver.

MA: Ok, I’m out of questions. This was really helpful. There’s a million other things that I’d love to ask, but you’re not going to answer them yet. I look forward to seeing the Powerset technology launch with a full web index and Microsoft’s ranking technology behind it. I think it’s going to be great. Ramez are you promising a full launch, or some kind of launch by the end of the year? You mentioned that earlier in the podcast.

RN: What I’m saying is that by the end of the year you will definitely see Powerset technology improving the experience for customers on Live Search.

MA: Ok. Alright guys, thanks very much for your time and congratulations to both of you.

BP: Hey thanks Mike. Bye.

RN: Bye.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

In May we reported that Powerset was in acquisition discussions with Microsoft and was hoping to bring another bidder to the table. Google was the likely candidate, but they have publicly dismissed the notion of contextual search as a revolutionary step forward. Microsoft, which is clearly interested in improving its search market share, turned out to be the best fit.

Rumors resurfaced last week about the imminent deal.

Powerset recently launched a showcase for its semantic search product, although they lacked the funds to do a full web index to prove out the product. As part of Microsoft, they won’t have that problem any longer. Now they just have to fight the bureaucracy to make sure the project continues to move forward.

The company had raised $12.5 million in venture financing, plus another $8 million or so in convertible debt as bridge financing. That means investors will get a decent return (but not a home run), and the founders and employees will also take some real money off the table.

We first covered Powerset in October 2006, and they were a TechCrunch40 company.

Update: Microsoft announcement is here, Powerset is here.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Our sources have been saying this deal is highly likely since May, but hasn’t actually been signed yet and could still be disrupted by the ongoing Microsoft-Yahoo negotiations. Dave Wehner, a Managing Director at investment bank Allen & Co. (he’s the guy who sold Bebo for $850 million to AOL), is representing Powerset in the deal.

Powerset debuted at TechCrunch40 last fall and opened a showcase of its technology to the public just last month.

Powerset has raised around $12.5 million in venture capital, and is rumored to have taken another $8 million or so in convertible debt as bridge financing.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Powerset, the natural language search engine that partially launched in May, has released a mobile version of their site that allows users to quickly search Wikipedia from their iPhone.

Since the release of the iPhone a number of sites including iPodia and Wapedia have released optimized versions of Wikipedia (though none actually made by the online encyclopedia). These sites reformat Wikipedia articles to better fit the iPhones screen while shrinking (or removing) images to conserve bandwidth.

What sets Powerset apart, and may make it the premier way to look up information from the iPhone, is the search engine’s ability to find both the relevant article and the exact passage that pertains to the search query. Even through the iPhone sports a relatively large screen, browsing through large amounts of text can still be a pain, which makes this feature even more valuable.

Powerset has lofty goals, aiming to use their natural search technology to overtake traditional search giants like Google. So far the company is only using Wikipedia for search results, so it’s hard to tell how well the technology will work once Powerset finally indexes the web, but for the time being it may well be the best reference tool on the iPhone.

You can watch a brief demo in the video below:

Powerset iPhone Web App Demo from officialpowerset on Vimeo.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

The company raised $3 million in equity at a $23 million post-money valuation. All previous investors participated, and new investors Marc Andreessen, SoftTech VC and Western Technology Investment also invested. They simultaneously closed a $1 million lease line with Western Technology Investment for server leases.

We don’t know much yet about Blekko, which was founded by former Topix founder/CEO Rich Skrenta. The company says they won’t be launching anything to the public until 2009. See our original post on Blekko for more background information.

See our coverage of Cuill as well, another hot stealth search startup we’re tracking.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Today marks another milestone for San Francisco based contextual search engine Powerset. They’ve launched a showcase for their user search experience – effectively the search engine minus the web crawl. For now, Powerset queries only Wikipedia and augments results with data from Freebase. The product launch comes just a day after reports that the company is being shopped to potential buyers by investment bank Allen & Co.

I have been able to test Powerset via their labs site for the last few weeks. I wrote about it last month, and the version that just launched is very similar.

There is no way to look at Powerset today and determine if it can be as disruptive to search as Google was when it launched almost a decade ago. That’s because it only queries Wikipedia, and so there is little need for proper ranking algorithms to sort the good from the bad results.

But what user can see is how effective a way it is to gather information quickly. For someone doing research, Powerset effectively removes a number of steps towards getting to the final information. It is particularly effective when the information needed is on many different web pages.

For example, a query on Powerset of “when did earthquakes hit tokyo” yields stunning results. Try this query at Google or even wikipedia to compare – instead of just picking out keywords that are in your query and on a web page, Powerset is actually making some sense of the content included in the wikipedia pages:

The way that Powerset returns queries means that answers are often found in the result snips, as above. They are also structuring a lot of the Wikipedia and (and already structured Freebase) data and inserting it into results. So a search for “Bill Clinton” shows results, but also shows Freebase structured data along with additional query refinements to get to more information. The important thing below isn’t the structured data in the results, its the fact that you can click on the action words and drill down into very specific queries (to find, for example, what bills he signed, or which Supreme Court justices he nominated, or who he slept with).

Powerset is indexing web pages much differently than normal search engines, which generally just record content to match against keyword queries. Instead, Powerset is trying to understand the content on the page so that it can be matched meaningfully to queries later. Even queries that don’t use matching words.

Indexing the web is expensive, though, and Powerset’s way of doing it requires even more time and computing power dedicated to a web page. That’s why they say they aren’t indexing the entire web yet – the company has raised just $12.5 million (plus another $8 million or so in bridge loans from investors). To index the web will require a new round of financing (see the first paragraph above about their sale/financing efforts).

Powerset is has taken a lot of criticism for their goal of trying to redefine how people search the web (including from us). But their lofty goals are what makes Silicon Valley so great – succeed or fail, Powerset is trying to do something pretty spectacular.

The company has also created a demo overview video – see below.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

But now the company may have to make a hard decision: sell now to one of the big Internet players looking for a point of differentiation in search, or take the risk of going it alone and possibly getting a huge, multi-billion dollar payoff down the road.

According to our sources, Powerset is exploring both options. They hired Dave Wehner, a Managing Director at investment bank Allen & Co. (he’s the guy who sold Bebo for $850 million to AOL, and is working on LinkedIn’s huge financing), to represent them in a possible sale or financing.

CNET is reporting today that Microsoft may be bidding for the company. According to our sources, those discussions have been going on for well over a month, and their most recent bid is “around $100 million.”

That probably won’t be enough to convince Powerset and their investors to sell. The big question is whether Google will step in to try and keep Powerset out of Microsoft’s hands, and start a real bidding war. That could drive the price significantly higher. Google, however, has publicly dismissed the notion of contextual search as a revolutionary step forward.

Whether that’s true or not is yet to be seen. But Powerset may find itself as a valuable chess piece in the emerging search war between Google and Microsoft. And if Google bets wrong, they could find their commanding lead in search eroded over time. A relatively small acquisition to keep Powerset out of Microsoft’s hands, even if just a hedging move, may suddenly be attractive to them.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Today, Henry Blodget & Co. at Silicon Alley Insider try to peg valuations on 25 private Web companies. Facebook is at the top of the list, but it is valued at $9 billion instead of the $15 billion that Microsoft’s investment put on the company. Why? Because everyone knows that the $15 billion is too high, so SAI decided to apply a 25X multiple on Facebook’s 2008 revenue forecast of $350 million. Does that make its valuation correct? Probably not. But in the absence of any true market pricing, anyone can go ahead and make a guess.

The same goes for any of the valuations on the SIA 25 list, which puts Wikipedia’s worth at $7 billion, Craigslist’s at $5 billion, Mozilla’s at $4 billion, LinkedIn’s at $1.3 billion, Ning’s at $560 million, RockYou’s at $325 million, and Spot Runner’s at $250 million. Note that three of the top five (Wikipedia, Craigslist, Mozilla) are essentially not-for-profits sitting on very valuable assets. The valuations for those three are based on what they would be worth if they were run differently with an eye towards maximizing revenues—which, of course, could impact how consumers interact with them, which in turn would impact their valuations.

Another 25 startups make up the contenders list, which includes Federated Media ($245 million), Yelp ($225 million), Meebo ($220 million), Mahalo ($150 million), Digg ($125 million), Etsy ($115 million), Powerset ($80 million), and Twitter ($75 million). A full list that changes dynamically every 20 minutes, based on changes in the Nasdaq, can be found here (although, exactly how the valuations are linked to the Nasdaq is never clearly explained)

Some of these valuations have more merit than others. Some have none whatsoever. For instance, SAI gets at its $125 million valuation for Digg by “splitting the difference” between a $200 million buyout rumor we reported and the $60-to-$80 million that Kara Swisher came up with. Splitting the difference between two rumors is not exactly the height of financial analysis.

But what are you gonna do? At least SAI acknowledges that the list is an imperfect work in progress. Don’t get too caught up in the actual numbers. It is more useful really as a starting point to think about relative valuation between different startups. Is Meebo really worth three times as much as Twitter? Is Ning worth as much as Slide? Let the parlor game begin.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

An early version of the search engine, which was demo’d to me yesterday at their offices, has been available to some users of their Powerlabs site. But for the most part, it’s been kept very quiet.

The early version of the service will serve as a showcase for the user interface and engine itself, but it will not have a full web index behind it. For now, Powerset will query only Wikipedia and Freebase. But when I tested the service I had something very similar to the “Aha!” feeling that ran through me the first time I ever used Google. In short, it is an evolutionary, and possibly revolutionary, step forward in search.

I’ll temper that statement since the company is not putting anything more than a tiny index of two sites behind the service for now. In particular, the fact that Powerset doesn’t have to bother with spam control and other relevance issues (which is what made Google so great when it launched), means it can’t yet be considered any kind of challenger in the search space. But anyone who uses it will be able to see the potential value of the engine when it is placed in front of a full web index.

For now the company is keeping specific features of the engine confidential, but I can say it has evolved significantly since a screen shot was released in mid-2007.

In preparation for the launch, some of the Powerset team have vowed not to shave until the product is released. They are chronicling their facial hair adventure on a site called Powerstache, which has been covered by Jessica Guynn at the LA Times.

Rumors have also been swirling around the company in general. A number of sources have said that Powerset is pitching for additional capital. And the company also appears to have put plans to hire a new CEO on hold – founder Barney Pell is still firmly in charge at the company.

Powerset is one of three new search engines that we’re keeping a close eye on. The other two, Cuill (pronounced “cool”) and Blekko, are still deep in stealth mode.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

But back to Blews. It’s a news aggregator (see Techmeme and about 45 others, including this gem), but it goes beyond mere clustering of stories to show what’s important right now based on who’s linking to what in near real time. Blews, which is only looking at political news, also tells you the bias of the links in to a story:

BLEWS uses political blogs to categorize news stories according to their reception in the conservative and liberal blogospheres. It visualizes information about which stories are linked to from conservative and liberal blogs, and it indicates the level of emotional charge in the discussion of the news story or topic at hand in both political camps. BLEWS also offers a “see the view from the other side” functionality, enabling a reader to compare different views on the same story from different sides of the political spectrum. BLEWS achieves this goal by digesting and analyzing a real-time feed of political-blog posts provided by the Live Labs Social Media platform, adding both link analysis and text analysis of the blog posts.

Here’s what all that looks like:

Liberal links are blue (rasberry) and on the left, conservative links are red (cherry) and on the right. The middle is the story itself in white (lemon). The dots around the edges suggest the emotional charge of the commentary, which can drip off of the Rocket Pop in very hot weather.

I note that no one on the team (Michael Gamon, Sumit Basu, Dmitriy Blenko, Danyel Fisher, Matthew Hurst and Christian Konig) is a user interface specialist or web designer.

Putting aside the UI, which is hard to do, the artificial intelligence behind Blews could be interesting. It is very hard to get a machine to decipher emotion and meaning from raw text unless they are doing mere keyword searches (see, for example, Powerset). Microsoft is calling this hard bit “detecting emotional charge.” If they’ve got it right, or are close, there are an unlimited number of potential applications for the technology.

As an aside, this somewhat reminds me of ScoutLabs, a startup we wrote about last December. Scout Labs helps brand marketers track commentary on their brands, and tries to decipher emotion towards that brand as well.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Circos is categorized under the ever expanding umbrella of semantic search engines, which currently includes the likes of Hakia, PowerSet, Kosmix, SemantiNet, Quintura, and TrueKnowledge. However, the engine is most like Kango, which has also taken on the task of categorizing hotels based on user reviews. VibeAgent also has a search engine for its own site that will search hotels based on qualities.

While Kango auto-generates tags after pouring through user reviews, Circo lets users search for any qualities they’re interested in. The engine then grades and ranks the results by each quality on an “A” through “F” scale based on how well the description fits for reviewers. For example, a hotel reviewers feel is spacious would rate highly if searching for openness, but poorly if you’re looking for a tiny room.

As with most search engines, Circos’ real test will be whether its application draws users away from other hotel and restaurant sites with less sophisticated search engines. Currently there are a bunch competing in the space. However, Circos says their technology can easily be extended to other categories since their algorithm does all the tough work of pulling the most relevant qualities from reviews. If hotels and restaurants don’t appeal, another category may hold their home run.

Circos is angel funded, based in San Mateo, and has eight employees (4 in Singapore).

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Skrenta left Topix last June. He started his new company, Blekko, almost immediately, along with five others from the Topix core team. They raised $2 million in seed funding in September from Baseline Ventures, two early Googlers (David DesJardins and Jeremy Wenokur), and the founding team.

The company is still deep in stealth and, apparently, working out of a garage in true startup style (see image below). The Blekko website, which today has nothing on it except a picture of a puppet created by Skrenta’s daughter, isn’t even close to having a landing page up, let alone the final product. But eventually Skrenta says they’ll launch a full scale search engine to compete with the big guys.

Skrenta, who’s very media savvy, won’t say much about how he’s going to tackle search (he’s not a fan of PageRank though:“PageRank wrecked the web. Google is the cause of all of this. and Google is going down with it.”). He says they are looking at improvements on the back end (indexing and query serving) as well as the user search experience itself. Beyond that, he says we have to wait. And it might be a long wait at that. The company, Skrenta says, may not have a public prototype available until 2009.

Normally an entrepreneur announcing they’re taking on Google with a six person team and just $2 million in funding would either be laughed at or ignored. In Skrenta’s case, he has proven himself more than once as capable of taking on big challenges and winning. This will be a company to watch, and speculate on, in 2008.

There are other promising search startups out there. Powerset, Cuill (we’ll be hearing more about them soon) and the upcoming Wikia Search Engine are all yet to launch. Mahalo is growing fast (but still tiny). Can anyone unseat Google? Perhaps not any time soon. But you don’t have to get much market share to be a huge winner in this space – every 1%, they say, is worth a cool billion dollars.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

We don’t think it’s a big advance to be able to type something as a question as opposed to keywords. Typing “What is the capital of France?” won’t get you better results than typing “capital of France.”

True, true. But he does acknowledge that there is some value in the technology:

We think what’s important about natural language is the mapping of words onto the concepts that users are looking for. . . . To give some examples, “New York” is different from “York,” but “Vegas” is the same as “Las Vegas,” and “Jersey” may or may not be the same as “New Jersey.” That’s a natural-language aspect that we’re focusing on. Most of what we do is at the word and phrase level; we’re not concentrating on the sentence. We think it’s important to get the right results rather than change the interface.

In other words, a natural-language approach is useful on the back-end to create better results, but it does not present a better user experience. Most people are too lazy to type in more than one or two words into a search box anyway. The folks at both Google and Yahoo know that is true for the majority of searchers. The natural-language search startups are going to find out about that the hard way. If Google doesn’t trounce them first.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Today they will announce a second fund, Founders Fund II. It’s much larger – $220 million. And unlike the first fund, the money comes mostly from outside investors. The new fund will allow Founders Fund to make 15-20 new investments, including pro-rata investments in follow on rounds.

A couple of investments have been made out of the new fund, they say, but have not yet been disclosed.

Founders Fund partners have deep connections in Silicon Valley, which help with deal flow (Peter Thiel, founder and former CEO of Paypal, Ken Howery, founder and former CFO of PayPal, Luke Nosek, founder and former Vice President of PayPal and Sean Parker, founder and former CEO or President of Napster, Plaxo and Facebook). But they also approach deals differently than most other funds.

Sean Parker said today in a phone interview that a glut in venture capital, combined with reduced capital needs of most startups, has led to a shift in balance of power between entrepreneurs and VCs. Founders Fund recognizes that shift and has evolved does deals a little differently because of it. For example, they invented and promote the issuance of a special class of stock, called Series FF, which allows entrepreneurs to take money off the table much earlier in their company’s lifecycle. They also allow significantly more liberal voting rights to founder board members than many other funds. See this article in the SF Chronicle earlier this year for more on how they do business.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

At the Web 2.0 conference, Pell gave an impressive demonstration of Powerset’s search technology, although it was restricted to a limited data set. How the search engine will do against the entire Web, which is a much bigger technical challenge, has yet to be seen.

But this shakeup does raise a big question. Why step down as CEO and leave a huge leadership gap (with no COO either) before you find a new CEO to take things over? Perhaps this was done more for internal reasons. Announcing everything all at once sends a signal to employees about the direction of the company, and minimizes future surprises. The CEO search also indicates that Powerset may finally be ready to open up its search engine to the general public sometime next year. Or perhaps Powerset’s board has become impatient with the company’s progress and wants new leadership. You can read Pell’s explanation about the transition here. (You can read our previous coverage here).

Crunch Network: CrunchBase the free database of technology companies, people, and investors

See the screen shot (click for larger view) – users are shown a query and a number of results and are asked to evaluate the relevancy of each result from five choices. In this case, the query is “revealing bikinis.” Users are asked to evaluate four sets of results within ten minutes, and are paid $0.02 for the effort.

The current batch of Powerset projects have run their course, and there are currently no other projects available on Mechanical Turk.

I spoke with Powerset CEO Barney Pell this evening who confirmed that they are using Mechanical Turk to get human feedback on search results. He says the results are not all Powerset generated – rather, they show results from Powerset, Google and others to see which users prefer for a given query. He also says this is an ongoing project, and new ones will be added soon.

Pell also said that Powerset plans to use Mechanical Turk over the long haul, even after launch. They’ll put actual user queries into Mechanical Turk in real time, add Powerset and competitor results and see which results people find more relevant. If results suggest Powerset isn’t more relevant, they’ll adjust their engine.

Powerset also uses the EC2 computing service, another web service offered by Amazon. They recently released some of their internal growth models that allow people to compare the relative costs of EC2 to building out a real data center.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Powerset

Powerset is a natural language search engine that can use everyday phrases and grammer to conduct more accruate web searches by understanding the search query and the pages it indexes. Parsing phrases and grammer theoretically produces better results because the egine has a better understanding of the searches intended goal than with just keywords alone. For instance, a Powerset search for “politicians who died in office” returns information on the subset of politicians who died in office, rather than a group of pages that ranked highly with the phrase.

Powerset presentation begins: talk about semantics and search, “we parse the web”. Natural language search.

Announcement: Powerset labs, where users can explore tech demos, share ideas, feed the learning engine and “improve your search karma”.

Demonstration of natural language queries with a social voting style feature. Touches of other sites

Demonstration of Powermouse (see screen shot), information is pulled from Wikipedia into a semantic index.

TC40 attendees will be amongst first in private beta.

Overall: tough sell in the search vertical, but interesting take. Great start to TC40.

Cognitive Code

Cognitive Code makes artificially intelligent user interfaces. Their main product is the SILVIA (Symbolically Isolated, Linguistically Variable, Intelligence Algorithms) platform, which can add a human-like artificially intelligent interface to nearly any digital device. The SILVIA platform can learn and converse in natural language to carry out tasks for the user. Potential applications include children’s digital toys and personal assistants.

Flagship product: “silvia platform” Symoblically isolated linguistically variable intelligence algorithm. Laymens terms: AI.

Demonstration with AI on the screen, the AI system is having a conversation with one of the Cognitive Code. A couple of bugs in the live demo, but pretty cool.

Uses include embedding in toys, phones, websites “unlimited uses.” First major target market is “smart toys.”

Clever idea, if they can pull it off we’re seeing the future of toys.

CastTV

CastTV is trying to build one of the web’s best video search engines by creating a rich index of contextual data about videos and an easy to use interface for searching them. The engine pieces together context for a video based on it’s metadata, the content surrounding it, and the content of pages linking to the video. Notably, CastTV also searches paid video searches such as Apple iTunes. Their user interface allows users to sort results by shows (to weed out non-relevant stuff), host (such as itunes, CBS Innertube, etc to focus on a favorite service provider), by date, relevance, prices, etc.

Presentation begins: CastTV doesn’t host videos, they index them.

Britney Spears video search compared, Google, Yahoo and CastTV: CastTV results are pitched as being better, more accessible etc

Colts Titans next example. CastTV is using smart clustering for results, pulling video from MSM and user generated content. Nice results, even if I have no interest in American Football

FAROO

FAROO is a peer-to-peer web search engine that has no centralized index and crawler. Each web page visited by users is automatically included into the distributed index. Ranking of search results is based on a distributed usage statistics of the web pages visited by FAROO users, which leads to a more democratic, user centric ranking. FAROO also shares advertising revenues up to fifty percent with its users. The search engine uses privacy-protected behavioral targeting to increase conversion rates.

Interesting concept, P2P in a strict sense. Results are only pages that have been visited by users…I cant’ help but think the SEO crowd is going to love this

The presenter claims that the algorithms actually prevent manipulation: he doesn’t know the people I know. Nice results though.

Indexing via a desktop P2P client, demonstrated version on Windows. Faroo beta opens today.

Viewdle

Viewdle is a white-label platform for indexing, searching and monetizing video. The technology they are developing lets video producers algorithmically extract metadata from news, shows, movies, and Internet video. This is much more effective than the old method of text-based metadata indexing. Viewdle’s most notable feature is their facial-recognition technology that can create a create a “real-time index of true on-screen appearances”. They plan on building one of the largest databases of people-in-video references. Reuters is currently testing out Viewdle’s technology with their videos news inventory by letting people search their catalog for specific people.

Demo starts with 2 minute demo video. Slick, we’ll see if we can get a copy.

The facial recognition is always an interesting concept, but I’m remind of Riya. More Britney Spears examples, although they are pulling data from others in the video as well, it looks a step from previous tech, particularly give it’s video they are scanning, not just pics.

Product: Top Chance, scans on criteria, including date. Popularity search includes total video time and when. Platform (presuming API) will also be released shortly to plugin widgets etc.

Expert Panel: Ryan Block Chris Anderson, Marc Andreessen, Om Malik, and Marissa Mayer

First question Marc Andreessen to Powerset, great question, how do you break out, API’s etc. Good response.

Chris Anderson: what are the advantages of the various products to the user

Faroo responds first: we are by the user, for the user, it’s good because “they are doing the search together”

Om Malik to Faroo: most P2P systems people turn off, how do you overcome that, also how do you seed the network?

Faroo: it’s not a problem…not a particularly good response.

Marissa Meyer wants to know about the video search startups, scaling etc…classic

CastTV: we’re scaling, focus. Viewdle “we reference a point” hence can scale to billions, using “fusion engine”

Discussion continues around AI and natural language tools.

Jason asks Om: which one is the most viable. Om: CastTV. One to last: Cognitive Code. Middle of the road pick: Powerset. Faroo is “interesting,” Viewdle will be “acquired soon”

Jason Calacanis to Marissa Mayer: will people switch away from Google. Reply: most people use more than one search engine according to stats. Google’s advantage is being a one stop shop. JC: what did you think of CastTV, MM: nice interface, clustering for duplicate issues is good tech.

Marc Andreessen: I don’t want to be obsessed with distribution…but I am, how do companies deal with it
Powerset: we’re very aware of this…uploading to users (???), embeding on external sites (Google custom search style I’d think).

Conclusion: speaking to Nick and we agree that CastTV was the winner in a very competitive group, good tech which just works with a practical use. Cognitive Code had the coolest product, but the demo wasn’t great which lost it for them.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

In a less than shining moment, Caitlin Upton, the 18 year old Miss South Carolina Teen, answered a fairly simple pagent question with a nonsensical answer:

Q: Recent polls have shown a fifth of Americans can’t locate the United States on a world map. Who do you think this is?

A: I personally believe that U.S. Americans are unable to do so because, uh, some people out there in our nation don’t have maps and, uh, I believe that our, uh, education like such as in, uh, South Africa and, uh, the Iraq and everywhere like such as, and I believe that they should, uh, our education over here in the U.S. should help the U.S., uh, should help South Africa and should help Iraq and the Asian countries, so we will be able to build up our future for our children.

Not one to miss a PR opportunity, yet-to-launch natural language search engine Powerset took a shot at parsing her answer so that queries could be run against it. Based on the query “Who does education help?” the index returned the result “Americans.” That’s an impressive result, given the nature of the data being queried.

The test shows the potential usefulness of Powerset as a search engine. The query does not match the content based on a keyword match, and the answer can only be determined via a contextual analysis of the data.

Powerset tends to look very good in demos against a limited index, as the above example shows. but it still has to prove that it can index and analyze large chunks of the web to become a viable competitor to Google and other search engines. That’s going to be their biggest challenge (and cost). Powerset still has much to prove as they prepare to launch.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

An example: they announced Powerlabs, a sandbox for users to suggest and give feedback on future Powerset features. People who sign up for Powerlabs are also promised early news, at least an hour before it is posted on the Powerset blog.

Another example: In May Powerset COO Steve Newcomb talked about how the company was predicting future growth, and posted data on their model on the company blog. When readers bravely requested that Powerset release the model itself, Newcomb complied, saying it would be made available this summer. In a post on his personal blog he said the reason for sharing the models was to show that the company intends to be open and give users unfettered access to information:

As I mentioned before, opening up our modeling techniques is part of a larger goal to begin the process of changing our image of a secretive stealth startup to a completely open company that gives you unfettered access to our product(s), the ability to help us design them and to provide insight into the way we think inside of Powerset.

Today, Powerset published the first in a series of models, with a Flash interface. Company-specific baseline assumptions have been removed or altered, but most of the industry assumptions remain intact.

Neal Mueller (Powerset Product Manager) walked me through the models and how they work. This first set helps a company that intends to index the web whether it is better to purchase, lease or create virtual servers on Amazon EC2. Assumptions about the size and refresh frequency of the index can be changed. Since the model is forward looking, it also makes assumptions about future server power and cost reductions from Moore’s Law.

All of the assumptions can be altered in the Flash interface, and the models can be embedded into other websites (although I could not get it to properly embed here).

Mueller says that at least two more dashboard models are coming – one for unique user forecasting and another one that they are not yet disclosing. The company is asking for feedback on the models, and will clearly take it seriously. Newcomb’s personal email is listed on the front page and he requests that feedback come directly to him.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

FactSpotter is text mining software that combines a linguistic engine that allows users to make queries in everyday language. FactSpotter looks for the keywords contained in a query along with the context those words have.

According to Xerox, FactSpotter is capable of combing through almost any document regardless of the language, location, format or type; take advantage of the way humans think, speak and ask questions; and discriminate the results highlighting just a handful of relevant answers instead of returning thousands of unrelated responses.

Frédérique Segond, manager of parsing and semantics research at XRCE said that the tool is more accurate because it delves into documents, extracting the concepts and the relationships among them. “By understanding the context, it returns the right information to the searcher, and it even highlights the exact location of the answer within the document”.

Whilst it sounds appealing, FactSpotter will not be coming to a browser near anyone, anytime shortly. Xerox plans to launch FactSpotter next year as part of the paid Xerox Litigation Service platform and has no plans for a wider or public release. Here’s betting that a Steve Jobs character comes along and steals the concept and turns into the next Google; history often does repeat itself.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Powerlabs is more than a sandbox to show off new product ideas that aren’t ready for prime time. They are encouraging people interested in Powerset to sign up for Powerlabs and create what is effectively a profile. Once registered, users will be able to see new product ideas and vote on them, as well as submit their own ideas to the community. Later on, users will gain points and influence within the community.

In the demo today the company showed me one idea that they will be putting into Powerlabs once it launches – a mashup of Powerset natural language results along with video results, keyword results and, interestingly, Freeweb results (the two companies are working together). See the screen shot to the right (click for larger view).

Powerlabs hasn’t launched yet, but you can register for it via email on the Powerset home page. They are sending out regular communications to those users, including news at least an hour before its posted on the Powerset blog or given to press. In an email today, Powerset revealed that they have acquired another company, although they did not give any further details.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Powerset should publish the model itself (without the specific Powerset assumptions of course) and let other startups tweak it for their own use. Most new companies don’t have the excel jockeys or the time to do this kind of work. Any competitive issues would be overshadowed by the considerable goodwill (and link juice) they’d get from doing this.

And that’s exactly what they are going to do. Newcomb says they’ll release the models in the next few weeks, minus Powerset-specific data. Don’t expect to see these on Google Docs, though (a perfect place to dump them, except that Google is who Powerset is gunning for). They’ll make the spreadsheets available for download or let people view them on the web via a Flash viewer.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

For those of you unfamiliar with Powerset, it is a new, well funded search engine that aims to allow users to write their queries in natural language. In a blog post in October 2006, CEO Barney Pell wrote out some of the ideas driving the company.

A couple of days ago the company showed a sample query and result on their blog. Normally boring stuff, but a lot of people are dying to get more information on the product. We’ve copied the screen shot above.

The query is “politicians who died in office.” This is a much better query that previous examples like “books by children” v. “books for children.” Those queries can be handled fairly well by Google by simply putting quotes around the query. But for “politicians who died in office” the results on Google won’t be as good. Context is required: Google has only six results for the query in quotes, and without quotes it loses its meaning and the results aren’t useful (notice the Powerset blog is the fourth result). The Powerset results are relevant and useful.

Hand picking a query here and there and showing a screen shot of results isn’t the same as killing Google. But it does show that Powerset has the potential of being extremely useful by attacking search from a different angle. I look forward to their launch.

Crunch Network: CrunchBase the free database of technology companies, people, and investors