DLA, PLSI, SVD, ICA, SDD, NNMF (non-negative matrix factorisation), LLE (locally linear embedding) and so forth are content-based, ie, they use the word-by-document frequency matrix to find similarity, which they’re all different to Google PageRank which is link-based (hubs & authorities of pages). These 2 types of algorithms are based on different theoretical foundations. However the dimensional reduction algorithms (DLA, PLSI, SVD, NNMF , SDD, LLE ) can be used in conjunction with a link-based algorithm such as PageRank to enhance websearch results (ie, make it more relevant). Google is reported to have adopted LSI into its search engine to work hand in hand with its PageRank.

The codes for LDA (latent dirichlet allocation), had been freely made available to the general public in the last 2 years or so, see the bottom of this page for download info.

We launched a prototype “semantic search engine” on Saturday on ESer.org.

We’re using a variation of probabilistic latent semantic indexing called latent dirichlet allocation to search through a part-of-speech tagged corpus of the English Wikipedia. Our search engine is using the June ‘08 version of WIkipedia, which can be downloaded as a ~40gb XML file.

The entire search engine is running on a two-year-old dual-core AMD operton 940 with 8GB of RAM. It’s using a erlang mnesia database (the entire site is written in erlang) running on a RAID-0 array of western-digital 320GB SATA disks (those are at least three years old).

The hosting bill is about $15/month. The hardware was free

The total time investment to create this prototype was about three weekends worth of coding.

The only other expense was some pizza

I sure hope Evri didn’t spend too much time or money doing this…

We’ve had about 1100 users, each running about 3.2 searches, since we launched on Saturday.

However, I do not think that the best application for PLA, PLSI, or SVD is web-search.

This class of algorithms is probably much more useful for mining private datasets. Therefore, we’re currently working on open-sourcing our code (after we’ve cleaned it up a bit) under GPLv2 on Sourceforge. We think a debugged-implementation of semantic-search/algorithmic-recommendations might be much more useful to people who want to mine their own private MySQL/Postrgesql databases and/or private intranets (specifically, windows file shares) — datasets that lack the link graph metadata of the web.

We should have the source code for ESer.org available for download within the next few weeks.

Please let us know if there are any specific data sources (other than MySQL and intranet shares) that you would like us to include support for out-of-the-box.

Cheers,

ESer.org

Like Linkedin + Alpha (yahoo)?

brilliant.

http://labs.google.com/sets

that said, evri probably had no problem shaking a few million out of the mouth-breather VCs for this talks-good premise…and it was likely fun to build it

I see Mahalo as web 0.5 and Evri as the 2.0 improvement. Of course now Web 2.0 is probably more of a slur than an accolade.

Whatever what one wants to call it, they’re pretty much the same thing. Online relationship matching, online product recommendation (eg, Amazon), online item popularity ranking or online site search do use the same algorithm, they’re only different in the domain of application. A good example here is the use of LSI (latent semantic indexing) in online product recommendation engine. LSI was primary first applied in text search search engine and from Neil Roseman’s description of his system, it is actually what LSI does, ie, match the relationship amongst terms in a corpus of documents to determine their similarities.

Now LSI has found applications in online sentiment analysis, image retrieval system (matching similar images), online product recommendation (similar to Amazon), online person attribute matching (ie, matching different people in a social network, etc… of how close their interests are).

So, Neil Roseman’s product is really search, but a slightly target domain from pure search, but actually, the algorithm is really a search algorithm and I wouldn’t be surprised if he is using LSI (or some variants of its, since there are a few variants of LSI).