Are you familiar with Very Short List? If you are, you can stop reading here. If not, check it out. I personally think it’s one of the best online services out there. The idea is simple – a daily email…

jt
http://www.FireMe.To/udi

1. They implement wisdom of the crowds phenomenon wrong – users must vote independently of each other, meaning they should not see what’s popular and what’s not. That kind of puts Digg in a catch 22 – in order to have a truly clever voting process, ALL or MAJORITY of users should vote on ALL or MAJORITY of content (up and/or down). That’s next to impossible to achieve and also defeats the purpose of having a page with most popular stories b/c people tend to vote for those and ignore the rest. On top of that, it has been shown that out of Digg’s 2.7mil users, about 35% are spammers, of the rest 65% about 95% are lurkers, of the remaining few percent another 95% are just blind voters, and only a fraction of percentage actually submits something. That turns Digg into a niche site at best and a self-promotion competition at worst. No wonder they couldn’t get the asking price of $300mil.

2. The number of news sources on Digg is VERY limited. I have personally used their API to run some stats my results show a perfect Pareto result – 21.7% of the sources accounted for 83% of the news. And that analysis covered the entire Digg dataset from Jan 2006 to April 2008. So much for the long tail. If you don’t believe me do it yourself. Just write a Ruby/Python/Perl whatever script to pull the popular stories from Digg going back to their beginning, put them in a DB and run some queries. You’d be shocked at how much hot air the Digg long tail is.

3. What Digg is doing with their latest feature is simply a somewhat primitive collaborative filtering. Showing me what other people vote for that overlaps with mine is OK but far from valuable. That’s not much different from showing me the front page. Similar voting patterns are far from similar tastes. What truly should be done is to build a model for EVERY user of what that user likes and then for upcoming news select the news based on CONTENT that would appeal most to every user individually. That’s where value is – finding what’s interesting to YOU. That’s what a recent MIT study called the future of the Web – on the fly personalization of the site that adjusts to you and to every other user’s preferences.

Some of my fellow grad students and I are building an engine that implements all of the above mentioned features, so hopefully we’ll manage to expose the sham that Digg is. I can’t believe that someone would be willing to pay $300mil for a niche site with a largely immmature user base, which on top of that is dominated by spammers. Until the technology industry realizes that building a vaporware business and selling it for a lot of money (only to collapse later) is NOT success, the bubbles will keep on coming, bursting and wasting a lot of money that would be better used somewhere else.

Good luck Digg, see you on the field soon!

It sounds like a fairly standard user-based collaborative filter. One problem with these is that they require a dense dataset to be effective, and if they are limiting it to 30 days of user activity, it is hard to see how the dataset will be dense enough. Put simply, user-based collaborative filters require a *lot* of data before they will work effectively. Unless you are Amazon or Netflix, you probably don’t have enough data for it to work well.

I have an interest to declare: I’ve been working on a collaborative filtering technology called SenseArray (http://sensearray.com/) for over a year now which is specifically designed to deal with a scarcity of data. It does this using a type of SVD collaborative filter, combined with the ability to use metadata about users (eg. browsers, operating system, geographic location, etc), and items being recommended (eg. keywords, categories, author, etc).

SenseArray already powers Newsgator’s website (http://newsgator.com/), and we’ll soon be rolling it out elsewhere for applications as diverse as selection of dating partners, to targeted advertising.

facebook provides a similar recommendation engine with its “People You May Know” feature. probably uses a similar algorithm, and it has been successful in getting people to “friend” me.

how did Reddit’s feature fail?

Overheard:
Arrington: Rose, want to be on my leadership panel at tc50?
Rose: Yes!
Arrington: Perfect, Heather will send you the contract – the terms are simple, we require 24 hours prior notice of any Digg announcement for the next 5 years. You will always be listed as “source”
Rose: Yes!