Last month news broke that a team of computer scientists had finally managed to improve Netflix’s recommendation algorithm by 10%, making them eligible to win the $1 million Netflix Prize, a competition that began back in 2006. The team, BellKor’s Pragmatic Chaos, is composed of two former leaders in the competition who banded together in attempt to finally break the 10% barrier and managed to succeed with a score of 10.08%. However, their announcement kicked off a 30 day window where other teams were invited to make their final submissions and potentially take the prize. Tonight, with just one day remaining, a team called The Ensemble has managed to overtake BellKor with a score of 10.09% — an improvement of .01% over the former leaders. You can see the full leaderboard here.
According to its homepage, the Ensemble is made up of two teams who chose to join forces: “Grand Prize Team” and “Opera Solutions and Vandelay United”. The Ensemble has just posted the following blog post to its official site:
We are happy today to have made a submission which surpasses Netflix’s 10% Grand Prize target. The Ensemble is the second team to do this in less than a month. We are very proud of our achievements, and those of our top competitor, Bellkor’s Pragmatic Chaos.
The competition will end tomorrow morning, so teams still have a little bit of time left to make their last-second submissions, but things are looking good for The Ensemble. This has to be absolutely brutal for team BellKor.
Update: With only minutes remaining in the last call period, BellKor’s Pragmatic Chaos submitted an entry that tied The Ensemble, with a score of 10.09%. Twenty minutes later, and only four minutes before the competition ended, The Ensemble struck back with a buzzer-beating 10.10% submission.
Thanks to TechCrunch intern Dan Romero for the tip.
Talk about a last-second victory.
we still don’t know if pragmatic chaos have a hidden ace for the last moment
That is a no mercy winning strategy. Well done.
Does this mean recommendations for Glitter will finally and mercifully stop once and for all?
Ouch.
Look for an 10% increase in drinking tonight from team BellKor.
++
Surely the 30-day window should start again from now.
This really is brutal for BellKor.
My thoughts exactly. The window should reset.
i would support the resetting of the window. this could possibly lead to more innovations from the two teams and others. in the end netflix would still win.
The 30-day window needs to restart!
Watch oiut for last minute submission from TeamZip FTW!
i ‘d say the window was a bad idea from the start. they should allow for one final window for everyone.
excellent example of standing material. Fortunately, the author simply genius.
Hmm, it’s anybody’s ballgame right. May the good one wins!
Yes, this team’s drinking capacity will definetly go up by at leeeast 10% …
)
Netflix could announce a $500,000 second prize, and really leave everyone feeling good. Half million is a drop in their marketing bucket.
What are you kidding? If you were an investor in Netflix would you want them throwing away $500k? Do you think that would get them thousands more users? No! Users don’t know about the contest.
Geesh, sometimes engineers have no business sense at all.
That’s less than 5 engineer’s for a year. I would bet dollars to doughnuts Netflix has way more than that working on their algorithms now. Plus the managers, hr, and other people to support the engineering effort. Just think of this as deferred payment for offsite work.
On top of being able to defer payment they got some of the smartest most published AI researchers to work and publish on this exact problem space.
While this may not get them hundreds of new members if it keep members around then it’s done it’s job. Keeping members is way cheaper than advertising to get new customers.
If I was a netflix board member I would think that was the deal of the century.
Sheesh business types. No long term business sense at all.
PS:
The test result set means nothing. The actual prize comes for fitting the real data set that netflix has. It’s totally possible one or the other has been over fitting the test data and they will not win.
And last I heard BelKor actually won the import part
i agree that the window should have been extended – this is pretty much equivalent to “snipe bidding” on ebay. however, without including it in the original rules, there probably isn’t much they can do.
buzzer beater for the win
If BellKor is smart and it sure seems they are, they wouldn’t have been sitting idle during this window since they know everyone is gunning for them. They would be working just as hard to improve their own algorithm.
They have been. Their last submission was 2:30 hours before The Ensemble’s latest submission.
This is an example of why businesses throwing innovation contests to the public is completely unjust. Someone usually loses pretty hard.
Sure, by entering the contest the entrants know the risk but in the end, the team Bellkor did a lot of work, quality work that reached the goal, and not only may they not get anything for it, the work may not get used.
It’s no worse than those logo websites where designers compete and may not get chosen. It’s spec work. Does anyone here like doing work on speculation?
Good for Netflix for pushing innovation and reaching out to the public. Bad for Netflix for using a contest format.
Members of BPC one the last two Progress Prizes (2007 and 2009). I would hardly say they got “nothing”.
That said, your premise is ridiculous. They knew the rules and they made the decision to compete. How is it “bad for Netflix for using a contest format”? If they hadn’t, they wouldn’t have had the level of competition and quality work that they did. The work that BPC will not go “unused”. There are plenty of applications that can use this technology, and BPC is free to sell it for that purpose.
Your definition of “justice” only makes sense in a weird world where no one works for compensation.
Unjust? Are you high? Clearly you have not clue what motivates researchers. I guarantee you the BellKor team enjoyed every minute of their participation. More importantly, the research community in recommendation systems benefited enormously from the Netflix prize. The research & results from many of the teams has advanced the field well beyond what would have happened without it.
Kudos to Netflix for sponsoring the prize. The challenge was incredibly well designed — not too easy, not too hard — just f’ing right! Whomever came up with the rules of the competition deserves some serious commendation.
You are also ignoring the fact that Netflix freed millions of dollars worth of data to the world. This data would never have been available, and Netflix could have used it to improve their algorithms by about 3-4% (still a huge amount) while their competitors would have got nothing.
As it stands now, their competitors can take advantage of the research done with free access to Netflix data, while Netflix foots a million dollar bill…
@campsteve – Competition is unjust? Maybe so, depends on how you consider and define justice, but that’s the real-world of business.
Competition is unfair? What absolute horseshit. If you don’t like it don’t enter.
I know of at least 2 developers of collaborative filtering software who decided not to enter this competition and do something else instead. By choice, go figure..
Also the teams who lose will still have developed some of the best recommendation software around and will have publicity for being so close. I’m sure they can develop businesses out of it without a huge problem.
Excellent example of the power and leverage of prizes! Also excellent example of how hard prizes are to design and why it is so important to make sure the rules set are solid and thought through.
Efficiency prizes should ALWAYS have follow on prizes to keep innovation moving. So, in this example, award the $1M to the best team that accomplishes the goal, and then set another $1M prize of x% more efficiency (7.5%?) in 6 months? This provides an incentive for the winning team and other teams that were close (and new teams) to keep innovating… not to mention the additional media and good will for NetFlix.
In each competition there should also be a second prize of $100,000? in order to reward the second place team and to give them some fuel to keep working.
Kudos to NetFlix again for using prizes.
Keith
* Formerly with the X Prize Foundation and Prize Capital, LLC.
Regardless of whether a contest format or not, does the actual construction of the “question” the Netflix Prize is trying to answer even have a bearing on the success of recommendations? We’ve been tackling this question in a series of articles here: http://blog.med...ound.com/?p=419.
UPDATE!!! Now they’re tied!!! Check the leaderboard now: http://www.netf...om//leaderboard
BellKor resubmitted today with 10.09% and a tied score of 0.8554
I wonder if they give it to The Ensemble for being the first to reach the number or if they split the prize equally.
Correct that – The Ensemble is up to 10.10 now, and has regained the lead.
It is not fully public yet, but “BellKor’s Pragmatic Chaos” is the winner.
Even though the leaderboard shows “The Ensemble” at the 1st place, it is only on the Quiz set, which approximates the important one – the Test set. However, on the Test set BellKor’s won.
OK. So now it is 9:48 PDT. Who won? Somebody, please?
I find the Netflix prize about as relevant as the X Prize.
It’s like the TREC conference was before they introduced the “Topic Distillation” track — TREC’s information retrieval products were embarrassingly bad compared to Google because TREC’s evaluation methodology couldn’t see that Google was better than previous web search engines.
Similarly, the Netflix prize is using a means of evaluation that won’t necessarily lead to a better user experience, and it’s pushing people towards using a particular data set which might not be the most enlightening about the subject.
I’m much more impressed by NanoCrowd’s reccomendation engine, which uses semantic information to rank movies by similarity — it digs into a richer source of meaningful information and produces a product that’s much better for finding interesting movies.
The flip side of it is, who cares about movie reccomendations anyway? I suppose Netflix does, because it means more rentals, but from my viewpoint, there are many more good movies to watch than there is time to watch movies. The people I know who watch movies all day are on disability or mooching from their parents, so they’re not a particularly desirable market.
There are a lot of areas where shopping is painful, online or not, and that’s where people should be focusing effort, not in areas where improvements aren’t going to change anything.