Netflix Boosts Prize Economics

By at least one metric – the number of people who have mentioned it to me – my brief post about Netflix appears to be my most popular one so far.

The post linked to a remarkable slide deck about the corporate culture that Netflix has embraced in its quest for excellence. Most memorable line: “adequate performance gets a generous severance package.” If you haven’t seen it, I encourage you to click on over. It’s worth your time.

Yesterday’s award of the first Netflix prize highlights another strength of Netflix’s culture: it clearly does not suffer from “not-invented-here” syndrome. Indeed, quite the reverse. A few years ago, Netflix realized that it had reached its limit in trying to improve the accuracy of its movie recommendation system. Even though users may rate dozens (or more) movies, it turns out to be difficult to predict what other movies they will like.

So Netflix decided to outsource this problem in an ingenious way: it offered a $1 million prize to any person or team that could improve the recommendation algorithm by at least 10%. Stated that way, the problem seems deceptively easy. But it took nearly three years before the winner – a team led by AT&T Research engineers – took home the prize.

As recounted in Netflix’s press release, this marathon ended in a race to the wire:

“We had a bona fide race right to the very end,” said [CEO Reed] Hastings. “Teams that had previously battled it out independently joined forces to surpass the 10 percent barrier. New submissions arrived fast and furious in the closing hours and the competition had more twists and turns than ‘The Crying Game,’ ‘The Usual Suspects’ and all the ‘Bourne’ movies wrapped into one.”

…

Netflix said “BellKor’s Pragmatic Chaos” edged out a team called “The Ensemble,” another collaboration of former competitors, with the winning submission coming just 24 minutes before the conclusion of the nearly three-year-long contest. The competition was so close and the submissions so sophisticated that it took a team of external and internal judges several weeks to validate the winner after the contest closed on July 26.

Happily, the resulting algorithm won’t be exclusive to Netflix:

The contest’s rules require the winning team to publish its methods so that businesses in many fields can benefit from the work done. The winning submission and the previously hidden ratings used to score the contest will be published at the University of California Irvine Machine Learning Repository. The team licensed its work to Netflix and is free to license it to other companies.

On the first day of my microeconomics class, I told my students that economics is all about incentives. As an example, I used the famous prize for a way to measure longitude, which inspired the invention of the chronometer (i.e., a clock of sufficient precision to measure longitude). Next time around, I will mention the Netflix prize as well.

P.S. Not one to rest on its successes, Netflix has already announced plans for a second Netflix prize. This one aims to find a better way to recommend movies to people based on demographic data (e.g., where they live) rather than movie ratings.