The Non-Intuitive Nature of Collective Intelligence

October 21, 2007 by sean.

Rob Brown on developing a bottom-up collective intelligence model for improving Netflix recommendations:

What was striking to me was that this system, iterating over a massive amount of sloppy, low precision data, could organize the model with such stunning precision. I could type in the names of two movies, and ask “how similar” they are, and the results were almost always exactly what I would expect. I could type the name of a movie, and get a list, in order, of the top 20 movies that are seen as most similar. And it did quite a good job at the assigned task, predicting how users would rate movies. Those who claimed the process couldn’t work, after seeing the results, were shocked.

The point, of course, is that this system is very evolution-like, in that lots of messy data, with very little apparent “intelligence,” processed by a simple iterative algorithm, can find sophisticated equilibria with a great deal of precision. Looking directly at the raw data, such as at an individual user’s set of ratings, would indicate a lot more slop than is apparent in the final model. The system doesn’t “know” that a movie is a science fiction movie, any more than natural selection “knows” why a particular mutation in the DNA increases the chance of an animal surviving to adulthood. Nonetheless, it works, against all intuition. [via karmatics.com]

Posted in collective intelligence | No Comments »