Collective Intelligence vs. The Wisdom of Crowds

David Edery, who was until recently part of the CMS staff and now works for Microsoft, has been generating some interesting discussion over on his blog, Game Tycoon, about how games might harness "the wisdom of crowds" to solve real world problems. It's an idea he's been promoting for some time but I only recently had a chance to read through all of his discussion. He starts by describing the growing academic interest that has been generated by James Surowiecki's The Wisdom of Crowds and then suggesting some of the challenges of applying these concepts in a real world context:

Despite a lasting surge in media, business, and academic interest, proven mechanisms via which to harness the wisdom of crowds remain in short supply. Idea markets have existed for many years, as have the "opinion aggregation" systems in websites (i.e. the user-generated product rankings found in Amazon.com). The chief obstacle is and always has been: how to properly incentivize the participants in a system, such that they generate meaningful, unbiased input.

There is, however, one well-known mechanism that does an amazing job of incentivizing people to think seriously and passionately about a given set of problems. A mechanism that compels people to meaningfully compete, against other people or against themselves, for no monetary benefit whatsoever. That's right -- video games.

For many years now, developers have been creating games that revolve around real-world problems such as resource development, political maneuvering, etc. One of the most famous of these is called SimCity; in it, players are taught to grapple with zoning issues, tax rates, etc. What if games that encouraged people to solve real-world problems (as a means of accomplishing larger objectives) were developed in tandem with corporate or government sponsors? Not "business games", but commercially-viable, entertaining games that consumers might not even recognize as out of the ordinary?

Imagine a SimCity-esq game in which the player is given the financial reins to a region. The game could be set in a real location (i.e. California), incorporate real world constraints (i.e you can't indulge in deficit spending forever), and could dynamically import the latest available real-world regional data via the Internet (i.e. demographic figures, current spending levels, etc). That way, when players begin a new game, they are immersed in a situation that closely resembles whatever situation California's politicians are currently grappling with. But here's the catch: once players get out of the tutorial phase, the game can begin recording their decisions and transmitting them to a central database, where they are aggregated into a form of "collective vote" on what actions to take (i.e. raise the sales tax or lower the sales tax). If the Wisdom of Crowds is correct, the collective choices of 100,000 game players in California (which would include knowledgeable people as well as many less-knowledgeable people) may very well be better than the choices of 1,000 Californian policy experts.

The idea of using games to collect the shared wisdom of thousands of players seems a compelling one -- especially if one can develop, as Edery proposes, mechanisms for linking game play mechanics with real world data sets. Indeed, Raph Koster -- another games blogger who has been exploring these ideas -- does Edery one better, pointing to a project which actually tested this concept:

What [Byron Reeves] showed was a mockup of a Star Wars Galaxies medical screen, displaying real medical imagery. Players were challenged to advance as doctors by diagnosing the cancers displayed, in an effort to capture the wisdom of crowds. The result? A typical gamer was found to be able to diagnose accurately at 60% of the rate of a trained pathologist. Pile 30 gamers on top of one another, and the averaged result is equivalent to that of a pathologist -- with a total investment of around 60-100 hours per player.

At the risk of being annoyingly pedantic, however, this debate keeps getting muddied because participants are blurring important distinctions between Surowiecki's notion of the Wisdom of Crowds and Pierre Levy's notion of Collective Intelligence. Edery uses the two terms interchangeably in his discussion (and to some degree, so does Koster), yet Surowiecki and Levy start from very different premises which would lead to very different choices in the game design process. Surowiecki's model seeks to aggregate anonymously produced data, seeing the wisdom emerging when a large number of people each enter their own calculations without influencing each other's findings. Levy's model focuses on the kinds of deliberative process that occurs in online communities as participants share information, correct and evaluate each other's findings, and arrive at a consensus understanding.

Here, for example, is how Surowiecki describes the contexts where his ideas about the wisdom of crowds apply:

There are four key qualities that make a crowd smart. It needs to be diverse, so that people are bringing different pieces of information to the table. It needs to be decentralized, so that no one at the top is dictating the crowd's answer. It needs a way of summarizing people's opinions into one collective verdict. And the people in the crowd need to be independent, so that they pay attention mostly to their own information, and not worrying about what everyone around them thinks.

Raph Koster picks up on this aspect of Surowiecki's model in his blog discussion:

The problems with this sort of approach, of course, are that people influence each other. When monolithic blocks appear within the group, you'll start to get inaccuracies. When apparently authoritative sources of information start broadcasting their impressions of reality, it'll distort the result. The results in markets are bubbles and crashes. The result, perhaps, in democracies, is ideological partisanship.

Koster extends this key point in a subsequent blog post:

Technically, Surowiecki's conception of "wisdom of crowds" is ONLY applicable to quantifiable, objective data. The very loosey-goosey way of using it to discuss any sort of collective discussion and opinion generation is a misrepresentation of the actual (and very interesting) phenomenon.

You can summarize the core phenomenon as "given a large enough and varied population offering up their best estimates of quantity or probability, the average of all responses will be more accurate than any given individual response."

But this is of very narrow application -- the examples are of things like guessing weight, market predictions, oddsmaking, and so on. The output of each individual must be in a form that can be averaged mathematically. What's more, you cannot use it in cases where one person's well-expressed opinion can sway another, as that introduces a subsequent bias into everything (which is why the wisdom of crowds doesn't always work for identifying the best product on the market, or the best art, or the like).

Using it for subjective things, such as opinions on politics, is a mistake for sure. And using it as a shorthand to describe the continuous editing and revision that appears on Wikipedia is also a mistake.

Wikipedia does not operate by wisdom of crowds. It operates by compromise and consensus, which is a very old mechanism (whereas the wisdom of crowds phenomenon is of relatively recent vintage).

The Wikipedia, as I discuss in Convergence Culture, depends on what Pierre Levy calls "collective intelligence." In the classic formulation, collective intelligence refers to a situation where nobody knows everything, everyone knows something, and what any given member knows is accessible to any other member upon request on an ad hoc basis. Levy is arguing that a networked culture gives rise to new structures of power which stem from the ability of diverse groups of people to pool knowledge, collaborate through research, debate interpretations, and through such a collaborative process, refine their understanding of the world. If Koster is suggesting that the "wisdom of crowds" works badly when confronted with the challenges of politics in a democratic society, Levy sees "collective intelligence" as a vehicle for democratization, feeling that it provides a context through which diverse groups can join forces to work through problems. As I suggest throughout Convergence Culture, there are all kinds of ethical and intellectual issues to be resolved before we can say we really inhabit the knowledge culture Levy describes.

The Wisdom of Crowds model focuses on isolated inputs: the Collective Intelligence model focuses on the process of knowledge production. The gradual refinement of the Wikipedia would be an example of collective intelligence at work.

In terms of games, think about Jane McGonigal's discussion of ARGS and the ways that a community of gamers can solve problems of enormous complexity simply by tapping expertise of individual members as needed. Here's how McGonigal defines the Alternate Reality Game:

An Alternate Reality Game is an interactive narrative or immersive drama, played out both online and in real world spaces, taking place over several weeks or months, in which hundreds, thousands, or tens of thousands of players come together online to real play, not role-play, forming unusually collaborative social networks, and working together to solve a mystery or problem, that would be impossible to solve alone.

McGonigal's essays and talks have identified a number of design techniques which insure that people need to collaborate in order to play the game and discuss the various mechanisms which have emerged to allow players to pool their knowledge as they work through complex challenges.

Compare this with what Edery says about tapping the wisdom of crowds through game play:

Crowd intelligence can fail (and fail spectacularly) when there's too much information passed between members of the crowd. Members start to alter their opinions based on the opinions of others, which skews the results. The online communities that build up around any popular game would seem to promote exactly this kind of skew.

In other words, one model sees the emergence of online communities as a bug which threatens the value of the game's research while the other sees online communities as a feature which enable us to process information in more complex ways than could be managed by any individual member. To tap the "wisdom of crowds", Edery has to find ways around all of those things which McGonigal and other advocates of "collective intelligence" are building into their ARGs:

* Use competition to discourage group-think. The scope of information-sharing is typically more limited when players (in any game genre) are working to best other players. Of course, blocks of information-sharing players will still form (in formal teams or otherwise) but that's not necessarily a critical problem.
* Online game communities typically form (the most persuasive) opinions about the objective aspects of a design mechanic; i.e. "you're better off using the shotgun than the pistol, except when you're fighting at a great distance." But if a challenge and its feedback mechanism both incorporate real-world data, as I suggested in my earlier article, it becomes harder for any individual (or the community as a whole) to form clear strategies around.
* Encouage population diversity to decrease the likelihood of groupthink. Distributing a game in different countries and courting players of different ages are both examples.

Both "collective intelligence" and "the wisdom of crowds" offer productive models for game design but we will get nowhere if we confuse the two. They represent very different accounts for knowledge production in the digital age and they will result in very different design choices.

Henry Jenkins is the co-founder of the MIT Comparative Media Studies Program.