Group Rethink

Common sense is uncommon in individuals and, at first blush, seems even more so in groups. No one expects crowds to produce useful thought. We fear the tyranny of the majority and mob rule, avoid peer pressure where we can, and immediately see the aptness of Charles Mackay’s 19th-century book title Extraordinary Popular Delusions and the Madness of Crowds.

But the idea of collective intelligence shouldn’t seem so far-fetched. After all, democracy is built on the principle that large groups know how to govern themselves. Commodities markets, which set prices on the basis of group knowledge, play a growing role in everyday business decisions. Even lone geniuses build on the work of others: for every Einstein there are Poincares and Lorentzes and Hilberts lurking in the background.

In fact, evidence of collective intelligence is all around us, and New Yorker writer James Surowiecki collects much of it in The Wisdom of Crowds. [Surowiecki wrote on technology and happiness for the January 2005 issue of Technology Review.] Surowiecki shows how groups can often outthink even the most knowledgeable experts. He offers proof after proof that “the value of expertise is, in many contexts, overrated.” By recounting how the stock market divined that booster rocket manufacturer Morton Thiokol was most to blame for the Challenger shuttle disaster (the official answer came six months later), or how the U.S. Navy found the sunken submarine Scorpion by aggregating the best guesses of a variety of experts, Surowiecki demonstrates that collective intelligence can be harnessed, and that it does not have to be unwieldy. Collections of experts, he concedes, are prone to the ills of groupthink, which can lead to debacles like the Bay of Pigs. But he argues that crowds with certain characteristics – notably, diversity of opinion, independence of opinion, decentralization, and a way to aggregate opinions to arrive at a collective decision – will generally outsmart their most brilliant members. This is true for specific problems and broad ones, Surowiecki says, and for crowds big and small. His premise quickly comes to seem intuitive.

Groups, then, can act as parallel-processing decision engines, pooling disparate knowledge to answer even hard questions in areas like public policy. What we lack, however, is a reliable way to build such decision engines. And Surowiecki’s book, unfortunately, offers no practical solutions.

But technologists, entrepreneurs, and venture capitalists do. The last several years have seen intense interest in developing technology that improves our connectedness (see “Tagging Is It,”). In part, that interest was spurred by the phenomenal success of open-source software, which is built by communities (see “How Linux Could Overthrow Microsoft,”). It also reflects the success of Google and eBay, which have profited by harnessing the collective behaviors of very large groups. Connecting technologies like online social networks and Web logs, or “blogs,” are familiar to many people, and wikis – group Web pages that any member may edit – soon will be (see “Larry Sanger’s Knowledge Free-for-All,” January 2005,). Technologists, then, are already attacking the problem of how to achieve a high group IQ.

Better communications tools are one ingredient. Indeed, Thomas W. Malone of MIT’s Sloan School of Management argues in The Future of Work that ever cheaper and more-useful communications technology will effect a revolution in the way businesses operate. E-mail is the obvious example, but Malone also points to artificial electronic markets, which can aggregate employees’ best guesses about sales, resource allocation, research and development efforts, and even pollution control. That last was done at BP, which used an internal futures market rather than a committee of experts when it wanted to find ways to reduce its emissions.

Malone says such markets, combined with blogs and other technologies that make it easier for employees to share information, will enable, for the first time in business history, “the economic benefits of large organizations, like economies of scale and knowledge, without giving up the human benefits of small ones, like freedom, creativity, motivation, and flexibility.” He is convinced that companies like Google, which uses internal blogs to keep management ranks flat, represent the future of industry. Tomorrow’s companies, he predicts, will be led not by dictatorial, alpha-ego CEOs but by “cultivators” who understand that productivity and profits soar when all of a company’s intellectual capital is being tapped.

Other institutions are also being remade through technologies that marshal collective intelligence. Dan Gillmor’s We the Media shows how blogging, the Short Message System (SMS), and corollary technologies like Really Simple Syndication (RSS) are creating a new and vital kind of journalism. (In the interest of full disclosure, I should mention that I am friendly with Dan.)

Gillmor believes that such technologies have brought us to a turning point in media history: with the removal of major barriers to information distribution, such as the need for a broadcast license or a printing press, more people can provide journalistic observation, and that makes for a better-informed populace.

It can also make for better media. An excellent illustration involves Jane’s Intelligence Review, the respected defense periodical. In 1999, Jane’s posted a draft of an article on cyberterrorism to the massively popular online discussion forum Slashdot, whose denizens vetted the document so thoroughly that Jane’s decided to rewrite it from scratch.

Gillmor does not claim that blogging, SMS, and the like are perfect. He is concerned about the ease with which technology can be used to promulgate untruths (he remembers the faked picture of John Kerry with Jane Fonda at an anti-Vietnam rally). He worries about “trolls” (people who post disingenuous, irrelevant, or obscene messages in order to get attention) and “spin doctors” (people who deliberately post misleading items). But he explains how the communal character of blog culture mitigates many potential excesses: bloggers who are uninteresting don’t get linked to; those who make false assertions can be pilloried. What frightens him more is the prospect that governments and the mainstream media will try to slow or even derail bloggers, SMS news services, and other emergent forms of journalism through defamation or copyright infringement laws.

Like many, Gillmor also believes that networked technologies could make the political process more democratic. For at least a decade, the Internet has been hailed as the antidote for big campaign contributors’ undue influence on electoral outcomes. Until recently, it was a quack cure. But the Howard Dean campaign’s successful use of the Internet in the 2004 Democratic primary race suggests that, as Gillmor puts it, “American politics was approaching a tipping point.” Joe Trippi, Dean’s former campaign manager, agrees.

In his book The Revolution Will Not Be Televised, Trippi contends that the Dean campaign would never have taken off had he not employed communications technology like Meetup, a website that facilitates real-world gatherings. He argues that in 2004, the Internet was to the presidential election what television was in 1956 – something present in 75 percent of homes but not truly understood by most politicians and political operatives. But by 2008, Trippi says, the Internet will be at the heart of the political process. America, in Trippi’s view, is now run by the 631 people who collectively raised between $100 million and $150 million for the Bush campaign. But technology will make it possible for ordinary citizens to band together, piling small donations into substantial political war chests and speaking with a voice every bit as powerful as those of the special-interest lobbies. The model for all this? The campaign of Howard Dean, of course.

What Trippi identifies as the Dean campaign’s strength could just as easily be described as its main problem: it was propelled by its supporters and therefore had no strategy for directing, or even much understanding of, the forces it had unleashed. The campaign had no clue, Trippi makes clear, that thousands of people would show up at Dean meetups at a time when the candidate was barely mentioned in the national press. Nor did it expect to see millions of dollars in small contributions coming in via the Internet.

Yet it’s beguiling to envision campaigns that are shaped as much by voters as by the candidates themselves. Trippi seems to be that rare political professional who would be happy to find that, in the next campaign cycle, he was out of a job.

An influence upon all these books is Howard Rheingold’s Smart Mobs, which looked at the way groups of people use cell phones and other wireless devices to organize collective action. Rheingold wrote that “the most far-reaching changes will come…from the kinds of relationships, enterprises, communities, and markets that the infrastructure makes possible.”

Creating a communications infrastructure that fosters a healthy democracy has been a concern of the United States since its founding. Newspaperman and intellectual Walter Lippmann once noted that the real trouble with both the press and representative democracy is “the failure of self-governing people to transcend their casual experience and their prejudice by inventing, creating, and organizing a machinery of knowledge.” That machinery may finally have arrived.

Michael Fitzgerald is a writer who lives outside Boston. He writes frequently for Technology Review, and his work has appeared in the Economist, the New York Times, and other publications.