Intelligent Machines
Software Frontier
The Cathedral and the Bazaar
It wasn’t until Frederick Jackson Turner published The Significance of the Frontier in American History in 1893 that Americans fully realized there had been a frontier. In what was probably the most significant and controversial piece of scholarship of the 1890s, Turner showed that the nation’s identity in the 19th century had hinged on westward expansion; now a new identity would be required.
“Open-source software” is another one of those concepts that gives historical unity to decades of bewildering change. The term didn’t enter circulation until early 1998, but in the two years since, the revolutionary idea behind it has caught hold in boardrooms and IT departments across Silicon Valley and beyond.
That idea, first set out in writing by freelance programmer Eric Raymond in 1997, is that free software developed by large, informal networks of expert programmers-working not for money but for the respect of their peers-is usually more elegant and serviceable than “closed” or proprietary software sold for profit. The concept had been part of hacker culture for years, but Raymond provided the most accessible and precise formulation.
In The Cathedral and the Bazaar, a 1997 white paper circulated on the Internet, Raymond explained how the spread of the Linux operating system had crystallized his thinking. (The new hardcover by the same name contains the white paper and two related essays.) Linux is a free version of the Unix operating system, and was written by University of Helsinki student Linus Torvalds with a legion of collaborators starting in 1991 (see “Programs to the People”). From the start Torvalds made the source code available over the Internet. As users tinkered with it, discovered bugs, and suggested fixes, a global virtual community of Linux developers sprang up. The approach worked. Today, Linux is running on nearly one-sixth of business server computers and is gaining fast, largely at the expense of Microsoft’s Windows NT.
Torvalds and his collaborators had set out to show that they could build a better operating system for free than IBM or Microsoft could with their billions. Some commentators explained their success as the result of the Internet’s ability to nurture fluid working communities. But ittook Raymond, who calls himself “the hacker culture’s tribal historian and residentethnographer,” to recognize that this was only part of the story. He came to believe that Linux had succeeded by turning the traditional software development model-based on hierarchy, central planning and secrecy-on its head.
Before Linux, Raymond writes, “I believed that the most important software needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with not a beta release before its time.” By contrast, the Linux community resembled “a great babbling bazaar of differing agendas and approaches.” Linux’s Internet archives were overflowing with versions of the system, and new releases appeared with alarming frequency. “Linus was treating his users as co-developers in the most effective possible way,”Raymond concluded. “Release early. Release often. And listen to your customers.”
As Raymond’s conversion took hold, he realized that the open-source faith had been around long before Linux. Similar processes had given rise to some of the crucial parts of the global computing infrastructure, such as TCP/IP, the Internet’s standard transmission protocol; SMTP for e-mail; and Apache, the software that powers more than 60 percent of Web servers. For hackers, Raymond says, building systems that benefit all is partly a form of play and partly a matter of prestige. Programmers volunteer to tackle complex problems because it’s exhilarating to find a good solution, but also because turning that solution over to the community is a philanthropic act that reflects well on the giver. Properly guided by a winsome visionary like Torvalds, in Raymond’s view, these forces can produce software that puts commercial products to shame. After all, open-source programs reflect the best efforts of the best minds, have been debugged by hundreds or thousands of reviewers-and they come with 24/7 “customer support” from the programming community.
“Factory software,” by contrast, offers none of these advantages. “The closedsource approach allows you to collect rent from your secret bits, but it forecloses the possibility of truly independent peer review,” Raymond writes. And worse: “When your key business processes are executed by opaque blocks of bits that you can’t even see inside (let alone modify) you have lost control of your business.”
The threat that customers might take back control by choosing open-source software has become a serious worry for software giants. In the words of the famous Halloween Memo, an internal strategy document leaked from Microsoft in 1998,”Recent case studies provide very dramatic evidence…that commercial quality can be achieved/exceeded by OSS [open-source software] projects…. The free idea exchange in OSS has benefits that are not replicable with our current licensing model and therefore present a long-term developer mindshare threat.”
Other companies are jumping on the bandwagon in order to avoid being run over. Raymond’s white paper played a direct role, for example, in Netscape’s decision in 1998 to make public the source code for its 5.0 Web browser. And while open-source software itself can’t be sold for a profit, an important new industry is springing up to service it. Red Hat, a Linux-support company, watched its market capitalization soar into the billions shortly after its initial public
offering last August.
Both as an overview of the opensource movement and as a case study in economic and cultural change, Raymond’s book is a great read. The author is far from impartial, but he is realistic; he admits there are areas, such as office applications, where closed-source software is still appropriate. In the end, however, he
believes that the entire computing infrastructure-operating systems, the Internet and communications software-will be open source,”cooperatively maintained by user consortia and by for-profit distribution/service outfits with a role like that of Red Hat.” If that happens, The Cathedral and the Bazaar could well be remembered as the most important book on the software frontier of the 1990s.
