Linus's World

The Prussian statesman Otto von Bismarck once summarized the unseemly side of politics with a quip: “Laws are like sausages. It’s better not to see them being made.”

The same would hold true for open-source software development – were it not for the fact that open source, by definition, implies transparency. In the case of the Linux kernel – the core of the widely-used open-source operating system – that openness means users are able not only to review the underlying source code and count the number of profanities in the source documentation, but also sit in on the internal debates that shape even the most minute design decisions.

Witness the drama surrounding what most Linux insiders now call the “BitKeeper fiasco,” an event that tested the authority and leadership skills of Linus Torvalds, 35, the original creator of the Linux kernel. When a key Linux development tool disappeared from the scene last spring, Torvalds created a replacement from scratch rather than watch the Linux community defect to a tool he didn’t like. In the process, he demonstrated the unique mix of brilliance and pigheadedness that have attracted legions of programmers to his cause since 1991.

The crisis was triggered in early 2005 when it was confirmed that Andrew Tridgell, an Australian hacker best known for reverse engineering the Windows NT networking protocol, had done the same for a a proprietary source code management tool called BitKeeper, published by South San Francisco-based BitMover.

Bitkeeper is a popular commercial program for managing distributed software development projects. Originally built by San Francisco programmer and entrepreneur Larry McVoy to suit Torvalds’ own management needs, BitKeeper had occupied a tenuous niche in the open source community. McVoy had modeled the software on a prior system developed for Sun Microsystems. Like its predecessor, BitKeeper offered peer-to-peer flexibility and a closed source license. Linux users could have free access, but only if they gave up the right to tinker with or copy the internal source code.

That more than a few Linux developers accepted this bargain is a testament to BitKeeper’s power. Offering speedier source code evolution and relieving kernel maintainers from the firehose-like torrent of unsolicited changes, BitKeeper turned what, until the end of 2001, had been a “push” process into a “pull” process. In other words, developer could use it to move changes back and forth quickly through the kernel system while still maintaining a clean, error-free source code tree. After fumbling numerous patches and irritating many top-level developers in the process, Torvalds officially deemed BitKeeper the best fit for a community itself built on the notion of small groups, loosely joined.

“Maybe I’m asocial or something, but I think people tend to naturally form pretty small groups of five to ten people,” says Torvalds, summarizing his embrace. “I really think that the thing that makes kernel development scale to bigger numbers is the network of these groups.”

An outspoken pragmatist on the issue of software licensing, Torvalds gambled on the kernel community’s willingness to trade transparency for efficiency. By March 2004, that gamble seemed to be paying off. In a press release issued by BitMover, the company measured a 130 percent increase in the number of files added to the Linux kernel during the two-year development cycle after the switch.

Within a year, however, the more idealistic wing of the open source community would have its say. Following a suggestion on the part of free software pioneer Richard M. Stallman, Aussie hacker Tridgell began writing a free software client that could pipe developer data through the BitKeeper system, effectively sidestepping BitKeeper’s closed nature. Rather than lose its tentpole product to the hacker hordes, BitMover promptly withdrew its Linux license, leaving Torvalds to handle the negative public relations fallout.

Torvalds responded with a unique mixture of hubris and humility. He was loath switch to a tool he considered less desirable, such as CVS (the Concurrent Versions System, developed by Stallman’s GNU Project). Instead, Torvalds sketched out the rough workings of a file system that would allow him and other managers to monitor source code changes and “roll back” to previous changes in the event of an errant design decision. In a grudging acknowledgment of his own stubborn unwillingness to use CVS, Torvalds dubbed the new system ‘Git.’

“ ‘Git’ can mean anything, depending on your mood,” warned Torvalds in the tool’s attendant README file, publicly accessible (of course) via the Kerneltrap.org website. “Stupid. Contemptible and despicable. Simple. Take your pick.”

Self-deprecation aside, the resulting e-mail traffic offers a rare glimpse at an alpha hacker defending his title. Insisting that developers treat Git not so much as a replacement for BitKeeper as the internal “plumbing” which others could use to build such a replacement, Torvalds offered withering scorn to any hacker who dared so much as suggest room for complexity.

“The fact is a lot of crap engineering gets done on the question of ‘what if?,’” he wrote in response to one security-minded inquiry. “You are literally arguing for the equivalent of ‘what if a meteorite hit my plane while it was in flight?’”

By May, the harsh e-mails were starting to dissipate as Git quickly took shape. A team of hackers recruited, for the most part, from the fringes of the kernel development team had delivered in one month a passable open source workaround. While not exactly competitive with BitKeeper, Git, much like the Linux kernel before it, seems well on its way to standard acceptance.

Looking back, kernel hacker H. Peter Anvin, sees it all as an emotional release. “When the BitKeeper fiasco broke, it turned what had previously been a political problem into a technical problem,” he says. “We’re a lot better at solving technical problems.”

As for Torvalds, he sees it as yet another tribute to the power of brutal honesty. “On the Internet, nobody can hear you being subtle,” Torvalds says. “I’ll happily be abrasive and opinionated if it helps get issues out in the open and gets people into the conversation. The real magic ingredient is being able to change your mind occasionally so that people know you’re an opinionated bastard, but that it might be worthwhile talking to you anyway.”