Software's Ultimate Sandbox

Building 9 did not look happy. It was another drizzly October day on the rain-drenched Microsoft campus in Redmond, Washington-and the mood outside only augmented the tempest within. Offices were being cleaned out, desks and file cabinets either missing or abandoned, boxes of equipment stacked in the halls. Many inhabitants of the sleek two-story structure had opted to work at home in the face of a triple horror: no computers, no e-mail, no Internet.

Since its 1991 inception, this foray into the future has gone full-out to woo the best and brightest computer wizards. The lab has grown rapidly, overrunning its headquarters and sprouting offshoots in San Francisco, Cambridge, England, and, as of last November, Beijing. While their names might not ring bells in Peoria, a swarm of digital legends now pad Building 31’s carpeted halls, among them laser printer inventor Gary Starkweather, personal computer pioneer Butler Lampson,

Multimedia oracle Linda Stone, and graphics gurus James Kajiya and Alvy Ray Smith. Microsoft’s digerati are busy shaping a future of startling 3-D images, machines that talk and respond to a person’s expressions, and lifelike Web-roaming agents. Among corporations, perhaps only IBM and Lucent Technologies boast larger computer science efforts, and the lab rivals programs at top universities such as Carnegie Mellon and MIT.

Seven-plus years into the lab’s existence, though, it’s getting to be put-up time. While researchers have placed code in a wide range of products, the lab has fallen short of its stated aim of providing true breakthroughs. And without real eye-opening advances to its credit, Microsoft Research has yet to answer the question on the minds of many computer industry watchers: Can a giant assailed for its lack of innovation-whose cornerstone products such as DOS, Windows and Internet Explorer spring largely from purchased technology-find a way to innovate from within?

not surprisingly, microsoft’s research managers answer “Yes.” With the goal of uniquely blending the free-thinking of academia with the business mantra of shipping products, the brain trust in Redmond believe they’re on track to follow in the footsteps of General Electric, AT&T, IBM and other major corporations that used their dominance to build precedent-shattering research labs. That may well be. But the upstart venture has not yet earned the full respect of industry watchers and rival organizations. The word is that Microsoft’s expanding galaxy of computer science stars still has a lot to learn about innovation before taking center stage in shaping the future of personal computing.

The whole Microsoft research enterprise started with polymath Nathan Myhrvold, now the company’s chief technology officer. Four years after graduating from UCLA at 19, Myhrvold earned a doctorate in mathematical and theoretical physics from Princeton; he then won a fellowship under Stephen Hawking at Cambridge University, where he studied quantum field theory in curved space-time. More recently, the 39-year-old Myhrvold has dabbled in mountain climbing and formula car racing. He even makes sporadic appearances as an assistant chef at Rovers, one of Seattle’s best French restaurants.

Myhrvold hooked up with Microsoft chairman Bill Gates in 1986, when Microsoft purchased Dynamical Systems, a Berkeley software company the mathematician had founded. Five years later, as head of advanced development, Myhrvold proposed creating a research laboratory to help Microsoft take charge of its future. His “vision statement” argued that the best way to ensure continuing access to strategic technologies was “to do it yourself.” That idea appealed to Gates’ paranoia. Even as his company grew dominant-ultimately provoking antitrust investigations that led to the company’s trial last fall in federal court-the software kingpin still worried about garage inventors killing his business with an unforeseen innovation. So in July 1991, Microsoft announced a new commitment to longer-term research.

The move hardly brought it into the ranks of the great industrial research organizations such as Bell Labs or IBM. Still, the creation of Microsoft Research marked a first for the software industry, which had always concentrated on developing its next product generation. By establishing the lab, Microsoft signaled its intent to be a player in such fields as speech recognition, futuristic user interfaces and 3-D graphics-cutting-edge technologies that might not bear fruit for years.

that’s a tall, broad order. but microsoft has set its priorities clearly. Among its prime goals: develop technologies that give people the ability to interact with computers using normal, everyday language. Leading the effort to capture this Holy Grail of computing is the natural language processing (NLP) group, whose 20 members make it one of the lab’s largest efforts. NLP aims to enable computers to grasp the meaning of any text-typed, e-mail, or voice converted to text-and then take action based on that understanding.

natural language processing is part of Microsoft’s overall goal of making computers-well, more like people. Machines of the future should be able to respond not only to spoken commands but to gestures and facial expressions. Indeed, it’s by fusing advances from a variety of research areas that managers expect to realize the full potential of their interdisciplinary lab. That hope is embodied in “flow.”

At Microsoft Research, two fields-vision and graphics-are converging rapidly to address the concept of flow. The basic idea behind the vision part of the challenge is to give computers the power to interpret and extract information from visual cues such as stored images or live camera feeds. Roughly half of Microsoft’s 14-member vision group is working mainly on constructing fresh viewpoints of a given scene from just a few images of the setting, as shot from different angles. The other half focuses on vision-based user interfaces. This involves such problems as watching for faces through a camera mounted atop the computer and determining whether there’s someone in front of the machine, where he or she is looking-even the expression on his or her face.

Graphics, of course, is heavily involved in techniques for assembling 3-D pictures. Senior graphics researcher Brian Guenter, for instance, is building an elaborate database of faces, facial muscle movements and expressions. When combined with speech recognition, natural language processing and vision technology, he hopes this archive will enable him to create virtual characters who communicate online via the spoken word-not text-and whose lips and expressions move in perfect sync with their voices.

Vision and graphics technologies might be combined like this: Users select a face as their online persona. They then sit at their computers, speaking into a microphone while a camera captures their expressions. Words, grimaces and smiles would all be broken into bits, transmitted over phone lines, and reconstructed in the virtual environment so that they appeared to emanate from the onscreen character. Such technology, Guenter believes, will prove a boon to online chat sessions and game-playing, where people want a degree of anonymity but desire more nuanced interactions than plain text affords.

The potential extends beyond the domain of online chat and game-playing. Kajiya, an Asian-American whose flowing mane and huge muttonchop sideburns make him look like a sci-fi zen master, also speaks of video-voice connections that will bring long distance, person-to-person communications into an entirely new dimension. The system would use programs that track gestures and eye movements and instantly redraw the screen image so that the person on the other end gets a different representation of the same scene, taking into account where his colleague is pointing or looking-just as they would in real life.

such projects impress and even charm. Even while pursuing its lofty dreams, however, the lab cannot escape comparisons to the one computer-science venture that got started with a similar blitz: Xerox’s Palo Alto Research Center, or PARC. Formed in 1970, Xerox PARC took only a few years to fashion the graphical interface, laser printer and other core technologies that have come to define personal computing.

Is Microsoft destined to make similar mistakes? Given the huge customer base locked into its existing products, would something truly revolutionary stand a chance of being embraced by Microsoft’s business groups? Some observers say no-at least not without the company’s experiencing the competitive hardships that rocked other labs in the early 1990s and forced researchers to work hand-in-hand with developers, marketers and customers to turn lab inventions into meaningful products. “I wonder if you can learn without going through the trials by fire,” says one top IBM research manager, adding that right now, anyway, “my impression is that Microsoft Research is much more like a sandbox for playing.”

There is a sandboxy aspect to the lab, almost as if its members are still simulating a great research effort before figuring out how to really do it. Computer industry research labs are famously casual-sports shirts and blue jeans. But Microsoft goes them one better: shorts and T-shirts. Rashid is a die-hard Star Trek fan who keeps a picture of himself with James “Scotty” Doohan on his office wall. Staffers hang with sci-fi writers such as Neal (Snow Crash) Stephenson and Greg Bear, who even created a character named Nathan Rashid in his book Slant.

And then there’s Myhrvold, who reportedly netted $104 million from stock options sales in 1997 and was recently featured in an aircraft ad because he was the 100th person to order a Gulfstream V jet. In Barbarians Led by Bill Gates, former Microsoft developer Marlin Eller and co-author Jennifer Edstrom complain that the fast-talking visionary throws out ideas without understanding the challenges involved. In particular, they cite his misplaced early 1990s evangelism for interactive TV, which has gone into hiding at Microsoft and just about everywhere else. “Talking to Myhrvold was a little like smoking dope,” they wrote. “It could give you insights’ but in the light of day those insights often didn’t make any sense.”

A more palpable portent of trouble may lie in the World Wide Web. It wasn’t until December 1995 that Gates proclaimed “a second PC revolution-the Internet.” To astute observers, being that late in the game was a symptom of big problems in the way research works at Microsoft. “They already have manifested to me a problem that we had been criticized for in the early days of PARC,” says John Seely Brown, the innovation guru who now directs the Xerox lab. “How is it conceivable,” he wonders, “that Gates did not understand the Internet until he did?”

Brown says that the evidence from Microsoft points in a disturbing direction: “Take the hypothesis that they have great researchers-and they’ve had them for some time.” These researchers, observes Brown, certainly grasped the Internet’s importance. Microsoft’s apparent lack of alertness, he concludes, suggests that mold-breaking ideas must be funneled through Myhrvold and Gates before they can diffuse out to the ranks. Such funnels, he says, slow the spread of research ideas and curtail innovation. His admonition: “Watch out for funnels.”

Arriving late at the Internet party may have been Microsoft’s most conspicuous miscue, but it isn’t the only one. Another ominous warning sign might be seen in Talisman, a system for rendering high-quality PC graphics. Microsoft touted it as a new standard. But Talisman proved to be a bust when it rolled out in late 1997. A big reason: the novel technology required software companies and graphics chip makers to adopt new procedures. Admits Kajiya, Talisman’s chief architect and now the lab’s assistant director, “It was just not natural for the market to do that.”

Despite such misfires, however, competitors should not easily dismiss Microsoft’s research arm. While Myhrvold has admitted attaching too little importance to the Internet in the early 1990s, Microsoft Research officials insist that the company was able to catch up rapidly because technologies developed in the lab were close at hand.

That may be viewing the experience through rose-colored glasses. Still, it’s hard to find a major company that wasn’t behind the Internet curve, albeit not as far as Microsoft. And no good lab can escape at least a few Talisman-like disappointments: If it doesn’t have some flops, it’s not thinking big enough. Besides, Microsoft’s research brain trust is keenly aware that what’s far more important than isolated successes and failures is a lab’s connection to the rest of its company-and its business objectives. They claim Microsoft’s research venture is well protected against becoming a software ivory tower.

Indeed, Microsoft deliberately established the lab on the main corporate campus rather than near an academic center la Xerox PARC. The goal: Keep business objectives high in researchers’ minds. Staffers routinely lunch with product group colleagues, and when technology is transferred, researchers often join developers in the business units until products are complete. Those close ties help ensure that the lab tackles problems important to the product side-and that product personnel know what research can offer, notes Richard Draves, manager of the lab’s operating systems group. Says he, “You can pick up a phone and call up a friend in just about any group in the company because you’ve worked with them in the past.”

Microsoft’s commitment to such fusion was readily apparent last summer, when virtually the entire speech research effort-20 people in all-was transplanted to Building 25 to help the speech products group introduce improved speech recognition technology into a variety of applications. Leader Xuedong Huang had a core team up and working even before boxes were unpacked and remodeling completed.

Close interactions with product groups is just one part of the puzzle, however. Despite any talk of funnels, lab personnel cite their strong bonds with Microsoft’s senior management as another key factor in keeping research on track. “At the very top ranks of the company people are technologists at heart-they love technology,” says Daniel Rosen, general manager of Microsoft’s New Technology group, which supplements Research by investing in or acquiring technologies from outside the company. In fact, Myhrvold and other members of Microsoft’s powerful executive committee hold doctorates in mathematics or computer science. And that expertise, Rosen says, helps pave the way for the fruits of research to be integrated into products and overall corporate strategy.

Finally, unlike Bell Labs and IBM, where studies range from hardware to software, biology to physics, Microsoft’s lab has a sharp and narrow focus. Outside of a four-person theory contingent, its 27 core groups focus on speech, natural language processing, graphics and other areas central to the future of personal computing. The result, lab managers believe, is a blue-sky lab with an “it’s-cool-to-ship” attitude. Researchers are expected to advance the state of the art by publishing papers and speaking at technical gatherings. Rashid and Ling proudly cite the fact that 10 of 50 invited papers at the prestigious Siggraph 96 conference were given by Microsoft’s graphics gurus. But they’re just as quick to boast the lab has contributed important code to virtually every company product. Proprietary creations include compression algorithms that allow more data to be stored on a disk, speech-recognition technology, an inference engine in Office 97 that makes troubleshooting easier, a video server and a host of development tools to facilitate programming.

Such feats-seemingly mundane in contrast to discovering laws of nature-mark a far different orientation from the old Bell LabsIBMXerox PARC style of throwing inventions over the wall to development groups. Research managers say their product-centric mindset stems from the dual philosophy governing the lab. On the grand level, researchers are mandated to do for their fields what DOS and Windows did for the PC: Set the standards on which everything else rests. But fundamental developments may take years. The interim aim is to spin out products that help justify the lab’s existence. Myhrvold sees such seedlings as a natural outgrowth of the lab’s more lofty ambitions. As he says in true guru-like fashion, “You get more small ideas by thinking big than by thinking small.”

it’s a good line. but while all the plans put in place to keep the lab’s free-thinking researchers closely attuned to corporate aims make perfect sense, they do little by themselves to dispel the cloud of doubt that hovers over Microsoft Research.

While such a reading of antitrust law is not entirely out of bounds, historical precedent suggests a less dire outcome. Similar antitrust actions at DuPont, General Electric, AT&T, IBM and Xerox-most of which have forced the companies to change business practices-have not curtailed research organizations, says David Hounshell, professor of technology and social change at Carnegie Mellon. In fact, Hounshell notes, it’s far more likely for antitrust pressures to increase the role of research. That’s because outside acquisitions and licensing deals typically fall under increasing scrutiny, making it more desirable for companies to innovate from within.

DuPont, for example, endured eight federal antitrust suits between 1944 and 1963. But, says Hounshell, “during this period it dramatically expanded its research programs. Its executives saw research, and particularly fundamental research, as the only course open to it because it was under antitrust pressure. I would gather that executives at Microsoft are probably thinking a lot the same way.” Witness Microsoft’s decision last fall to open a lab in China and the mandate to reach 600 research staffers by 2000.

The other great mystery centers on the lab itself. Microsoft’s upstart research arm clearly has a lot going for it. For one thing, as the universities and government labs responsible for many fundamental advances in computing scale back research, it is filling a vital void. Then, too, no one is contesting the ability of the people inside Building 31 to do some truly innovative work. “Nathan [Myhrvold] is a very imaginative guy, and he and Rashid are hiring some incredibly excellent people,” says Xerox PARC’s Brown. “There’s no question in my mind that they’re building up a firstclass research operation.”

But people alone will not carry the lab to greatness-and the lab’s missteps on the Talisman and Internet fronts have to raise some red flags. “While they’ve certainly gotten a talented team of people together, there’s not much you can point to in the way of output-and output has to be the measure of success of any research organization,” notes James C. McGroddy, former director of IBM Research. McGroddy speaks on this topic with authority; he spearheaded IBM’s campaign during the early 1990s to make the scientifically esteemed organization more relevant to Big Blue’s businesses.

In the end, McGroddy and others who take a harder line still give the Soft at least a fighting chance of pulling off its research dreams. Michael D. Garey, director of mathematical sciences research at Lucent’s Bell Labs, says that more and more he finds himself competing with Microsoft for top talent. He sums up the lab’s prospects this way: “If they do it right, if they hire people and let a decent number of them do research that gets published in serious journals, then maybe one of these days the reality will match the PR story, and they will take their place among the premier industrial research labs.”