Microsoft’s HoloLens Will Put Realistic 3-D People in Your Living Room

Demonstrations of augmented-reality displays typically involve tricking you into seeing animated content such as monsters and robots that aren’t really there. Microsoft wants its forthcoming HoloLens headset to mess with reality more believably. It has developed a way to make you see photorealistic 3-D people that fit in with the real world.

With this technology, you could watch an acrobat tumble across your front room or witness your niece take some of her first steps. You could walk around the imaginary people just as if they were real, your viewpoint changing seamlessly as if they were actually there. A sense of touch is just about the only thing missing.

That experience is possible because Microsoft has built a kind of holographic TV studio at its headquarters in Redmond, Washington. Roughly 100 cameras capture a performance from many different angles. Software uses the different viewpoints to create a highly accurate 3-D model of the person performing, resulting in a photo-real appearance.

The more traditional approach of using computer animation can’t compare, according to Steve Sullivan, who works on the project at Microsoft. He demonstrated what Microsoft calls “video holograms” at the LDV Vision Summit, an event about image-processing technology, in New York on Tuesday. More details of the technology will be released this summer.

“There’s something magical about it being real people and motion,” he said. “If you have a HoloLens, you really feel these performances are in your world.”

Microsoft is working on making it practical and cheap enough for other companies to record content in this form. It might one day be possible to visit a local studio and record a 3-D snapshot of a child at a particular point in life, said Sullivan.

Demonstrations of the technology included holographic videos of theatrical and acrobatic performances. Sullivan also showed how the format could be used in sports instruction. Someone looking for help with golf technique, for example, could wear a HoloLens to examine recordings of a pro golfer swinging different clubs at full or reduced speed, viewing the instructor up close and from different vantage points.

Microsoft has also recorded catwalk models using its system. That could help Internet shoppers by showing them how an item of clothing looks and hangs more realistically than is possible with still photos or 2-D video, said Sullivan. It should also be possible to use captured performances as the basis for animated characters in games or other applications, he said.

HoloLens uses a novel holographic display technology that can trick the eye into perceiving 3-D objects more effectively than conventional stereoscopic displays (see “Microsoft Making Fast Progress with HoloLens”). Sensors in the headset allow the device to figure out how to present virtual objects so they fit in with the real world. Sullivan showed how holographic videos can also be played back in 2-D on a tablet, in a special player that lets you drag your fingers to change your viewpoint on the action.

Several companies are working on ways to capture live action such as sports or movies for viewing on more conventional 3-D headsets like the Oculus Rift. But they capture only the 3-D view from the position of the camera at the time of shooting. Devices like the Rift also use a less sophisticated method of tricking the brain into perceiving 3-D objects, and they cannot mix virtual content with the real world.

A startup called Magic Leap, backed by Google, is developing its own wearable augmented-reality device based on display technology that’s similar to Microsoft’s (see “10 Breakthrough Technologies 2015: Magic Leap”). So far, however, Magic Leap’s demonstrations of its technology have involved animated content, not live action recorded in three dimensions.