VR is still a novelty, but Google’s light-field technology could make it serious art

I recently got a private tour of a NASA space shuttle’s cockpit, a quirky mosaic-covered LA home, and a peaceful chapel with light streaming through ornate stained-glass windows—all without leaving my chair.

That chair was in an office at Google’s Silicon Valley headquarters, and I was wearing an HTC Vive virtual-reality headset on my face. But because these places were filmed with a high-resolution prototype camera that reproduces some of the key cues we use to understand depth in the real world, it felt more like actually being there than anything I’ve experienced with any other live-action VR. Which is to say it was pretty damn cool.

I could peer around the seats in the space shuttle Discovery, revealing buttons and switches on the walls of the cockpit that were previously obscured. As I looked closely at mirrored bits of tile on the outside of the mosaic house, I glimpsed reflections of other tiles in the background and saw a dizzying display of shapes and patterns. In the chapel, I gazed at the floor, and the colorful sunbeams moved as I did.

Starting Wednesday, Google is releasing these images (which are just stills, though they feel like mini worlds to explore) as part of a free app called Welcome to Light Fields that shows the company’s vision for the future of truly immersive, lifelike virtual reality. The app, which will be available through the video-game platform Steam, is meant to work with HTC’s Vive, Oculus’ Rift, and Windows mixed-reality headsets.

As the name suggests, the images in the app were created by recording and re-creating light fields, a term that refers to all the light traveling in all directions in a given volume of space. Google used a number of small, interlinked cameras to capture lots of images as viewed from the perspective of a roughly beach-ball-size sphere. Using those photos—that spherical patchwork—its system can accurately reproduce the rays of light wherever a viewer might look within that fairly small space.

For many headset users, Google’s app will be the first experience of live-action VR that they can actually explore by moving their head around. And though the app includes only stills, not videos, it gives a sense of what may someday be possible: virtual exploration of historic sites and art, and eventually hyperrealistic VR films.

“If we can do a really good job of capturing the real world, then we really give you a teleportation machine that can help you be anywhere, and that’s pretty powerful,” says Paul Debevec, a senior scientist with Google’s Daydream virtual-reality group, who worked on the camera and app.

Like so much about virtual reality, that idea is still mostly a promise. The technology has yet to become popular with consumers, for many reasons—headsets are bulky, expensive, and silly-looking, and the VR experience doesn’t always look or feel all that real.

Debevec and others working on VR see light fields as a potential solution for that last problem. Most cameras don’t capture depth data, so if you’re looking at video in VR—of, say, swiftly swimming technicolor fish—and you move your head up or down, the whole undersea world will move with you. This is the status quo; it’s obnoxious, and it can cause motion sickness.

With Google’s light-field camera, this doesn’t happen. Rather, just like in real life, when I shifted my head, things that were closer to me moved more than those that were farther away. The phenomenon is called motion parallax, and Google may be the first company to capture it in a widely available live-action app.

To make the device, Debevec and his team modified an existing rig, made up of a circle of 16 connected GoPro cameras, that Google introduced for professional VR content creators in 2015. But instead of building a spherical camera as some others (Facebook among them) have done, they connected the cameras in a vertical arc and placed the whole thing on a tripod with a motor and a battery. This let the contraption spin slowly in a circle and capture a sphere’s worth of images. To produce an even fuller picture, a low-tech accessory—a bearing and a piece of twine—can help it slowly drop as it rotates.

A typical circle takes about 30 seconds to complete, Debevec says, though the contraption can rotate more slowly to capture low-light shots without having motion blur muck up the results. The sphere of images it can capture is just about two feet in diameter, roughly the size of a beach ball—a size he says was limited by the distance between the pilot and commander seats on the space shuttle.

Google’s light-field camera and Welcome to Light Fields app are still very much in the experimentation stage. Because the area captured is so small and your head is essentially starting out in the center of it, you can’t move that far in any direction while investigating the scenes. I sat in a chair the whole time, and if I moved more than a few inches to one side, the scene before me started to blacken in that direction, which was frustrating.

It’s also not clear how best to include people in these scenes. Just two of the images I saw did so, and both were disturbing and creepy. In one scene, the owners of the mosaic tile house sat stock still but moved their eyes with the rotating camera, giving the effect that their gaze followed mine. Perhaps this will be fixed in time, but for now, the uncanny valley comes into effect.

Most of Google’s camera work is beautiful, though, and worth checking out. I can see it working especially well in a museum or educational setting. Unlike so much of what you can see from inside a headset today, it feels like a virtual glimpse of a world that’s real.