Since its inception, 360 video has given us a glimpse of what was possible with a panoramic view of video that created a new paradigm of experiential media. It was interesting in the beginning, but I have realized its limitations, and I am now looking to take it to the next level.

One major limitation of 360 video is that it is an overall flat experience, since it positions the viewer in a static location with limited freedom of movement and low levels of interactivity. Another drawback is that it takes the viewer out of the narrative by revealing the production crew and equipment in the scene since it captures the full 360° field of view.

Volumetric video capture—touted by Microsoft as “holograms” —resolves these issues. It contains spatial data and enables viewers to walk around and interact with people and objects within CG environments, similar to what is experienced in real-time rendered VR games. It is far more immersive than 360 video footage because it captures the movements of real people in three dimensions. Users can view these movements from any angle by using positional tracking, which is standard on desktop and standalone Virtual/Augmented/ Mixed Reality systems, and on any mobile device enabled with ARKit from Apple or ARCore from Google.

The production workflow for mixed reality (MR) capture involves fundamentally different video technology from what is standard right now. Currently, MR production requires a capture studio which usually features a greenscreen stage equipped with an array of 100+ cameras. These camera arrays use photogrammetry/videogrammetry and infrared scanning, which converts live video images into three-dimensional objects.

The camera array captures both in color for textures and infrared to create the 3D model of the person. An algorithm generates the mesh of the subject and extracts the 3D models, animations, and textures, which can be integrated into a game engine such as Unity or compressed into a streaming-ready MP4 video file.

Once the performer’s or presenter’s body is fully digitized and all spatial viewing perspectives are recorded, they can be displayed as avatars, allowing for more personal interaction and viewing. These human representations have added depth and are quite lifelike. This is much more compelling than non-VR/ AR/MR 360 since it allows for a more intimate connection and brings the power of presence closer to that of reality.

Despite the current skepticism about the viability of VR/AR/MR, volumetric video streaming and capture will soon begin to have a great impact on consumer media and especially the enterprise market. This technology will eventually replace standard video recording and live streaming, and it will be used for various types of training, presentations, communication, collaboration, and live events.

Volumetric video simply adds another dimension to storytelling and visualization because of the creative freedom it allows within fully optimized VR/AR/MR.

Volumetric will take communication and interactive collaboration to a whole new level, allowing people to emotionally and intellectually connect on a deeper level with the content they’re consuming. It will also engender viewer engagement through realistic shared experiences.

[This article appears in the January/February 2018 issue of Streaming Media Magazine as “Volumetric Live Action Capture and Streaming Will Take Us Beyond 360 Video.”]