The quest to digitize the physical world in three dimensions has evolved rapidly. Traditional photogrammetry, which uses algorithms to stitch 2D photos into textured polygon meshes, is highly effective for static, opaque objects. However, it fails catastrophically when dealing with transparency, reflections, or complex fine details like hair and foliage.
Neural Radiance Fields (NeRFs)
NeRFs revolutionized volumetric capture by shifting the paradigm from polygons to neural networks. Instead of building a mesh, a NeRF uses a multilayer perceptron (MLP) to memorize the way light travels through a specific volume of space. By training on a sparse set of 2D images, the network can synthesize novel views of complex scenes, perfectly preserving specular highlights and glass reflections.
Limitations of NeRFs
- Training Time: Early NeRF models took days to train on a single scene, though modern implementations (like Instant NGP) have reduced this to minutes.
- Rendering Speed: Querying a neural network for every ray of light during rendering is computationally heavy, making high-resolution, real-time playback in VR challenging.
- Dynamic Scenes: Capturing moving subjects (4D NeRFs) requires immense synchronization of camera arrays and exponential increases in training time.
3D Gaussian Splatting
Introduced as a breakthrough alternative, 3D Gaussian Splatting bypasses the neural network during the rendering phase. The scene is represented by millions of microscopic 3D ellipses (Gaussians), each possessing properties of position, scale, opacity, and spherical harmonics (which control how the color changes depending on the viewing angle).
This method maintains the photorealistic advantages of NeRFs—handling reflections and transparencies beautifully—but can be rasterized by standard GPUs at hundreds of frames per second. Gaussian Splatting provides real-time performance natively, making it highly suitable for integration into spatial computing headsets and traditional game engines.
Studio Requirements for Volumetric Video
Capturing high-fidelity human performances volumetrically still requires dedicated infrastructure. A standard volumetric studio typically employs 50 to 100 synchronized 4K cameras arranged in a dome configuration, heavily diffused lighting to minimize harsh shadows, and substantial edge-compute clusters to process terabytes of data per minute of recording.
As algorithmic processing improves, we anticipate the hardware requirements will scale down. Sparse-view capture algorithms are already demonstrating the ability to generate acceptable volumetric proxies from as few as three or four camera angles, paving the way for broadcast-level volumetric streaming in live sports and remote communication.