Authors
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, Andrea Tagliasacchi
University of British Columbia; Simon Fraser University; University of Toronto; Google Research
Portals
Abstract
With the advent of Neural Radiance Fields (NeRF), neural networks can now render novel views of a 3D scene with quality that fools the human eye. Yet, generating these images is very computationally intensive, limiting their applicability in practical scenarios. In this paper, we propose a technique based on spatial decomposition capable of mitigating this issue. Our key observation is that there are diminishing returns in employing larger (deeper and/or wider) networks. Hence, we propose to spatially decompose a scene and dedicate smaller networks for each decomposed part. When working together, these networks can render the whole scene. This allows us near-constant inference time regardless of the number of decomposed parts. Moreover, we show that a Voronoi spatial decomposition is preferable for this purpose, as it is provably compatible with the Painter's Algorithm for efficient and GPU-friendly rendering. Our experiments show that for real-world scenes, our method provides up to 3x more efficient inference than NeRF (with the same rendering quality), or an improvement of up to 1.0~dB in PSNR (for the same inference cost).
Contribution
- We highlight the presence of diminishing returns for network capacity in NeRF, and propose spatial decompositions to address this issue
- We demonstrate how a decomposition based on Voronoi Diagrams may be learned to optimally represent a scene
- We show how this decomposition allows the whole scene to be rendered by rendering each part independently, and compositing the final image via Painter’s Algorithm
- In comparison to the NeRF baseline, these modifications result in improvement of rendering quality for the same computational budget, or faster rendering of images given the same visual quality
Related Works
Image-space neural rendering; Neural volumetric rendering