Authors
Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan T. Barron
Google Research; MIT; UC Berkeley
Portals
Summary
We optimize a Neural Reflectance and Visibility Field (NeRV) 3D representation from a set of input images of a scene illuminated by known but unconstrained lighting. Our NeRV representation can be rendered from novel views under arbitrary lighting conditions not seen during training.
Abstract
We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.
Contribution
- In this work, we present a method to train a NeRF-like model that can simulate realistic environment lighting and global illumination. Our key insight is to train an MLP to act as a lookup table into a visibility field during rendering
Related Works
Neural Radiance Fields; Implicit geometry representations
Comparisons
Overview
Given any continuous 3D location as input, NeRV outputs the volume density as well as The visibility to a spherical environment map surrounding the scene, which is multiplied by the direct illumination at that point and added to the estimated indirect illumination at that point to determine the full incident illumination. This is then multiplied by the predicted BRDF, and integrated over all incoming directions to determine the outgoing radiance at that point.