Authors
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, Hendrik P. A. Lensch
University of Tubingen; Google Research
Portals
Summary
This paper proposed a novel method to decompose image collections from multiple views taken under varying or fixed illumination conditions. By introducing physically-based lighting and appearance representations, this method can relight an object, and information can be extracted from the neural volume.
Abstract
Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model the radiance field of an object, most of these techniques only enable view synthesis and not relighting. Additionally, evaluating these radiance fields is resource and time-intensive. We propose a neural reflectance decomposition (NeRD) technique that uses physically-based rendering to decompose the scene into spatially varying BRDF material properties. In contrast to existing techniques, our input images can be captured under different illumination conditions. In addition, we also propose techniques to convert the learned reflectance volume into a relightable textured mesh enabling fast real-time rendering with novel illuminations. We demonstrate the potential of the proposed approach with experiments on both synthetic and real datasets, where we are able to obtain high-quality relightable 3D assets from image collections. The datasets and code is available on the project page: https://markboss.me/publication/2021-nerd/
Contribution
- We propose a neural reflectance decomposition (NeRD) technique that uses physically-based rendering to decompose the scene into spatially varying BRDF material properties
- We also propose techniques to convert the learned reflectance volume into a relightable textured mesh enabling fast real-time rendering with novel illuminations
Related Works
Neural scene representations; BRDF estimation
Comparisons
NeRF, NeRF-A
Overview
In comparison to NeRF, NeRD architecture mainly differs in the second finer network. NeRD uses decomposition network as a finer network which stores the lighting independent reflectance parameters instead of the direct view-dependent color. Also, the sampling network in NeRD differs from NeRF as we learn illumination dependent colors as NeRD can work with differently illuminated input images.