Authors
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, Hendrik P. A. Lensch
University of Tübingen; Google Research; Microsoft Azure AI
Portals
Summary
Our neural-PIL based technique decomposes images observed under unknown illumination into high-quality BRDF, shape and illuminations. This allows us to then synthesize novel views (targets shown in insets) and perform relighting or illumination transfer.
Abstract
Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics. Neural approaches such as NeRF have achieved remarkable success in view synthesis, but do not explicitly perform decomposition and instead operate exclusively on radiance (the product of reflectance and illumination). Extensions to NeRF, such as NeRD, can perform decomposition but struggle to accurately recover detailed illumination, thereby significantly limiting realism. We propose a novel reflectance decomposition network that can estimate shape, BRDF, and per-image illumination given a set of object images captured under varying illumination. Our key technique is a novel illumination integration network called Neural-PIL that replaces a costly illumination integral operation in the rendering with a simple network query. In addition, we also learn deep low-dimensional priors on BRDF and illumination representations using novel smooth manifold auto-encoders. Our decompositions can result in considerably better BRDF and light estimates enabling more accurate novel view-synthesis and relighting compared to prior art. Project page: https://markboss.me/publication/2021-neural-pil/
Contribution
- We aim to replace the costly illumination integration step within these rendering approaches with a learned network
- We also present a smooth manifold auto-encoder (SMAE), based on interpolating auto-encoders, that can learn effective low-dimensional representations of light and BRDFs
Related Works
Coordinate-based MLPs;BRDF estimation;Illumination estimation
Overview
1) Our coarse network uses a latent illumination estimate to predict a view-dependent color and density. 2) Pre-trained networks restrict the possible BRDF representation (BRDF-SMAE) and the incident, lighting (PIL) to lower-dimensional spaces. A single evaluation of Neural-PIL returns a pre-filtered, illumination cone according to surface roughness. Using that, the BRDF estimate, and a surface, normal (the unit-norm gradient of our density estimate), we render the shaded color c.