Authors
Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan
Apple; HKUST
Portals
Summary
NeILF is capable of modelling the joint illumination of direct/indirect lights from different sources. Estimated incident lights at point x1 and point x2 well explain the mixed lighting of the scene, including an environment map with high-radiance sun light, two near-range point lights, and two near-range area lights.
Abstract
We present a differentiable rendering framework for material and lighting estimation from multi-view images and a reconstructed geometry. In the framework, we represent scene lightings as the Neural Incident Light Field (NeILF) and material properties as the surface BRDF modelled by multi-layer perceptrons. Compared with recent approaches that approximate scene lightings as the 2D environment map, NeILF is a fully 5D light field that is capable of modelling illuminations of any static scenes. In addition, occlusions and indirect lights can be handled naturally by the NeILF representation without requiring multiple bounces of ray tracing, making it possible to estimate material properties even for scenes with complex lightings and geometries. We also propose a smoothness regularization and a Lambertian assumption to reduce the material-lighting ambiguity during the optimization. Our method strictly follows the physically-based rendering equation, and jointly optimizes material and lighting through the differentiable rendering process. We have intensively evaluated the proposed method on our in-house synthetic dataset, the DTU MVS dataset, and real-world BlendedMVS scenes. Our method is able to outperform previous methods by a significant margin in terms of novel view rendering quality, setting a new state-of-the-art for image-based material and lighting estimation.
Contribution
- Representing scene lightings by the neural incident light field, where occlusions and indirect lights of the scene can be naturally handled
- A differentiable framework for joint material and lighting estimation, which significantly outperforms previous state-of-the-art methods in various datasets
- A bilateral smoothness and a Lambertian assumption to constrain the roughness and the metallic, reducing the material-lighting ambiguity during the network optimization
Related Works
The Rendering Equation; Differentiable Rendering; Material and Lighting Estimation
Comparisons
PhySG, SG-ENV, NeRFactor, NeRF