Authors
Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Marc Habermann, Christian Theobalt
Max Planck Institute for Informatics, Saarland Informatics Campus; MIT
Portals
Summary
Our method takes multiple views of a scene under one unknown illumination condition as input and allows novel-view synthesis and relighting (corresponding environment maps in green insets) with intricate multi-bounce illumination (orange insets).
Abstract
Given a set of images of a scene, the re-rendering of this scene from novel views and lighting conditions is an important and challenging problem in Computer Vision and Graphics. On the one hand, most existing works in Computer Vision usually impose many assumptions regarding the image formation process, e.g. direct illumination and predefined materials, to make scene parameter estimation tractable. On the other hand, mature Computer Graphics tools allow modeling of complex photo-realistic light transport given all the scene parameters. Combining these approaches, we propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function, which implicitly handles global illumination effects using novel environment maps. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. To disambiguate the task during training, we tightly integrate a differentiable path tracer in the training process and propose a combination of a synthesized OLAT and a real image loss. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art and, thus, also our re-rendering results are more realistic and accurate.
Contribution
- A method for recovering the radiance transfer field from images of objects under an unknown light condition, hence enabling free-viewpoint relighting with realistic global illumination
- A neural precomputed radiance transfer (PRT) field for multi-bounce global illumination computation and neural implicit surface rendering
- A novel supervision strategy leveraging a differentiable ray-tracer for physically based scene reconstruction, multiple light bounce rendering, and a new synthetic OLAT supervision
Related Works
Precomputed Radiance Transfer; Inverse Rendering and Relighting