Authors
Renjiao Yi, Chenyang Zhu, Kai Xu
National University of Defense Technology
Portals
Abstract
We present a learning-based approach to relight a single image of Lambertian and low-frequency specular objects. Our method enables inserting objects from photographs into new scenes and relighting them under the new environment lighting, which is essential for AR applications. To relight the object, we solve both inverse rendering and re-rendering. To resolve the ill-posed inverse rendering, we propose a weakly-supervised method by a low-rank constraint. To facilitate the weakly-supervised training, we contribute Relit, a large-scale (750K images) dataset of videos with aligned objects under changing illuminations. For re-rendering, we propose a differentiable specular rendering layer to render low-frequency non-Lambertian materials under various illuminations of spherical harmonics. The whole pipeline is end-to-end and efficient, allowing for a mobile app implementation of AR object insertion. Extensive evaluations demonstrate that our method achieves state-of-the-art performance. Project page: https://renjiaoyi.github.io/relighting/.
Contribution
- A weakly-supervised inverse rendering pipeline trained with a low-rank loss. The correctness and convergence of the loss are mathematically proven
- A large-scale dataset of foreground-aligned videos collecting 750K images of 100+ real objects under different lighting conditions
- An Android app implementation for amateur users to make a home-run
Related Works
Inverse Rendering; Image Relighting
Overview
Overview of our method. At training time, Spec-Net separates input images into specular and diffuse branches. Spec-Net, Normal-Net and Light-Net are trained in a self-supervised manner by the Relit dataset. At inference time, inverse rendering properties are predicted to relight the object under novel lighting and material. The non-Lambertian render layers produce realistic relit images.