Authors
Abhimitra Meka, Christian H
MPI Informatics; Saarland Informatics Campus; Google; Stanford University
Portals
Summary
Given only two observations (color gradient images) of an actor, our method is able to relight the subject under any lighting condition. Our approach generalizes to unseen subjects, viewpoints, illumination conditions and can handle dynamic performances.
Abstract
We present a novel technique to relight images of human faces by learning a model of facial reflectance from a database of 4D reflectance field data of several subjects in a variety of expressions and viewpoints. Using our learned model, a face can be relit in arbitrary illumination environments using only two original images recorded under spherical color gradient illumination. The output of our deep network indicates that the color gradient images contain the information needed to estimate the full 4D reflectance field, including specular reflections and high frequency details. While capturing spherical color gradient illumination still requires a special lighting setup, reduction to just two illumination conditions allows the technique to be applied to dynamic facial performance capture. We show side-by-side comparisons which demonstrate that the proposed system outperforms the state-of-the-art techniques in both realism and speed.
Contribution
- A capture system that enables 4D reflectance field estimation of moving subjects
- A machine learning-based formulation that maps spherical gradient images to the OLAT image corresponding to a particular lighting direction
- A task-specific perceptual loss trained to pick up specularities and high frequency details
- A sliding window based pooling loss that robustly handles the small misalignments between the spherical gradient images and the groundtruth OLAT image
Related Works
Parametric Model Fitting; Image-Based Relighting; Learning-Based Techniques
Comparisons
Fyffe et al. 2009, Shu et al. 2017, Yamaguchi et al. 2018
Overview
Deep Reflectance Fields – given only two observations (color gradient images) of an actor, our method is able to relight the subject under any lighting condition. Our approach generalizes to unseen subjects, viewpoints, illumination conditions and can handle dynamic performances. Our network receives as input a pair of gradient images and a lighting direction. Via a U-Net architecture, it regresses the OLAT image that is corresponding to that particular lighting configuration.