Authors
Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu
Michigan State University; Qualcomm Technologies Inc.
Portals
Summary
We introduce a novel face-relighting method that produces geometrically consistent shadows. By proposing a differentiable algorithm based on the principles of ray tracing that directly uses the face geometry for modeling hard shadows, our method produces physically correct hard shadows which the state-of-the-art face relighting method cannot produce.
Abstract
Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.
Contribution
- We propose a single image face relighting method that can produce geometrically consistent hard shadows
- We introduce a novel differentiable algorithm to estimate facial cast shadows based on the estimated geometry
- We achieve SoTA relighting performance on 2 benchmarks quantitatively/qualitatively under directional lights
- Our differentiable hard shadow modeling improves the estimated geometry over models that use diffuse shading
Related Works
Face Relighting; Differentiable Rendering and Ray Tracing
Comparisons
SfSNet, DPR, SIPR, Nestmeyer
Overview
Given a single image It and target lighting direction, our model generates a relit image with geometrically consistent cast shadows. The geometric consistency is achieved thanks to our shadow mask estimation module, which estimates shadow mask Mshadow using depth map (the face geometry). Mshadow incorporates non-diffuse cast shadows into our shading.