Authors
Valentin Deschaintre, Yiming Lin, Abhijeet Ghosh
Imperial College London
Portals
Summary
Our method aims at acquiring both the 3D shape (surface normals and depth) and spatially varying reflectance of an object using practical acquisition involving frontal flash illumination and single view acquisition. To tackle this highly ill-posed problem, we propose to leverage linear polarization cues in surface reflectance, providing strong initial cues to our deep network
Abstract
We present a novel method for efficient acquisition of shape and spatially varying reflectance of 3D objects using polarization cues. Unlike previous works that have exploited polarization to estimate material or object appearance under certain constraints (known shape or multiview acquisition), we lift such restrictions by coupling polarization imaging with deep learning to achieve high quality estimate of 3D object shape (surface normals and depth) and SVBRDF using single-view polarization imaging under frontal flash illumination. In addition to acquired polarization images, we provide our deep network with strong novel cues related to shape and reflectance, in the form of a normalized Stokes map and an estimate of diffuse color. We additionally describe modifications to network architecture and training loss which provide further qualitative improvements. We demonstrate our approach to achieve superior results compared to recent works employing deep learning in conjunction with flash illumination.
Contribution
- We propose the first method for joint 3D shape and SVBRDF estimation combining polarization cues (under flash illumination) and deep learning
- We publish a new synthetic dataset with diffuse polarization effects for supervised learning
- Analysis of polarization cues under different lighting and practical acquisition protocol for high quality results
- An improved deep network architecture and training loss for 3D object appearance acquisition
Related Works
Practical Appearance Acquisition; Polarization; Reflectance Estimation; Surface Normals Estimation
Overview
Our network architecture has a general U-Net shape. We divide the decoders into three different branches, each handling a related set of output map(s), specifically: normal and depth, diffuse albedo, roughness and specular albedo. We introduce res-blocks on the skip connections between the encoder and the different branches of the decoder, allowing the network to adapt the information forwarded to the different branches of the decoder.