Authors
Zongrui Li, Qian Zheng, Boxin Shi, Gang Pan, Xudong Jiang
Nanyang Technological University; Zhejiang University; Peking University
Portals
Abstract
Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by the unknown light. Although the ambiguity is alleviated on non-Lambertian objects, the problem is still difficult to solve for more general objects with complex shapes introducing irregular shadows and general materials with complex reflectance like anisotropic reflectance. To exploit cues from shadow and reflectance to solve UPS and improve performance on general materials, we propose DANI-Net, an inverse rendering framework with differentiable shadow handling and anisotropic reflectance modeling. Unlike most previous methods that use non-differentiable shadow maps and assume isotropic material, our network benefits from cues of shadow and anisotropic reflectance through two differentiable paths. Experiments on multiple real-world datasets demonstrate our superior and robust performance.
Contribution
- We propose a differentiable shadow handling method that facilitates exploiting shadow cues with global shape-light information to solve UPS. Experimental results demonstrate its effectiveness in shadow map recovery and surface normal estimation.
- We introduce an anisotropic reflectance model that describes both isotropic and anisotropic materials to improve performance on general materials. Experimental results demonstrate its effectiveness on surface normal estimation for objects with a broad range of isotropic and anisotropic materials.
- We propose the DANI-Net that simultaneously optimizes shape, anisotropic reflectance, shadow map, and light conditions in an unsupervised manner, propagating inverse rendering errors through two paths involving the shadow cues and anisotropic reflectance, respectively. DANI-Net achieves state-of-the-art performance on several real-world benchmark datasets.
Related Works
Unsupervised calibrated photometric stereo; Uncalibrated photometric stereo; Shadow handling in photometric stereo; Neural reflectance representation in 3D vision
Comparisons
LL22, SCPS-NIR
Overview
Framework overview of DANI-Net. DepthMLP takes the positional code as the input and outputs depth wi. MaterialMLP takes the same positional code as the input and outputs ? d i and c k i . The shadow sij is calculated based on Eq. (3). Spatially varying anisotropic specularity ? s ij is obtained through Eq. (5). Rendered images are generated through Eq. (2). The inverse rendering loss LIR measures the rendering error between rendered images and observed images, and backpropagates to the light calibration and shape estimation through the differentiable shadow path (in red and green colors) and the anisotropic reflectance path (in blue and green colors).