Authors
Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong
The University of Hong Kong; FNii and SSE, CUHK-Shenzhen; Nanyang Technological University; MIT-IBM Watson AI Lab
Portals
Summary
Our method takes multi-view multi-light images as input, and is able to reconstruct accurate surface and faithful BRDFs based on our shadow-aware renderer. Specifically, we only take images under sparse views with each view illuminated by multiple unknown single directional lights.
Abstract
Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer. After optimization, the reconstructed object can be used for novel-view rendering, relighting, and material editing. Experiments on both synthetic and real datasets demonstrate that our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods. Our code and model can be found at this https URL.
Contribution
- We introduce a neural inverse rendering method for multi-view photometric stereo, which jointly optimizes shape, BRDFs, and lights based on a shadowaware differentiable rendering layer
- We propose to regularize the surface normals derived from the radiance field with normals estimated from multi-light images, which significantly improves surface reconstruction, especially for sparse input views (e.g., 5 views)
- Our method achieves state-of-the-art results in MVPS, and demonstrates that incorporating multi-light information appropriately can produce a far more accurate shape reconstruction
Related Works
Single-view Photometric stereo (PS); Multi-view Photometric Stereo (MVPS); Neural Rendering
Comparisons
NeRF, PhySG, NeRFactor, NeRF, NRF, UNISURF
Overview
Given multi-view multi-light images, we first obtain guidance normal maps via uncalibrated photometric stereo (UPS) to regularize the neural density field, which encourages accurate surface reconstruction. We then perform neural inverse rendering to jointly optimize surface normals, BRDFs and lights based on the initial shape.