Authors
Xilong Zhou, Nima Kalantari
Texas A&M University
Portals
Summary
We propose a novel optimization-based method for estimating the reflectance parameters given a single image. Test-time optimization for single image reflectance estimation is susceptible to overfitting as this is a highly ill-posed problem. To combat this issue, we introduce a novel approach that incorporates the test-time optimization into the training process. Specifically, we train our network by minimizing the error between the ground truth and network’s output after the test-time optimization. Through this strategy, we ensures that our network learns a prior that is suitable for test-time optimization. Here, we show the results of our network before (initial) and after (optimized) a few iterations of test-time optimization, as well as the rerendered images using the optimized reflectance parameters. We provide comparisons against several state-of-the-art methods on a large number of scenes in the supplementary materials.
Abstract
In this paper, we propose a novel optimization-based method to estimate the reflectance properties of a near planar surface from a single input image. Specifically, we perform test-time optimization by directly updating the parameters of a neural network to minimize the test error. Since single image SVBRDF estimation is a highly ill-posed problem, such an optimization is prone to overfitting. Our main contribution is to address this problem by introducing a training mechanism that takes the test-time optimization into account. Specifically, we train our network by minimizing the training loss after one or more gradient updates with the test loss. By training the network in this manner, we ensure that the network does not overfit to the input image during the test-time optimization process. Additionally, we propose a learned reflectance loss to augment the typically used rendering loss during the test-time optimization. We do so by using an auxiliary network that estimates pseudo ground truth reflectance parameters and train it in combination with the main network. Our approach is able to converge with a small number of iterations of the test-time optimization and produces better results compared to the state-of-the-art methods.
Contribution
- We propose a novel training strategy for single image SVBRDF estimation to achieve robustness to overfitting during the testtime optimization
- We present a learned reflectance loss to compliment the rendering loss during the test-time optimization
- We extensively compare our approach against several stateof-the-art methods on various datasets, including our new dataset of real images with ground truth
Related Works
Direct-Estimation Methods; Optimization-Based Methods; Meta-Learning