Authors
Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, Shuang Zhao
University of California, Irvine; Carnegie Mellon University
Portals
Abstract
Physics-based differentiable rendering, the estimation of derivatives of radiometric measures with respect to arbitrary scene parameters, has a diverse array of applications from solving analysis-by-synthesis problems to training machine learning pipelines incorporating forward rendering processes. Unfortunately, general-purpose differentiable rendering remains challenging due to the lack of efficient estimators as well as the need to identify and handle complex discontinuities such as visibility boundaries. In this paper, we show how path integrals can be differentiated with respect to arbitrary differentiable changes of a scene. We provide a detailed theoretical analysis of this process and establish new differentiable rendering formulations based on the resulting differential path integrals. Our path-space differentiable rendering formulation allows the design of new Monte Carlo estimators that offer significantly better efficiency than state-of-the-art methods in handling complex geometric discontinuities and light transport phenomena such as caustics. We validate our method by comparing our derivative estimates to those generated using the finite-difference method. To demonstrate the effectiveness of our technique, we compare inverse-rendering performance with a few state-of-the-art differentiable rendering methods.
Contribution
- The differentiation of full path integrals with respect to arbitrary scene parameterizations (§5.1), resulting in our differential path integrals comprised of completely separated interior and boundary components that can be estimated independently using different Monte Carlo estimators
- A reparameterization of the path integral (§5.2) that minimizes the types of discontinuities to be handled by the boundary integral
- New unbiased Monte Carlo methods that estimate, respectively, the interior and boundary components of our differential path integrals (§6). Our technique greatly outperforms previous methods for complex scene geometries and light transport effects
Related Works
Path-space rendering; Derivatives for rendering; Physics-based differentiable rendering; Derivatives for vision; Automatic differentiation
Overview
We introduce path-space differentiable rendering, a new theoretical framework to estimate derivatives of radiometric measurements with respect to arbitrary scene parameters (e.g., material properties and object geometries). By directly differentiating full path integrals, we derive the differential path integral framework, enabling the design of new unbiased Monte Carlo methods capable of efficiently estimating derivatives in virtual scenes with complex geometry and light transport effects. This example shows a dinning room scene lit by the sun from outside the window. On the right, we show the corresponding derivative image with respect to the vertical location of the sun. (Please use Adobe Acrobat to view the teaser images to see them animated.)