Rendering - Paper Copilot

3D rendering is the 3D computer graphics process of converting 3D models into 2D images on a computer. Rendering is the final process of creating the actual 2D image or animation from the prepared 3D scene. This can be compared to taking a photo or filming the scene after the setup is finished in real life.

NeILF: Neural Incident Light Field for Physically-based Material Estimation

Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan

Apple; HKUST

Portals

Abstract

We present a differentiable rendering framework for material and lighting estimation from multi-view images and a reconstructed geometry. In the framework, we represent scene lightings as the Neural Incident Light Field (NeILF) and material properties as the surface BRDF modelled by multi-layer perceptrons. Compared with recent approaches that approximate scene lightings as the 2D environment map, NeILF is a fully 5D light field that is capable of modelling illuminations of any static scenes. In addition, occlusions and indirect lights can be handled naturally by the NeILF representation without requiring multiple bounces of ray tracing, making it possible to estimate material properties even for scenes with complex lightings and geometries. We also propose a smoothness regularization and a Lambertian assumption to reduce the material-lighting ambiguity during the optimization. Our method strictly follows the physically-based rendering equation, and jointly optimizes material and lighting through the differentiable rendering process. We have intensively evaluated the proposed method on our in-house synthetic dataset, the DTU MVS dataset, and real-world BlendedMVS scenes. Our method is able to outperform previous methods by a significant margin in terms of novel view rendering quality, setting a new state-of-the-art for image-based material and lighting estimation.

Related Works

The Rendering Equation; Differentiable Rendering; Material and Lighting Estimation

Comparisons

PhySG, SG-ENV, NeRFactor, NeRF

Acquiring the Reflectance Field of a Human Face

Paul Debevec,Tim Hawkins,Chris Tchou,Haarm-Pieter Duiker,Westley Sarokin,Mark Sagar

University of California at Berkeley; LifeF/X, Inc.

Portals

Abstract

We present a method to acquire the reflectance field of a human face and use these measurements to render the face under arbitrary changes in lighting and viewpoint. We first acquire images of the face from a small set of viewpoints under a dense sampling of incident illumination directions using a light stage. We then construct a reflectance function image for each observed image pixel from its values over the space of illumination directions. From the reflectance functions, we can directly generate images of the face from the original viewpoints in any form of sampled or computed illumination. To change the viewpoint, we use a model of skin reflectance to estimate the appearance of the reflectance functions for novel viewpoints. We demonstrate the technique with synthetic renderings of a person's face under novel illumination and viewpoints.

Related Works

Facial Modeling and Animation; Reflectometry; Image-Based Modeling and Rendering

Real-Time Rendering, Fourth Edition

Tomas Akenine-M

Portals

Physically Based Rendering: From Theory To Implementation, Third Edition

Matt Pharr, Wenzel Jakob, and Greg Humphreys

Portals

Abstract

Photorealistic computer graphics is ubiquitous today, with applications that include entertainment—notably, movies and video games; product design; and architecture. Over the past decade, physically-based approaches to rendering have become widely used, where an accurate modeling of the physics of light scattering is at the heart of image synthesis. These approaches offer both visual realism and predictability. Physically Based Rendering describes both the mathematical theory behind a modern photorealistic rendering system as well as its practical implementation. A method known as “literate programming” combines human-readable documentation and source code into a single reference that is specifically designed to aid comprehension. The ideas and software in this book show the reader how to design and employ a full-featured rendering system capable of creating stunning imagery.

Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography

Paul Debevec

University of California at Berkeley

Portals

Abstract

We present a method that uses measured scene radiance and global illumination in order to add new objects to light-based models with correct lighting. The method uses a high dynamic range image-based model of the scene, rather than synthetic light sources, to illuminate the new objects. To compute the illumination, the scene is considered as three components: the distant scene, the local scene, and the synthetic objects. The distant scene is assumed to be photometrically unaffected by the objects, obviating the need for reflectance model information. The local scene is endowed with estimated reflectance model information so that it can catch shadows and receive reflected light from the new objects. Renderings are created with a standard global illumination method by simulating the interaction of light amongst the three components. A differential rendering technique allows for good results to be obtained when only an estimate of the local scene reflectance properties is known. We apply the general method to the problem of rendering synthetic objects into real scenes. The light-based model is constructed from an approximate geometric model of the scene and by using a light probe to measure the incident illumination at the location of the synthetic objects. The global illumination solution is then composited into a photograph of the scene using the differential rendering technique. We conclude by discussing the relevance of the technique to recovering surface reflectance properties in uncontrolled lighting situations. Applications of the method include visual effects, interior design, and architectural visualization.

Related Works

Optical Film Compositing; Global Illumination; Compositing Objects into Photography

BRDF: Geometrical Considerations and Nomenclature for Reflectance

F.E. Nicodemus, J.C. Richmond, J.J. Hsia, I.W. Ginsberg, T. Limperis

NATIONAL BUREAU OF STANDARDS

Portals

Abstract

A unified approach to the specification of reflectance, in terms of both incident- and reflected beam geometry, is presented. Nomenclature to facilitate this approach is proposed. Under specified conditions-including uniform irradiance, a uniform, isotropic, plane surface, and allowance for edge effects due to sub-surface scattering-the geometrical reflecting properties of a reflecting surface are readily characterized or specified in terms of the bidirectional reflectancedistribution function (BRDF). The BRDF is a derivative, a distribution function, relating the irradiance incident from one given direction to its contribution to the reflected radiance in another direction. Nomenclature (concepts, terms, symbols, and units) for categorizing and specifying reflectance quantities for a variety of different beam configurations (both incident and reflected beams) is described, and all are defined and interrelated in terms of the BRDF. The conditions under which the formalism can be applied, including situations involving considerable sub-surface scattering, are carefully established. The entire treatment is limited to the domain of classical geometrical-optics radiometry and does not take into account interference and diffraction phenomena, such as are frequently encountered with highly coherent radiant flux. The other radiation parameters such as wavelength, (temporal) modulation, and polarization and the effects of fluorescence (or phosphorescence) are discussed briefly.

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan

Google; UC Berkeley

Portals

Abstract

The rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. The straightforward solution of supersampling by rendering with multiple rays per pixel is impractical for NeRF, because rendering each ray requires querying a multilayer perceptron hundreds of times. Our solution, which we call "mip-NeRF" (a la "mipmap"), extends NeRF to represent the scene at a continuously-valued scale. By efficiently rendering anti-aliased conical frustums instead of rays, mip-NeRF reduces objectionable aliasing artifacts and significantly improves NeRF's ability to represent fine details, while also being 7% faster than NeRF and half the size. Compared to NeRF, mip-NeRF reduces average error rates by 17% on the dataset presented with NeRF and by 60% on a challenging multiscale variant of that dataset that we present. Mip-NeRF is also able to match the accuracy of a brute-force supersampled NeRF on our multiscale dataset while being 22x faster.

Related Works

Anti-aliasing in Rendering; Scene Representations for View Synthesis

Comparisons

NeRF, JaxNeRF, SRN, Neural Volumes, LLFF, NSVF

Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering

Brent Burley

Walt Disney Animation Studios

Portals

Deep Reflectance Fields: High-Quality Facial Reflectance Field Inference From Color Gradient Illumination

Abhimitra Meka, Christian H

MPI Informatics; Saarland Informatics Campus; Google; Stanford University

Portals

Abstract

We present a novel technique to relight images of human faces by learning a model of facial reflectance from a database of 4D reflectance field data of several subjects in a variety of expressions and viewpoints. Using our learned model, a face can be relit in arbitrary illumination environments using only two original images recorded under spherical color gradient illumination. The output of our deep network indicates that the color gradient images contain the information needed to estimate the full 4D reflectance field, including specular reflections and high frequency details. While capturing spherical color gradient illumination still requires a special lighting setup, reduction to just two illumination conditions allows the technique to be applied to dynamic facial performance capture. We show side-by-side comparisons which demonstrate that the proposed system outperforms the state-of-the-art techniques in both realism and speed.

Related Works

Parametric Model Fitting; Image-Based Relighting; Learning-Based Techniques

Comparisons

Fyffe et al. 2009, Shu et al. 2017, Yamaguchi et al. 2018

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker

University of California, San Diego; Adobe Research

Portals

Abstract

We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. To train this network, we augment the SUNCG indoor scene dataset with real-world materials and render them with a fast, high-quality, physically-based GPU renderer to create a large-scale, photorealistic indoor dataset. Our inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -- allowing us to jointly reason about shape, lighting, and reflectance. Experiments show that our framework outperforms previous methods for estimating individual scene components, which also enables various novel applications for augmented reality, such as photorealistic object insertion and material editing. Code and data will be made publicly available.

Related Works

Single objects; Large-scale scenes; Datasets; Differentiable rendering

Category: Rendering

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Related Works

Portals

Portals

Abstract

Portals

Abstract

Related Works

Portals

Abstract

Portals

Abstract

Related Works

Comparisons

Portals

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Related Works