TOG - Paper Copilot

Ultra-high Resolution SVBRDF Recovery From a Single Image

Jie Guo, Shuichang Lai, Qinghao Tu, Chengzhi Tao, Changqing Zou, Yanwen Guo, Jie Guo

Nanjing University; Zhejiang University

Portals

Abstract

Existing convolutional neural networks have achieved great success in recovering Spatially Varying Bidirectional Surface Reflectance Distribution Function (SVBRDF) maps from a single image. However, they mainly focus on handling low-resolution (e.g., 256 × 256) inputs. Ultra-High Resolution (UHR) material maps are notoriously difficult to acquire by existing networks because (1) finite computational resources set bounds for input receptive fields and output resolutions, and (2) convolutional layers operate locally and lack the ability to capture long-range structural dependencies in UHR images. We propose an implicit neural reflectance model and a divide-and-conquer solution to address these two challenges simultaneously. We first crop a UHR image into low-resolution patches, each of which are processed by a local feature extractor to extract important details. To fully exploit long-range spatial dependency and ensure global coherency, we incorporate a global feature extractor and several coordinate-aware feature assembly modules into our pipeline. The global feature extractor contains several lightweight material vision transformers that have a global receptive field at each scale and have the ability to infer long-term relationships in the material. After decoding globally coherent feature maps assembled by coordinate-aware feature assembly modules, the proposed end-to-end method is able to generate UHR SVBRDF maps from a single image with fine spatial details and consistent global structures.

Related Works

SVBRDF Recovery from Multiple Images; SVBRDF Recovery from Single Images; Vision Transformers

Comparisons

HANet, Guided

SpongeCake: A Layered Microflake Surface Appearance Model

Beibei Wang, Wenhua Jin, Miloš Hašan, Ling-Qi Yan

Nankai University; Nanjing University of Science and Technology; Adobe Research; University of California, Santa Barbara

Portals

Abstract

In this paper, we propose SpongeCake: a layered BSDF model where each layer is a volumetric scattering medium, defined using microflake or other phase functions. We omit any reflecting and refracting interfaces between the layers. The first advantage of this formulation is that an exact and analytic solution for single scattering, regardless of the number of volumetric layers, can be derived. We propose to approximate multiple scattering by an additional single-scattering lobe with modified parameters and a Lambertian lobe. We use a parameter mapping neural network to find the parameters of the newly added lobes to closely approximate the multiple scattering effect. Despite the absence of layer interfaces, we demonstrate that many common material effects can be achieved with layers of SGGX microflake and other volumes with appropriate parameters. A normal mapping effect can also be achieved through mapping of microflake orientations, which avoids artifacts common in standard normal maps. Thanks to the analytical formulation, our model is very fast to evaluate and sample. Through various parameter settings, our model is able to handle many types of materials, like plastics, wood, cloth, etc., opening a number of practical applications.

Related Works

Layered material models; Microflake models; Microfacet multiple scattering; Single scattering in volumetric layers; Neural networks for material appearance

The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting

Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, Matthew DuVall, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, Sean Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul Debevec, and Shahram Izadi

Google

Portals

Abstract

We present "The Relightables", a volumetric capture system for photorealistic and high quality relightable full-body performance capture. While significant progress has been made on volumetric capture systems, focusing on 3D geometric reconstruction with high resolution textures, much less work has been done to recover photometric properties needed for relighting. Results from such systems lack high-frequency details and the subject's shading is prebaked into the texture. In contrast, a large body of work has addressed relightable acquisition for image-based approaches, which photograph the subject under a set of basis lighting conditions and recombine the images to show the subject as they would appear in a target lighting environment. However, to date, these approaches have not been adapted for use in the context of a high-resolution volumetric capture system. Our method combines this ability to realistically relight humans for arbitrary environments, with the benefits of free-viewpoint volumetric capture and new levels of geometric accuracy for dynamic performances. Our subjects are recorded inside a custom geodesic sphere outfitted with 331 custom color LED lights, an array of high-resolution cameras, and a set of custom high-resolution depth sensors. Our system innovates in multiple areas: First, we designed a novel active depth sensor to capture 12.4MP depth maps, which we describe in detail. Second, we show how to design a hybrid geometric and machine learning reconstruction pipeline to process the high resolution input and output a volumetric video. Third, we generate temporally consistent reflectance maps for dynamic performers by leveraging the information contained in two alternating color gradient illumination images acquired at 60Hz. Multiple experiments, comparisons, and applications show that The Relightables significantly improves upon the level of realism in placing volumetrically captured human performances into arbitrary CG scenes.

Related Works

Image-based Rendering; Model Based; Volumetric Capture; Machine Learning

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

Thu Nguyen-Phuoc, Feng Liu, Lei Xiao

Reality Labs Research, Meta

Portals

Abstract

This paper presents a stylized novel view synthesis method. Applying state-of-the-art stylization methods to novel views frame by frame often causes jittering artifacts due to the lack of cross-view consistency. Therefore, this paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. Specifically, we adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation for their capability to render high-quality novel views for a variety of scenes. However, as rendering a novel view from a NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that goes beyond an off-the-shelf GPU capacity. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps. Such a method enables us to make full use of our hardware memory capacity to both generate images at higher resolution and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor and dynamic scenes, and synthesizes high-quality novel views with cross-view consistency.

Related Works

Image and video style transfer; 3D style transfer; Novel view synthesis

Comparisons

AdaIN, WCT, LST, MCCNet, ReReVST, StyleScene

Reflectance and Texture of Real-World Surfaces

Kristin J. Dana, Shree K. Nayar, Bram Van Ginneken, Jan J. Koenderink

Columbia University; Utrecht University

Portals

Abstract

In this work, we investigate the visual appearance of real-world surfaces and the dependence of appearance on the geometry of imaging conditions. We discuss a new texture representation called the BTF (bidirectional texture function) which captures the variation in texture with illumination and viewing direction. We present a BTF database with image textures from over 60 different samples, each observed with over 200 different combinations of viewing and illumination directions. We describe the methods involved in collecting the database as well as the importqance and uniqueness of this database for computer graphics. A related quantity to the BTF is the familiar BRDF (bidirectional reflectance distribution function). The measurement methods involved in the BTF database are conducive to simultaneous measurement of the BRDF. Accordingly, we also present a BRDF database with reflectance measurements for over 60 different samples, each observed with over 200 different combinations of viewing and illumination directions. Both of these unique databases are publicly available and have important implications for computer graphics.

Neural Light Transport for Relighting and View Synthesis

Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T. Barron, Ravi Ramamoorthi, William T. Freeman

Massachusetts Institute of Technology; Google; University of California, San Diego

Portals

Abstract

The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires.

Related Works

Single observation; Multiple views; Multiple illuminants; Multiple views and illuminants

Comparisons

Diffuse Base, Barycentric Blending, Deep Shading, Xu et al., Relightables

Deep Reflectance Fields: High-Quality Facial Reflectance Field Inference From Color Gradient Illumination

Abhimitra Meka, Christian H

MPI Informatics; Saarland Informatics Campus; Google; Stanford University

Portals

Abstract

We present a novel technique to relight images of human faces by learning a model of facial reflectance from a database of 4D reflectance field data of several subjects in a variety of expressions and viewpoints. Using our learned model, a face can be relit in arbitrary illumination environments using only two original images recorded under spherical color gradient illumination. The output of our deep network indicates that the color gradient images contain the information needed to estimate the full 4D reflectance field, including specular reflections and high frequency details. While capturing spherical color gradient illumination still requires a special lighting setup, reduction to just two illumination conditions allows the technique to be applied to dynamic facial performance capture. We show side-by-side comparisons which demonstrate that the proposed system outperforms the state-of-the-art techniques in both realism and speed.

Related Works

Parametric Model Fitting; Image-Based Relighting; Learning-Based Techniques

Comparisons

Fyffe et al. 2009, Shu et al. 2017, Yamaguchi et al. 2018

Cook-Torrance: A Reflectance Model for Computer Graphics

Robert L. Cook, Kenneth E Torrance

Cornell University; Lucasfilm Ltd.

Portals

Abstract

A new reflectance model for rendering computer sythesized images is presented. The model accounts for the relative brightness of different materials and light sources in the same scene. It describes the directional distribution of the reflected light and a color shift that occurs as the reflectance changes with incidence angle. A method for obtaining the spectral energy distribution of the light relfected from an object made of a specific real material is presented, and a procedure for accurately reproducing the color associated with the spectral energy distribution is discussed. The model is applied to the simulation of a metal and a plastic.

Light Diffusion in Multi-Layered Translucent Materials

Craig Donner, Henrik Wann Jensen

University of California, San Diego

Portals

Abstract

This paper introduces a shading model for light diffusion in multi-layered translucent materials. Previous work on diffusion in translucent materials has assumed smooth semi-infinite homogeneous materials and solved for the scattering of light using a dipole diffusion approximation. This approximation breaks down in the case of thin translucent slabs and multi-layered materials. We present a new efficient technique based on multiple dipoles to account for diffusion in thin slabs. We enhance this multipole theory to account for mismatching indices of refraction at the top and bottom of of translucent slabs, and to model the effects of rough surfaces. To model multiple layers, we extend this single slab theory by convolving the diffusion profiles of the individual slabs. We account for multiple scattering between slabs by using a variant of Kubelka-Munk theory in frequency space. Our results demonstrate diffusion of light in thin slabs and multi-layered materials such as paint, paper, and human skin.

Tag: TOG

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Related Works

Portals

Abstract

Related Works

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Portals

Abstract