Authors
Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Milo
University of California, San Diego; Adobe Research
Portals
Abstract
We propose NeuMIP, a neural method for representing and rendering a variety of material appearances at different scales. Classical prefiltering (mipmapping) methods work well on simple material properties such as diffuse color, but fail to generalize to normals, self-shadowing, fibers or more complex microstructures and reflectances. In this work, we generalize traditional mipmap pyramids to pyramids of neural textures, combined with a fully connected network. We also introduce neural offsets, a novel method which allows rendering materials with intricate parallax effects without any tessellation. This generalizes classical parallax mapping, but is trained without supervision by any explicit heightfield. Neural materials within our system support a 7-dimensional query, including position, incoming and outgoing direction, and the desired filter kernel size. The materials have small storage (on the order of standard mipmapping except with more texture channels), and can be integrated within common Monte-Carlo path tracing systems. We demonstrate our method on a variety of materials, resulting in complex appearance across levels of detail, with accurate parallax, self-shadowing, and other effects.
Contribution
- A neural method which can represent a wide variety of geometrically complex materials at different scales, trained from random queries of the continuous 7-dimensional multiscale BTF. These queries can come from real or synthetic data
- A neural offset technique for rendering complex geometric appearance including parallax effects without tessellation, trained in an unsupervised manner
- Ability to learn appearance from a small number of queries per texel (200-400), due to an encoder-less architecture
Related Works
Prefiltering and Mipmapping; Bidirectional Texture Functions; Neural Reflectance; Other Neural Material Methods
Overview
Overview of our neural architecture. Left: The neural offset module (more detail in Fig. 4) takes a uv-space location and incoming direction, and predicts a new (offset) uv-space location to simulate parallax effects. Middle: The neural texture pyramid is queried using trilinear interpolation to obtain a 7-channel feature vector. Right: The feature vector, with incoming and outgoing directions, is fed to a material-specific multi-layer perceptron, which predicts the RGB reflectance value.