Authors
Xilong Zhou, Milo
Texas A&M University; Adobe Research
Portals
Summary
We present a category-specific generative model for spatially-varying materials, whose results are seamlessly tileable and optionally conditioned on patterns controlling the material structure. Our model can be used to generate new materials, or inverted to find materials matching target photographs by optimization. Here we show examples of tile, leather, stone and metal classes, either directly generated (1, 2, 4, 6, 8) or reconstructed from photographs (3, 5, 7, 9, 10) and, in some cases, conditioned on an input structure pattern (1, 3, 5, 10). The insets show, where applicable, the target photograph, condition pattern and a rendering of the synthesized material. The generated maps include height fields that are displacement-mapped in the rendered scene.
Abstract
Recent methods (e.g. MaterialGAN) have used unconditional GANs to generate per-pixel material maps, or as a prior to reconstruct materials from input photographs. These models can generate varied random material appearance, but do not have any mechanism to constrain the generated material to a specific category or to control the coarse structure of the generated material, such as the exact brick layout on a brick wall. Furthermore, materials reconstructed from a single input photo commonly have artifacts and are generally not tileable, which limits their use in practical content creation pipelines. We propose TileGen, a generative model for SVBRDFs that is specific to a material category, always tileable, and optionally conditional on a provided input structure pattern. TileGen is a variant of StyleGAN whose architecture is modified to always produce tileable (periodic) material maps. In addition to the standard "style" latent code, TileGen can optionally take a condition image, giving a user direct control over the dominant spatial (and optionally color) features of the material. For example, in brick materials, the user can specify a brick layout and the brick color, or in leather materials, the locations of wrinkles and folds. Our inverse rendering approach can find a material perceptually matching a single target photograph by optimization. This reconstruction can also be conditional on a user-provided pattern. The resulting materials are tileable, can be larger than the target image, and are editable by varying the condition.
Related Works
Material Acquisition; Guided Material Generation and Acquisition; Generative Models
Comparisons
Overview
The conditional version of TileGen is trained on a dataset of SVBRDF parameter maps with corresponding condition patterns. Our conditional generator has a CollageGANlike encoder that maps the input pattern ? into features ? at the start of the StyleGAN2-based decoder (green); the decoder also receives the latent code ? (via a mapping network) and noise. The encoder and decoder have been modified to only use tileable operations. The resulting SVBRDF maps, together with the condition, are randomly translated and fed to a StyleGAN2 discriminator. Differences between conditional model and unconditional model are shown in light blue. In unconditional model, we do not have input patterns and encoder-decoder. See Sec. 3 for more details.