Authors
Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, Zhaopeng Cui
Zhejiang University; Google
Portals
Summary
We propose a novel semantic-driven image-based editing approach, which allows users to edit a photo-realistic NeRF with a single-view image or with text prompts, and renders edited novel views with vivid details and multi-view consistency.
Abstract
Despite the great success in 2D editing using user-friendly tools, such as Photoshop, semantic strokes, or even text prompts, similar capabilities in 3D areas are still limited, either relying on 3D modeling skills or allowing editing within only a few categories. In this paper, we present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image, and faithfully delivers edited novel views with high fidelity and multi-view consistency. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space, and develop a series of techniques to aid the editing process, including cyclic constraints with a proxy mesh to facilitate geometric supervision, a color compositing mechanism to stabilize semantic-driven texture editing, and a feature-cluster-based regularization to preserve the irrelevant content unchanged. Extensive experiments and editing examples on both real-world and synthetic data demonstrate that our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes.
Contribution
- We propose a novel semantic-driven image-based NeRF editing approach, called SINE, which allows users to edit a neural radiance field simply on just a single view of the rendering. SINE leverages a prior-guided editing field to encode fine-grained geometry and texture changes over the given pre-trained NeRF, thus delivering multi-view consistent edited views with high fidelity
- To achieve semantic editing functionality, we develop a series of techniques,including cyclic constraints with a proxy mesh for geometric editing, the color compositing mechanism to enhance texture editing, and the feature-cluster-based regularization to control the affected editing area and maintain irrelevant parts unchanged
- Experiments and editing examples on both real-world/synthetic and object-centric/unbounded 360? scenes data demonstrate superior editing capabilities and quality with effortless operations
Related Works
Neural rendering with external priors;Neural 2D & 3D scene editing
Comparisons
EG3D, EditNeRF
Overview
We encode geometric and texture changes over the original template NeRF with a prior-guided editing field, where the geometric modification field F?G transformed the edited space query x into the template space x ? , and the texture modification field F?T encodes modification colors m? . Then, we render deformed template image Iˆo and color modification image Iˆm with all the queries, and use a color compositing layer to blend Iˆo and Iˆm into the edited view Iˆ.