Authors
Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, Guanbin Li
Sun Yat-sen University; University of Pennylvania
Portals
Summary
e 1: Our approach DreamEditor allows users to edit 3D scenes with text prompts. DreamEditor achieves precise and high-quality editing that maintains irrelevant regions unchange
Abstract
Neural fields have achieved impressive advancements in view synthesis and scene reconstruction. However, editing these neural fields remains challenging due to the implicit encoding of geometry and texture information. In this paper, we propose DreamEditor, a novel framework that enables users to perform controlled editing of neural fields using text prompts. By representing scenes as mesh-based neural fields, DreamEditor allows localized editing within specific regions. DreamEditor utilizes the text encoder of a pretrained text-to-Image diffusion model to automatically identify the regions to be edited based on the semantics of the text prompts. Subsequently, DreamEditor optimizes the editing region and aligns its geometry and texture with the text prompts through score distillation sampling [29]. Extensive experiments have demonstrated that DreamEditor can accurately edit neural fields of real-world scenes according to the given text prompts while ensuring consistency in irrelevant areas. DreamEditor generates highly realistic textures and geometry, significantly surpassing previous works in both quantitative and qualitative evaluations.
Contribution
- We introduce a novel framework for text-guided 3D scene editing, which achieves highly realistic editing results for a wide range of real-world scene
- We propose to use a mesh-based neural field to enable local modification of the scene and decouple texture and geometric features for flexible edit
- We devise a stepwise editing framework that first identifies the specific regions requiring editing according to text prompts and then performs modifications exclusively within the selected regions. This systematic procedure ensures precise 3D editing while minimizing unnecessary modifications in irrelevant regions
Related Works
Text-guided image generation and edit; Text-to-3D generation; Neural Field Editing
Comparisons
Instruct-NeRF2NeRF, NeRF-Art, DreamFusion, DreamBooth
Overview
Our method edits a 3D scene by optimizing an existing neural field to conform with a target text prompt. The editing process involves three steps: (1) The original neural field is distilled into a mesh-based one. (2) Based on the text prompts, our method automatically identifies the editing region of the mesh-based neural field. (3) Our method utilizes the SDS loss to optimize the color feature, geometry feature, and vertex positions of the editing region, thereby altering the texture and geometry of the respective region. Best viewed in color.