Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
This repository contains the Gen-3Diffusion model presented in the paper Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy.
Gen-3Diffusion addresses the challenging problem of creating realistic 3D objects and clothed avatars from a single RGB image. It leverages a pre-trained 2D diffusion model and a 3D diffusion model, synchronizing them at both training and sampling time. This synergy allows the 2D model to provide strong generalization for shapes, while the 3D model enhances multi-view consistency, leading to high-fidelity geometry and texture in generated 3D objects and avatars.
Key Insight :raised_hands:
- 2D foundation models are powerful but output lacks 3D consistency!
- 3D generative models can reconstruct 3D representation but is poor in generalization!
- How to combine 2D foundation models with 3D generative models?:
- they are both diffusion-based generative models => Can be synchronized at each diffusion step
- 2D foundation model helps 3D generation => provides strong prior informations about 3D shape
- 3D representation guides 2D diffusion sampling => use rendered output from 3D reconstruction for reverse sampling, where 3D consistency is guaranteed
Pretrained Weights
Our pretrained weights can be downloaded from Hugging Face.
mkdir checkpoints_obj && cd checkpoints_obj
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/model.safetensors
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/model_1.safetensors
wget https://huggingface.co/yuxuanx/gen3diffusion/resolve/main/pifuhd.pt
cd ..
The avatar reconstruction module is same to Human-3Diffusion. Please skip if you already installed Human-3Diffusion.
mkdir checkpoints_avatar && cd checkpoints_avatar
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/model.safetensors
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/model_1.safetensors
wget https://huggingface.co/yuxuanx/human3diffusion/resolve/main/pifuhd.pt
cd ..
Sample Usage (Inference)
The following commands illustrate how to use the model for image-to-3D object and avatar generation. Please refer to the GitHub repository for full installation and setup instructions.
# given one image of object, generate 3D-GS object
# subject should be centered in a square image, please crop properly
# recenter plays a huge role in object reconstruction. Please adjust the recentering if the reconstruction doesn't work well
python infer.py --test_imgs test_imgs_obj --output output_obj --checkpoints checkpoints_obj
# given generated 3D-GS, perform TSDF mesh extraction
python infer_mesh.py --test_imgs test_imgs_obj --output output_obj --checkpoints checkpoints_obj --mesh_quality high
# given one image of human, generate 3D-GS avatar
# subject should be centered in a square image, please crop properly
python infer.py --test_imgs test_imgs_avatar --output output_avatar --checkpoints checkpoints_avatar
# given generated 3D-GS, perform TSDF mesh extraction
python infer_mesh.py --test_imgs test_imgs_avatar --output output_avatar --checkpoints checkpoints_avatar --mesh_quality high
Citation :writing_hand:
If you find our work helpful or inspiring, please feel free to cite it:
@inproceedings{xue2024gen3diffusion,
title = {{Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy }},
author = {Xue, Yuxuan and Xie, Xianghui and Marin, Riccardo and Pons-Moll, Gerard.},
journal = {Arxiv},
year = {2024},
}