Official implementation of MicroDreamer: Zero-shot 3D Generation in ~20 Seconds by Score-based Iterative Reconstruction.
display.mp4
[10/2024] Add a new mesh export method from LGM
The codebase is built on DreamGaussian. For installation,
conda create -n MicroDreamer python=3.11 conda activate MicroDreamer pip install -r requirements.txt # a modified gaussian splatting (+ depth, alpha rendering) git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization pip install ./diff-gaussian-rasterization # The commit hash we used # d986da0d4cf2dfeb43b9a379b6e9fa0a7f3f7eea # simple-knn pip install ./simple-knn # nvdiffrast pip install git+https://github.com/NVlabs/nvdiffrast/ # The version we used # pip install git+https://github.com/NVlabs/nvdiffrast/@0.3.1 # kiuikit pip install git+https://github.com/ashawkey/kiuikit/ # The version we used # pip install git+https://github.com/ashawkey/kiuikit/@0.2.3 # To use ImageDream, also install: pip install git+https://github.com/bytedance/ImageDream/#subdirectory=extern/ImageDream # The commit hash we used # 26c3972e586f0c8d2f6c6b297aa9d792d06abebbImage-to-3D:
### preprocess # background removal and recentering, save rgba at 256x256 python process.py test_data/name.jpg # save at a larger resolution python process.py test_data/name.jpg --size 512 # process all jpg images under a dir python process.py test_data ### training gaussian stage # train 20 iters and export ckpt & coarse_mesh to logs python main.py --config configs/image_sai.yaml input=test_data/name_rgba.png save_path=name_rgba ### training mesh stage # auto load coarse_mesh and refine 3 iters, export fine_mesh to logs python main2.py --config configs/image_sai.yaml input=test_data/name_rgba.png save_path=name_rgbaImage+Text-to-3D (ImageDream):
### training gaussian stage python main.py --config configs/imagedream.yaml input=test_data/ghost_rgba.png prompt="a ghost eating hamburger" save_path=ghost_rgbaCalculate for CLIP similarity:
PYTHONPATH='.' python scripts/cal_sim.pytotal_1.mp4
total_2.mp4
This work is built on many amazing open source projects, thanks to all the authors!
@misc{chen2024microdreamerzeroshot3dgeneration, title={MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction}, author={Luxi Chen and Zhengyi Wang and Zihan Zhou and Tingting Gao and Hang Su and Jun Zhu and Chongxuan Li}, year={2024}, eprint={2404.19525}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2404.19525}, }