Fine-tuning stable diffusion to generate planet/moon textures.
Cherry-picked best of several generations with varying checkpoints, guidance scales, and seeds.
| Prompt | Texture | Render |
|---|---|---|
| A planet with a surface of turquoise and gold, marked by large, glittering rivers of molten metal and bright, shining seas. |  |  |
| A small, rocky planet with a sandy, yellow surface, characterized by its large, winding canyons and massive, towering volcanoes |  |  |
| A planet with a surface covered in lush, green vegetation and large bodies of water, with a strikingly colorful atmosphere. |  |  |
| A frozen moon with a pristine white surface adorned with deep blue ice crevasses and towering ice peaks. |  |  |
| A dense, greenish gas giant, surrounded by a hazy, nebulous atmosphere, with faint, swirling bands visible across its surface. |  |  |
| A gas giant with a turbulent, stormy surface, displaying a mesmerizing swirl of teal, navy, and violet hues. |  |  |
Follow the instructions on sdxl-lora-planet-textures. You can also find several pre-trained models here.
LoRA enabled training on an NVIDIA 3090 Ti.
import torch from diffusers import DiffusionPipeline, AutoencoderKL vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16) pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", vae=vae, torch_dtype=torch.float16, variant="fp16", use_safetensors=True) pipe.load_lora_weights("sshh12/sdxl-lora-planet-textures") pipe.to("cuda") prompt = "A dwarf planet exhibiting a striking purple color, with a surface peppered with craters and towering ice formations" negative_prompt = 'blurry, fuzzy, low resolution, cartoon, painting' image = pipe(prompt=prompt, negative_prompt=negative_prompt, width=1024, height=512).images[0] imageSee scripts/generate_images.py for an example of advanced usage (including using an upscaler).
Cherry-picked best-of-4. It tends to struggle with prompts involving oceans or continents as that's pretty overfit to Earth. Generally, this model is fairly overfit to existing objects in our solar system.
| Prompt | Texture | Render |
|---|---|---|
| a red-ish moon full of large volcanos and craters. fictitious |  |  |
| a large gas giant with multi-color rainbow bands. fictitious |  |  |
| a dark grey cratered planet with large white icy poles. fictitious |  |  |
- Generate a dataset with the scripts in the repo or use sshh12/planet-textures
- Clone https://github.com/justinpinkney/stable-diffusion @
f1293f9795fda211d7fffdb84cd308424c2a184band applyv1/stable-diffusion.patch - Train the model. I used a
NVIDIA RTX A6000on LambdaLabs. If you do everything correctly the first time, the expected cost is $12.
from huggingface_hub import hf_hub_download ckpt_path = hf_hub_download(repo_id="CompVis/stable-diffusion-v-1-4-original", filename="sd-v1-4-full-ema.ckpt") !(python main.py \ -t \ --base ../v1/planet-diffusion.yaml \ --gpus "1" \ --scale_lr False \ --num_nodes 1 \ --check_val_every_n_epoch 10 \ --finetune_from "$ckpt_path" \ data.params.batch_size=1 \ lightning.trainer.accumulate_grad_batches=8 \ data.params.validation.params.n_gpus=1 \ )Feel free to contact me (using GitHub issues) for the original weights or you run into issues setting this up.
!(python scripts/txt2img.py \ --prompt 'your prompt here' \ --outdir '../outputs/' \ --H 512 --W 1024 \ --n_samples 2 \ --config '../v1/planet-diffusion.yaml' \ --ckpt 'logs/2023-06-29T00-13-09_planet-diffusion/checkpoints/epoch=000249.ckpt')