Come on, come on, let's adapt the conversion script to SD 2.0

[piEsposito]

Is your feature request related to a problem? Please describe.
It would be great if we could run SD 2 with cpu_offload, attention slicing, xformers, etc...

Describe the solution you'd like
Adapt the conversion script to SD 2.0

Describe alternatives you've considered
Stability AI's repo is not as flexible.

[averad]

🤗 Diffusers with Stable Diffusion 2 is live!

anton-l commented (#1388 (comment))
diffusers==0.9.0 with Stable Diffusion 2 are live!

Installation
pip install diffusers[torch]==0.9 transformers

Release Information
https://github.com/huggingface/diffusers/releases/tag/v0.9.0

Contributors
@kashif
@pcuenca
@patrickvonplaten
@anton-l
@patil-suraj

📰 News

• Weights for the 768x768 model is up! - Averad

  • https://huggingface.co/stabilityai/stable-diffusion-2

• add v prediction #1386 - by patil-suraj and patrickvonplaten has been merged into --main by patil-suraj

• Weights for the 512x512 models are up! - patrickvonplaten

  • https://huggingface.co/stabilityai/stable-diffusion-2-base
  • https://huggingface.co/stabilityai/stable-diffusion-2-inpainting

✏️ Notes & Information

Related huggingface/diffusers Pull Requests:

• SD2 docs #1424
• added initial v-pred support to DPM-solver #1421
• Add tests for Stable Diffusion 2 V-prediction 768x768 #1420
• Allow to set config params directly in init #1419
• Fix ONNX conversion and inference #1416
• Fix all stable diffusion #1415
• fix clip guided #1414
• Add SD2 inpainting integration tests #1412
• [MPS] call contiguous after permute #1411
• fix stable-diffusion V2 on MPS #1410
• Upscaling fixed #1402
• Add the new SD2 attention params to the VD text unet #1400
• Support SD2 attention slicing #1397
• StableDiffusionUpscalePipeline #1396
• add v prediction #1386 - by patil-suraj and patrickvonplaten

👇 Quick Links:

• https://github.com/Stability-AI/stablediffusion
  • requirements.txt
  • txt2img
  • img2img
  • Gradio Scripts
  • Streamlit Scripts
• https://huggingface.co/stabilityai/stable-diffusion-2
• https://huggingface.co/stabilityai/stable-diffusion-2-base
• https://huggingface.co/stabilityai/stable-diffusion-2-depth
• https://huggingface.co/stabilityai/stable-diffusion-2-inpainting
• https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler

👁️ User Submitted Resources:

• Test Version of convert_original_stable_diffusion_to_diffusers.py by hafriedlander
• Example of accessing the penultimate text embedding layer by hafriedlander
• NovelAI Improvements on Stable Diffusion

💭 User Story (Prior to Huggingface Diffusers 0.9.0 Release)

Stability-AI has released Stable Diffusion 2.0 models/workflow. When you run convert_original_stable_diffusion_to_diffusers.py on the new Stability-AI/stablediffusion models the following errors occur.

convert_original_stable_diffusion_to_diffusers.py --checkpoint_path="./512-inpainting-ema.ckpt" --dump_path="./512-inpainting-ema_diffusers"

Output:

Traceback (most recent call last):
File "convert_original_stable_diffusion_to_diffusers.py", line 720, in <module> 
        unet.load_state_dict(converted_unet_checkpoint)
File "lib\site-packages\torch\nn\modules\module.py", line 1667, in load_state_dict
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for down_blocks.0.attentions.0.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.proj_out.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
.... blocks.1.attentions blocks.2.attentions etc. etc.

[devilismyfriend]

trying to but likely I won't be able to do it lol

[0xdevalias]

Semi-Related:

• Please make a Dockerfile Stability-AI/stablediffusion#4
• Add an example Google Colab / Jupyter Notebook file Stability-AI/stablediffusion#9
• [Feature Request]: SD 2.0 Support AUTOMATIC1111/stable-diffusion-webui#5009 [dupe]
• [Feature Request]: Support for new 2.0 models | 768x768 resolution + new 512x512 + depth + inpainting AUTOMATIC1111/stable-diffusion-webui#5011
• Stable Diffusion 2 #1392
• Stable Diffusion 2 and Apple Silicon/MPS #1398
• CLIP Guided StableDiffusion V2 #1404
• Add support for Stable Diffusion 2.0 models Haidra-Org/AI-Horde#86
• Add support for Stable Diffusion 2.0 models JoePenna/Dreambooth-Stable-Diffusion#112
• Add stable diffusion 2 TheLastBen/fast-stable-diffusion#599
• Add support for Stable Diffusion 2.0 models ShivamShrirao/diffusers#143
• https://github.com/Sygil-Dev/nataili/issues/67
• Add support for Stable Diffusion 2.0 models Sygil-Dev/sygil-webui#1686

[devilismyfriend]

after looking at it I'm not sure it has anything to do with the script, seems like the u2net on diffusers needs to have 4 dimensions on the tensor size.

[AugmentedRealityCat]

needs to have 4 dimensions

So I guess this will take time...

[devilismyfriend]

needs to have 4 dimensions

So I guess this will take time...

maybe not, I'm not that knowledgeable on the subject but I assume a unet2D needs to be 4D, or maybe you can just artificially add it idk

[0xdevalias]

rudimentary support for stable diffusion 2.0

MrCheeze/stable-diffusion-webui@069591b

Originally posted by @152334H in AUTOMATIC1111/stable-diffusion-webui#5011 (comment)

[hafriedlander]

https://github.com/hafriedlander/diffusers/blob/stable_diffusion_2/scripts/convert_original_stable_diffusion_to_diffusers.py

Notes:

• Only tested on the two txt2img models, not inpaint / depth2img / upscaling
• You will need to change your text embedding to use the penultimate layer too
• It spits out a bunch of warnings about vision_model, but that's fine
• I have no idea if this is right or not. It generates images, no guarantee beyond that. (Hence no PR - if you're patient, I'm sure the Diffusers team will do a better job than I have)