Stable Diffusion with OV TorchDynamo backend (#1338)

openvinotoolkit · Jan 19, 2024 · d3545d0 · d3545d0
1 parent 4e70504
commit d3545d0
Show file tree

Hide file tree

Showing 8 changed files with 452 additions and 1 deletion.
diff --git a/.ci/ignore_treon_docker.txt b/.ci/ignore_treon_docker.txt
@@ -44,6 +44,7 @@
 271-sdxl-turbo
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
+276-stable-diffusion-torchdynamo-backend
 301-tensorflow-training-openvino
 305-tensorflow-quantization-aware-training
 404-style-transfer-webcam
diff --git a/.ci/ignore_treon_linux.txt b/.ci/ignore_treon_linux.txt
@@ -47,4 +47,5 @@
 271-sdxl-turbo
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
+276-stable-diffusion-torchdynamo-backend
 404-style-transfer-webcam
diff --git a/.ci/ignore_treon_mac.txt b/.ci/ignore_treon_mac.txt
@@ -44,4 +44,5 @@
 271-sdxl-turbo
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
+276-stable-diffusion-torchdynamo-backend
 404-style-transfer-webcam
diff --git a/.ci/ignore_treon_win.txt b/.ci/ignore_treon_win.txt
@@ -45,4 +45,5 @@
 270-sound-generation-audioldm2
 271-sdxl-turbo
 272-paint-by-example
-273-stable-zephyr-3b-chatbot
+273-stable-zephyr-3b-chatbot
+276-stable-diffusion-torchdynamo-backend
diff --git a/.ci/spellcheck/.pyspelling.wordlist.txt b/.ci/spellcheck/.pyspelling.wordlist.txt
@@ -32,6 +32,7 @@ autoregressive
 autoregressively
 AutoTokenizer
 backend
+backends
 Baevski
 BasicUNet
 BEiT
@@ -251,6 +252,7 @@ imagenet
 Imagenet
 ImageNet
 Imagenette
+ImgPipeline
 impactful
 IMU
 IMUs
@@ -610,6 +612,9 @@ SSD
 SSDLite
 sst
 StableDiffusionInpaintPipeline
+StableDiffusionPipeline
+StableDiffusionImg
+StableDiffusionImg2ImgPipeline
 STFT
 stateful
 streamable
@@ -621,6 +626,7 @@ subchunks
 subdirectories
 subdirectory
 subgraph
+subgraphs
 sublicense
 subpackage
 subtask
@@ -654,6 +660,8 @@ tokenizers
 tokenizes
 tokenizing
 Tongyi
+TorchDynamo
+torchdynamo
 TorchMetrics
 TorchScript
 torchvision
@@ -718,6 +726,7 @@ waveforms
 Wav
 WavLM
 WebGL
+WebUI
 WER
 WIKISQL
 WikiTable

diff --git a/README.md b/README.md
@@ -225,6 +225,7 @@ Demos that demonstrate inference on a particular model.
 | [273-stable-zephyr-3b-chatbot](notebooks/273-stable-zephyr-3b-chatbot)<br> |  Use Stable-Zephyr as chatbot assistant with OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/cfac6ddb-6f22-4343-855c-e513269cf2bf width=225> |
 | [274-efficient-sam](notebooks/274-efficient-sam/)<br>| Object segmentation with EfficientSAM and OpenVINO™ | <img width="225" src="https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/15d0a23a-0550-43c6-9ca9-f689e772a79a"> |
 | [275-llm-question-answering](notebooks/275-llm-question-answering)<br> | LLM Instruction following pipeline | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/daafd702-5a42-4f54-ae72-2e4480d73501 width=225> | 
+| [276-stable-diffusion-torchdynamo-backend](notebooks/276-stable-diffusion-torchdynamo-backend/)<br> | Image generation with Stable Diffusion and OpenVINO™ `torch.compile` feature | <img src=https://user-images.githubusercontent.com/32199725/270969617-ee88eb42-3b0d-42f4-ac71-992c40451849.png width=225> |
 
 <div id='-model-training'></div>
 

diff --git a/...s/276-stable-diffusion-torchdynamo-backend/276-stable-diffusion-torchdynamo-backend.ipynb b/...s/276-stable-diffusion-torchdynamo-backend/276-stable-diffusion-torchdynamo-backend.ipynb
diff --git a/notebooks/276-stable-diffusion-torchdynamo-backend/README.md b/notebooks/276-stable-diffusion-torchdynamo-backend/README.md
@@ -0,0 +1,25 @@
+# Image Generation with Stable Diffusion using OpenVINO TorchDynamo backend
+
+This notebook demonstrates how to use a **[Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-2-1)** model for image generation with [OpenVINO TorchDynamo backend](https://docs.openvino.ai/2023.1/pytorch_2_0_torch_compile.html). The `torch.compile` feature enables you to use OpenVINO for PyTorch-native applications. It speeds up PyTorch code by JIT-compiling it into optimized kernels.
+
+By default, Torch code runs in eager-mode, but with the use of torch.compile it goes through the following steps:
+
+1. Graph acquisition - the model is rewritten as blocks of subgraphs that are either:
+    * compiled by TorchDynamo and “flattened”,
+    * falling back to the eager-mode, due to unsupported Python constructs (like control-flow code).
+2. Graph lowering - all PyTorch operations are decomposed into their constituent kernels specific to the chosen backend.
+3. Graph compilation - the kernels call their corresponding low-level device-specific operations.
+
+## Notebook Contents
+
+This notebook demonstrates how to run stable diffusion using OpenVINO TorchDynamo backend.
+
+Notebook contains the following steps:
+
+1. Create PyTorch models pipeline using Diffusers library.
+2. Import OpenVINO backend using `torch.compile`.
+3. Run Stable Diffusion pipeline with OpenVINO TorchDynamo backend.
+
+## Installation Instructions
+
+If you have not installed all required dependencies, follow the [Installation Guide](../../README.md).