The Modular Diffusers #9672

yiyixuxu · 2024-10-14T19:00:44Z

TO-DOs before merge

PAG needs to be a guider on its own
One more guide on how to write a PipelineBlock; and how to assemble SequentialPipelineBlocks and AutoPipelineBlocks

TO-DOs

guider needs to be a ConfigMixin
ModularPipelineBlocks.from_pretrained needs to work with official blocks
API docs
a simple test case
Merge the concept of ModularLoader into ModularPipeline
Flux/Wan/ITX
We overrode changes in this PR https://github.com/huggingface/diffusers/pull/11652/files, need to make sure the updated code works for this case

Documentations

Documentations are live here

HuggingFaceDocBuilderDev · 2024-10-14T19:07:16Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yoland68 · 2024-10-16T17:20:48Z

Very cool!

… completely yet (look into later)

…oisestep

oozzy77 · 2024-10-30T14:50:49Z

hi this is very interesting! I'm making a Python pipeline flow visual scripting tool, that can auto-convert functions to visual nodes for fast and modular UI blocks demo. Itself is a pip package: https://pypi.org/project/nozyio/

I wanted to integrate diffusers with my flow nodes UI project but found its not very modular. But this PR may change that! Looking forward to see how this evolves.

github: https://github.com/oozzy77/nozyio happy to connect!

yiyixuxu · 2024-10-30T20:58:40Z

@oozzy77 thanks!
do you want to join a slack channel with me? if you want to experiment building something with this PR I'm eager to hear your feedback and iterate base on that

oozzy77 · 2024-10-31T20:10:20Z

Hi super willing to join slack channel with you! What’s the workspace channel I should join？or you can invite me ***@***.***

…

On Thu, Oct 31, 2024 at 4:59 AM YiYi Xu ***@***.***> wrote: @oozzy77 <https://github.com/oozzy77> thanks! do you want to join a slack channel with me? if you want to experiment building something with this PR I'm eager to hear your feedback and iterate base on that — Reply to this email directly, view it on GitHub <#9672 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMBK3ZHNSKN56N262LBH3WLZ6FCBNAVCNFSM6AAAAABP5SYMXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGM3DQMBYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

yiyixuxu · 2024-10-31T23:07:43Z

@oozzy77 I sent an invite!

…_index

…into modular-diffusers

a-r-r-o-w

Thanks @yiyixuxu! The other review comments address most of the PR already. To me, I already was exposed to a full implementation early-on so it's hard for me to review as user perspective, but the docs are great and give a wide coverage of everything.

docs/source/en/modular_diffusers/getting_started.md

src/diffusers/hooks/_helpers.py

* update * fix

…into modular-diffusers

Co-authored-by: Aryan <[email protected]>

…f all the guiders

…into modular-diffusers

* [CI] Fix big GPU test marker (#11786) * update * update * First Block Cache (#11180) * update * modify flux single blocks to make compatible with cache techniques (without too much model-specific intrusion code) * remove debug logs * update * cache context for different batches of data * fix hs residual bug for single return outputs; support ltx * fix controlnet flux * support flux, ltx i2v, ltx condition * update * update * Update docs/source/en/api/cache.md * Update src/diffusers/hooks/hooks.py Co-authored-by: Dhruv Nair <[email protected]> * address review comments pt. 1 * address review comments pt. 2 * cache context refacotr; address review pt. 3 * address review comments * metadata registration with decorators instead of centralized * support cogvideox * support mochi * fix * remove unused function * remove central registry based on review * update --------- Co-authored-by: Dhruv Nair <[email protected]> * fix --------- Co-authored-by: Dhruv Nair <[email protected]>

…into modular-diffusers

.github/workflows/nightly_tests.yml

add

33f85fa

add dataflow info for each block in builder _repr_

52a7f1c

yiyixuxu added 8 commits October 16, 2024 20:56

add img2img support - output does not match with non-modular pipeline…

e8d0980

… completely yet (look into later)

update img2img, result match

ad3f9a2

add from_pipe + run_blocks

ddea157

controlnet

af9572d

fix controlnet

2b6dcbf

combine controlnetstep into contronetdesnoisestep

70272b1

refactor guider, remove prepareguidance step to be combinedd into den…

46ec174

…oisestep

update pag guider - draft

f1b3036

a-r-r-o-w mentioned this pull request Oct 25, 2024

[docs] add docstrings in pipline_stable_diffusion.py #9590

Merged

6 tasks

yiyixuxu added 7 commits October 26, 2024 21:17

refactor guider

540d303

up

6742f16

add

005195c

fix so that run_blocks can work with inputs in the state

024a9f5

remove img2img blocksgit status consolidate text2img and img2img

37e8dc7

refactor, from_pretrained, from_pipe, remove_blocks, replace_blocks

8b811fe

style

c70a285

yiyixuxu mentioned this pull request Nov 6, 2024

Adaptive Projected Guidance #9626

Closed

add autostep (not complete)

ffc2992

asomoza mentioned this pull request Nov 30, 2024

Analysis of Classifier-Free Guidance Weight Schedulers #7756

Open

2 tasks

yiyixuxu added the roadmap Add to current release roadmap label Dec 4, 2024

hlky mentioned this pull request Dec 5, 2024

Detail-Daemon diffusers #10112

Open

update/refactor

ace53e2

yiyixuxu added 2 commits July 7, 2025 20:52

make sure modularpipeline from_pretrained works without modular_model…

6521f59

…_index

Merge branch 'modular-diffusers' of github.com:huggingface/diffusers …

e0083b2

…into modular-diffusers

a-r-r-o-w approved these changes Jul 7, 2025

View reviewed changes

a-r-r-o-w reviewed Jul 7, 2025

View reviewed changes

src/diffusers/hooks/_helpers.py Outdated Show resolved Hide resolved

yiyixuxu and others added 16 commits July 8, 2025 06:15

components manager: use shorter ID, display id instead of name

863c7df

add a guide on components manager

a2da000

Copied-from implementation of PAG-guider (#11882)

be5e10a

* update * fix

Merge branch 'modular-diffusers' of github.com:huggingface/diffusers …

04171c7

…into modular-diffusers

Apply suggestions from code review

e6ffde2

Co-authored-by: Aryan <[email protected]>

update remove duplicated config for pag, and remove the description o…

5f3ebef

…f all the guiders

add link to components manager doc

59abd95

addreess more review comments

f95c320

Merge branch 'main' into modular-diffusers

79166dc

add experimental marks to all modular docs

cb9dca5

add more docstrings + experimental marks

d27b654

style

595581d

Merge branch 'modular-diffusers' of github.com:huggingface/diffusers …

73c5fe8

…into modular-diffusers

big doc updategit status!

a935bea

Merge branch 'modular-diffusers' of github.com:huggingface/diffusers …

2b006a2

…into modular-diffusers

yiyixuxu commented Jul 9, 2025

View reviewed changes

.github/workflows/nightly_tests.yml Outdated Show resolved Hide resolved

a-r-r-o-w and others added 5 commits July 9, 2025 09:58

Merge branch 'main' into modular-diffusers

9106f9c

Merge branch 'main' into modular-diffusers

cf0f8e5

update more modular pipeline doc

2104bef

update doc

65ba892

up

01300a3

yiyixuxu merged commit f33b89b into main Jul 10, 2025
31 of 32 checks passed

github-project-automation bot moved this from In Progress to Done in Diffusers Roadmap 0.35 Jul 10, 2025

a-r-r-o-w deleted the modular-diffusers branch July 10, 2025 09:36

vladmandic mentioned this pull request Jul 12, 2025

Create modular pipeline from existing pipeline #11915

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

The Modular Diffusers #9672

The Modular Diffusers #9672

Uh oh!

yiyixuxu commented Oct 14, 2024 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Oct 14, 2024

Uh oh!

yoland68 commented Oct 16, 2024

Uh oh!

oozzy77 commented Oct 30, 2024 •

edited

Loading

Uh oh!

yiyixuxu commented Oct 30, 2024

Uh oh!

oozzy77 commented Oct 31, 2024 via email

Uh oh!

yiyixuxu commented Oct 31, 2024

Uh oh!

a-r-r-o-w left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

The Modular Diffusers #9672

The Modular Diffusers #9672

Uh oh!

Conversation

yiyixuxu commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TO-DOs before merge

TO-DOs

Documentations

Uh oh!

HuggingFaceDocBuilderDev commented Oct 14, 2024

Uh oh!

yoland68 commented Oct 16, 2024

Uh oh!

oozzy77 commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiyixuxu commented Oct 30, 2024

Uh oh!

oozzy77 commented Oct 31, 2024 via email

Uh oh!

yiyixuxu commented Oct 31, 2024

Uh oh!

a-r-r-o-w left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yiyixuxu commented Oct 14, 2024 •

edited

Loading

oozzy77 commented Oct 30, 2024 •

edited

Loading

a-r-r-o-w left a comment •

edited

Loading