Detail-Daemon diffusers

[NicholasCao]

Describe the solution you'd like.
Detail-Daemon: https://github.com/Jonseed/ComfyUI-Detail-Daemon

How to implement Detail-Daemon in diffusers, as seen in https://github.com/Jonseed/ComfyUI-Detail-Daemon. Will there be a better official component in the future?

[hlky]

This should currently work by passing the custom sigmas that are created to pipelines with sigmas argument. We can consider this as an example for upcoming Modular Diffusers. We are also planning on separating "sampler"/Scheduler and "noise schedule"/sigmas, so this kind of functionality (modifying sigmas) could fit in there.

[NicholasCao]

sigmas = np.linspace(1.0, 1 / num_inference_steps, num_inference_steps)
sigmas[10:20] = sigmas[10:20] * 0.95
pipeline(sigmas=sigmas, xxx)

So, it can be implemented this way in Diffusers? @hlky

[hlky]

Yes, see #10080 for a similar use case. Let me know if you encounter any pipelines that don't support passing sigmas.

[NicholasCao]

@hlky got！thx！

[blepping]

Yes, see #10080 for a similar use case. Let me know if you encounter any pipelines that don't support passing sigmas.

Unfortunately this won't be enough (unless I'm misunderstanding, which is possible since I'm not a Diffusers person). Detail Daemon accomplishes its effect with a mismatch between the schedule used for sampling and the sigmas the model gets called with. For example, given x is current latent, sigma is the current sigma and sigma_next is the next sigma, a normal Euler step would look like:

denoised = model(x, sigma)
dt = sigma_next - sigma
d = (x - denoised) / sigma
x_next = x + d * dt

and a "Detail Daemon" step would look like:

dd_adjustment = -0.05
denoised = model(x, sigma * (1.0 + dd_adjustment))
dt = sigma_next - sigma
d = (x - denoised) / sigma
x_next = x + d * dt

If the user is able to replace the model with a wrapper function that performs the adjustment then that would probably be the easiest way to do it.

[hlky]

Thanks @blepping! We'll be able to do that with modular Diffusers when its ready #9672

cc @yiyixuxu for interest in another use case of modular

[blepping]

Thanks @blepping! We'll be able to do that with modular Diffusers when its ready #9672

Definitely looks like a big improvement! (And the guider stuff looks pretty similar to the ComfyUI approach.) I have to confess, I've actually avoided touching Diffusers stuff as much as possible since it seems like every project has to reinvent the world. If that modular stuff lands I may have to take another look!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[hlky]

This has been added to Modular Diffusers #9672 (comment)

[blepping]

This has been added to Modular Diffusers #9672 (comment)

Seems good! I'd suggest crediting the original implementation at https://github.com/muerrilla/sd-webui-detail-daemon/ though - that's where the idea came from.

Also you might want to make it possible to set start/end times. Generally you wouldn't want that offset applied for the entirety of sampling.