Image and Video Generation with Flows

[EngEmmanuel]

I am interested in doing image and video generation using mean flow and other variants. Ideally, I would be able to use both UNet and Transformer architectures but UNet is my priority. This would involve using different bits of code you have graciously contributed e.g. this repository, 'imagen-pytorch' and 'x-transformers'

How do you think it's best to go about this? For example, I could try and tweak the Unet3D class from 'imagen-pytorch' to work with different flow classes you have written in this repo. Or instead, try and take the flows code over to the imagen-pytorch set up instead. I would also like to be able to use classifier free guidance.

Where is best to start? I can try to contribute if you can provide some pointers, please

[lucidrains]

@EngEmmanuel hey, thanks for your interest

so i'd recommend just using x-transformers as it has enough functionality to support DiT, which is SOTA (attention is the best we have for now). the wrappers in this repository should be enough for you to do something like what i did here

[lucidrains]

@EngEmmanuel feel free to reach out over Signal and get me caught up on the ongoings in OSS flow video. i'm sort of out of the loop there

[EngEmmanuel]

For my use case, it's easier to do generation at full scale (32 x 1 x 224 x 224) and I just can't afford the attention memory requirements with my humble GPU quota. I want to use binary masks for the images/frames at train and test time and trying to do that in a latent space seems difficult

1. Do you already have something like a ~ (2+1)D ViT implementation somewhere? I checked x-transformers and vit_pytorch (perhaps not very well) but couldn't seem to find one

2. '(attention is the best we have for now)' I assume this means that implementing a (2+1)D Unet with CFG and other forms of conditioning is probably low on your priority list? I might have a go at it (no promises, I'm still learning this) if it will be a while

I am not very familiar with OSS Flow Video either, sorry :/