Potential for simple video transfer [Demo showcase]

[Petas3]

Hi,

spent last 2-3 weeks figuring out how this all works and putting some real simple stuff together into a pipeline so videos with reasonable stability can be produced - I think we reached that point. ControlNet was the final piece (perfect timing), all in all its a combination of ~8-10 free tools/AIs.

Following video is produced from ~80 unrelated source images used to train model - hope to reduce this to 1 source image, with user guided expansion to <10 ai generated supplemental sources. Made on a trained model ~aprx 8 hours - hope to reduce under 30m when I get the hang of it. Input for the video is unrelated 3D model animation that looks bit of familiar, but not really close as I made the model in like 5 minutes. Theres tons of supplemental data pregenerated and the generation is basically automated and only requires the guidance video and 1 seed frame - the rest of the conversion is completely unguided.

Hope to improve this in the following weeks including automation of the fine tuning, theres a ton of adjustments done to the init params so it 'works'. Note its generated in full 30 fps to showcase 'stability' and then interpolated to 60. If it was done in 5fps with human oversight the result would ofc be better, but that is not the point.

ex11-2x-RIFE-RIFE4.0-60fps.mp4

Thanks for the amazing tool. Feel free to use this for promo.

[ljleb]

Ah I can't see your video on my side...

For extremely quick textual inversion, keep an eye on this: https://arxiv.org/pdf/2302.13848.pdf (paper reports ~0.05 sec using a single image)
possible that it only works with objects or characters? I hope it can also learn styles or general image information.
might be able to get extremely consistent results if introduced into a video pipeline

[Petas3]

Looks promising, like a super embedding ^3, only faster and supposedly better.

[AugmentedRealityCat]

Ah I can't see your video on my side...

I have the same problem, but I've identified the cause and found a solution. Basically, this video has been encoded in HEVC, and that codec is not included for free with Windows 10 - you have to pay something like 2$ to get it. And not that's not the solution.

The solution is free: it's VLC ! Just right-click on the video above and then save as before loading it in VLC.

[Petas3]

Yes VLC should play it, I will change the codec for the future outputs. Thanks for pointing this out, didn't occur to me the Nvidia encoder default is no good.

[Petas3]

Stay tuned, new full feature showcase of very very simple control net video (simple batch, no feedback) is rendering.

Full video: https://www.youtube.com/watch?v=-B0h254Gn1A

Video preview added, omg, how is this possible:
https://user-images.githubusercontent.com/58807841/223585966-dcb6d8f3-b32d-41cd-9c97-cc521c59712b.mp4

Teasers added (showcase of light styling with controlnet):

[Petas3]

Improved version of the pipe in 2x the FPS and 2x the res using additonal LORA to increase stability. Includes some pretty good post processing. If you preprocess externally you get much better results as the annotators are kind of bad compared to the current models available in repos.
https://youtu.be/f27FV39K0-M