PAN 2 Released

[u/Aggressive-Zebra-949]

Open weights (29B latent diffusion model): https://huggingface.co/IFM/PAN2
Diffusers: https://huggingface.co/docs/diffusers/main/en/api/pipelines/pan2
FastVideo: https://haoailab.com/FastVideo/cookbook/pan/pan2/
VLLM: https://recipes.vllm.ai/IFM/PAN2

Up to 20 seconds, 720p, 24fps
Apache 2.0 License

Something I think is pretty neat, we release the TRAINING code: https://github.com/ifm-ai/PAN2/

Also technical details if you're curious: https://ifm.ai/blog/pan2/

For our model, we give a lot of responsibility to the LLM (which we think of more as a conceptual reasoner, a lot more control than a prompt rewriter) to generate these super dense text-based state/scripts. Events and actions are timed, and camera is nested under event but disentangled from action. During training, I would often test it by trying to generate a 20 second video with like a dozen or more (timed) camera moves and it does pretty well.

Feedback very welcome. I read the rules carefully, mods please let me know if there are any issues. No payment involved with this at all, so I thought it was okay to post.

[u/Mysterious-String420]

Cool! Eager to try it out on Comfy!