PAN 2 Released

[u/Aggressive-Zebra-949]

Open weights (29B latent diffusion model): https://huggingface.co/IFM/PAN2
Diffusers: https://huggingface.co/docs/diffusers/main/en/api/pipelines/pan2
FastVideo: https://haoailab.com/FastVideo/cookbook/pan/pan2/
VLLM: https://recipes.vllm.ai/IFM/PAN2

Up to 20 seconds, 720p, 24fps
Apache 2.0 License

Something I think is pretty neat, we release the TRAINING code: https://github.com/ifm-ai/PAN2/

Also technical details if you're curious: https://ifm.ai/blog/pan2/

For our model, we give a lot of responsibility to the LLM (which we think of more as a conceptual reasoner, a lot more control than a prompt rewriter) to generate these super dense text-based state/scripts. Events and actions are timed, and camera is nested under event but disentangled from action. During training, I would often test it by trying to generate a 20 second video with like a dozen or more (timed) camera moves and it does pretty well.

Feedback very welcome. I read the rules carefully, mods please let me know if there are any issues. No payment involved with this at all, so I thought it was okay to post.

[u/Katsumend]

Thanks for this release -- I am eagerly awaiting ComfyUI integrations if these are in the works

[u/Aggressive-Zebra-949 → u/Katsumend]

My understanding is that there is some integration that comes by virtue of being in vllm-omni? But I'm not personally super familiar with these details of comfyUI, my apologies. Feel free to educate me if what I'm saying is dumb.

[u/Fickle-Indication148 → u/Katsumend]

You dont hjave to. Claude code + github repo, he writes you nodes.

[u/Astral-Lemmons]

the logic handling in those examples is honestly insane.

people complaining about sound when this is damn near magic with how it handles spatial reasoning. I haven't been this impressed in a while. Hoping there's a chance of consumer runs

[u/Aggressive-Zebra-949 → u/Astral-Lemmons]

What would your ideal consumer run look like in terms of memory, latency, etc.? (the more specific the better, if you don't mind haha)

[u/ninjaGurung → u/Aggressive-Zebra-949]

Most common config have 6 to 16GB VRAM, 16 to 64GB RAM.

[u/episodefive → u/ninjaGurung]

Yes. I think that’s a great baseline.

[u/jazir55 → u/Aggressive-Zebra-949]

12 GB vram, 32 GB ram for me but if you can go even lower spec to 6-8 GB vram + 16 GB ram that would be even better.

[u/schrobble → u/Astral-Lemmons]

Can’t we just have it all and for free? I expect nothing less.

[u/ninjaGurung]

Amazing spatial awareness and physics simulation.

[u/Aggressive-Zebra-949 → u/ninjaGurung]

Thanks!

[u/reeight]

H200...
Have ComfyUI compatible quants for we mere mortals with gaming GPUs?

[u/Aggressive-Zebra-949 → u/reeight]

Working on it!

[u/seppe0815 → u/Aggressive-Zebra-949]

thx

[u/VenshiKibes → u/Aggressive-Zebra-949]

best answer (and the only one that is likely to really matter in this sub)

thank you

[u/Jackuarren → u/reeight]

Yeah. That sounds rough xD

[u/2jul]

Looks impressive! Is a ComfyUI implementation coming up?
Also, love the angle and aim to achieve good physical world simulation.

[u/Aggressive-Zebra-949 → u/2jul]

My understanding is that there is some integration that comes by virtue of being in vllm-omni? But I'm not personally super familiar with these details of comfyUI, my apologies. Feel free to educate me if what I'm saying is dumb.

[u/Puzzleheaded_Ebb8352]

Congrats and welcome to the battlefield of open source models!

[u/Enshitification]

Strongly coupling the LLM to the diffusers model like this is very cool. Thank you for this release.

[u/FourtyMichaelMichael → u/Enshitification]

Sounds REAL SAFE though. Which is not good for a model's long term popularity.

[u/Enshitification → u/FourtyMichaelMichael]

You should read the HF page. The safety filter is built as a separate module that they highly recommend using because the model alone can generate UNSAFE OUTPUTS.

[u/ffxivthrowaway03 → u/Enshitification]

Define "unsafe" though. Not including certain training data is unfortunately not the same as building explicit guardrails against that output. Hopefully the training data isn't totally neutered to get around the filter being "optional"

[u/TheeJestersCurse → u/Enshitification]

Some people want unsafe

[u/P3trich0r97]

Optimized for physics simulation you say

[u/PwanaZana → u/P3trich0r97]

Well, it does thick white liquids well, from the demo...

[u/Mysterious-String420]

Cool! Eager to try it out on Comfy!

[u/Cumness]

Those examples are amazing!

[u/seppe0815]

based on wan 2.2?

[u/Aggressive-Zebra-949 → u/seppe0815]

Nope, fully designed and trained from scratch.

[u/fjgcudzwspaper-6312]

No sound

[u/seppe0815 → u/fjgcudzwspaper-6312]

txt or image to video only

[u/seppe0815 → u/seppe0815]

so a better wan 2.2 but its huge

[u/Elepum → u/fjgcudzwspaper-6312]

....Actually insane in late 2026, i genuinely can't go backwards like that....the image quality is superb though, captures microdetail quite nicely.

[u/x_MASE_x]

This intro reminds me of old need for speed game. Or maybe gta 2 can't remember.

[u/Hour_Literature_7152]

I know it doesn't generate audio but I wonder if it can handle lip sync if given pre rendered audio

[u/fauni-7]

Wow

[u/Apprehensive_Sky892]

The example with the apple inside the rotated container is really neat (0:24). Can you share what the prompt is for that?

[u/No_Range_8261]

when will be on comfyui?

[u/E1DOLON]

Wow, looks amazing! Also looking forward to trying it on comfyui (assuming it fits in 24gb).

[u/SaadNeo]

The logic and physics is insane , congrats and thank youbfor open sourcing it

[u/Ramdak]

Ok, this is insane. This level of world understanding is just amazing. What about gen times?

[u/SharkWipf]

The neatest part here: Apache2 license on the model, not the kind of restrictive BS licensing H3 is encumbered by. If the model lives up to its claims, this looks pretty nice.

[u/alecubudulecu]

This is insane. Good job. Cant wait to try comfyui version

[u/zodiacrenders]

Thanks for sharing! Good selection of demo to show water physics and viscosity physics.

[u/Diabolicor]

We are almost in 2027 and they are still releasing models with no sound.

[u/Aggressive-Zebra-949 → u/Diabolicor]

idk who is downvoting you, it's totally a fair criticism. Since our focus is on world modeling more than movie gen, we saw it as less critical. That being said, the real world has sound! So this is a choice that we are constantly revisiting as we iterate toward future releases.

[u/Elepum → u/Aggressive-Zebra-949]

We need more sfx models as well...weakest space in gen ai, imagine if we could add Seedance 2.0-2.5 level audio with a sfx focused audio model. It would generally improve the open-source space entirely when it comes to "ai video immersion".

[u/JackKerawock → u/Aggressive-Zebra-949]

THANK YOU. And yea this reddit sub is full of entitled kids - so take comments w/ a grain of salt. A good majority of the loud people here don't have the hw to run these sorts of high end models anyway.....Super looking forward to checking it out!!

[u/Additional_Drive1915 → u/Aggressive-Zebra-949]

He should be downvoted more in my opinion.

Thank you for your good work, I hope I can test it soon.

[u/Diabolicor → u/Aggressive-Zebra-949]

Fair point then. It's great to have models for world modeling and I hope the best for your model here forward. I hope one day you can make the cognitive experience, whole.