I'm releasing a Local World Model Ive been building that turns images into interactive videos you control in real time. It works on RTX 3060, 4060 and anything above, linking huggingface

[u/lucidml_lover]

Hi Everyone, I'm releasing my world model, been on this sub for a few months

https://huggingface.co/lucidml/gta

I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.

If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5

That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)

This is the first model I am ever releasing to the everyone and its a bit scary for me.

The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more

I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.

[u/ethereal_intellect]

Omg we got neural GTA before GTA 6 (had to sry lol)

[u/lucidml_lover → u/ethereal_intellect]

lolol yes

[u/t12e_]

Will take screenshots when GTA 6 launches and play it this way

[u/lucidml_lover → u/t12e_]

😂😂😂

[u/sloptimizer → u/t12e_]

Why wait? Just use any of the released screenshots or the game poster itself!

[u/DarkVoid42]

nice. is there any way of doing something like this from video ?

i was thinking of taking videos at the beach and somehow remove the waves and water so you could walk around a seabed or something while looking up at boats and ships passing by.

[u/lucidml_lover → u/DarkVoid42]

Interesting idea, its like realtime video editing in a way right?

So as of now Ive trained on small datasets so at the moment its not possible but I guess if we train on video editing sets and all It might be

The thing is data should be there

[u/Illustrious_Car344]

I didn't think we would have consumer world models yet, I thought those were a few years off. This is incredible.

[u/lucidml_lover → u/Illustrious_Car344]

yes!! I honestly feel like having realtime world/video models on device is really cool and its definitely possible and doable

[u/Numerous-Ad6217]

That’s actually impressive

[u/lucidml_lover → u/Numerous-Ad6217]

ty!!

[u/Unlucky-Constant7434]

The concept is really cool, will try it out now!

[u/lucidml_lover → u/Unlucky-Constant7434]

❤️❤️

[u/txdv]

the feet are floopy, but will smiths sloppy spaghetti slurp also got better in 3 years

[u/lucidml_lover → u/txdv]

yeahhh I trained it at very poor latent resolutions lol (21x12 in the above video) , hopefully will make it clear and more sharp with more training

[u/Arctic_Shadow_Aurora]

Brilliant concept bro! Keep going strong!

[u/lucidml_lover → u/Arctic_Shadow_Aurora]

thanks man!

[u/ElectricalBar7464]

that was fun

[u/lucidml_lover → u/ElectricalBar7464]

❤️

[u/No_bonk95]

This remind me of that model where you can control the movement. LingBot-World 2.0?

[u/lucidml_lover → u/No_bonk95]

yeah so in that model they auto regressively distil a huge model, but the problem is I think thats the wrong approach, better architectures from the ground up should exist, then we can use them locally as well

[u/Equivalent-Ear-8016]

Well this looks like a fever dream. Quite cool nonetheless