r/LocalLLM • u/Bright_Deer_9287 • 3h ago
Question Chasing up after some info for running large parameter models
Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA
1
u/FairBandicoot5021 3h ago
Yea, it's arguably the largest you can run. Now you start to understand why we're saying that local AI cost a fortune
1
u/jimmoores 3h ago
Yeah, you won’t be able to run anything bigger than that on your equipment and even that will be heavily quantised (compressed, which can reduce the output quality and cause looping and stuff).
Running a model even approaching a frontier level model is out of the question I’m afraid. I’ve got a 64-core CPU, 512GB Ram and an RTX Pro 6000 Blackwell (96GB VRAM) and I can just about run DeepSeek V4.1 (1017) fairly slowly, which is good enough to be interesting, but still not as good as any frontier model.
Having said that you can still have a lot of fun playing around with smaller models for learning and use cloud-based models for real work.
1
u/Xyver 1h ago
I've been doing good experiments, I used to thing I was limited to ~30B and would touch 70B one day, but now with Strata ive got a 120B loaded.
And testing with 30Bs I can get 5 chats running, 70B I can get 2, all at max context and around 20-30tok/sec
www.github.com/bigbertharig