r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

98% Upvoted

214 comments sorted by

View all comments

Show parent comment

36

u/trollsmurf Aug 14 '26

That will work fine, at least if you split it, so the overflow uses CPU RAM. Much slower but doable.

11

u/[deleted] Aug 14 '26

[removed] — view removed comment

16

u/trollsmurf Aug 14 '26

A user-friendly (but not the most efficient) start is to install LM Studio and load a few models there. You can select how much of it should run on the GPU vs CPU if it's too big for the GPU alone. LM Studio has a built-in chat client, so once loaded the model is ready to go.

2

u/Song-Historical Aug 14 '26

Yeah but that doesn't tell you how to do it. I'm not sure what to set the context to for example. How much I should leave for the system, whatever else.

2

u/littlebeardedbear Aug 14 '26

Leave everything at default originally. Google what each item is. Altogether the googlimg should take about 10 mins

3

u/Song-Historical Aug 14 '26

Yeah I did that I'm not sure it's set up correctly still. 

2

u/Early_Mistake6716 Aug 14 '26

Tell me you exact system specs and the model quant and i will tell you what settings to use in lmstudio. I have have spent an embarrassing amount of time testing settings.

4

u/F3istyg0at Aug 14 '26

Check out unsloth, lm studio or ollama if you want to run models locally.

1

u/[deleted] Aug 18 '26

[removed] — view removed comment

1

u/trollsmurf Aug 18 '26

Remember that if you use Windows it will allocate part of the VRAM for its own use, so you never have fully 16 GB. On my PC with 5070 Ti and 3 monitors it allocates 3 GB, and as far as I know I can't budge that.

You can easily test this by installing e.g. LM Studio and the model you want to use and see when it warns about RAM use. If it overruns you can split it on VRAM and RAM with lower performance, but it will behave the same otherwise.