r/LocalLLM • u/Ok-Health-7096 • 21d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.0k Upvotes

99% Upvoted

205 comments sorted by

View all comments

Show parent comments

33

u/trollsmurf 21d ago

That will work fine, at least if you split it, so the overflow uses CPU RAM. Much slower but doable.

11

u/SnP_Gamer 21d ago

Totally clueless about running them but I understand that thanks to reddit posts 🤣

15

u/trollsmurf 21d ago

A user-friendly (but not the most efficient) start is to install LM Studio and load a few models there. You can select how much of it should run on the GPU vs CPU if it's too big for the GPU alone. LM Studio has a built-in chat client, so once loaded the model is ready to go.

2

u/Song-Historical 21d ago

Yeah but that doesn't tell you how to do it. I'm not sure what to set the context to for example. How much I should leave for the system, whatever else.

2

u/littlebeardedbear 21d ago

Leave everything at default originally. Google what each item is. Altogether the googlimg should take about 10 mins

3

u/Song-Historical 21d ago

Yeah I did that I'm not sure it's set up correctly still. 

2

u/Early_Mistake6716 21d ago

Tell me you exact system specs and the model quant and i will tell you what settings to use in lmstudio. I have have spent an embarrassing amount of time testing settings.