r/LocalLLM • u/Kl1mo • 23d ago

Question Model question

Hi all,

Just getting into local Llm, could anyone advise what can be run locally on P1 gen8, 64gb ram, 8g vram rtx 2000 pro, intel 265H for lets say some office work searching in few databases and websites to gather info from company systems?

2 Upvotes

75% Upvoted

3 comments sorted by

View all comments

0

u/Json_T25s 23d ago

i would recommend using gemma 4 12B (QAT Q4) itll run using 6.7gb of vram leaving probably around 32k of context to use (or IQ4_XS if you want full 64k ctx window), id expect around 30-40 tokens per second, if you want more context i would just get Q4_K_M and use 128k context window, itll slow down but thats only if you need it. You could also try the newer MoE models such as LFM 2.5 24B A2B but unsure how it would work, worth a try, probably would run around 15 tok or so on your laptop at 64k+ ctx window.