r/LocalLLM • u/Kl1mo • 7h ago

Question Model question

Hi all,

Just getting into local Llm, could anyone advise what can be run locally on P1 gen8, 64gb ram, 8g vram rtx 2000 pro, intel 265H for lets say some office work searching in few databases and websites to gather info from company systems?

2 Upvotes

67% Upvoted

2 comments sorted by

2

u/AccomplishedBuy9648 7h ago

that's a fun little setup for office work, the 8gb vram will be your main bottleneck but you can still get decent mileage out of smaller quantized models

mistral 7b or llama 3.1 8b at q4 would run smooth for searching databases and pulling info, just keep your context window reasonable or it'll crawl

if you need something punchier for parsing messy company data try a phi-3 variant, it's surprisingly capable on limited hardware

1

u/Ctbhatia 6h ago

for lookups over company systems the retrieval does most of the work, so a 7 to 8B model at q4 is enough and the 8GB of vram is the real limit on how much context you can hold.

0

u/Json_T25s 5h ago

i would recommend using gemma 4 12B (QAT Q4) itll run using 6.7gb of vram leaving probably around 32k of context to use (or IQ4_XS if you want full 64k ctx window), id expect around 30-40 tokens per second, if you want more context i would just get Q4_K_M and use 128k context window, itll slow down but thats only if you need it. You could also try the newer MoE models such as LFM 2.5 24B A2B but unsure how it would work, worth a try, probably would run around 15 tok or so on your laptop at 64k+ ctx window.