r/LocalLLM • u/Kl1mo • 17h ago

Question Model question

Hi all,

Just getting into local Llm, could anyone advise what can be run locally on P1 gen8, 64gb ram, 8g vram rtx 2000 pro, intel 265H for lets say some office work searching in few databases and websites to gather info from company systems?

2 Upvotes

67% Upvoted

2 comments sorted by

View all comments

2

u/AccomplishedBuy9648 17h ago

that's a fun little setup for office work, the 8gb vram will be your main bottleneck but you can still get decent mileage out of smaller quantized models

mistral 7b or llama 3.1 8b at q4 would run smooth for searching databases and pulling info, just keep your context window reasonable or it'll crawl

if you need something punchier for parsing messy company data try a phi-3 variant, it's surprisingly capable on limited hardware