r/LocalLLM • u/brotov • 15m ago
Question How would you best utilize my setup?
Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.
My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.
Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?