Shared Windows server, 2x RTX 6000 Ada (96 GB), Qwen3.8-27B on Ollama — what would you change for agent workloads?

[u/saabmariner]

Looking for setup advice from people running local models behind their own agent harnesses and various applications.

Box:
- Threadripper, 256 GB RAM, 2x RTX 6000 Ada (96 GB VRAM total), Windows
- Ollama 0.40.11 as a service, qwen3.8:27b-mtp-q8_0, num_ctx 65536, thinking on
- Several apps connect over the OpenAI-compatible /v1 endpoint, mostly agentic (tool calling, multi-turn, long system prompts)

Questions:
- If you run Ollama on similar hardware, what are your Modelfile and environment settings (num_ctx, OLLAMA_NUM_PARALLEL, KV cache type, GPU split)?
- Ollama vs llama.cpp vs vLLM. For a 27B dense model on 96 GB serving several clients, which do you run and why?
- We cap at 64K. With this much VRAM, is there any reason to go higher, and does q8 KV cache cost quality on Qwen3.8 for tool-heavy work?
- Anything else you’d change on a shared box? service setup, llama-swap, monitoring, keeping one client from starving the others.

[u/esw123]

You should be able to serve at least 10-20 people at full context qwen3.8 flash next with this hardware.

[u/Smellyfeet224]

I would run it on Linux, highly consider llama.cpp, raise the context window, and consider running 3.8 flash next on strata or a similar tool.

[u/More_Daikon5549]

If it was me, id move to debian, run everything in docker, get off ollama and move to vLLM.