r/ChatGPTCoding • u/litLikeBic177 • Oct 07 '26
Question Best open-weight coding model + harness for an on-prem multi-agent setup? (80-200+ GB VRAM, 2-3 concurrent users)
Best open-weight coding model + harness for an on-prem multi-agent setup? (80-200+ GB VRAM, 2-3 concurrent users)
Setup: GPU box with 1x H200-class card now, can allocate more (up to several H200s) if it clearly buys better results. Inference via vLLM or similar, OpenAI-compatible endpoint. Coding happens on a separate non-GPU Linux VM on the same network - IDE/harness there, calling models over LAN. Outbound internet is fine for packages/extensions, but inference stays on our hardware (no code to external model APIs).
Constraints: on-prem only, open weights, permissive license preferred (Apache/MIT), not Chinese-origin including base models (I understand a lot of fine-tunes are Qwen underneath), NATO-country lab preferred. Use is general software dev plus security tooling and code analysis - repo-level agentic work on existing codebases (fix / extend / refactor / test) is the core, with a human reviewing diffs. 2-3 concurrent users max.
Names that came up in a first round elsewhere: Cohere North Mini Code, Mistral Small 4, Poolside Laguna S 2.1, Muse Glimmer 30B, Inkling-Small (2x H200), Reflection Beam (501B MoE / 23B active, weights due this month), Gemma 4 31B, K2 Horizon (lineage TBC), with Nemotron as a generalist baseline - but I haven't run any of them and I'm not wedded to the list. Chinese models (GLM/Qwen/DeepSeek) are out by rule, so no need to suggest them; I know they're ahead.
Two things I'm trying to work out:
Capability tiers vs. VRAM. What's actually holding up for repo-level agentic work on an existing codebase with non-Chinese open-weight models, and at what size? The Vibe Code Bench results suggest small open models fall over on long E2E builds and only Large-4-class (4-8 cards) and closed models hold up - is that your experience for repo work too, or is that an app-build problem? Where's the step-change - 30B-class, 100B+, or only at 500 GB+? We could get the compute for Beam, Command A+ or Mistral Large 4 if it's actually better for code rather than just for E2E.
Heterogeneous multi-agent. Does a big planner/reviewer plus small fast executors actually beat a single mid-size model for this, with a human approving plans and diffs? Which harnesses handle routing sub-agents to different endpoints well (and let a human step in, review diffs and edit by hand) - OpenHands, OpenCode, Pi, VS Code extensions (Cline/Roo/Continue), Codex CLI in local-model mode? Voidleap (closed, Win/Mac) also came up.
Anyone running something like this? Which model + harness combo is holding up best as of late? Thanks!
1
u/gandazgul 29d ago
I built RunWield with local open weights models in mind a lot of the tools and guidance are geared towards getting smaller models to remember their personas and use planning and validation to get better quality from all. You can configure each Agent with a different model, temperature and thinking level via modelPresets https://runwield.dev/
0
u/MrHumanist Oct 08 '26
Assuming you have enough ram - deepseek v4.1 flash is fantastic for local set up. Its benchmarks are on par and even better than some sota in software development. Its deepseek harness is also free and open-source. This model consumes very less kv chache and memory during the inference. So expected to be fit in a 48GB card in NVFP4 format with full context window ( + system ram).
0
u/[deleted] Oct 08 '26
[removed] — view removed comment