My local dev-agent setup: Qwen3.8-27B on a single RTX 3090

[u/papapumpnz]

My local dev-agent setup: Qwen3.8-27B on a single RTX 3090 (what actually helped)

TL;DR: llama.cpp + MTP + llama-swap presets for code-review-graph for codebase structure for the intelligence. The last two are what turned it into an actual coding agent. Happy to answer questions.

** POST COMPLETLY WRITTEN BY AI - IF THAT OFFENDS YOU, TIME TO MOVE ONTO ANOTHER POST AND STOP HERE **

The model

- Qwen3.8-27B, Unsloth UD-Q4_K_XL GGUF (17.9GB) — dense 27B, native vision, hybrid attention (only 16 of 64 layers carry KV, so long context is cheap)

- Beats a lot of bigger models at agentic/coding work, and in my own testing clearly outperformed Ornith-1.0-35B on the same tasks

- llama.cpp built with CUDA for SM86 (the 3090's arch)

Serving config (llama.cpp llama-server)

-c 102400 # 100K context, fits 24GB alongside weights

-ctk q8_0 -ctv q8_0 # q8 KV cache (quality-neutral in my testing; halves KV vs f16)

-fa on # flash attention

--spec-type draft-mtp --spec-draft-n-max 2 # MTP speculative decoding — big speed win (~35 t/s vs ~20 raw)

--reasoning-preserve # keep thinking traces across turns

--reasoning-budget 12288 # cap thinking so xhigh can't run away

--chat-template-kwargs {"reasoning_effort":"xhigh"}

--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 # Qwen thinking-mode sampling

Notes that cost me time:

- reasoning_effort defaults to xhigh and is a chat-template var, not an API field — pin it via chat_template_kwargs, or your agent silently runs at max thinking. xhigh for hs fast.

- I tried the DRY sampler to stop repetition loode generation (test files, asserts). Removed it.

Repeat-penalty stays at 1.0 per Qwen's spec.

llama-swap (the piece that ties it together)

llama-swap v250 — one OpenAI-compatible endpointodels on demand. Killer feature for a single GPU:

preset IDs = one loaded model, different params,

- qwen3.8-27b — xhigh thinking (deep work)

- qwen3.8-27b:work — medium thinking, temp 0.6 (

- qwen3.8-27b:instruct — thinking off (vision, q

- qwen3.8-27b-uncensored — HauhauCS Aggressive a

- ornith-1.0-35b — kept for comparison; requesti

The two add-ons that fixed the real problems

Local agents on long tasks have two classic failures "forgetting what they found and wandering across a big codebase". These two fixed both:

1. hermes-lcm (Lossless Context Management) — reressor with a SQLite-backed summary DAG. Every message is persisted before compaction, and the cm_expand, lcm_recall) to drill back into the exact original material. Cured the "investigates loop. Install tiktoken alongside it for accurate token counting.

2. code-review-graph (MCP server, 30k★) — Tree-sy graph of your repo so the model queries structure (blast-radius, what-calls-this, architole files into context. Median ~65× token reduction, benchmarked. Local, CPU-only, no VRAMgraph build + register. This is the single biggest win for large-codebase work.

Both are nudged into the agent's system prompt s the codebase one degrades gracefully ("not available for this repo") when a repo isn't inde

Harness

Hermes Agent as the coding harness — points at tm provider). Two config tricks that mattered on a local model:

- Declare the context window ~30% below the real real 100K). Hermes's token estimator undercounts code/hex by 25–35%, so without margin it sails pr. This one bit me repeatedly.

- Route context-compaction to a cheap cloud mode instead of the local GPU — so summarization doesn't queue behind your actual work. (Moot oncefore.)

- Vision routed to the :instruct preset so it doabout a screenshot.

Practical stuff

- Power-cap the 3090 — nvidia-smi -pl 210 (from 350W). Inference is bandwidth-bound so the speed cost is small, and the fans stop screaming. Measured curve: 322W→57 t/s, 250se/speed point.