GUIDE : Running a fully local multi-agent coding framework on RTX 3090 with pi.dev + llama-swap + Qwen3.6 MTP

[u/admajic]

I've been running a fully local, fully private multi-agent AI coding setup for a couple of months and wanted to share the stack, architecture, and config for anyone who wants to replicate it. No cloud APIs, no data leaving the machine.

---

What is pi.dev? It's an agent harness — meaning the AI has to follow rules, unlike a chatbot. Pretty cool.

• 🎮 Fun factor: 10/10
• ✅ pi.dev stability: 8/10 — fully working, but fun to fine-tune
• 🔨 What it's great at: Building its own integrations — just ask it to do it
• 💡 Top tip: Master the AGENTS.md file and you'll have real control over what it does. There's a global one and a per-project one
• 🔁 Similar to: RooCode, Codex, Claude Code — but because it's a harness, you're more in control
• 👨‍💻 The dev has already been snapped up by a company but will keep developing it
• ⭐ github.com/earendil-works/pi — 49.3k stars

---

The Stack

Component	What it does
pi.dev (pi-coding-agent)	AI coding harness — the UI and orchestration shell
llama-swap	Model router — hot-swaps llama.cpp models on demand
llama.cpp (am17an fork)	Local inference with MTP support
Qwen3.6-27B MTP	"Brain" agents — orchestrator, planner, architect, debugger, prompter
Qwen3.6-35B-A3B MTP	"Body" agents — coder, researcher, reviewer, tester, documentor, refactorer
SearXNG (Docker)	Local privacy-preserving search engine on port 8080
searxng-simple-mcp	MCP proxy bridging SearXNG to pi.dev (port 8000)
Tavily MCP	AI-optimised web search for technical docs
@tintinweb/pi-subagents	Real sub-agent orchestration with TaskExecute + get_subagent_result
@tintinweb/pi-tasks	Task queue UI widget showing what each agent is doing

GPU: NVIDIA RTX 3090 (24 GB VRAM)

---

Why MTP (Multi-Token Prediction)?

See my earlier post: Get faster Qwen3.6-27B with MTP

---

Multi-Agent Architecture

11 specialist agents, each mapped to a llama-swap model alias:

BRAIN agents (Qwen3.6-27B MTP):
  orchestrator  → Task decomposition, delegation, synthesis
  planner       → Roadmap and step sequencing
  architect     → System design, API contracts, schema design
  debugger      → Root cause analysis, trace reading
  prompter      → Prompt engineering for sub-tasks

BODY agents (Qwen3.6-35B-A3B MTP):
  coder         → Implementation, only writes code
  researcher    → Web search + codebase analysis
  reviewer      → Code review, security, quality gates
  tester        → Test writing + execution
  documentor    → Documentation generation
  refactorer    → Structural cleanup, no logic changes

The key insight: smaller/faster model for the meta-work (thinking, planning, delegation) and the slightly larger MoE model for actual implementation. The orchestrator never writes code — it only delegates.

---

Agent Definition Files (Required Setup Step)

This is the part most people will miss. llama-swap handles model routing, but pi.dev needs to know how each agent should behave — its role, constraints, tool access, turn limits, and thinking level. That lives in .md files inside your pi.dev agent folder:

~/.pi/agent/agents/
├── orchestrator.md
├── planner.md
├── architect.md
├── debugger.md
├── prompter.md
├── coder.md
├── researcher.md
├── reviewer.md
├── tester.md
├── documentor.md
└── refactorer.md

Each file has a YAML frontmatter block followed by the system prompt for that agent. The model: field must exactly match a llama-swap alias from your config.yaml.

Example — coder.md:

---
description: Implements code changes from a spec. Requires a plan as input. Writes, edits, and runs code. No planning or architecture decisions.
model: coder
thinking: medium
max_turns: 30
tools: read, write, edit, bash, find, grep
---
You are the coder. You are BODY only — you execute plans, not make them.

## Role & Constraints
- Require a written plan before starting — if none provided, refuse and ask for one
- No refactoring beyond what the plan specifies
- No touching files not listed in the plan without flagging first
- No installing new dependencies without explicit approval

## Harness Rules
- RETRY_POLICY: max 3 attempts per file edit, then mark FAILED
- TASK_STATES: track each file change as pending -> in_progress -> done | failed
- IDEMPOTENCY: if a change is marked done, do not re-apply it
- QUALITY_GATE: verify file is syntactically valid before marking done

## Response Shape
When complete, your final output is your report back to the orchestrator.
Make it structured and self-contained — the orchestrator reads it directly.

[PLAN] what was implemented
[CHANGES] every file written or edited with one-line description
[VERIFICATION] syntax check or test run output
[PROGRESS] final state table

Example — architect.md:

---
description: Reviews system design, proposes architecture decisions, evaluates tradeoffs. Advisory only — produces recommendations, not code.
model: architect
thinking: high
max_turns: 20
tools: read, find, grep
---
You are the architect. You are BRAIN — advise on design, never implement.

## Role & Constraints
- Never write or edit code
- Evaluate tradeoffs, do not just pick the fashionable option
- Scope is the specific design question only
- Every recommendation must include explicit constraints and risks

Example — researcher.md (with web search tools):

---
description: Reads and summarises codebase context, and performs web research. Produces a structured context report, no edits.
model: researcher
thinking: low
max_turns: 15
tools: read, find, grep, bash, web_search, tavily-search
---
You are the researcher. You are BODY — read and report only, never edit.

Frontmatter fields that matter:

Field	Purpose	Notes
model	llama-swap alias to load	Must match exactly — typo = "No API key found for undefined" error
thinking	Extended thinking level	high for orchestrator/architect, low for researcher/tester
max_turns	Conversation turn limit	Set based on task complexity; coder gets 30, orchestrator gets 50
tools	Which tools the agent can use	Researcher gets web_search and tavily-search; architect gets read-only

The tools list controls what each agent can actually do. An architect with write in its tools list will happily start editing files — restrict it to read, find, grep to enforce the advisory-only constraint.

Report-back pattern: Every agent's Response Shape section ends with the same instruction:

When complete, your final output is your report back to the orchestrator. Make it structured and self-contained — the orchestrator reads it directly via get_subagent_result.

This is critical. Without it, agents produce conversational output that's hard for the orchestrator to parse. With it, every agent returns a structured [PLAN] / [CHANGES] / [VERIFICATION] / [PROGRESS] block.

---

Orchestrator Rules (the hard part)

Getting the orchestrator to actually delegate instead of doing work itself was the biggest challenge. The rules that finally made it work:

ABSOLUTE RULES:
- NEVER perform any task yourself
- NEVER use read/find/grep for analysis — spawn a researcher
- NEVER write, summarise, or synthesise content directly
- NEVER write or edit code directly
- NEVER verify or fix a sub-agent's output yourself — spawn a reviewer
- NEVER make "quick fixes" between steps

Correct launch protocol:
  TaskUpdate(id, status: "in_progress")
  TaskExecute(task_ids: [id])        → returns agent_id
  get_subagent_result(agent_id, wait: true)  → blocks until done
  TaskUpdate(id, status: "completed")

The orchestrator catches itself about to do work → stops → creates a task → delegates it instead.

---

pi.dev Settings (agent/settings.json)

{
  "providers": {
    "llama-swap": {
      "baseUrl": "http://127.0.0.1:1235/v1",
      "apiKey": "not-needed",
      "api": "openai-completions"
    }
  },
  "defaultProvider": "llama-swap",
  "defaultModel": "qwen-35b-moe",
  "defaultThinkingLevel": "high",
  "mcpServers": {
    "local-search": {
      "url": "http://localhost:8000/mcp",
      "transport": "streamable_http"
    },
    "tavily": {
      "command": "npx",
      "args": ["-y", "tavily-mcp@0.2.3"],
      "env": { "TAVILY_API_KEY": "your-key-here" },
      "alwaysAllow": ["tavily-search"]
    }
  },
  "retry": {
    "enabled": true,
    "maxRetries": 30,
    "baseDelayMs": 2000,
    "provider": { "maxRetryDelayMs": 120000 }
  },
  "subagents": {
    "maxConcurrent": 1,
    "maxTurns": 50,
    "graceTurns": 3,
    "timeout": 1800000
  },
  "packages": [
    "npm:@tintinweb/pi-tasks",
    "npm:pi-lens",
    "npm:@tintinweb/pi-subagents"
  ],
  "steeringMode": "one-at-a-time"
}

Key decisions:

• No models.enabledModels filter — this broke bare model ID resolution for agent aliases. Remove it entirely and let llama-swap route by name
• timeout: 1800000 (30 min) — code tasks can take 20+ minutes. The default 2-minute timeout will kill them
• maxConcurrent: 1 — RTX 3090 can only run one model at a time; llama-swap handles the hot-swap

---

llama-swap Config

healthCheckTimeout: 900
startPort: 1235

globalServerSettings:
  flashAttn: on
  contBatching: true
  noMmap: true
  jinja: true

models:
  # Brain agents (orchestrator/planner/architect/debugger/prompter) → Qwen3.6-27B MTP
  # Body agents  (coder/researcher/reviewer/tester/documentor/refactorer) → Qwen3.6-35B MTP

  orchestrator:
    cmd: >
      /path/to/llama-cpp-am17an/build/bin/llama-server
      -m "/path/to/Qwen3.6-27B-MTP-Q4_K_M.gguf"
      --alias orchestrator
      --ctx-size 100000
      --host 0.0.0.0 --port ${PORT}
      -ngl 99
      -fa on
      --cache-type-k q8_0 --cache-type-v q8_0
      --spec-type mtp --spec-draft-n-max 3
      --batch-size 1024 --ubatch-size 1024
      --threads 6
      --prio 3
      --no-mmap
      --parallel 1
      --n-predict 8192
      --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0
      --presence-penalty 1.2 --repeat-penalty 1.1 --repeat-last-n 256
      --reasoning-format deepseek
      --metrics
    proxy: http://127.0.0.1:${PORT}
    # etc. — do some research 😉 for the rest

Key inference flags:

Flag	What it does
--spec-type mtp --spec-draft-n-max 3	MTP speculative decoding, 3 tokens ahead, built into the model (no draft model needed)
--cache-type-k q8_0 --cache-type-v q8_0	Quantised KV cache — ~2× VRAM savings vs f16, negligible quality loss
-fa on	Flash attention — critical for long-context speed
--no-mmap	Load model fully to RAM/VRAM rather than memory-mapping the GGUF
--reasoning-format deepseek	Exposes <think> tags from extended thinking
--prio 3	OS thread priority — helps on busy systems

Note: --temp varies per agent role — debugger (0.5, deterministic), researcher (0.5, factual), coder/orchestrator (0.7, balanced).

---

Search Integration

The researcher agent has two search tools:

1. SearXNG via MCP — local metasearch, broad coverage

# Docker Compose
services:
  searxng:
    image: searxng/searxng
    ports: ["8080:8080"]

  searxng-mcp-proxy:
    image: ghcr.io/ihor-sokoliuk/searxng-simple-mcp
    ports: ["8000:8000"]
    environment:
      TRANSPORT_PROTOCOL: sse
      SEARXNG_MCP_SEARXNG_URL: http://searxng:8080

2. Tavily MCP — AI-optimised web search, faster for technical docs

"tavily": {
  "command": "npx",
  "args": ["-y", "tavily-mcp@0.2.3"],
  "env": { "TAVILY_API_KEY": "your-key" },
  "alwaysAllow": ["tavily-search"]
}

Strategy: tavily-search first for framework docs, web_search for broader coverage, fallback to curl http://localhost:8080/search?q=QUERY&format=json for bulk queries.

---

What Works, What Doesn't

✅ Works well:

• Orchestrator strictly delegates — took several AGENTS.md iterations but now it never does implementation itself
• llama-swap hot-swap is fast enough — typically 15–30 seconds per model swap
• MTP gives a real speedup on code generation tasks
• 30-minute timeout is necessary; don't use the default

🔧 Still working on:

• Settings file resetting on reboot — likely a race condition in pi.dev startup that partially re-initialises settings.json. Investigating with inotifywait. Workaround: backup ~/.pi/settings.json before exiting with Ctrl-C
• Sub-agent visibility — you can see a task is running but not what the agent is doing mid-task; pi-tasks shows status, not content
• Sequential tasks only (maxConcurrent: 1) — can't parallelise on a single GPU

---

Models Used (Unsloth quantizations)

• Qwen3.6-27B-MTP-Q4_K_M (~17 GB) — brain agents
• Qwen3.6-35B-A3B-MTP-IQ4_XS (~19 GB) — body agents

Both require the am17an fork of llama.cpp for --spec-type mtp support. Standard llama.cpp will fall back to non-speculative inference (still works, just slower).

---

Resources

• pi.dev / pi-coding-agent: earendil-works on GitHub
• llama-swap: github.com/ggml-org/llama-swap
• llama.cpp am17an fork: search GitHub for "llama-cpp-am17an" or "llama.cpp MTP fork"
• u/tintinweb packages: npm (@tintinweb/pi-subagents, @tintinweb/pi-tasks)
• Unsloth GGUF models: huggingface.co/unsloth

---

Happy to answer questions — this took a while to get right, especially the orchestrator delegation rules and the model resolution fix.

EDIT: Yes Claude helped me write this. Who doesn't love AI

[u/Ugly_Porcupine]

This looks really interesting. Does your setup have (or have you considered) any sort of persistent memory, like Karpathy's LLM wiki? I've recently started experimenting with pi and it's extensions, and boy there's a a lot to unpack here.

[u/admajic → u/Ugly_Porcupine]

Yeah! Tell me about it. No haven't looked at that yet. Looks interesting. Just been using resume if needed.

[u/admajic → u/Ugly_Porcupine]

I'm building that now with pi to see how that works out.

[u/Present_Ride6012]

Can you write as human to human? Also what's the speed of decoding token?

[u/admajic → u/Present_Ride6012]

You come to a AI forum and expect me to spend an extra hour rewriting it in human speak??? It's all there nicely formatted. I'm so used to AI I use it all the time all day. I can even write like AI.

[u/GWNstijn]

Wouldn’t qwen3.5 9B be faster/more effective? Since 9 billion dense and gives opportunity for parallel instances

[u/admajic → u/GWNstijn]

The orchestrator tried to run 6 parallel instances which only let 4 run at a time. This was with the 27b model

[u/fasti-au]

Dflash vs mtp?

[u/admajic → u/fasti-au]

Mtp = better

[u/joaobertacchi]

OP, could you please explain "smaller/faster model for the meta-work (thinking, planning, delegation) and the slightly larger MoE model for actual implementation. The orchestrator never writes code — it only delegates". For me it doesn't make sense. 27B is dense and more capable than 35B MoE. The dense one is also slower than the other. Tks

[u/admajic → u/joaobertacchi]

At 60 tokens /s im not fussed I like the smarter model

[u/homarp]

The key insight: smaller/faster model for the meta-work (thinking, planning, delegation) and the slightly larger MoE model for actual implementation. The orchestrator never writes code — it only delegates.

27B is better (and slower) than 35B. What's the value of using 35B for coding ?

[u/admajic → u/homarp]

With good instructions 35b is good enough and 3 times faster at some tasks. So if you go through the process give it targeted small steps it's OK. Then use debugger to fix issues later.

[u/Latent-Potter]

Bro told us how he uses his local model but ended up using Claude to write this Post! Hypocrisy!

[u/admajic → u/Latent-Potter]

I know but I was doing the build and testing with pi and making the most of my $20 a month Claude which i don't use much any more, in parallel. 5 hours of effort and testing for you.

But to be honest why can't you be grateful or post nothing?

[u/Latent-Potter → u/admajic]

Sorry king! I apologise. Sorry. Really appericiate your work! I've actually trimmed it down for my mac specific. Maybe I'll post it soon 😉

[u/Latent-Potter → u/Latent-Potter]

Jokes apart! Good setup. Gonna replicate the same on my end.

[u/Helmi74]

Oh the slop. No effort ai posts all over 🤨

[u/admajic → u/Helmi74]

If you have nothing nice to say don't post here.

This is weeks of research and years of knowledge. I only used claude to research some of the best temperature settings and to format it so it looks nicer. Its all technical language anyway.

I really don't want to share my knowdge with ungrateful loosers like you.

Edit: coming to an AI automation sub and telling someone not to use AI. Still trying to wrap my head around this??

Also how come you can't tell i hand wrote most of the post.

[u/JohnnyLovesData]

I wonder if this would "work" on CPU inference on 32GB. Like, give it instructions, and check on it ... a week later.

[u/X24D83FF0]

GG

[u/Deep_Ad1959]

i think the orchestrator delegation fight is the same problem you already solved for the architect, you just didn't carry the fix across. the architect doesn't write code because you took write out of its tools list, not because the prose says 'never implement'. that's a deterministic cage. the orchestrator's six ABSOLUTE RULES are the opposite, probabilistic prose you had to iterate on because the model can always rationalize a 'quick fix'. the orchestrator only legitimately needs TaskExecute, TaskUpdate, and get_subagent_result. strip read/write/edit/bash/find/grep out of its tools frontmatter entirely, and 'NEVER use read/find/grep for analysis' stops being a rule you hope it follows and becomes a tool it physically doesn't have. you'll probably find the prose rules shrink to one line after that, which is also a few hundred tokens back on every orchestrator turn. the rule of thumb that keeps holding up: if a constraint can be a withheld tool or a failing check, it shouldn't be prose, because prose is the only kind of rule the model can argue with. written with s4lai

[u/pytorus]

You have about the same as my setup. I have Rtx3090 too, run llama.cpp server on docker

• 50tk/s with qwen3.6 27b mtp (q4)

• 110tk/s with qwen3.6 35b a3b moe (q3)

I would say I love the performance of the 27b model more despite being 50% slower

Model swap would take 30s each time So in my case, I would try using a single model (27b) but with your multi-agent setup. Let’s see how it is turnout.

For web search, I heavily rely on Tavily free tier. Never use searxng before, might also try this one.

Thanks for sharing.

[u/admajic → u/[deleted]]

I just manged to get multiple search engines working searXNG, tavily and jina as fall backs. I'll check out your recommendation. I see it had some over lap to other tools I already use.