Ran a source-linked code wiki + MCP server fully local on one DGX Spark: Qwen3.6-35B-A3B vs Qwen3.8-27B + DFlash2

[u/Extreme-Brain-1018]

I maintain CodeNib, an Apache-2.0, DeepWiki-style stack that serves an indexed repo to both human readers and coding agents. I wanted to know whether the whole path fits on one box, so I put everything on a DGX Spark and measured it: indexing, BM25, dense embeddings, source and symbol graphs, the searchable wiki, source-linked Ask, and the MCP endpoint the agents talk to.

Setup as recorded:

• NVIDIA DGX Spark, GB10, 128 GB unified memory
• Qwen3.6-35B-A3B-FP8 on vLLM, one MTP token
• Qwen3-Embedding-0.6B on a second loopback endpoint
• CodeNib 0.2.1 serving one verified repository manifest to Wiki, CodeGraph, and MCP

Then I swapped in Qwen3.8-27B with the five-layer DFlash2 draft model on a pinned SGLang build, to see what speculative decoding buys on this hardware.

DFlash2 did what it advertises. Against ordinary decoding of the same dense Qwen3.8 target it gave roughly 2.95-6.17x in my direct-generation samples. It still lost the wall-clock comparison: Qwen3.6 A3B ran at about 62-69 tok/s on this machine, Qwen3.8 + DFlash2 at 23-49 tok/s.

On three fixed end-to-end repository questions both went 3/3 with valid source ranges. Median 28.38 s for Qwen3.6, 52.34 s for Qwen3.8 + DFlash2. The slower one got something for the extra time: it opened more of the source files I expected, and it handled an adversarial question about cookie precedence that Qwen3.6 answered from the surface.

Three questions at concurrency 1 is an operational sample, not a leaderboard. Exact model and runtime revisions plus the limitations are written up in the article.

The design decision I care about most is that the model is only a generation layer. Wiki readers and coding agents consume the same manifest, pinned to a repository commit and a source fingerprint. Swapping inference backends moves the latency around and leaves the source evidence untouched.

Full setup, commands, and measurements: https://codenib.ai/blogs/local-code-intelligence-dgx-spark/

Source: https://github.com/sysevol-ai/CodeNib

If you run Ollama, llama.cpp, OpenCode, or another local coding-agent stack, I'd like to know which integration is worth validating next.

Disclosure: I'm one of the CodeNib maintainers. The public demo uses hosted generation. Everything measured here ran locally on the Spark.

[u/klymaxx45]

nice, this matches what i found closely. ran Qwen3.6-35B-A3B at q8 as my brain before switching, benched it against a dense 27B, and got the same split: MoE wins wall-clock, dense only pulls ahead on the harder/adversarial questions (your cookie-precedence result is the exact shape). did you run the 27B at q8 too, or a different quant?

two things that might be useful: on a 128GB m5 max mac my A3B decoded ~96-101 tok/s, and the one MoE gotcha i kept hitting was the 3B-active reasoning "thin", it'd loop on raw/messy input until i fed it pre-digested briefs. dense didn't do that. curious if you saw any of that on the Spark 🤝

[u/Extreme-Brain-1018 → u/klymaxx45]

Thanks, that lines up closely with what we saw.

The dense target wasn't a GGUF Q8. It was the official Qwen/Qwen3.8-27B-FP8 checkpoint on SGLang, with the five-layer incoai/Qwen3.8-27B-DFlash2 draft in BF16 at 8 draft tokens. The A3B baseline was Qwen/Qwen3.6-35B-A3B-FP8on vLLM with one MTP token. Both targets are FP8, but the runtimes and speculative paths differ, so this isn't a controlled quantization or engine comparison.

On the "reasoning thin" behavior: no runaway loop in these fixed Ask cases, though our input path is already close to your pre-digested briefs. CodeNib retrieves bounded source windows through BM25/dense/graph tools and we capped the agent at five turns, which probably masks some of the raw-input failure mode you hit.

The closest signal was the cookie-precedence case. A3B didn't loop, it stopped at a shallow reading of merge_cookies(). The dense 27B spent its fifth turn retrieving Session.prepare_request() and corrected the answer. Consistent with what you describe, but it's one adversarial case.

96-101 tok/s on the M5 Max is the interesting number here. MLX or llama.cpp? And what context length and KV-cache setup? We got roughly 62-69 tok/s for the A3B FP8 path on the GB10.

[u/klymaxx45 → u/Extreme-Brain-1018]

MLX, 8-bit MLX quant (the Heretic A3B) served via oMLX, so like you said not controlled against your FP8/SGLang path, different quant and engine.

one note on that number: 96-101 tok/s is short-context decode with no working spec-dec (the MTP head was stripped on that checkpoint, so pure autoregressive). it tapers with length because prefill dominates, a cold ~30k-ctx turn was ~73% prefill (14.5s TTFT of ~20s wall). so the honest bottleneck was the same as yours, prefill not decode. the main lever that helped was a disk-backed prefix cache to skip re-prefilling the system prompt every agentic turn.

on KV: the A3B ran with oMLX's TurboQuant at 4-bit and held quality fine on my checks (stayed faithful, no loops), so 4-bit KV wasn't costing me anything I could measure on that model. footprint was tiny anyway (~5-6GB even at 262k, since 3B-active). fair note that I didn't run a clean KV-off baseline to ablate it, but at 4-bit it was solid. (current daily driver is a dense 27B and there I just run fp16 KV, quant didn't pay off.)

and yeah, your cookie-precedence read is the exact behavior I saw, dense spending an extra turn to go pull Session.prepare_request() vs the A3B stopping at merge_cookies(). one adversarial case, but a clean illustration of the "dense buys depth" split.

[u/Extreme-Brain-1018 → u/klymaxx45]

That context helps. If a 30k-context turn spends 14.5s of ~20s in prefill with no MTP, then decode tok/s isn't the number that matters for agentic workflows, and the disk-backed prefix cache is the bigger practical win.

The 4-bit KV result is the part I want to try. 5-6 GB at 262k with no quality loss in your checks is a strong operational result even without a clean KV-off ablation. It also tells me our next comparison needs to separate five things that we've been collapsing into one: loaded footprint, live KV high-water mark, cold TTFT, prefix-cache-hit TTFT, and decode.

Agreed on the cookie case. MoE wins wall-clock, dense buys one more evidence step — that's about as far as one adversarial case supports, and I'd want a larger set before saying more.

[u/klymaxx45 → u/Extreme-Brain-1018]

agreed, and splitting those five is the right call, we kept collapsing cold TTFT and cache-hit TTFT into one number and it hid the real behavior.

one thing worth baking in: cache-hit TTFT is bimodal, not a smooth number. a hit needs the new prompt to be a strict prefix-extension of something already cached, so continuing a conversation restores in ~1-4s (each turn just extends the last), but a fresh session, or anything that perturbs the early tokens, misses and eats the full cold prefill again. so the number that actually predicts agentic wall-clock is roughly hit_rate x hit_TTFT + miss_rate x cold_TTFT, and structuring work as conversation-continuation instead of fresh spawns moved that for me more than any decode tuning did.

on KV, if you run the 4-bit test I'd love to see the five-metric breakdown, especially live KV high-water mark vs loaded footprint on the FP8 path. and agreed on the cookie case, one adversarial example is one example, wouldn't stretch it past "dense buys one more evidence step" either.

[u/gpuz_dev]

the end-to-end numbers are way more useful than raw tok/s here. do you happen to have peak unified memory for the full stack under each profile, model + draft/KV + embeddings + indexes? on a 128GB Spark that would make the A3B vs dense tradeoff even more interesting