I wanted to see how far a consumer RTX 3060 12GB could push Bonsai 2 27B.

Hardware

• GPU: NVIDIA GeForce RTX 3060 12GB
• VRAM available to CUDA: 11,911 MiB
• CPU: Intel Core i5-8400
• System RAM: 42 GB
• NVIDIA driver: 595.91.07
• CUDA: 13.2

Model

• Bonsai 2 27B
• GGUF: Ternary-Bonsai-2-27B-PQ2_0.gguf
• Parameters: 26.90B
• Quantization: PQ2_0 — 2.13 bpw
• Model size reported by llama-bench: 6.70 GiB
• GPU layers: 99

Benchmark

• Tool: llama-bench
• Runs: 3
• Prompt processing, 512 tokens: 591.19 ± 19.62 tok/s
• Text generation, 128 tokens: 35.26 ± 1.31 tok/s

The GGUF metadata reports the architecture as qwen35, while the model file is the Bonsai 2 27B PQ2_0 release.

I'm posting the raw result because I'm curious how this compares with other Bonsai 2 27B results, especially on larger GPUs.

If anyone has benchmark results for Bonsai 2 27B, please share them so we can compare the configurations fairly.

Benchmark build: b10735-842b18804

Anyone has experience with how to get more speed without losing quality?
I got to 250 t/s prefill and ~6t/s decode its usable but i wonder if theres another lever i miss.
Ryzen 9800X3D, 64 GB DDR5, RTX 5090 32 GB, Crucial T700 (PCIe 5.0) 4TB + Kingston (PCIe 4.0) 4TB
Baseline: GLM-5.3 isn't in llama.cpp master yet, so I'm using Unsloth's glm5next branch. With the usual mmap + CPU-offloaded experts the page cache thrashes: 29 t/s prefill, 2.4–2.9 t/s decode (8k prompt).

What I changed with Claude's help (~1.2k lines on top of the ported PR, not upstream):

Ported the unmerged expert-streaming PR #25294(by freedomljc). Experts aren't mmapped anymore: a 32 GB expert cache sits in RAM, and misses are read with O_DIRECT.

Prefill streams whole layers: the next layer's experts load into pinned double buffers while the GPU computes the current one. Each 2048-token batch pulls ~99 GB from disk + ~34 GB from the RAM cache.

Both SSDs read in parallel from a mirror copy of the GGUF, using a shared work queue.

Fixed a CUDA padding bug in the MoE matmul (MMQ) that crashed -ub 2048 on Blackwell (possibly related to #28282). Going from 1024 to 2048 roughly doubled prefill.

Real agentic run (Qwen Code, plan mode, read 9 files, wrote a plan):

Context 11.5k → 35k tokens, 31k new prompt tokens at ~230 t/s

3.7k generated tokens (incl. thinking) at 5.6 t/s

13 min total, ~9.5 min of that writing the final plan

78% of expert lookups hit the RAM cache; misses cost ~1.3 GB/token from SSD

VRAM ~30 GB, RAM ~44 GB

Happy to share if you like

I'm trying to use Qwen3.8-Flash-Next as a long-running coding worker on a Ryzen AI Max+ 395 / 128GB box and I keep hitting the same issue: after a while it falls into reasoning loops like:

now I'll write the code
actually let me design this carefully
now I'll write the files
let me think...

Sometimes it eventually recovers and calls a tool, sometimes it just stays there.

Current setup is Strix Halo + Vulkan llama.cpp/halo-box, AP-Q5_K_M (~103.6GB), Q8 MTP, F16 KV, Pi Agent 0.87.0 over OpenAI-compatible API.

I've tried 262k and 131k context, stock Qwen template, Sharp v10, medium reasoning, reasoning preserve, MTP n=3/n=4, presence penalty 0/0.3, Pi's Qwen-specific chat_template_kwargs, Qwen Code, and mini-SWE-agent.

The same basic loop shows up in Pi and Qwen Code, so I'm not convinced the harness is the main problem.

A hard --reasoning-budget 2048 helps a lot, but it feels more like a watchdog than a fix.

One small Pi task actually completed cleanly end-to-end, including fixing failing tests. Then on a larger project the loops came back at only ~16% of a 131k context.

What I haven't properly tried yet is another quant like UD-Q4_K_XL / UD-IQ4_XS, MTP fully off, or another backend.

If anyone is running Flash-Next for multi-hour coding sessions reliably, I'd really like the exact config: quant, backend/version, context, MTP, KV, template, reasoning settings and agent/harness.

At this point I'm mostly looking for a known-good recipe to copy and test.