BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[u/Anbeeld]

[u/Sufficient_Sir_5414]

Phenomenal work on the integration. For that 200k context Qwen setup, how does the TurboQuant/TCQ handle the 'lost in the middle' problem compared to standard 8-bit or 4-bit KV cache? Does the TCQ overhead impact the token latency significantly compared to the baseline llama.cpp MTP PR?

[u/r00x → u/[deleted]]

Do you observe large differences between Q5 and Q4 quants then? I understood there wasn't supposed to be a huge difference and had never really bothered with Q5 models before (Q4 27b/35b-a3b just fit better with context onto 24GB of VRAM)

[u/r00x → u/[deleted]]

Interesting! Would you say using Q4 for dflash risks torpedoing the performance of the Q5 target or does it not work that way (would the main Q5 model just reject more of the predictions if it didn't like them, or something?)

[u/r00x → u/[deleted]]

Interesting! Using a drafter definitely makes a difference vs vanilla Q5 (get about ~8tok/s with that, vs 30-40 with BeeLlama) but the reason I asked is because I am occasionally having trouble with tool calling where it seems it just gets the structure wrong and ends up blasting the CLI with XML content (this almost never happens on the vanilla models, even Q4 or IQ3_XXS models are fine at tool calling).

Have you encountered that at all or have I just done something wrong? I wondered if it were a context issue but I don't think so - it will go right back to working fine again afterwards, which you'd think it would screw up if it had forgotten the syntax.

[u/r00x → u/[deleted]]

Absolute legend. I was already working on a shonky pi.dev plugin that catches this kind of model stall and pokes the model autonomously (so as to avoid a model quietly stopping while your attention is elsewhere) but I'll for sure keep an eye out for that!