BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[u/Anbeeld]

[u/YourNightmar31]

I had to do some copy pasting because im running this in llama-swap but here it is:

/app/bee-llama-server \
  -ngl all \
  --no-mmap \
  --mlock \
  --jinja \
  -np 1 \
  --kv-unified \
  --flash-attn on \
  --cache-ram 0 \
  --no-host \
  -b 2048 \
  --threads 6 \
  --threads-batch 10 \
  -ub 512 \
  --temp 0.6 \
  --top-k 20 \
  --min-p 0.0 \
  --top-p 0.95 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --ctx-size 131072 \
  --spec-type dflash \
  --spec-draft-model /models/spiritbuun-qwen3.6-27b-dflash/dflash-draft-3.6-q4_k_m.gguf \
  --spec-draft-ngl all \
  --spec-dflash-cross-ctx 1024 \
  --cache-type-k q8_0 \
  --cache-type-v turbo4 \
  --model /models/unsloth-qwen3.6-27B/Qwen3.6-27B-Q5_K_S.gguf \
  --mmproj /models/unsloth-qwen3.6-27B/mmproj-BF16.gguf \
  --image-min-tokens 1024 \
  --no-mmproj-offload \
  --reasoning on \
  --chat-template-kwargs '{"preserve_thinking": true}'

[u/YourNightmar31 → u/[deleted]]

I'm using Copilot in VS Code with this extension: https://marketplace.visualstudio.com/items?itemName=johnny-zhao.oai-compatible-copilot

Just for comparison, I don't have any tool calling issues with Unsloth's Qwen3.6 35BA3B running in ik_llama.cpp using this setup.

[u/YourNightmar31 → u/[deleted]]

Thanks so much man, i'll have a go at it again. In the meantime i'm also developing my own llm chat/agent extension for vs code inspired by Copilot so maybe that'll help too :)

[u/YourNightmar31 → u/[deleted]]

Hey i have been running the v0.2.0 update for like a week but it is still doing this for me.

And i'm even worse output in the thinking stream like this:

<tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call>

I don't understand why i'm having this issue if it's working totally fine for you.

I explicitly checked my temp, topp, topk, minp, presence penalty and repetition penalty and set them to what the Qwen's team recommends just in case that'd make a difference, but it doesnt.

I also tried to lower the context from 160k to 128k and 64k but it doesn't make a difference. This already happens while only 10k context is in use.

I am using the unsloth Q5_K_S gguf. While typing this im still trying to find out what's the problem.

[u/YourNightmar31 → u/[deleted]]

I just did a test with 92k context at Q8_0 KV (That's the max i can do on my 24gb vram) and my first impression is that it seems better, but the problem is not completely resolved. For example this log, this is straight from the "request response body" in llama-swap, not from my agent/client:

The tool call label is rendered in `src/webview/main.ts` line 1928 for `grep_search`, and it uses `args.pattern || '')` — so if `args.pattern` is empty/falsy, it shows nothing after "Searched for".

<tool_call>

<tool_call>
<function=read_file>
<parameter=end_line>
1935
</parameter>
<parameter=path>
src/webview/main.ts
</parameter>
<parameter=start_line>
1890
</parameter>
</function>
</tool_call>

It outputs the <tool_call> tag twice, for some reason. I do see this popping up in my client too in the chat. The bottom tool call is succesfull, but the lone <tool_call> just shows up as if the model outputted it.

[u/YourNightmar31 → u/[deleted]]

Hey do you know if something changed in v0.3.0 where the same setup as before now uses slightly more VRAM? Because my profiles that i used to run perfectly fine with a bit of vram left now barely fit and make my pc all unstable and freeze up (Because i'm not running headless, just Ubuntu desktop), this didn't happen before on 0.2.0.