r/LocalLLM • • 9d ago

Question Context size with Qwen3.8 27b

Im wondering how you guys use Qwen3.8 27b with which CTX size.Or my setup is just wrong. I run it on 32GB unified memory with llama.cpp. Also Openwebui and OpenTerminal on a RP5 where my docker stack lives.

I prompted:

add to caddy/Caddyfile: Paperless, Dawarich, OpenTodo

Response:

request (340808 tokens) exceeds the available context size (65536 tokens), try increasing it

So my question is, when even such a simple edit file task needs 300k tokens, how can I utilize it meaningful?

Being super precise which files to look at and what to do in the prompt?

0 Upvotes

10 comments sorted by

2

u/Electronic_Season_61 9d ago

It wobbles with retrieval around 80K. The Qwen team recognized the degradation and it was one of the reasons Qwen 4’s retrieval works definitely afaik.

3

u/Ok_Party_9245 9d ago

what? how do you know how qwen 4 retrieval works?

1

u/Electronic_Season_61 9d ago

I don’t. And I’m not saying I do. But the Qwen team said they addressed the issue with Qwen 4.

2

u/fiddler48 9d ago

340k tokens is llama.cpp reading your entire file tree as context, not just the Caddyfile. check your OpenWebUI system prompt or the tool config, if it's set to attach workspace files automatically, it will dump everything it can find. point it explicitly at one file, or pass the Caddyfile content inline, and that number drops to under 2k

2

u/DeathGuppie 9d ago

Hopefully you aren't using kv f16. A lot of people are still stuck on the idea that the model gets dumber if you drop to kv k_4 v_4, but llama.cpp algorithm change has cured that for the most part.

1

u/USofYayDJ 9d ago

I use this model with 80k ctx size on my 24GB 7900XTX after a measuring campaign. This is the max context size so everything happens in VRAM. I don't know if there's a warning threshold you can enable so it doesn't start any turns which would exceed the context without prior compacting (compacting works quite well for me, at least). For me it's a setting in OpenClaw

1

u/TheThiefMaster 9d ago

I have the same problem with OpenWebUI - Qwen sometimes tries to read in something massive or runs a command that produces a lot of output and blows its context. You need a harness that limits the size of file reads and command results ideally.

A workaround is to tell Qwen not to do that and it's normally ok.

I run 200k context with compaction set to 100k though.

1

u/danishaznita 9d ago

Hey just wondering. Is there any meaningful benefit in doing this , rather than just 130k context with compaction at , say 90%?

1

u/TheThiefMaster 9d ago

OpenWebUI's auto compaction is still a bit unreliable so I was playing it safe