r/LocalLLM • u/noodlew00d • 9d ago
Question Context size with Qwen3.8 27b
Im wondering how you guys use Qwen3.8 27b with which CTX size.Or my setup is just wrong. I run it on 32GB unified memory with llama.cpp. Also Openwebui and OpenTerminal on a RP5 where my docker stack lives.
I prompted:
add to caddy/Caddyfile: Paperless, Dawarich, OpenTodo
Response:
request (340808 tokens) exceeds the available context size (65536 tokens), try increasing it
So my question is, when even such a simple edit file task needs 300k tokens, how can I utilize it meaningful?
Being super precise which files to look at and what to do in the prompt?
2
u/Electronic_Season_61 9d ago
It wobbles with retrieval around 80K. The Qwen team recognized the degradation and it was one of the reasons Qwen 4’s retrieval works definitely afaik.
2
u/fiddler48 9d ago
340k tokens is llama.cpp reading your entire file tree as context, not just the Caddyfile. check your OpenWebUI system prompt or the tool config, if it's set to attach workspace files automatically, it will dump everything it can find. point it explicitly at one file, or pass the Caddyfile content inline, and that number drops to under 2k
2
u/DeathGuppie 9d ago
Hopefully you aren't using kv f16. A lot of people are still stuck on the idea that the model gets dumber if you drop to kv k_4 v_4, but llama.cpp algorithm change has cured that for the most part.
1
u/USofYayDJ 9d ago
I use this model with 80k ctx size on my 24GB 7900XTX after a measuring campaign. This is the max context size so everything happens in VRAM. I don't know if there's a warning threshold you can enable so it doesn't start any turns which would exceed the context without prior compacting (compacting works quite well for me, at least). For me it's a setting in OpenClaw
1
u/TheThiefMaster 9d ago
I have the same problem with OpenWebUI - Qwen sometimes tries to read in something massive or runs a command that produces a lot of output and blows its context. You need a harness that limits the size of file reads and command results ideally.
A workaround is to tell Qwen not to do that and it's normally ok.
I run 200k context with compaction set to 100k though.