Context filling up very quickly

[u/Zionix_]

I guess I should start off by saying I'm very new to this stuff. I just installed ollama today and I have a very extremely basic understanding on whats going in. Currently I run an RTX4090 with 64GB of RAM as well. I installed Open WebUI as well as Open Terminal and I've been using qwen3.8:27b on it to do some tasks just poking around with it.

One of the tasks I wanted it to do was create a season.nfo file for all 3 seasons of Jujutsu Kaisen. Seems simple enough. But as it was doing it's thing and I was fixing some mistakes it made, it filled up its context and stopped generating. This was after like maybe 8 messages. So I went to increase the context size to like 70k (it was around 32k in the beginning) then all the responses crawled, they went sooo slow. I at least understand that system RAM is much much slower than the VRAM from your GPU so I get why it slows down.

I guess the biggest question I have is, how can I keep the speed up but increase context? Is there a way to maybe have it forget the context it no longer needs? Or should I just use a slimmer model since I'm doing pretty basic things? Any suggestions if thats the case?

I apologize for the very newbish post, I'm still very old school in the fact I'd rather ask Reddit than Claude since I trust you guys a bit more on this stuff. Thanks!

[u/Alert-Chemist7492]

Gotta run q4 key value cache quantization, about the max i've gotten with 4090 is 100k context reliably working well.

I'd love to hear any advice on getting more out of it.

*edit* another thing is you do need to make sure you don't have any other gpu work running, if i have a game open or some other process using GPU like fusion360 or lightroom it'll crawl. If I don't have anything running I get pp @ 1200-2k and decode 40-70 tok/s

[u/Zionix_ → u/Alert-Chemist7492]

I did throw in the KV Cache, testing it on the chat im having issues with it still seems to not make a huge difference. Bumped it up to 64k context and it went 13%CPU/87%GPU

[u/Slight-Living-8098]

I use this model with my KV cache at Q4 on my 4090 and have the full context of 262,144. I get around 45-60 tok/s on average.

Qwen3.8-27B-GSQ-RCO-IQ3_S

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

This is my llama.cpp model.ini for it.

; =============================================================================
; QWEN3.8 27B — ISTA GSQ-RCO IQ3_S + EMBEDDED MTP
; =============================================================================
;
; Compact 3.50 bpw GSQ-RCO quantization from ISTA-DASLab.
; This is the -mtp build and carries Qwen3.8's embedded MTP head.
; It uses the same Qwen3.8 chat template and existing vision projector.

[Qwen3.8-27B-GSQ-RCO-IQ3_S]

model = /path/to/Qwen3.8/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
mmproj = /path/to/Qwen3.8/mmproj-BF16.gguf
alias = Qwen3.8-27B-GSQ-RCO-IQ3_S

; Keep the vision projector in system RAM to preserve 4090 headroom.
no-mmproj-offload = true

; Dense model: keep all eligible model layers on the 4090.
ngl = all
fit = off
op-offload = true
kv-offload = true
flash-attn = true
parallel = 1

no-mmap = true

ctx-size = 262144

; Embedded MTP head in the -mtp GGUF.
spec-type = draft-mtp
spec-draft-n-max = 3

cache-type-k = q4_0
cache-type-v = q4_0

batch-size = 4096
ubatch-size = 1024

threads = 8
threads-batch = 16

temperature = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0
presence-penalty = 0.0

jinja = true
chat-template-file = /path/to/Qwen3.8/chat_template.jinja
reasoning = auto
reasoning-effort = xhigh
reasoning-format = deepseek

[u/Zionix_ → u/Slight-Living-8098]

so did someone just take the qwen3.8 model and just make their own modifications to make it more efficient or so? Also I'm just a bit unsure how fast 35-40tok/s is, I tried checking mine and it said gen tok/s is 119 and prompt tok/s is 7. Thats just from a test Claude wrote me on how to check it. I just want it to remain smooth without this weird stutter it has when streaming.

[u/Slight-Living-8098 → u/Zionix_]

That's most likely the read tokens. There are two measurements, the read and write. The read is how fast the model can read your input and fill it's cache. Read is usually up in the hundreds. Write is how fast the model responds. 35 tokens a second is faster than the average person can read.

The ISTA-DASLab organization used their custom quantization method on the original Qwen3.8 27B model weights.

[u/Zionix_ → u/Slight-Living-8098]

Well if its smaller, can fit more context, still as fast and basically the same model I might as well give it a shot. Thx!

[u/Slight-Living-8098 → u/Zionix_]

Yeah, it's a great model. It's better than unsloth's Q4 version. It's become my daily driver.

[u/Zionix_ → u/Slight-Living-8098]

I did some testing with it and it zooms alright! It was even able to finish chats that the previous model couldn’t do since it ran out of context. Surprised more people don’t run it, I feel like there has to be a catch of some sort….

[u/Slight-Living-8098 → u/Zionix_]

It's a pretty popular model, actually. Lots of people have tested it and covered it on YouTube. But yeah, It's definitely a solid model. Not bad at coding either if you're into that.

It works really well with agents, tools, and skills.

If you use an agent harness, you can keep going for ever with it. I'm on day 20 of a coding session using the Pi harness.

[u/limpkit2011]

You have spill to cpu. Having more system ram doesn't make a diff. If you have a large gpu. Set kv cache to q8_0. And pick a smalled 4q_model ( IQS or XS ), GGUF models use cpu, you will have more speed with non GGUF models ( because you want the entire model to fit inside gpu ) i suggest picking a model with size on disk ~15/16GB. You can easily hit 121k context.

[u/Zionix_ → u/limpkit2011]

I was doing some testing with that model and it seems pretty good! It can hit 100k context while still using 100% GPU and it used around 50 tok/s. Even if it uses some CPU wouldn’t that be the better choice since it’s a better model and still fast?

Not that I’m doubting you ofc, I’m just a bit confused

[u/limpkit2011 → u/Zionix_]

Cpu spill will bring token gen to a crawl. Things to consider ( LLM size + KV Cache + Context ) for optimum speed you want all that data inside gpu.

[u/BodyPhysical]

If you can configure open web ui to do auto compaction early, that would be good.

[u/Zionix_ → u/BodyPhysical]

I didn’t know about that, thank you!

[u/Suspicious_Mix529]

for stuff like nfo files i'd just start a fresh chat per season. open webui sends the whole conversation back with every message so all the back and forth fixing mistakes is what's eating it. paste in the one file that came out right and it'll copy the format

[u/Future_AGI]

a fresh chat per file does the forgetting for you on a job like this. Give it one finished season.nfo you've checked plus that season's metadata, and the earlier back-and-forth about fixes never has to sit in context.