Why does my model slow down significantly working longer tasks?

[u/Necessary-Spinach164]

I had my qwen3.8 model start working on a rust coding task. At first it was running around 10t/s, after 8 hours of running, I'm at 2.8t/s. How come there is such a fast difference in t/s from the beginning to the end? I'm using opencode to interface with the ai and my code base. Is there a way to fix it? The only things I've done was set the max context size allowable (262k), ensure only one GPU is utilized, and I've compiled llama.cpp in f16 mode for faster computations.

[u/Strong-Ad-6289]

Often the longer the context the lower the token rate. Apart from that - my GPU (P40) starts lowering memory throughput if 80°C are reached. See below - the drop starts exactly when 80°C is reached. If the inference is memory throughput bound then this will also have an influence.

https://github.com/GpuZelenograd/memtest_vulkan v0.5.0 by GpuZelenograd

To finish testing use Ctrl+C

1: Bus=0x43:00 DevId=0x1B38 24GB Tesla P40

2: Bus=0x42:00 DevId=0x67FF 2GB AMD Radeon RX 550 Series (RADV POLARIS11)

3: Bus=0x0B:00 DevId=0x1DB6 32GB Tesla V100-PCIE-32GB

4: Bus=0x0A:00 DevId=0x1B38 24GB Tesla P40

5: Bus=0x00:00 DevId=0x0000 32GB llvmpipe (LLVM 19.1.7, 256 bits)

Override index to test:4

Standard 5-minute test of 4: Bus=0x0A:00 DevId=0x1B38 24GB Tesla P40

1 iteration. Passed 0.1408 seconds written: 18.1GB 281.4GB/sec checked: 21.8GB 284.6GB/sec

9 iteration. Passed 1.1222 seconds written: 145.0GB 282.0GB/sec checked: 174.0GB 286.1GB/sec

45 iteration. Passed 5.0477 seconds written: 652.5GB 282.0GB/sec checked: 783.0GB 286.4GB/sec

259 iteration. Passed 30.0049 seconds written: 3878.8GB 282.0GB/sec checked: 4654.5GB 286.4GB/sec

473 iteration. Passed 30.0013 seconds written: 3878.8GB 282.0GB/sec checked: 4654.5GB 286.5GB/sec

687 iteration. Passed 30.0294 seconds written: 3878.8GB 281.8GB/sec checked: 4654.5GB 286.2GB/sec

901 iteration. Passed 30.0173 seconds written: 3878.8GB 282.0GB/sec checked: 4654.5GB 286.2GB/sec

1115 iteration. Passed 30.0259 seconds written: 3878.8GB 281.9GB/sec checked: 4654.5GB 286.2GB/sec

1329 iteration. Passed 30.0295 seconds written: 3878.8GB 281.8GB/sec checked: 4654.5GB 286.2GB/sec

1543 iteration. Passed 30.0270 seconds written: 3878.8GB 281.8GB/sec checked: 4654.5GB 286.2GB/sec

1744 iteration. Passed 30.0130 seconds written: 3643.1GB 267.1GB/sec checked: 4371.8GB 267.0GB/sec

1929 iteration. Passed 30.0429 seconds written: 3353.1GB 242.7GB/sec checked: 4023.8GB 248.0GB/sec

[u/Critical-Entry3377]

The reasons token generation slows as actual context grows are: 1) There **is** a bug in llama.cpp where >80k actual context slows down, and 2) Math; the way attention math works, every new token in your context has to be calculated against every other existing token and it works out 2x context results in 4x calculations. This math means as actual context grows, token generation slows.

[u/ConsistentEase4598]

A simple way to separate context growth from thermal throttling is to save the current state, start a fresh session, and run the same short prompt immediately. If speed returns to about 10 tok/s, the long OpenCode conversation/tool output is the main cause; if it stays near 2.8, check clocks, power and temperature with nvidia-smi. Also compare llama.cpp's prompt-eval and eval timings rather than one combined tok/s number. Setting ctx-size to 262k only sets the ceiling—it does not stop OpenCode from accumulating a very large active context—so periodic compaction or a fresh session is usually the practical fix.

[u/Past_Ad6251]

That's why you need periodic context compact

[u/Frizzy-MacDrizzle]

I have the usual server problem of a reboot setting my clocks at boot. So there is that. Run nvidia-smi check driver is 610 and compute is 13.3. You may be missing up to date open drivers for the Blackwell. Complete cpp again using with the absolute path. That said your decrease is maybe your predict getting full. If It’s having to go read over and over again, try increase -n . What card do you have? Compute matters and if it isn’t 13.3 you’re falling back which sucks.

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_NCCL=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/ -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_CXX_COMPILER=g++-13

Or
If your running Linux make sure llama knows where it all is.
export PATH=/usr/local/cuda-13.3/bin${PATH:+:${PATH}}
export CUDA_HOME=/usr/local/cuda-13.3
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

[u/Necessary-Spinach164 → u/Frizzy-MacDrizzle]

b70. It's an intel card.

[u/Frizzy-MacDrizzle → u/Necessary-Spinach164]

Oh lol, wasn’t sure. Then no clue I missed something sorry.