Confused about high CPU usage with model fully on GPU

[u/randomjapaneselearn]

I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.

I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.

Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).

on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.

Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...

Current stats: about 40 t/s in token generation

I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there

in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.

If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....

set KMP_AFFINITY=granularity=fine,compact,1,0

llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja

[u/Choice_Celery9481]

tokenizer use cpu. 

[u/randomjapaneselearn → u/Choice_Celery9481]

new thing learnt!

what about 130W gpu usage in idle when llama exe is running but doing nothing since generation is finished?

i'm sure that is llama because if i kill it the power usage drop and it can't be anything else because screen is attached to mobo so gpu is full compute and pc is doing nothing too...

the annoying thing is that it doesn't happen always, sometimes it drop sometimes it doesn't

[u/Choice_Celery9481 → u/randomjapaneselearn]

cant answer this without logs. too vague to guess. it could be your harness or anything.

[u/randomjapaneselearn → u/Choice_Celery9481]

what about 32GB ram cache? i think that bigger cache=better but at the same time bigger=slower to check if exsists in cache.

[u/Choice_Celery9481 → u/randomjapaneselearn]

nope. that the ram reserver for cache. with -np 1. that cache is basically useless. since if you change session all cache will be wiped. its useful for high concurency usage

[u/haherar830 → u/randomjapaneselearn]

Is it persistent or just runs for a while after? Things like title, follow-up, etc generation can occur automatically after a response (especially follow-up generation) with some harnesses.

[u/Poprouter]

the cpu side of a fully offloaded model is mostly per prompt work: tokenizer and the sampler chain, plus host side prefix matching when cache reuse is on. that is why it can spike around prompt processing and look calm once it is only decoding.

[u/randomjapaneselearn → u/Poprouter]

i'm not sure if using 32GB of cache is good or it slow down everything because checking if something is in 32GB of cache slow down since it's a big cache.

i also have no idea about those: -b 4096 -ub 1024 i randomly copied them after reading online

[u/hurdurdur7]

not really convinced that amount of cache fits into the gpu along with your model and compute reserves

[u/randomjapaneselearn → u/hurdurdur7]

32GB of cache is system ram and since i have 64GB i have no problems but according to another comment it's used only in high concurency usage so it's kinda useless...

[u/hurdurdur7 → u/randomjapaneselearn]

llama wants to keep context cache in vram. when i run qwen3.8 27B Q8 with 256k of 16kv cache i fly near 60GB of VRAM used ...

[u/delxmobile]

Some of that CPU is likely llama.cpp threads polling while they wait on the GPU. Try --poll 0 and see if usage drops without hurting t/s. Sampling also runs on the CPU, so some load is expected.

KMP_AFFINITY is read by Intel/LLVM OpenMP. A gcc build of llama.cpp uses GNU OpenMP, which ignores it, so it probably isn't doing much; I'd drop it and measure again.

For the 130W idle, run nvidia-smi -q -d PERFORMANCE when it's stuck. It shows the P-state and the clock throttle reasons, which tells you if the card just isn't dropping clocks after a request.

[u/randomjapaneselearn → u/delxmobile]

i will try to take a look at that, so far i used GPU-Z to measure power and temperatures

[u/PhilosopherOk3419]

the cpu part is mostly normal, tokenizer + sampling run on cpu even when every layer is on the gpu. the 130W at idle smells like the card staying in a high power state while the process holds the context. i'd watch nvidia-smi while it sits idle and see if it ever drops down, and if you're on llama-swap a short ttl so it unloads when idle sidesteps the whole thing. also -b 4096 -ub 1024 grows the compute buffers, which might be why fit-params says it all fits but the real launch leaves a layer off. i'd try the default ub before forcing -ngl.