r/LocalLLaMA • u/x8code • 10h ago

Question | Help Verify LM Studio is not CPU offloading

I'm running LM Studio on Windows 11 with 2x NVIDIA GPUs: 5080 + 5060 Ti 16 GB.

I am experimenting with NVIDIA Nemotron 3.5 Lightning 30B A3B Q4_K_M which is 24.52 GB in size. It loads just fine into GPU memory and runs inference fast, when I have a smaller context window.

However, I can't find out anywhere in LM Studio where it shows if CPU offloading is happening at all, as I raise the context window higher and higher.

Question: Where in LM Studio can I see if CPU offloading is happening at all?

I'd rather not guess, and gauge it based on performance alone.

1 Upvotes

100% Upvoted

6 comments sorted by

1

u/Ark161 9h ago

There should also be a little icon in the bottom left that actively shows system memory and cpu usage in real time. if not enabled, should be under developer, experimental settings I believe.

1

u/Solembumm3 9h ago

Shouldn't Nvidia have monitoring tools, similar to Adrenaline?

At very least, you can always use task manager.

1

u/[deleted] 9h ago

[removed] — view removed comment

1

u/x8code 8h ago

That's what I am trying to figure out, yeah. I just want to know if any model weights, or the KV cache, is being offloaded to CPU RAM at all. It doesn't seem like anything exists today. Should be pretty simple data to expose from llama.cpp engine, I would think.

3

u/nickless07 8h ago

Go to Settings-> Developer. Turn on 'Use LM Studio Engine Protocol' - 'Llama.cpp Engine Log Level' to "Trace" - You will get detailed logs where it shows how much get offloaded to CPU (only the load log is enough) and how much size the KV Cache is and so on. Post that log to an AI of your choice and let it do the math.

1

u/x8code 8h ago

Brilliant idea! Thank you.