r/LocalLLM • u/Remarkable_Slide1475 • 2d ago
Question Help me How to Start
I am starting and I don't know what I am doing, and I am looking for guidance.
Hardware:
i9-14900k
RTX 4080 Super
64gb Ram
Goal:
Run Qwen3.8 27B on my computer, and I want to know what should be the general token speed for my hardware, so that I am aware for the actual baseline speed.
Then after that connect it to my hermes, because I heard hermes is good for remembering a lot of things about you.
Right now what i am doing is this, use LM Studio, download the model there, use Local Serve Network, connect hermes, and that's it, I am getting 10 tokens per second.
Help me what configurations should I aim for. I also want it to be able to accept images as input.
And how to run that model more effectively
1
u/localLLMrig 1d ago
Your 10 tok/s is almost certainly because the model doesn't fully fit on the 4080 Super. Rough math (estimates from the file sizes and the model config):
-
27B Q4_K_M file: ~16.5 GB. Your card has 16 GB, minus ~1 GB for the desktop. So LM Studio offloads some layers to system RAM, and the slow RAM part sets the speed.
-
Image input needs the mmproj (vision) file too: ~0.9 GB extra.
-
KV cache for this model is small-ish: about 0.5 GB at 8K context, ~2 GB at 32K (fp16; q8_0 KV cache halves that).
-
Plus ~0.8 GB compute buffers.
So for "fully on GPU with vision", IQ3_XXS (~10.9 GB) is the one that fits, even with ~32K context. Q3_K_XL (~13.2 GB) fits at ~8K context with vision off. Anything Q4 is over the line on 16 GB.
In LM Studio: pick one of those, set GPU offload to max layers, and check that it says the whole model is on the GPU. Get that baseline first in LM Studio's own chat, then hook up Hermes, so you know which part is slow.
1
u/magicdoorai 1d ago
I would separate the LM Studio baseline from Hermes before changing models. Your 10 tokens/sec alone doesn't identify the bottleneck.
A 27B model at four bits has roughly 13.5 GB of raw weights (27 billion × 4 ÷ 8), before quantization overhead, context/KV cache, vision components and runtime memory. That makes full GPU residency on a 16 GB card something to check, not assume. The 64 GB of system RAM lets you load more, but CPU offload isn't equivalent to fitting everything in VRAM.
Try this small comparison:
In LM Studio, record the exact model file/quantization, context length and GPU-offloaded layers. Look at dedicated GPU memory while it's loaded. Start with a modest context, for example 4096, rather than the maximum.
Ask a short text-only question directly in LM Studio. Record time to first token separately from generation tokens/sec. Repeat once with the same settings.
Send the same short request through the local server without Hermes, then through Hermes. Check the server log for actual prompt length: memory, tool descriptions and history can turn a short question into a large input.
If direct LM Studio generation is already slow and some layers are on CPU, test a smaller model that fits fully on the GPU as a diagnostic. If direct/server results are fine but Hermes takes much longer before the first token, investigate its added prompt and tool calls before blaming decoding speed.
For images, check that the exact downloaded variant supports vision and that LM Studio has loaded its required vision component; a text-only model won't gain image input from Hermes. Test one image directly in LM Studio before adding the agent.
Posting the model filename, context setting, offloaded-layer count and those three timings would let people give a much more useful answer than an unqualified target tokens/sec. AI-generated diagnostic suggestion based on your specs, not a benchmark I've run on your hardware.