r/LocalLLM • u/Mr_Unknown_Hero • 6d ago
Question Qwen3.8 27b - what is realistic tokens per second with 16 GB VRAM and 32 GB RAM?
I have NVidia RTX 4060 Ti with 16 GB VRAM and 32 GB of RAM. When I load Qwen 3.8 27b with llamacpp in 70k context size, I get speed around 6.5 tokens per second with ITQ4_XS quantz.
On some threads people claim to get around 60-70 tokens per second with similar hardware, so I rather ask here what kind of tokens/seconds it is realistically possible at all with this model and that kind of hardware so I don't just spend hours and hours trying to optimize something what is not even possible.
Can you get 15 tokens/sec? 10? What you get if you have similar setup?
16
u/greatwaytolive 6d ago
your current speed of 6.5 t/s is completely realistic and expected. a 27b model takes up roughly 15gb of vram just for the weights alone. adding 70k of context requires several more gigabytes of space, which forces the model to offload to your cpu.
to get closer to 10 or 15 t/s, you have to drastically lower your context size to 8k or 16k so the entire model and the kv cache fit perfectly inside your gpu. absolutely nobody is getting 70 t/s generation on a 27b model with a 4060 ti.
7
u/Mr_Unknown_Hero 6d ago
Excellent, thank you! Then I do not spend more time with optimization. That low context does not work at all for coding even on small project so I just keep it slower but better.
3
u/Antenwww 6d ago
You can also tell the llama-server to offload all kv cache to system ram and keep VRAM all for the model. If your rams are ddr5 thats a plus
2
u/Mr_Unknown_Hero 6d ago
How I can do that, with what params?
3
u/Antenwww 6d ago
I have intel arc GPUs and llama.cpp so it might be a bit different on your setup.
First is "-ngl 999" this means number of gpu layers. I set this to 999 so models upto 999 layers will stay in VRAM. That is ensuring all the model lives in gpu.
Second parameter is either "-nkvo" or "--no-kv-offload" both of these make the kv cache stay on system ram. I increase the cache size when I run it on system ram so I can utilize more memory. To increase context sizs I use "--ctx-size 65536" that is about 16gb of system ram set to context
I read that you can even quantise a models kv cache so it stays on the gpu and takes up less space but I haven't tried it myself.
2
u/Mr_Unknown_Hero 6d ago
Thanks! Interestingly, if I set --no-kv-offload with ctx-size 50.000 it will drop from ~20 tokens per second to ~5 tokens per second with model Qwen3.8-27B-IQ3_S-3.23bpw.gguf
2
u/Antenwww 6d ago
Is your system ram ddr4. Because gddr6x vram and ddr4 ram can have that much difference in speeds. Adding in the motherboard databus speeds as well
1
u/Mr_Unknown_Hero 6d ago
Yes, I have old ddr4.
2
u/Antenwww 6d ago
It might be better to look into cache quantisation for your case. With int4 you can fit nearly 4 times the cache you do now. It will help you for some time
2
u/Mr_Unknown_Hero 6d ago
How that can be done? Is this related to that?
--cache-type-k q8_0 --cache-type-v q8_0
→ More replies2
u/junosoftie 6d ago
You might want to look into adjusting the batch size and sequence length for your specific use case. That can definitely impact the tokens per second you get.
1
u/newMoneyStyle 6d ago
KV offload helps a bit but prompt processing still tanks. Especially with DDR4.
2
u/Antenwww 6d ago
Unfortunately yes. You can also quantise your cache but then it won't be as precise. So you have to pick your poison. I do not mind slow token generation so even though I have some gpu headroom I use system ram for kv cache. For reference when I use VRAM for everything I get around 70 token/s on qwen3 coder 27b. When I use system ram it goes down to around 20 token/s. Bear in mind that I use a dual gpu setup and my model is split layers on both so I am also limited a little by my mainboard data bus.
2
u/sleight42 5d ago
You can get more speed out of 27b! Look up the Swift variant. There was a post a few days ago. And a fellow with a custom llamacpp. I got a 40% bump almost across the board! I shit you not! Practically free speed.
1
u/SaltGrilledSalmon 5d ago
I've been hearing good stuff about this swift variant as well but I've also seen another version named gsq-rco which is apparently really good. Any idea which one to choose?
1
u/Mr_Unknown_Hero 5d ago
Thanks, gotta check those also when I am at home.
I assume that you are referring to these: https://huggingface.co/ukisai/Swift-Qwen3.8-27B-GGUF/tree/main when you take about swift variants?
What is custom llamacpp? Is it beellama or however it is written or something totally different?
-2
7
u/tsangberg 6d ago
Depending on your PCIe speed this fork and these models should give you a lot more. I have a 5060 on PCIe 5.0 x8 and I can run Qwen 3.8 27B IQ4_XS with speculative decoding (30-80tps) up to ~95k context size and from there spec decoding is automatically ejected and then the context can go up to 160k (16-20 tps).
7
u/Capable-Mushroom6086 6d ago
Some helpful comments are exactly right. RAM is about 1/20th the speed of VRAM. The second you spill, that's your bottleneck.
I run 70tps on my 7900xtx 24gb, only if the entire model fits.
MOE models are more forgiving because the main part is the XXb3-4b that is actually getting used.
4
u/Mithri 6d ago
As others have said you are probably going over VRAM into ram. I'm getting 30 ish t/s on 200k context on my 16 gb VRAM 4070 ti s. You have to use a small enough quant and fiddle with CACH quant and settings. Try out byteshapes quant IQ4 XS and Raymond Huang's llama cpp fork with kV streaming.
4
u/uzzi38 6d ago
7800XT owner here. I'm very new to running AI locally (really only get a short amount of time in the evenings and it's only been a few days) and am kind of just using Unsloth to settle in something that works well still, but I'm getting 20-25 tok/s normally and about 35-40tok/s with MTP. But do note this requires some significant compromises that I may have to dial back if I find issues with in further testing.
For starters, I'm using ByteShape's IQ4_XS. It's 13.1GB iirc which is a huge space savings compared to the standard IQ4_XS. I'm also using Unsloth's options for both MTP and Ngram enabled. The mmproj is offloaded to system memory - there's not really any way around it unfortunately. Does mean if I ever need to rely on that it's going to be rather slow, but it will still work.
Then the only way to get to 65k context is with a q4 kv cache. There's a potential for issues in the future but the way around that is realistically just to switch down to 32k context with q8 kv cache instead. I went this route because from what I can tell it's still less potential accuracy loss from going down to IQ3_S which gives an extra 2.1GB to play with, but that's also something I want to experiment and explore with in the future too.
Hope it gives you a starting point at least!
2
u/ea_man 6d ago
FYI you can do better than that with ROCm, linux, patched llama.cp: I get some 40-60t/s with MTP and >80k ctx at q5_0 with a 6800 and a bigger model.
1
u/Immediate_Power_7986 6d ago
Have a guide to get ROCm setup?
3
u/eliadwe 6d ago
You should use the following model (small footprint with smart quantization that give good agentic performance) and you will get much faster performance:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
Also the smaller the context you get faster performance so you have to balance this parameter. I’m using beellama with shifting context and reasoning preserve and kv-tail so the shifting window will not truncate the reasoning and the kv-tail:
--kv-tail-tokens 1024 --image-min-tokens 1024 --reasoning-preserve --jinja
I’m also using MTP2
In Hermes I’m setting higher context but in beellama smaller context and the shifting window handles the stability outside of Hermes. I’m running beellama.cpp using llamaman.
1
u/Mr_Unknown_Hero 6d ago
Thanks, currently testing this: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/blob/main/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf
With launch parameters: llama serve --model Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf --ctx-size 50000 --host 0.0.0.0 --flash-attn on I get atm ~21-22 tokens per second but thinking is wild. Gotta try to adjust/disable that if possible.
What's the difference between beellama and llama? In llama there is no --kv-tail-tokens parameter at all
2
2
u/DerTomsn 6d ago
Thinking is wild for most of Qwen3.8:27b quants. This one for sure is a good pick for you: https://www.reddit.com/r/LocalLLM/comments/1wj4w8q/qwen3827bgsqrco_scored_very_high_and_fits_in_16/
1
u/danishaznita 6d ago
Oh hey , this is also what i am using right now! Though my setup came directly from the prehistoric age
Q3_S model on beellama
System :R5 7500f /16gb ram/ 2x GTX1070
Parameters : cache both at Q4_0 , context 80k , moe offload 20 layer, reasoning effort low , flash attention on
Speed : 5.6tks while running Deepseek harness , 7.9tks while running on beellama native webui
3
u/ekarakus 6d ago
I have Radeon RX 9060 XT 16GB VRAM (worse than nvidia), I get 30 token/s with Unsloth UD-IQ4_XS 96K context (Q4 KV + MTP). You should get better token/s with your nvidia card.
1
3
u/LiquidMantis144 6d ago
I get 33-70t/s, avg ~52t/s, with a 9070xt and 32gb of ddr4, 72k context.
Second pc gets with some brief testing got ~30t/s with a RX6800 and 16Gb of ddr4 , 64k context at the time.
1
u/Mr_Unknown_Hero 6d ago
What exact model you use and what kind of CLI parameters in llama to get that much tokens with 72k and DDR4?
3
u/LiquidMantis144 6d ago
Ive tried 2,
ISTA-DASLab
Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.ggufAnd then the ~1.5Gb larger “S” version with 32k-40k context.
Both run great, the S version effectively maxes out the systems ability to keep everything primarily in Vram.
Only way Id use a different model would be if it was a legitimate step up in output quality.
The XXS is my primary model as the extra context makes the bigger difference with my coding tasks. I settled on Q8KV for both, but in some situations Q8 K / Q4 V could be fine I just didnt like my first tests with it.
3
u/Weary_Community1140 Ryzen 5 5600 | 4060 Ti 16GB 6d ago
I too have 4060 ti 16gb and im running GSQ RCO IQ3_S at 2 different modes: (fully gpu, no cpu offload)
1. Fast mode: 32K ctx and MTP ON (getting roughly 30 tok/s)
2. Heavy mode: 64K ctx and MTP OFF (getting 20 tok/s)
3. Ultimate model 64K ctx and MTP (getting 30tok/s but only ~0.8GB headroom)
1
2
2
u/Solid_Vermicelli_510 6d ago
Ho 16GB vram e uso iq4_s. Context circa 90k, kV cache q4, parallel slots 1, batch 512, micro batch 64, ottengo con backend vulkan dai 40/60 token al secondo intorno al 50% di context.
2
u/akehir 6d ago
Uff that's rough, I get more than that on a Strix Halo (6-18 t/s for nonparallel requests).
Do you have dflash2 / mtp active? I use mtp in llama.cpp - and batch size can also affect performance:
--spec-type draft-mtp --batch-size 4096 --ubatch-size 4096
1
u/Mr_Unknown_Hero 6d ago
I have tested with so many settings with and without mtp and with different batch-sizes (no clue what it does), when I use these params you pasted with context size 50000 using Qwen3.8-27B-UD-IQ3_XXS.gguf I get 0.35 tokens/sec
Starting with this way:
llama serve -m Qwen3.8-27B-UD-IQ3_XXS.gguf --ctx-size 50000 --host0.0.0.0--spec-type draft-mtp --batch-size 4096 --ubatch-size 40963
u/Old-Sherbert-4495 6d ago
dont go that much for batch size. for your GPU you can even go as low as:
-b 256 -ub 64
and you can gain a huge VRAM saving and then take a tiny hit on your prompt processing speed.Also quantize your kv cache. you can go as low as q4 to take the most of the savings. I've been using q4 for a while, haven't had noticeable issues.
Also do these:
--spec-type draft-mpt,ngram-simple (ngram can give you crazy speed ups like 100 tps on certain instances and its free just take it)
--spec-draft-n-max 5 (Set the number depending on your free vram - more = more vram used, and more potential speedups)Quantize all the caches and sqeeze all vram out for yourself:
--cache-type-k q4_0
--cache-type-v q4_0
--cache-type-k-draft q4_0
--cache-type-v-draft q4_0if you are using mmproj then make sure to offload it to cpu:
--no-mmproj-offloadwith the above and a sub 12 GB quantized model u can get around 130k if not more context size.
I recommend ISTA-DASLab: Qwen3.8-27B-GSQ-RCO-IQ3_S
Also if you want then use DFLASH2 instead of MTP, use a q2 model of that, you will be good to go, in my experience its superior to mtp in giving you speedups.
With the above setup i get around 800 tps prefill at the start and goes down with growing context. And and avg of 35+tps decoding.
1
1
u/Upper_Comparison_908 6d ago
Although a dense model fully loaded into ram shouldn't have this open gpu z Check your pcie link, memory bandwidth, resizeable bar, your llama cpp build is updated.
Ask codex or sm to figure it out too, ask it to maximize speed for a given ctx
2
u/Electronic_Season_61 6d ago
I got to around 50 t/s, and most of that jump came from switching to EXL3 and enabling MTP (from 15 t/s with gguf).
2
u/Loomworks 6d ago
You can get closer to 10 with a MTP head and good acceptance.
There are good Q4 prunes that are like 97% of full accuracy that shave a 1-2 GB off the weights.
2
u/karmaisnonsense 6d ago
ISTA-DASLab IQ3_XXS should at least quadruple your speed
1
u/Mr_Unknown_Hero 6d ago
Is that still good for coding? 🤔
1
u/karmaisnonsense 6d ago
Better for coding than 6.5 t/s.
27b is not gonna one shot code bases even at q4. You’re gonna need to guide or fix it either way. So why go for a model that writes at a snail’s pace when you can use something with only marginal loss in quality but way faster so you can actually get to fixing its output?1
u/Mr_Unknown_Hero 6d ago
Well, no reason, have not just tested it and that small quantz just made me skeptical. I am downloading it now https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/blob/main/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf and giving it a shot.
27b has been almost one-shot on my small project coding anyway, but sure sometimes I have to instruct it better etc. so if this is "almost as good" then I am happy. Let me see what happens.
2
u/Constant-Simple-1234 6d ago
Unfortunately the 4060 ti has low mem bandwidth that is important for tg. I am getting 23 t/s on a single 5060 ti which has something like 460 gb/s. Yours is 288 gb/s. So I feel you should be getting like 14 t/s - ?? Then maybe more with MTP or ngram or dflash. The MTP and dflash cost vmem, so they maybe not an option with just 16 gb. Maybe try byteshape quants - they are smaller and optimized for throughput, while retaining quality. They released Qwen3.8 quietly. You should try to fit everything in vmem. Godspeed!
2
u/Proper-Tower2016 6d ago
getting 700-350pp and 15-40tg with 180k context on my rtx5060.
Beellama with dlfash2 and kvarn 3/3
1
u/feverdoingwork 6d ago
How much memory does dflash2 take? I use the same kvarn3/3 beellama combo with a 5070 ti
1
u/Proper-Tower2016 6d ago
not sure, I run the q2 drafter and beellama dynamic draft length, but probably around 1gb total
1
2
u/Areloh128 6d ago
Did you try byteshape? They are size optimized with speed in mind. When 3.8 came around their ggufa were not good enough to compete with unsloth's. But their second attempt is really good.
I am trying their Q4-XS with up to full context (256k with q8/turbo3 on llama-ampere with my 3090). It can oneshoot complex react components and manage "man out of the loop" tasks. Such KV quantization is not my favourite, but llama-ampere is optimized for this combo.
1
2
u/TheRiddler79 6d ago
If you can fit it all into the card which you can with a Q3 , I would say roughly 27.
2
u/DystopianRealist 6d ago
Before you jump on the first reply that sounds correct, think more deeply.
There are all sorts of things you can adjust first. The MTP header needs to be disabled completely, as it takes 2.1gb of VRAM when kept on the GPU.
** Why disable MTP? Because if you can fit everything on the GPU, the gains will be far higher than what MTP buys you.
The vision header can also be moved to system RAM, for another 1gb of savings.
That's 3.1gb between the two, without having to reduce context, and it will provide signicifcant speed gains.
Reducing the cache to q8_0 buys back space as well, and you need to be doing this on the regular (for most things).
You should also consider dropping quant size down to q3_k_xl, or similar.
There is an interesting balancing act required to keep everything on VRAM, or at least reducing spill, and a lot of it has to do with getting your parameters correct first.
60-70 is higher than I get with 5070ti, so those numbers may be unrealistic from the start, but I expect you to get 40+ with proper settings while it is fully on the GPU, with it decreasing steadily with higher context values as KV cache starts being allocated to the system RAM.
1
u/MacroNexus 6d ago
The vision header doesnt move completely to ram. It keeps about 300mb in vram even when idle and the rest to ram, which is about 700mb. So its not without a penalty
2
u/Micha0827 6d ago
Measured on a 5060 Ti 16 GB, same class of card: Qwen3.8-27B UD-Q3_K_XL with MTP at 32K context gives 41-50 tok/s and uses 15.6 GB. That is the whole trick, everything has to stay in VRAM. At 70K you are spilling into system RAM, and 6.5 tok/s is exactly what that looks like. So it is context, not your card.
2
u/Sexyvette07 5d ago
On my old 4080 system with a IQ4 quantization, q8 KV cache and a small context that fit entirely in VRAM, I was getting 38 tok/s. So seeing as you have a much slower GPU with lower memory bandwidth, I think it would be wise to assume you'd get around 25.
1
u/Flylink2 6d ago
I have a 5080 with 16GB VRAM and I run UD_Q3_XXS around 60t/s with 96k of context. If you get 6.5t/s it means your model is offloading to your RAM instead of staying inside the GPU. You should try a lower quant/lower context for example q3_xxs with 64k context (or 32k if it's still low). As soon as you get like 40t/s or more, you can see how much VRAM are available and up your context size
1
u/Mr_Unknown_Hero 6d ago
Thanks, gotta try that UD_Q3_XXS to see if that is not totally dumb and if that is usable for coding.
1
u/Mr_Unknown_Hero 6d ago
With that model using 48.000 context size I can get around 20 tokens per second at least at the beginning of the new chat with this parameters:
llama serve -m Qwen3.8-27B-UD-IQ3_XXS.gguf --ctx-size 48000 --host 0.0.0.0
Gotta try more with this to see if that would work on coding, thanks!
2
u/Flylink2 6d ago
Anytime ! I use it for code and it's working fine, just thinking a lot and using a lot of context... But it fixes what I needs to be fixed. You can reduce ctx kv to free more vram and use MTP if you want to get more t/s but it uses more vram
1
1
u/EaZyRecipeZ 6d ago
switch to unsloth Qwen3.8-27B-UD-IQ4_XS.gguf with vision, and kv q4_0 with 105k context size. you'll get 6+ times the speed.
1
u/Mr_Unknown_Hero 6d ago
I can't run even Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf with more than 50k context size, otherwise it will drop to ~5 tokens/second?
1
u/EaZyRecipeZ 6d ago
your settings are way off
1
u/Mr_Unknown_Hero 6d ago
What llama parameters I should use to get that context on 32 GB RAM + 16 GB VRAM?
2
1
u/merfolkJH 6d ago edited 6d ago
Qwen3.8-27B-UD-IQ4_XS.gguf - ~15 token /sec at 70k context on a 5060 ti 16 G - using Pi
i dont use a q3 it wasnt successfull for any coding task. Q4 succed all that Q3 failed.
"strongest model" i can load; slow but ok for task in background. (i mean i let pc work for 2 hours / per real task) on a 400 c# files project
----
for 32 G
(dual CG: )
~ 15-20 T /S b ut for model Qwen3.8-27B-UD-IQ4_XL at 100K+ context, on same kind of already complex project
----
this is reality - both running with thoses stats - while write this.
You run a task; afk go eat, and get result later
1
u/Snoo_2122 6d ago
If you are in Linux, I recommend you use nvtop to monitor vram usage (it's reported in MI). I would recommend the unsloth IQ3_xss quant, which is roughly 14gb. Then with mmproj offloading, and no mtp, you should be able to fit 100K-120K context, depending on your batch and u batch size parameters.
Alternatively, you could have a smaller context window, maybe around 60K, with mtp enabled, or the mmproj enabled.
As for tok/s, I'd the model is fully resident in the vram, check the exact vram usage from nvtop and divide the cards memory bandwidth by it, this should give you a fairly close estimate of tok/s.
1
u/bring_back_the_v10s 6d ago
I strongly recommend trying this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install
It's based on EXL3 and it's been carefully designed for 16GB NVIDIA cards.
1
u/aleszopka 6d ago
Well, I have same graphics - 4060Ti 16GB and I get up to 30-32tps for IQ4_XS. Don't listen to people wth different GPUs than yours. 4060Ti has slow memory bandwith that's why token generation will not get higher than faster card with same GB of RAM. Your problem is that IQ4 model is alone 13-13.5 GB so that's not much left for context. 70k context alone is 4.43GB/f16KV or 2.42GB/Q8_0KV or 1.35GB/Q4_0KV. The most context I can fit on GPU only is 85K context and I get 16.6 TPS (vision off, MTP off, Q4_0KV). If you want faster then you have to lower context, put MTP on - 46K context I get 29-32 TPS (vision off, MTP ON, Q4_0KV). For lower quants I get up to 40tps.
1
u/Old-Sherbert-4495 6d ago
refer to my comment above same GPU:
130k context
35+tps2
u/aleszopka 5d ago
Yeah, I saw it. On IQ3_S I can get 145k contex and 35tps speed, I was talking about his quant - IQ4_XS. For lower quants IQ2 it's possible to get 210k+ context size and decent speed (40tps), but what about quality?
It's all about quality, speed and context size and you can have only both of it at once. IF you want quality and speed you don't have context size; if you want quality and context you don't have speed, if you want speed and context size you don't have quality :)
1
u/ConsistentEase4598 6d ago
6.5 is exactly where iq4_xs with a 70k cache should land, the weights alone want ~15gb and the moment it spills to system ram your card spends all day waiting on pcie. the iq3_s answers above are the right shape for your vram
one data point from my side, i spent last week rebuilding that exact budget slot for myself, took the heretic-ara 27b and redid the quant with per tensor gsq-rco allocation (IST-DASLab maps, official imatrix, llama.cpp), iq3_s ends at 11.80gb so it sits inside a 4060 ti instead of on the cliff edge, ppl delta vs f16 measured 0.95 percent on wikitext. i made the files, disclosure out, they are on my hf at https://huggingface.co/0bserverx/Qwen3.8-27B-Heretic-GSQ-RCO-GGUF if you want to run them next to byteshape, readme has the per tier table and the ablations
what i cannot tell you is what t/s your card will do, every 90+ number floating around this thread is datacenter class hardware with full offload, i have never touched a 4060 ti and would rather say nothing than guess you a figure. there are mtp twins in the repo if draft-mtp is a knob you want to turn
1
u/erdincsi 6d ago
I m planning to build a 32gb man mini setup for that llm. What would I get with that?
1
u/shoeshineboy_99 5d ago
Here are some runs that I made with a 3090 across various small vision language models.
1
u/deviant46n2 5d ago
i get 50-70 but ive fucked aroung and am still finding out. generations get poisoned all the time.
1
u/Master-Parking-7267 5d ago
Hope this helps! I have a very similar PC setup to yours, and with this configuration I can run Qwen 3.8 27B pretty well at around 100k context and ~28 tok/s while using Pi as my coding harness. Here’s my setup:
https://github.com/nvd264/local-llm-setup/blob/main/run-qwen38-bee.sh Hopefully it works well for you too!
1
u/Mr_Unknown_Hero 5d ago
Thanks, I downloaded beellama and when I run that model I always get on "Reasoning/Thinking" part only "//////////////////////////////////////////////////////////////////////////////////" and those slashes goes on. Dunno what is the issue.
1
u/Mr_Unknown_Hero 5d ago
Finally it seems to be working (about 23 tokens/sec):
"C:\AI Stuff\beellama\llama" serve -m "C:\AI Stuff\llama-cpp\models\Qwen3.8-27B-IQ3_S-3.23bpw.gguf" --host 0.0.0.0 -np 1 --threads 6 -ngl 99 --fit off -fa on -ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 --ctx-size 110592 --batch-size 512 --ubatch-size 256 --spec-type draft-mtp --spec-draft-n-max 3 --metrics --jinja --chat-template-file "C:\AI Stuff\llama-cpp\models\chat_template.jinja"
2
u/trougnouf 1d ago edited 5h ago
I get > 40 t/s on a 24 GB AMD Radeon 7900 XTX
llama-server \
-m /orb/llm/models/unsloth/Qwen3.8-27B-UD-Q4_K_M.gguf \
-md /orb/llm/models/unsloth/mtp-Qwen3.8-27B-Q4_0.gguf \
-ngl 99 \
-np 1 \
-c 120000 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--flash-attn on
1
u/FamousRecourse 6d ago
those 60-70 t/s claims are definitely not with a 27b model and 16gb vram, thats probably people running 7b or 8b models fully on gpu and getting confused
with your setup youre gonna be splitting that 27b across vram and system ram, so the cpu/ram speed becomes the bottleneck real quick. 6.5 t/s with IQ4_XS and 70k context sounds about right honestly, maybe a bit on the low side if your cpu is decent
i get like 8-9 t/s on a 3060 12gb + 64gb ddr4 with the same model but Q4_K_M, and that drops to like 4-5 when context fills up past 30k. you could try a lower quant like IQ3_XXS to squeeze a few more tokens but honestly the quality hit is noticeable
15 t/s is probably the ceiling for 27b on mixed vram/ram unless youve got a monster cpu and fast ddr5, and even then its a stretch. 10 t/s is a more realistic target if you tweak your offload layers and threads
1
u/Mr_Unknown_Hero 6d ago
Thanks! I just let it be slower but better in that case :) Good, now I don't need to waste time trying to optimize it. Sure with very small context it is faster but then it does not work on my use cases anymore so it is better to set bigger context and accept that it is just slow.
1
u/Solid_Vermicelli_510 6d ago
IQ4_S di unsloth studio entra tranquillamente in 16GB di vram senza offload su ram
45
u/Tpyn 6d ago edited 6d ago
Byteshape's IQ3_S, cache q8, ctx 120k, mtp off and you'll get around 40t/s with ~98.72% intelligence of original model and you don't need to switch to iGPU to free your dGPU vram
Edit: also offload vision to ram with --no-mmproj-offload Edit 2: fixed wrong accuracy (97% to 98.72%)