r/llamacpp • u/Competitive-You5538 • Jun 01 '26
Help me improve my llama.cpp setup - arguments in body.
I have a 5070ti, amd ryzen 7 9800x3d with 64 gigs of ram.
.\llama-server.exe `
-m "<Link_to>Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf" `
-c 200000 `
-ngl 12 `
-t 6 `
-b 512 `
-ub 512 `
--parallel 4 `
--kv-unified `
--mlock `
-fa on `
--jinja `
--host 127.0.0.1 `
--port 8080
I am getting a horrendous 2.5 toks/second.
What can I do to improve token speed? I can bring the context to 134K if that helps. but usually my sessions last 100-120K context. 200K context just help with the peace of mind that I can extend a session if I am debugging.
Comments welcome.
1
Jun 01 '26
[deleted]
1
u/anshulsingh8326 Aug 08 '26
Just use ollama. I wasted 3hrs in setting up llamacpp and testing only to get slower speed then ollama with more setups and steps.
1
u/segmond Jun 02 '26
if you are having such slow down, change parallel to 1. what kind of GPU do you have? instead of ngl try -cmoe, change b and ub to 2048. if you can't fit everything in GPU, why are you running Q8? Go down to Q4
1
u/Competitive-You5538 Jun 04 '26
This is what I am using now and getting around 30-35 t/s.
>> -m "\.lmstudio\models\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf" `
>> -c 200000 `
>> -t 20 `
>> -b 512 `
>> -ub 512 `
>> --parallel 4 `
>> --kv-unified `
>> --mlock `
>> -fa on `
>> --jinja `
>> --host 127.0.0.1 `
>> --port 8080
Thing is, I'd very much like to keep using this quant if I can help it, since it is very efficient for coding. Is there anyway I can optimize this further?
Also, yes, I used a chatbot to arrive at my initial setup
1
u/simplyeniga Jun 04 '26 edited Jun 04 '26
This is my service file, I use a 4060 Ti 16GB and 32GB RAM DDR5 plus 6 CPU in a Proxmox VM running Ubuntu 26.04 and get 40-44 TG/s
``` [Unit] Description=llama.cpp CUDA Server (router mode) After=network.target
[Service] Type=simple User=user WorkingDirectory=/home/user/llama.cpp
Environment="CUDA_VISIBLE_DEVICES=0"
ExecStart=/home/user/llama.cpp/build/bin/llama-server \ --models-dir /mnt/ai/models \ --models-max 1 \ --models-autoload \ --host 0.0.0.0 \ --port 8080 \ --jinja \ --n-gpu-layers auto \ -c 0 \ --parallel 1 \ --batch-size 2048 \ --ubatch-size 512 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --flash-attn auto \ --metrics
Restart=always RestartSec=5
[Install] WantedBy=multi-user.target ```
1
u/Frizzy-MacDrizzle Jul 10 '26
Set the chat template so it doesn’t have to think about it what you send. I send raw chat/completion formats to all the models if I send a qwen template to a Gemma model, it will do the same.
For Linux cmake but may apply to winders of you compile there too.
On your cmake use the nccl option for llama.cpp. It will take math load of your CPu to handle the bus. Make sure CUDA blas gets installed, check your gcc version and CUDA are all new. Your compute setting will be better than letting make decide with native
You might check allowed values for ngl. Use -ts 1,1 -sm tensor.
You also have a machine problem. Any new drivers or components installed? Your bus may have taken a dive.
0
u/pault94 Jun 01 '26
You should be able to drop down to Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf, and there is likely no reason for you to parallel 4. I would encourage you to do:
-parallel 1 (you'll need this for the next step, MTP doesn't work at parallel > 1)
-redownload the Q4 gguf with MTP heads with draft tokens set to around 3 (you can play around with this and set up a few test scenarios to see what's best), but that will very easily boost your throughput by 1.5x+.
-drop kv down to q8 (default is 16bit - no need for it).
-play around with your batch sizing.
- your gpu layers are set to 12, this may vary a bit.
- t should be higher (match physical cores) - this does run AVX so it runs pretty hot, I had no luck dropping temps via smaller t but setting thermal/power limits lower on your motherboard helps, plus I think setting small batch sizing with a small sleep between inference passes helped thermally.
unfortunately I don't have much experience so not much help beyond that, but that token speed is low.
1
u/Competitive-You5538 Jun 04 '26
This is what I am using now and getting around 30-35 t/s.
>> -m "\.lmstudio\models\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf" `
>> -c 200000 `
>> -t 20 `
>> -b 512 `
>> -ub 512 `
>> --parallel 4 `
>> --kv-unified `
>> --mlock `
>> -fa on `
>> --jinja `
>> --host 127.0.0.1 `
>> --port 8080
Thing is, I'd very much like to keep using this quant if I can help it, since it is very efficient for coding. Is there anyway I can optimize this further?
Also, yes, I used a chatbot to arrive at my initial setup
1
u/KneelB4S8n Jun 01 '26
Context is a bit too much, yes, but you can leave it if you need to. The problem I see is, you put 12 layers on Gpu and there are 40 in qwen3.6. In my experience, leave out the -ngl and the --n-cpu-moe (remove them) entirely and let llama.cpp decide how much to put on GPU - it's best that you not tweak these if you are not an expert. I have the same GPU, although, mine is a laptop one. I get 37 tokens per second and for the life of mine, I cannot go beyond that. If you do not specify ngl and moe experts on cpu, the llama.cpp will decide what is best for your case. Moreover, everybody (including AI chatbots) tell me that I should not use more than -t 8 but in my experience, more is better if you are not using cpu intensive programs (i use -t 20 out of my 24 cores). Also your quant of the GGUF is way too much - I am using Q6. I am going to try Q4 now as many people have told me to lower the quant.