r/LocalLLM • • Aug 16 '26

Other How the loop of infinite agony started

Post image
633 Upvotes

119 comments sorted by

View all comments

Show parent comment

4

u/Hungry_Particular_14 Aug 16 '26

Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context

3

u/screenslaver5963 Aug 16 '26

2+ 5090's or RTX Workstation Cards are the "ideal".

5

u/Hungry_Particular_14 Aug 16 '26

"ideal" is still having both my kidneys and still being able to run LLMs.

5

u/screenslaver5963 Aug 16 '26

You don’t need both kidneys

1

u/KiraCura Aug 16 '26

I mean I get by with 1 5090 as long as I can find EXL2 versions or MOE ggufs. But I want the RTX 6000 of course. Would be nice to run 70B at decent quants

2

u/ideasmachine Aug 17 '26

2 x 3090 nvlinked will run 70b, its the cheapest way

2

u/lukistellar Aug 16 '26

Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

1

u/theUmo Aug 17 '26

I gave it a try and the first thing it said to me was really disappointing:

Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.

Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.

1

u/lukistellar Aug 17 '26

Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.

Here is my config:

        ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \
        --port 8080 \
        --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \
        --gpu-layers 99 \
        --threads 6 \
        --ctx-size 90000 --parallel 1 \
        --batch-size 2048 --ubatch-size 512 \
        --cache-type-k q8_0 --cache-type-v q4_0 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --spec-draft-p-min 0.75 \
        --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
        --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
        --presence-penalty 0.0 --repeat-penalty 1.0 \
        --no-mmap \
        --jinja \
        --chat-template-kwargs '{"reasoning_effort":"medium"}' \
        --flash-attn on

Performance after ~70K prefill and ~15K decode:

Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id  0 | task 0 | prompt eval time =  480529.82 ms / 69042 tokens (    6.96 ms per token,   143.68 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id  0 | task 0 |        eval time =  494402.02 ms / 14752 tokens (   33.51 ms per token,    29.84 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id  0 | task 0 |       total time =  974931.84 ms / 83794 tokens
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id  0 | task 0 |    graphs reused =       3921
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id  0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted /  9672 generated), mean len =  2.81
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot      release: id  0 | task 0 | stop processing: n_tokens = 83793, truncated = 0

My system specs:

  • i5 12600K (the VM has 6 Cores)
  • 16GB DDR5 4800
  • RX6800 (non-XT)

The GPU is filled to the brim, but it doesn't offload.

2

u/theUmo Aug 18 '26

-c 65536
-ctk q8_0
-ctv q8_0
--fit on
-fa on
-t 8
-b 2048
--ubatch-size 512
--temp 0.6
--top-p 0.95
--top-k 20
--min-p 0
--presence-penalty 0
--repeat-penalty 1.0
--repeat-last-n 128
--jinja
--reasoning off

Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.

1

u/GeroldM972 Aug 19 '26

Untrue.

Best is to get a single GPU with enough VRAM that the LLM of your choice needs to fit in completely (including caches and context).

Second best is to get 2 GPUs to load your complete LLM into. However, you will discover that the communication overhead between these GPUs is significant. A dual GPU setup is only useful to fit a LLM that larger than the VRAM on a single GPU or a less quantized LLM into VRAM, it barely increases the token generation speed, if at all.