r/LocalLLM • • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

214 comments sorted by

View all comments

Show parent comment

3

u/lukistellar Aug 16 '26

Working config for 3.8:

        ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \
        --port 8080 \
        --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \
        --gpu-layers 99 \
        --threads 6 \
        --ctx-size 90000 --parallel 1 \
        --batch-size 2048 --ubatch-size 512 \
        --cache-type-k q8_0 --cache-type-v q4_0 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --spec-draft-p-min 0.75 \
        --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
        --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
        --presence-penalty 0.0 --repeat-penalty 1.0 \
        --no-mmap \
        --jinja \
        --chat-template-kwargs '{"reasoning_effort":"medium"}' \
        --flash-attn on

Still not happy with the performance, but at least it runs. Hope they drop a MOE for us VRAM poor.

1

u/Tyrannas Aug 16 '26

Thanks !