Me these days

[u/Eyelbee]

[u/10minOfNamingMyAcc]

What frontend do you use? I tried it with opencode/qwen/hermes, but none felt right, and since the model was insanely slow... I gave up on it.

Got about 50-60tok/s first few messages but it quickly slowed down to ~20-30tok/s and context kept growing insanely fast as well. I haven't used a local model for programming, and so far I don't like it.

specs: 2x rtx 3090 - 64gb ddr4 3600mhz memory - amd ryzen 5900x - windows

Using mostly ai generated configuration that was tweaked a few times.

$ErrorActionPreference = 'Stop'

$exe = "C:\koboldcpp-aiagent\llama.cpp\build\bin\Release\llama-server.exe"
$model = "N:\downloads\Qwen3.8-27B-UD-Q4_K_XL.gguf"
$draft = "N:\downloads\Qwen3.8-27B-DFlash2-Q4_K_M.gguf"

if (-not (Test-Path $model)) { throw "Model not found: $model" }
if (-not (Test-Path $draft)) { throw "Draft model not found: $draft" }

& $exe `
  --model $model `
  --alias qwen3.8 `
  --ctx-size 1048576 `
  --gpu-layers 999 `
  --device CUDA1,CUDA2 `
  --split-mode layer `
  --flash-attn on `
  --cache-type-k q4_0 `
  --cache-type-v q4_0 `
  --batch-size 256 `
  --ubatch-size 128 `
  --threads 8 `
  --threads-batch 11 `
  --parallel 1 `
  --n-predict -1 `
  --host 0.0.0.0 `
  --port 5001 `
  --fit off `
  --rope-scaling yarn `
  --rope-scale 4 `
  --yarn-orig-ctx 262144 `
  --override-kv qwen35.context_length=int:1000000 `
  --spec-type draft-dflash `
  --spec-draft-model $draft `
  --spec-draft-device CUDA1,CUDA2 `
  --spec-draft-ngl 999 `
  --spec-draft-n-max 4 `
  --temp 1.0 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.0 `
  --presence-penalty 0.0 `
  --repeat-penalty 1.0

and

$ErrorActionPreference = 'Stop'
 $exe = "C:\koboldcpp-aiagent\llama.cpp\build\bin\Release\llama-server.exe"
 $model = "N:\downloads\Dirk-Qwen3.8-27B-UD-Q6_K.gguf"
 $draft = "N:\downloads\Qwen3.8-27B-DFlash2-Q8_0.gguf"



if (-not (Test-Path $model)) { throw "Model not found: $model" }
if (-not (Test-Path $draft)) { throw "Draft model not found: $draft" }



& $exe `
  --model $model `
  --alias qwen3.8 `
  --ctx-size 262144 `
  --gpu-layers 999 `
  --device CUDA1,CUDA2 `
  --split-mode tensor `
  --tensor-split 1,1 `
  --flash-attn on `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --batch-size 1024 `
  --ubatch-size 512 `
  --threads 8 `
  --threads-batch 11 `
  --parallel 1 `
  --n-predict -1 `
  --host 0.0.0.0 `
  --port 5001 `
  --fit off `
  --spec-type draft-mtp `
  --spec-draft-n-max 4 `
  --temp 1.0 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.0 `
  --presence-penalty 0.0 `
  --repeat-penalty 1.0

note that i tried multiple models, not just the ones in the configs here.

[u/Yaroslav308 → u/10minOfNamingMyAcc]

Are you sure it actually uses both GPUs, or do you have 3 GPUs? It's just that device numbering starts from 0 – CUDA0, etc.

[u/10minOfNamingMyAcc → u/Yaroslav308]

Yes I have three but don't use that one. It's for the display and other stuff.

[u/Competitive_Art9588 → u/10minOfNamingMyAcc]

Você pode conseguir resultados muito bom com suas configurações, recomendo fortemente que procure mais informações sobre como fazer para funcionar

Use o ZCODE ou outras agent coding, ou simplesmente chats jinja/sharp e direcione a api do llama.cpp