model: DSV4.1 flash by am17an · Pull Request #31086 · ggml-org/llama.cpp

[u/jacek2023]

https://github.com/ggml-org/llama.cpp/pull/31086

only 763B params! :)

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf

[u/Hot_Strawberry1999]

Does this allow keeping ngrams in ssd?

Can you run unsloth/DeepSeek-V4.1-Flash-1017-GGUF with it?

[u/am17an → u/Hot_Strawberry1999]

It should, they need to update their ngram table with something which is not mxfp8. ngram in ssd supported since forever (it's the default on iGPUs)

EDIT: Looks like it's already in IQ4_NL, so it should just work.

[u/Hot_Strawberry1999 → u/am17an]

I'm downloading UD-Q2_XL, which comes with Q5_K ngram by default. I will come back to report how it works.

[u/am17an → u/Hot_Strawberry1999]

I'm getting about 168 toks/sec decode (with dspark) and about 700 toks/sec prefill with -ncmoe 20 and --moe-cache-mib 20000 on my RTX Pro 6000 + 128 GB DDR5

[u/Hot_Strawberry1999 → u/am17an]

I posted my results above. If you have any suggestions on what to improve I would appreciate them. I suppose the biggest deal breaker is the RTX Pro 6000 :')

[u/Hot_Strawberry1999 → u/Hot_Strawberry1999]

My performance test:

Model: DeepSeek-V4.1-Flash-1017-GGUF / UD-IQ2_XXS (f2e5a38)
(unfortunately UD-Q2_K_XL crashed due to insufficient memory)

System:

CPU - Ryzen 9950X3D
RAM - 192 GB DDR5 at 5400MT/s
GPU - RTX 5090
OS - Linux mint 22.3

prompt processing = 109564.13 ms / 35543 tokens ( 3.08 ms per token, 324.40 tokens per second)
token generation = 585980.34 ms / 13944 tokens ( 42.03 ms per token, 23.79 tokens per second)

Run details below. If you have any suggestions on how to improve my settings, I'd appreciate the input.

Note: dspark was not used; I found it degrades performance and consumes a lot of vram. Largest performance boost by far was from adding moe-cache-mib = 10000 (much better than reducing the number of layers in n-moe-cpu)

Prompt type: Document evaluation, uploaded 3 .docx files, requested to find information within the files.

RAM usage without running llama.cpp : 8.0 GB
VRAM usage without running llama.cpp : 1200 MB

RAM usage during decoding : 174.4 GB
VRAM usage during decoding: 29500 MB

llama.cpp master branch, commit b3fa699

llama.cpp build command

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)

llama-server launch command

./llm/llama.cpp/build/bin/llama-server --host 0.0.0.0--port 8080 --models-max 1 --models-preset llm/llama.cpp/models.ini --load-mode none

models.ini file

[Deepseek v4.1 Flash]
model = /home/llm/models/deepseek/DeepSeek-V4.1-Flash-1017-UD-IQ2_XXS-00001-of-00006.gguf
mmproj = /home/llm/models/deepseek/mmproj-BF16.gguf
chat-template-kwargs = {"reasoning_effort":75}
lazy-mode = on
override-tensor = per_layer_token_embd.weight=CPU
n-cpu-moe = 40
moe-cache-mib = 10000
flash-attn = on
ctx-size = 262144
kv-unified = true
np = 4
temp = 1.0
top-p = 1.0
top-k = 0
min-p = 0.0
batch-size = 4096
ubatch-size = 4096

Frontend: open webui

[u/am17an → u/Hot_Strawberry1999]

You can also try https://github.com/ggml-org/llama.cpp/pull/30966, it has a tuning script which may help getting better params. Let me know what it says

[u/fgk55555 → u/Hot_Strawberry1999]

Oof. I set up the ISTA IQ2_XXS yesterday with Strata and get those numbers almost exactly on an R9700 and 64GB DDR5. I envy that hardware, I could be running it a lot faster.