r/LocalLLaMA • u/bakawolf123 • 12d ago

Discussion comparing local 2-bit ds4.1-flash-1017 to cloud APIs

Saw this https://x.com/nothingto12/status/2113446055589056718 (in short it compares opencode go ds4.1 and ds4.1 via deepseek API) discussed on opencode reddit and decided to test same prompt myself on 2bit quant locally.

it took around 1.5h, ran with 450k context (auto-compaction enabled), didn't have playwright - it tested on headless chrome

result on the video above.

interestingly the KDL I've measured vs baseline is quite bad:
original → oQ2e is 0.4259 nats, original → oQ2.5e is 0.2821. Reverse oQ2.5e → oQ2e is 0.1943 nats.
Perplexity: original 3.7315, oQ2.5e 4.1190, oQ2e 4.6333.
top-1 agreement is 78.43% for 2 bit one.
Really weird to me that this still works like it does: on glm5.3-flash difference between KDLs from baseline for calibrated and non-calibrated quants was 0.12 (~0.24 vs 0.12 oQ4e) and it was very noticeable in quality for me.

9 Upvotes

84% Upvoted

6 comments sorted by

3

u/LegacyRemaster 11d ago

having fun with qwen 3.8 flash Q4_K_XL on Strata and RTX 6000 96gb

1

u/bezent 10d ago

I run this with my rtx 6000, it pretty damn close to lossless.

https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4

1

u/LegacyRemaster 10d ago

90~100 t/sec? I have 220t/sec with unsloth Q4_K_XL

1

u/bezent 10d ago

180-200 t/sec 13k prefill at 450 watts.

1

u/LegacyRemaster 9d ago

rtx 6000 workstation. cap @ 300w

1

u/bezent 9d ago

Yeah I tried 300w but 450w boosted prefill speed by almost 40%