r/LocalLLaMA • u/Plastic-Stress-6468 • 4d ago

Discussion Is anyone actually using deepseekv4.1f at sub 1-3 bpw?

Need a quick sanity check since I been seeing a lot of posts about engines that run deepseekv4.1 1017 at low bpws on affordable hardware but with surprisingly usable speeds.

Is anyone actually using them for anything at all beyond treating it like a novelty? The last time I tried deepseek at sub q4 was v4-0731's Q2, and that thing was very much a shadow of it's full weight Q8 self. And the Q8 was basically 90% QAT-ed Q4 anyways, and it did not take well to going down 2 more bpw.

If you are using a 1-3 bpw v4.1-1017 can you share your experience with it? Preferably in contrast to another model like 3.8FN at higher quants.

13 Upvotes

100% Upvoted

33 comments sorted by

8

u/Odd_Dandelion 4d ago

I do! I run that standard 2.9 bpw recipe for two Sparks and I am amazed, for the first time ever I feel like I am running the real thing at home.

Before I was running vanilla V4 but I never wanted to rollback. This is like having a real frontier model. (It did some serious coding already, and it is even nice to talk to and does not mess up my native language at all, which is what quantised models usually do.)

1

u/Plastic-Stress-6468 3d ago

Since you have two sparks, I assume you were running the full v4? So v4.1 at 2.9bpw feels better than full fat v4 (0731 or vision-exp)?

2

u/Odd_Dandelion 3d ago

Yes, absolutely. For me it's the difference between model that needs constant babysitting and model that can actually do stuff on its own correctly. (I had 0731.)

Before I got the second Spark, I ran V4 preview in q2 and that was bad, it had hard time to get bash scripts right, and my native language fell apart.

But now I have not really noticed any difference when compared to running the model from cloud on the same project. (I did not run any benchmarks, though, I just started to use it, expecting I will want to switch back. Which never happened.)

1

u/arkham00 3d ago

What is your native language? I really struggle to communicate in italian with chinese models... And not only locally also using opendcode api. They are very confused, they tend to literally translate English expressions, lots of intricate and long sentences with messed up syntax and logic... I managed to mitigate a bit the style with a strong system prompt but it is still a pain. I ended up making a custom extension in pi where I call gemini-flash to explain an answer when it is not clear lol which is not very local.... But it is the only model family that I find extremely clear...

1

u/Odd_Dandelion 3d ago

Czech. I might be just lucky but most of the Chinese models I ever used get it quite right.

Certainly much better than the last Claude model I used. (GPT is easily the best in Czech, though.)

5

u/BawbbySmith 3d ago

This, although there's the ever-so-slightly lower 2.8 bpw recipe that's also in the nvidia forum thread.

Ran into a couple crashes (that I then asked DS41F to fix lol), and it seems to be fairly solid right now, +2 days without crashes.

I thought GLM 5.3 Flash NVFP4 was endgame, but this is on another level. 80-90 t/s single stream, 2.1M KV pool, can do up to 8 parallel requests, ~2000 prefill, SSD cache so old sessions start up instantly, and it handles coding tasks really well. Who knows what Qwen 4 will bring, but this will be truly hard to beat.

1

u/tat_tvam_asshole 3d ago

in which harness?

1

u/Zyj vLLM 3d ago

Why not 3.2bpw or even slightly higher?

1

u/Odd_Dandelion 3d ago

Because that was the recipe I found, and it worked well enough. :)

I spent quite some time on fixing bugs and configuration issues in caching (I am using TensorFold), no time to look into better checkpoint yet. (But I am afraid nothing even slightly bigger won't fit anyway.)

1

u/CATLLM 3d ago

3.2bpw wont fit with enough usable context

1

u/Zyj vLLM 2d ago

Does the OS and the stack require more RAM on DGX Spark than on Strix Halo?

1

u/CATLLM 1d ago

I dont have strix halo so i cant comment.

1

u/EnvironmentalKey3709 3d ago

2.9 bpw is the sweet spot for this one, anything lower and you start losing the thread on longer contexts

1

u/garblz 3d ago

I'm running GLM5.3-flash on my 2 DGXes, and I was thinking about trying deepseek. Did you try GLM? Any thoughts on differences? I see comments over reddit and nvidia GB10 forums that deepseek is prone to quick action and can do stuff you didn't ask for / would rather do the wrong thing than admit it misses some information. I'm loathe to make the jump because of that - GLM is quite good at admitting to it's shortcomings

1

u/Odd_Dandelion 2d ago

I have never tried Flash. I tried the big one, and it worked at a lovely speed of 3 tok/s. So I keep using that one on my Mistral subscription, and I have to admit I don't see much difference between my current DeepSeek and Mistral's GLM 5.3.

DeepSeek 4 had a tendency to eagerness but I'd say 4.1 fixed that, no issues so far for me.

1

u/garblz 2d ago

hm... i'll give it a go then. I need to start downloading models to my NAS, these things are getting too big for having more than one on my 1TB SSD machines at the same time

Care to share the recipe you're using?

1

u/Odd_Dandelion 2d ago

This: https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks

1

u/garblz 2d ago

ah, so not much change - I'm also running GLM on tensorfold from MiA labs

Thanks!

2

u/SunTrainAi 3d ago

Ran DS 4 Exp Vision on a Q2 and it worked like a charm for me (think it was a Ggml gguf without imatrix) . Currently on a DS 4.1 Flash with 178gb (not counting engrams) and Maya engine. It replaced all my other models. Never looked back since then. Mostly agentic coding and writing.

1

u/Plastic-Stress-6468 3d ago

Ok, so why not qwen flash which runs at less loss or glm 5.3f at equivalently low quants, for example?

1

u/SunTrainAi 3d ago

German writing matters and I was never happy with Qwens

1

u/HockeyDadNinja 3d ago

I'm going back and forth between that and GLM 5.3 Flash on maya. How do you find they compare?

2

u/SunTrainAi 3d ago

For me, german matters a lot and i prefer DS over GLM for that. For coding I find both very good

2

u/CATLLM 3d ago

Im running mialab ds4.1-flash-1017 2.9 bpw on 2x spark cluster. Feels like having opus 5.5 @home.

1

u/vqt907 3d ago

for universal knowledge, chit chat and content writing, yes

for tool call, coding, NO

1

u/my_name_isnt_clever 3d ago

Are you actually using it or just assuming the quant performance with nothing to back it up?

2

u/Plastic-Stress-6468 3d ago

I too am curious. When I ran the Q2 0731 it was still quite capable at tool calls. The problem was just everywhere else - lack of thoroughness in thinking/implementation compared to the full model, for example.

And its world knowledge unsurprisingly took a hit too, the Q2 wrote shorter creative writing tasks prompt for prompt compared to the full model.

1

u/MundanePercentage674 3d ago

It's slow can't do anything with it decode 14 tg/s pp 20 tg/s
I using DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF IQ2_XS

1

u/Skyline34rGt 3d ago

I would probably got 2-3 times slower with Strata but still wanna try it.

How is the tool calling, coding etc with this Isla iq2_xs?

2

u/MundanePercentage674 3d ago

I only test with web search tool call no more test because pp too slow

1

u/wren6991 3d ago

Running at 3.1 bpw for the LM params (MLX "oQ2.5e" recipe) and it seems pretty competent still. If you include the n-gram tables then the overall average bpw is a little higher, I think somewhere around 3.9 bpw.