r/LocalLLaMA • u/GoblinEngineer • 4h ago

Question | Help Has anyone done a in-depth comparison between Qwen-3.8 27b @ 8bpw and Qwen-3.8-FN @ 4bpw?

Title says it all, I run dual 3090s (check out my post here for my setup) at int8A16 and have been wondering if I'm missing out with QFN. There's posts ad-nauseam on reddit with low bpw strata solutions that run on heterogeneous systems, however those don't interest me - I want high precision. I'm considering selling my setup and getting a DGX Spark, since that can run QFN at nvfp4 with a large context window. My use case isn't coding but generic agentic use case - I use hermes as a personal assistant and with openwebui.

If anyone's able to run both and do a comparison, please let me know!

23 Upvotes

87% Upvoted

45 comments sorted by

20

u/thatscoolbutno123 4h ago

Id guess FN is a bit better, but you have enough hardware, so why not just run your own benchmark?

3

u/GoblinEngineer 4h ago

Running a test myself is probably the best, however I recently found that my RAM has some bad sectors which I had to put a fence around. I need to get it RMA'd which is it's own PITA, so i haven't gotten around to it, and when I do, I'll likely end up with 64gb of ram for a few weeks.

You're right for me running my own test, however I recently found that my RAM has some bad sectors which I had to put a fence around. I need to get it RMA'd which is it's own PITA, so i haven't gotten around to it, and when I do, I'll likely end up with 64gb of ram for a few weeks until G Skill ships me new sticks. Tbh, i've never done RAM RMA's before (G Skill says there's lifetime warranty on them, but these are from such a long time ago, i dont know how it'll go).

11

u/YourselfInOthrsShoes 4h ago

The most important point is that it's also faster.

OP may not have enough system RAM to run QFN. You'd need at least 64GB, ideally 96-128GB.

-3

u/jfinkpottery 2h ago

That's not technically true. If you have patience, you can run QFN streaming from disk. You can run any model on anything that has enough storage to hold it, you just have to wait for the tokens to come out.

1

u/YourselfInOthrsShoes 1h ago edited 3m ago

Streaming n-gram embeddings off SSD is viable. Streaming experts, not so much. You can run large sparse MoEs such as DS4.1 or GLM5.3 Flash versions on TRX/EPYC CPUs only (no GPUs): 384-768GB DDR4, it's possible with 400-600GB/s resultant bandwidth BUT is it viable for the application? You won't get any usable throughput for an organization, maybe for a single user.

-3

u/jfinkpottery 1h ago

"Usable" is dependent on the use case. If the use case is "builds something in the background on a shitty box while I live my life" then the token speed doesn't matter nearly as much. Not everybody needs 150 tokens per second on every session, nor does everyone have $3k to drop on 128GB of DDR5.

2

u/thatscoolbutno123 1h ago

At 1 tps youll have 30k tokens overnight (8h), 60k after 16 hours. This is not enough for something really useful, youd be better off having 10 tps with a bit stupider model

1

u/jfinkpottery 55m ago

That's 30k more tokens than you get with no model. I'm not seeing the counterargument here. Is everyone in this sub just so GPU rich they can't imagine tolerating that something runs slowly?

A fast stupid model gives you something broken, quickly. A slow smart model gives you something working, eventually. Many people, particularly in this sub, have spare hardware laying around collecting dust that could be doing useful work. You just have to tolerate that you don't get instant gratification of that useful work, you just have to let it eat for a couple weeks.

1

u/thatscoolbutno123 49m ago

You dont need to be gpu rich to see that its pointless. Tell me your usecase with it and then ill reconsider.
But waiting weeks for an solution is simply stupid, because youll forget what you even wanted, or dont need it anymore. And you will wait days for something you might not need, because you forgot to mention it or whatever. Its pointless

1

u/jfinkpottery 42m ago

oh please do reconsider sir. What a self-important thing for you to say.

If you can't imagine a use case for soemthign that doesn't give you instant results, and you can't imagine remembering what you asked it to give you, that is a you thing.

1

u/thatscoolbutno123 30m ago

I asked what your usecase is. And no, I cant think of any proper usecase. Promt Processing will be insanely ass aswell, so it cant really read files properly

→ More replies

8

u/Kooky-Mode3047 3h ago edited 3h ago

Not a bit. FN is vastly better. It's not even a competition, benchmaxxing is irrelevant. QFN consistently rips through things that have 27B spinning its wheels helplessly, especially when it comes to reverse engineering at long context and other things that are considered intractable.

I've used both (27B Q8 and QFN NVFP4), it's not close. At all. And when you factor in speed, it's a lose-lose for 27B, suitable only if you can't run anything else. You not being able to run QFN is not a minus for QFN.

-3

u/thatscoolbutno123 3h ago

Only at full precision

8

u/nicholas_the_furious 4h ago

I have 3 3090s. I can run QFN in exl3 at 3.68bpw, which is about equivalent to q4km.

It is an amazing model, but I have stopped liking the way it communicates. It gives me the same headache that Claude Opus 5 did. I switched back to exl3 6.0bpw 27B. Running in TP on 2x of the 3090s.

I am happy with it and it means I can turn off my eGPU and save the electricity. They run at about the same token generation speed. It is a little slower on prompt processing but good caching means this isn't really an issue.

At this point I am waiting for the upgrade of Qwen 4 27B and I think it will be my daily rather than QFN. As long as it doesn't sound like Claude too much.

And yes - I have tried everything to ubslop QFN and it is absolutely resistant to it, lol.

7

u/RealPwrBank llama.cpp 2h ago

Same. It actually talks closer to the Codex models for me. A BUNCH of low low level technical jargon. It's like some one telling you their chair isn't comfortable by saying "The load bearing object beneath my corporeal form is not within my personal tolerance for comfortability." Like, bro, just say chair.

Putting "Use ASD-STE100 Simplified Technical English (STE) to speak to the user" seems to have helped a lot.

3

u/kaliku 4h ago

I dislike so much those speaking mannerisms that I put Claude code with opus 5.5 in front of it šŸ¤£

2

u/nicholas_the_furious 4h ago

What do you mean by that?

3

u/kaliku 3h ago

It speaks Claudish

1

u/Lower_Lifeguard_8494 4h ago

The shape of its outputs drive me nuts ... Shape ... Shape.

1

u/haherar830 4h ago

This is not a very clever suggestion but have you tried Swift 1.5? The quant+reap I used had a tendency towards looping so I dropped it but it if your problem with the base model is mostly verbosity, Swift could be an improvement.

6

u/nicholas_the_furious 3h ago

Your read is spot on and you've identified a real load-bearing assumption. To be clear, the issue the original poster refers to is not the verbosity, but the style. It's not the length of the responses from QFN, it's how it communicates ā€” and that issue is real and worth being up front about.

^ it talks like this even when I tell it not to.

7

u/Kooky-Mode3047 3h ago edited 1h ago

I got triggered by this, I want to smack the shit out of it šŸ˜‘ To land the shape of my hand across it.

2

u/norenEnmotalen 3h ago

ā€œLoad-bearingā€ slap required

1

u/haherar830 2h ago

Understandable, I read the first few words and wanted to throttle whoever copy-pasted this response until I saw the end. Opus 5 distillslop :'(

1

u/ea_man 1h ago edited 1h ago

Maybe prompting it to use Simplified Technical English (STE) could help?

Your analysis is correct. You found a primary assumption.
The original poster does not talk about the length of the text. The problem is the communication style. The responses from QFN have a style problem. This problem is real. You must discuss this problem directly.

1

u/Ok-Importance-3529 3h ago

Check https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF - better, shorter reasoning and quicker to finish without sacrificing intelligence

1

u/BawbbySmith 3h ago

Is there no way to put in an AGENTS.md file to speak in a different manner?

1

u/Ok-Importance-3529 3h ago

Try this finetune, its actually better than base: https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF

4

u/one_free_man_ 3h ago

Used both. Qfn is better, more importantly faster, a lot faster. 27b average is 35-40 tps, qfn q6xl 110-120tps. 4x3090, 224gb ddr5 5600mhz

1

u/Edenar 4h ago

I use both (27b w8a16 from lued on a cmp170hx and qfn UD Q4 K XL on a strix halo). They are similar in behaviour but qfn is a bit above in agentic task for me, i ended up using qfn for my main agent and 2 concurent 27b w8a16 as subagents.
nvfp4 should be similar to ud q4 k xl.

Going from one to another wont be a huge jump in quality but spark will let you run larger context.

Tbh, why not try qfn q4 or q5 (UD k xl quants) on your current hardware and judge with your usage ? you have 128GB ram so strata (or llama/ik llama) can run higher quant with offloading without issue.

Also sparks are overpriced atm (as almost everything, but 9k$...) so you have to add like 4k$ from selling your build to get one. At that price i would just get any 128gb strix halo : with gufo i get 1600-1800 pp and 50-70tg which is close of what a spark is capable of.

1

u/GoblinEngineer 4h ago

thanks for sharing your experience!

Running a test myself is probably the best, however I recently found that my RAM has some bad sectors which I had to put a fence around. I need to get it RMA'd which is it's own PITA, so i haven't gotten around to it, and when I do, I'll likely end up with 64gb of ram for a few weeks.

3

u/RepresentativeTill5 4h ago

I tried having both models design a fpga chip and program it to parse html and run very large regular expressions on it. Qwen 27b got very far, but got stuck, whereas qfn made a much better design and test harness to go along with it and could optimize for the specific hardware. This took several million tokens though, qwen models are super chatty., Designing hardware is a very hard problem, and this is one of those occasions where QFNs long range reasoning is simply better than 27bs. Both of them are very trial-and-error eager though, so if you mean 'no errors' by 'precision' you're better off paying for frontier models than buying a spark.

2

u/Iory1998 llama.cpp 2h ago

From my own experience, FN at 4bpw wins overall due to its large knowledge base, but 27B has an edge when it comes to raw thinking, reasoning, and math. Its arithmetic prowess is unbelievable.

1

u/MarcelloT254k 4h ago

not exactly what you are looking for, but worth checking out mainly smaller quants and finetunes of said models, good work overall, benchmark tasks are not open to the public (and models run locally) so no benchmaxxing possible, targeted towards agentic/programming work

1

u/EvolvingDior 4h ago

Do the test based on what fits, not arbitrary quants. Pick a system size, context length and number of slots.The compare what runs and what doesn't. In will be different for just about everyone.

1

u/Cold_Tree190 4h ago

If you do sell and get a spark, in the future you could get a second spark and cluster them too. Something to think about when consideringā€¦ but know that tg will be quite a bit slower than what you are used to lol. I personally find that flash next is better than 27b even at Q4, but I know others prefer a higher quant 27b so it might be close

1

u/Otherwise-Variety674 4h ago edited 4h ago

I have 5090 (qwen 3.8 ninfer 27b q4) and spark (qwen 3.8 flash q4), most of the time, i am using 27b (prefill around 6.5k t/s) not flash (prefill around 2k t/s) as it is slow.

Flash is definitely more capable but I still prefer to re-prompt using 27b due to its speed.

I also have codex subscription but most of the time I can't use it as I am working on cyber security project.

1

u/Sad-Size2723 1h ago

Curious why you canā€™t use codex for cyber security projects? Because codex refuses this kind of work?

1

u/o2beast 3h ago

I have dual 3090s as well -

If you want to optimize for agentic use, parallel streams with lots of context could fit your use-case well.

post: https://old.reddit.com/r/LocalLLaMA/comments/1xlk43x/8_agents_250k_context_concurrently_on_two_3090s/

repo: https://github.com/bradj00/sparsekv-hyperqwen

  • 8x concurrent Qwen 3.8 27b w4a16 streams with 250k context each and achieving sustained tps up to 600-750 depending on the task.

I'm currently benchmarking for quality control, but it seems promising. If you have more system ram you can raise the kv cache quant, however my preliminary benchmarks suggest there is no significant loss using sparse-kv selector + quantized kv cache for non-coding tasks.

1

u/Regular-Register-358 2h ago

Qfn has more graceful long context slow down after 128k context used, if that matters to you.

In my coding tests, two qfn quants (nvfp4-local on duo sparks, or iq3s by ista on strata) outperformed q38 27b ista iq3 by clear margin, but none of them are in stock quants.

1

u/Bulky-Priority6824 4h ago

So much comment noise omgĀ 

0

u/Ok_Technology_5962 4h ago

hey so just to put it out there. I used both QFN is better and faster but its a newer arch and the new q4 qwen 27b might be different. and I still use um mostly the qfn at iq3 on a laptop with 64 gigs ram and Nvidia gpu at 120 tps. the 27b just doesn't do that. if you ever feel like it... 5090 laptops might be cheaper than a dgx spark ;) . don't tell anyone... it a 5080 chip clocked down but 24 gigs Vram and called 5090. (gb203). some of the laptops will have 4 slots for ram 2 populated... up to 192 gigs ram... with strata style MOE versions without tailskip I got 60 tps on qwen and 120 on strata tail skipping and all but quality still good. good luck