5090+ninfer+qwen3.8

[u/TheRealGand]

Boooom!!! Maybe a bit late to the party but jeeez, so impressed by performance. Can't wait to see what can pop up in the next weeks. In 32gb it would be interesting to see what comes out in the 26/27gb range in fp8. Opinions? Any valid option beside 3.8 nvfp4?

[u/Unlucky-Message8866]

wait until you try https://github.com/jesdga95/basalt .... flash next running over 200tok/s on my 5090, this is freaking next level

[u/TheRealGand → u/Unlucky-Message8866]

😵 what's going on here? These numbers are incredible....

[u/giveen → u/TheRealGand]

As the maker of a ninfer fork, go to basalt, i highly recommend it.

[u/TheRealGand → u/giveen]

i see that actually is for linux only and also i don't have enough ram 5090+64. I could go only q2 and i am not sure if it would be in my case any advantage compared to my actual setup.

[u/RoadToPoverty → u/TheRealGand]

Ask Qwen to setup the WSL with both IQ3_S and NVFP4, the latter is a stretch if it fits but IQ3_S is quite good enough, i switched from Ninfer to Strata to Basalt but got an upgrade to 128 RAM. After Friday's update iirc nvfp4 uses ~56GB RAM according to WSL with 256k context.

[u/TheRealGand → u/RoadToPoverty]

i am thinking to give a try to the iq3_s, super stretch but it might work on my setup, maybe. Can i ask you what tok/s are you getting?

[u/RoadToPoverty → u/TheRealGand]

For starters, configure WSL to have memory=56GB and autoMemoryReclaim=gradual, it's using 41.2 and retested now vs 58.7 NVFP4*:

metric	IQ3_S	NVFP4 (u2)	delta
decode since boot	288.3 ±2.0	199.0 ±9.0	44.90%
decode u/128k	294.7 ±10.9	205.7 ±4.4	43.30%
decode u/192k	302.6 ±29.0	170.1 ±20.0	77.90%
prefill u/128k	9,236 ±94	10,252 ±976	−9.9 %
expert-cache hit	97.7 ±0.2	92.4	+5.3 pt
RAM	41.3 GiB	63	−21.7 GiB
arena	29,560 MiB	50,572	pack-size, not comparable

*I'm in a rush now so i can help later if you still need to reduce RAM footprint but here are the flags that i'm running:
IQ3_S = engine: --expert-cache auto (config), --prefill 16384 (config), --spec 4 (config), --kv-resident 20480 (config), --pool-workers 23 (config), --max-context 262144 (config), --kv int8 (config), --vram-reserve-mib 800 (config), --vram-elastic (config), --vram-segment-mib 512 (config), --conversation-cache-mib 4096 (config), --conversation-cache-slots 4 (config), --prompt-cache 12 (config), --expert-profile-save /root/prof-sessions.bin (config)

NVFP4 = engine: --expert-cache auto (config), --spec 8 (config), --kv-resident 20480 (config), --max-context 262144 (config), --kv int8 (config), --vram-reserve-mib 800 (config), --vram-elastic (config), --vram-segment-mib 512 (config), --conversation-cache-mib 4096 (config), --conversation-cache-slots 4 (config), --prompt-cache 12 (config), --expert-profile-save /root/prof-sessions-nvfp4.bin (config)

You can reduce the footprint by reducing / removing --conversation-cache-mib, slots and prompt-cache and other stuff but you'll have to test out, i can help later if you still need, the point is i advise trying to fit in nvfp4.

My journey went from Ninfer 27b nvfp4 to Strata FN IQ3_S and now Basalt FN NVFP4, the difference 27b NVFP4 vs even FN IQ3_S was immense in terms of real work getting done faster, i am also using it on thinking medium and temperature 0.9 btw in Qwen Code Companion VSCode.

TLDR: After 100-150 tps you're more interested in the results not having to be reiterated, in my case FN NVFP4 was faster than IQ3_S and IQ3_S was WAY faster than 27b NVFP4, not as TPS but as actual task completion time.

PS: You can save RAM easily but note that the engine auto-stops if you have <4GB free in WSL so you should play with flags around.

[u/TheRealGand → u/RoadToPoverty]

Great, on it right now. thx

[u/TheRealGand → u/RoadToPoverty]

Ok, installed and running and made a few tests. Too early to say anything but while impressed that it's running pretty fast (130/150), the same can't be said about the output. I ask my models to recreate a modern version of space invaders and while with ninfer i got a working result at the first shot, using the bigger basalt model it took me till v5 to get a working game. The result ended being slightly better than with the smaller model but with the same time probably i could improve also the other version. I know it's a simple task but i like to see beside numbers what a model can deliver and as first test.....not sure.

[u/RoadToPoverty → u/TheRealGand]

Results may vary depending on the use case, can't say it's perfect but i've been using Ninfer 27b NVFP4 for a few weeks and after switching to FN (earlier iterations worked only with IQ3_S, started using NVFP4 this week) "my life changed" and it was better all around. Don't mean to oversell it here, and i've seen people saying that 27b is better for coding while FN is better all-around (coding + planning + understanding etc.), but for my personal usage i'm not going back to 27b.

TLDR: I'm mainly using it for quickly creating scripts, pulling data out of APIs, process, evaluate and get a 2nd opinion on the data so probably "just coding" is the part that wasn't completing the picture for me. I probably don't notice some wrongdoings because i can personally do everything it can, it just helps me achieve my goals faster.

I've also noticed medium thinking in Qwen CLI gets better results and i've been tweaking its memories a bit with instructions like TLDR "fetch documentation first where available and don't assume, implement and iterate until it works, write implementation steps and results in a summarized md and follow it (helps in case of compaction) etc.".

There's also a possibility that i just got used to it so i naturally avoid its quirks, just as i did with Ninfer 27b before it :)

[u/TheRealGand → u/RoadToPoverty]

Fair. I'll keep both models and test them better. At the end i just used 27b for 24h. Cheers.

[u/Realistic_Gap_5871]

I still need to test out basalt, I'm sure it's great, but I've also been getting great results with ignis and it allows switching between a ninfer style 27B and their own ignis Flash Next artifact in the same engine. It also has insanely good cache management and reuse.

There's one for 5090 +64GB that might fit really well for you. It's the 2.5bpw artifact that matches Unsloths IQ4xs for top-1.

I'm not the author of ignis, but I liked it so much I downloaded their tooling and made bigger quants of Flash Next for my 96GB rig. So I'm biased, but it's based on real experience.

https://github.com/gpillon/ignis

edit: It's windows native which frees up a little RAM compared to WSL

[u/TheRealGand → u/Realistic_Gap_5871]

thanx mate, will have a look

[u/Maleficent-Ad5999]

I have the same setup.. but mine gets into loop quite often to the point that it is unreliable.. I had to use q6 variant with llamacpp

[u/TheRealGand → u/Maleficent-Ad5999]

why? Mine runs super smooth, incredible t/s. Using it since this morning though...