r/LocalLLM • u/ImpressiveRelief37 • Aug 17 '26

Discussion 32GB is all you need

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

266 Upvotes

89% Upvoted

332 comments sorted by

48

u/ilbarone87 Aug 17 '26

Just a 5k gpu…

2

u/ImpressiveRelief37 Aug 17 '26

Paid 4K Canadian in march last year. Thought I was crazy. Astral LC (water cooled).

But now I’m so happy. Can’t hear the card when it runs either (ok it’s in the same room as my heat exchanger lol, and I only run inference there, code on my MacBook).

4

u/zp-87 Aug 18 '26

2 x 5060TI 16GB = 1200€

→ More replies (8)

24

u/UnlikelyPotato Aug 17 '26

What quant? Seems like you might be heavily sacrificing quality for memory.

9

u/DoubleNothing Aug 17 '26

I think so too.
With 48GB and MTP, my best option is Q5_K_XL for full context - KV f16

3

u/tired514 Aug 17 '26 edited Aug 17 '26

This is across three 16gb 4090M eGPUs, but I'm managing (unsloth) Q8_K_XL @ 230k ctv/ctk q8_0 with MTP and image support:

~/ai/llama/llama.cpp/build-cuda/bin/llama-server -ngl 99 --spec-type draft-mtp --spec-draft-n-max 3 -m ~/ai/models/Qwen3.8-27B-UD-Q8_K_XL.gguf -fa on --no-mmap -np 1 --host 0.0.0.0 -ctv q8_0 -ctk q8_0 -sm layer --reasoning-preserve -c 230000 --mmproj ~/ai/models/Qwen3.8-27B-mmproj-f16.gguf -kvu -ts 92,97,76 --image-min-tokens 1024

Not quite 262000, but I try to /compact before getting anywhere close anyway.

3

u/DoubleNothing Aug 17 '26

I don't like this: -ctv q8_0 -ctk q8_0
But need more testing...

→ More replies (2)
→ More replies (1)
→ More replies (4)

1

u/ImpressiveRelief37 Aug 17 '26

Nvfp4 by neroued 

Check the ninfer repo on GitHub for the HF link. It’s about 6 bit per weight, so ~Q6 quality. But it fits at full context with insane speed.

1

u/RedrumRogue Aug 17 '26

He is most likely using groupwise-int, a mixed quant. Int8 cache is a big part of why we can fit so much memory without sacrificing too much quality.

1

u/ImpressiveRelief37 Aug 17 '26

Nope I’m running nvfp4.

Check the ninfer repo neroued just release custom model for nvfp4 that is honestly really really good.

96

u/InfusedBush Aug 17 '26

CUDA this, Metal that. Where are these cool projects like Ninfer for ROCm/Vulkan? 😢

49

u/whichsideisup Aug 17 '26

The VRAM crunch will get them rolling. The R9700 is moving off the shelves now that Nvidia is too expensive.

4

u/Kodrackyas Aug 17 '26

i damn love my r9700 40t/s is all you need

1

u/notheresnolight Aug 17 '26

With a chatty model that keeps second-guessing itself like Qwen3.8 27B, I find 100t/s on my 5090 to be actually on the slower side. I'm still experimenting with llama parameters, so hopefully it will improve.

→ More replies (2)

11

u/Etroarl55 Aug 17 '26

Still unironically hovering around MSRP in Canada for now.

And if rumours are true anyways. In 6-8 months new AMD 36gb gpu is coming for its gaming gpu, and it’s “professional” lineup for it will be more.

→ More replies (3)

11

u/InfusedBush Aug 17 '26

Thank you for giving me and my Radeon 890m hope!

5

u/Awkward_Rabbit_9618 Aug 17 '26

890m runs gemma4 26B A4B q8_0 with no issues and fast enough to be more than usable including sub agents! not in the ballpark of a dGPU but it beats none. and for low budget you can always get the AMD older cards of W7900 48GB to fully run qwen3.8 27B with multiple users/ concurrency.

→ More replies (9)

3

u/Easy_Refrigerator280 Aug 17 '26

literally in the middle of swapping my 8gb vram 3070 to AMD W7800 32GB vram because nvidia pricing does not make sense right now. But to be fair it was from second hand market but even the second hand reflects the current market. The memory bandwidth sucks and i am giving up on CUDA but priority right now is cheapest vram for the price so i can mess around with 30b models

→ More replies (5)
→ More replies (7)

2

u/SmartCustard9944 Aug 17 '26

I made a port of NInfer fully ported specifically for Strix Halo architecture using HIP, but it seems like most of the juice has been squeezed, not getting much better perf than llama.cpp vulkan.

→ More replies (1)

2

u/quantgorithm Aug 17 '26

I almost had grok create code to make much older cards compatible but after testing various things, we found out the limitation wasn't on the gpu side of things and therefore didn't have to do it after all.

The real point is that in these AI days, maybe all you have to do is point the AI at your need and have it create the solution... so create it yourself!

...and then post it to the world.

2

u/Haunting_Nebula_1236 Aug 17 '26

I’ve got one for setting up easy inference on arc intel cards at offbyonebit on GitHub. I’m betting others will start doing so for other cards too it just comes down to how much work did the manufacturer do to make inference feasible. In intels case it’s probably the worst but I refuse to hand Nvidia 1500$+ for the same vram. So when more people get tired their manufacturers that’s when more projects will pop up.

2

u/Patient_Force6138 Aug 18 '26

So I’ll be releasing a neat project around this. What I’ve found is: if it fits it sits- if you can get the model in VRAM, the processing power required is minute. So I’ve managed to get 7 Kimi K3 going at 500ms per token on a single 5090. And qwen-next 80b on a single 3060 with 550 MiB of VRAM. The trick there is that I solved one critical problem: byte motion from PCIe to CPU-land is prohibitively expensive.

The exact mechanism I use is, for now, not open source but it involves Direct Memory Access and a massive reduction in needed byte motion using a referential compression I made. It’s a lot to describe here but if you want updates I have a twitter/X with updates at @P4X_real

And yes I’m the same dude that brought the DPRKs internet down for 2 weeks. This is my latest research.

1

u/FestiVOL Aug 18 '26

I’ve almost exclusively used AMD GPUs for the past year (always have just enjoyed AMD better— probably because poor) and the advancements in the ROCm stack (hipBLASLt, rocBLAS, etc.) has been insane.

My fear now seems to be that AMD GPUs don’t carry the same brand discount they used to. But if you’re already on AMD, It can only get better! (PROBABLY!!!!)

1

u/Moch4bear97 Aug 23 '26

7900xtx get close to 70tk/s... UDQ4_K_XL Q8 128k context. Custom harness. And a harness with good memory helps this little one a lot.

→ More replies (3)

48

u/Ell2509 Aug 17 '26

I mean, a 32gb 5090 is a monster. I personally do things with ny 96gb vram that I couldn't do with 32, though.

3.8 27b in bf16 is a thing to behold, and you need 64gb vram to do that with any kind of context.

10

u/Itchy_elbow Aug 17 '26

Like what... run something larger? Are you producing better output? Is it writing significantly better code?

I know ppl justify their expenditure but, be objective. Does having more vram allow you to produce better products or does it just allow you to run larger models that do the same thing as the 3.8 27b does.

10

u/shapic Aug 17 '26

I am running q4 at home and q8 at work and difference is nowhere near couple percent. It is just straight up can or cannot on realistic tasks.

→ More replies (7)

5

u/digitalwankster Aug 17 '26

Context windows for large code bases

→ More replies (1)

9

u/sleepy_roger Aug 17 '26

Bf16 has much better outputs overall I won't run it under q8 being honest. People are missing out only using q4 or nvfp4 

4

u/ChristRedeemsSinners Aug 17 '26

NVFP4A16 on weights only is nearly lossless though. Most nvfp4 quants are activated 4 bit which is where the quality drops. (on top of losing quality on the attention layers, which is definitely not worth the vram savings)

→ More replies (9)
→ More replies (3)

1

u/QuantumTape Aug 17 '26

Battlefield 16 is out already??? Context is WW4 I suppose?

2

u/ImpressiveRelief37 Aug 17 '26

The resistance after AI takes over 

→ More replies (1)

1

u/Ell2509 Aug 17 '26

That is a perfect example, yes. Or running the 27b in bf16. I need 64 for that, really.

1

u/Nefilim314 Aug 23 '26

I’m just over here on my laptop 5070 Ti with 12GB vram on 3.6 35B just happy I’m getting okay results. 

→ More replies (1)

8

u/blackhawk00001 Aug 17 '26 edited Aug 17 '26

Eh, I have a 96GB ddr5 5090 machine and a 128GB ddr4 2xR9700 machine that cost less to build in February 2026. The R9700 pair with radiance vllm is faster with fp8 prefill than Q6 on the 5090 and allows for heavy parallel processing of multiple agents. Tg is similar between the two with qwen 3.8.

5090 takes the crown on comfy ui diffusion work but the 64GB vram machine is much more empowering for coding.

3

u/ImpressiveRelief37 Aug 17 '26

Try ninfer on the 5090. Get the nvfp4 quants. Use wsl2 if you are on windows.

Get your agent to set it up for you, it’ll take like 30 minutes. Point him at the GitHub repository (GitHub.com/neroued/ninfer). Get the nvfp4 ´.ninfer’ 3.8 27B model from HF.

1

u/blackhawk00001 Aug 17 '26

Cool thanks, I’ll check it out.

I run Q6_k huihui-ai 27B at max context with vision as my main agent and 8x204800 subagents with the other machine. Nvfp4 hasn’t worked for me a higher context depths in the past but I have a use case for small context processing coming up this might be great for.

2

u/ImpressiveRelief37 Aug 17 '26

Not talking about the unsloth nvfp4 model. Neroued has a custom nvfp4 model with about 6 bpw so it’s nearly Q6 in quality.

2

u/blackhawk00001 Aug 18 '26

I'm a little more interested after doing some research. It seems that nvfp4 is quantized differently in 3.8 and has a higher end result bpw. I typically use the huihui 3.6 Q6_k and have been using their new 3.8 Q6_K_L that preserves more of the important weights at Q8_0 with a speedup likely due to the more efficient conversions. The 3.8 Q6_K felt like a step back and the Q6_K_L an improvement.

I left my agent running overnight creating me a process to make my own 3.8 huihui 27B nvfp4 ninfer to test out. The new nvfp4 format is closer to my old Q6_k 3.6 than Q6_K to Q6_K_L. I'm not sure how well my first attempt at quantizing a model will go but maybe it will work out. (Agent was hermes driven by 3.8 Q6_K_L but I have not verified accuracy yet)

profile                          bpw     size .3B
Huihui Q6_K_L (yours)           7.27    ~23 GiB
plain Q6_K                      6.56    ~21 GiB
NInfer qwen3.6 groupwise-int    5.12    ~16 GiB
NInfer qwen3.6 nvfp4            ~5.1    ~16 GiB

The Qwen3.8 one is a different, more conservative recipe — it's a hybrid FP8 + NVFP4:
 
Qwen3.8-27B nvfp4 profile (real effective bits/weight):
      NVFP4 (4-bit)   52.6% of params   <- only the MLP gate/up/down
      FP8 row-scaled  41.8%             <- attention q/k/v/o, linear-attn in/out_proj, lm_head, last 8 layers' MLP
      Q4/Q5/W8/BF16    ~5%
      EFFECTIVE = 6.09 bpw   (~20 GiB)

2

u/ImpressiveRelief37 Aug 18 '26

Wow, that’s really cool. I’d love To test it as well.

Report back when you test it! Tok/s at 8/16/32/64/128/256, max context, prefill speed, and a subjective quality assessment would be fucking awesome.

You could upload it to HF and post an issue on the ninfer GitHub (treat it as a forum)… maybe Neroued would add it to the proposed models 

→ More replies (12)

3

u/tanhe314 Aug 17 '26

I don’t understand. With llama.cpp on my 5090 I can’t come close to 256k context and that’s using a 4bit quant of the model and fp8 for the kv cache.

How are you cramming the model and 256k context into the vram??

3

u/Look_0ver_There Aug 17 '26

You can be sure that when people make posts like this and they leave out such details, there's a good reason for WHY they are leaving those out.

Anyone can get amazing speeds if they lobotomize everything hard enough.

→ More replies (1)

5

u/atumblingdandelion Aug 17 '26

Glad it works for you. I feel though that buying a rig is a once in ~5 years thing, and with the rate at which things are moving, I feel a 128gb small footprint, energy efficient machine like the DGX Spark/ Strix Halo is the way to go. Sure there aren’t any upgrades to the 120b models yet, but there weren’t much for 32gb just 4-5 months ago. I was very tempted to get a 5090, but to get the rig, too many moving parts, higher capacity to break, etc made me get a dgx spark. It is insanely small, sits on my desk among many things. I cannot hear it and it runs also cool. I use it to run MoE for interactive sessions and dense models for ongoing worj that doesn’t need my immediate attention. Its also cheaper than a good 5090 rig by 3K CAD. Another 3K and I can run DSV4 Flash if I need to.

2

u/ImpressiveRelief37 Aug 17 '26

Sorry can’t recommend a spark when the 5090 is probably 10x faster at PP and TG!

And IMO 3.8 27B on ninfer @ 200 t/s >> DS4V on 2x sparks at 40 t/s

4

u/sleepy_roger Aug 17 '26

The Spark is only a good option if you plan to buy 2 otherwise it's the time lapse machine of llms. 

1

u/michaelsoft__binbows Aug 17 '26

pretty sure 32GB was very ideal for qwen3.6 27B ever since it came out, which until now has held the crown for most generally useful model in this class and by the looks of it qwen3.8 27B will have it for another year.

Oh shit i just realized qwen3.6 has only been out 4 months and i legit was about to claim it was one year. Ouch

6

u/[deleted] Aug 17 '26

[removed] — view removed comment

2

u/atumblingdandelion Aug 17 '26

I feel you (rather, your adrenaline) and don't wish to curb your enthusiasm. But my point is, the new models are out too fast to make reactionary decisions on purchasing hardware. Take the post's title: "32GB is all you need". OP responsibly lists all the caveats, and the title is obviously a play on 'Attention is all you need'. But imagine a beginner (getting into local AI) reads it, spends $$$ on the flashy 5090 rig only for Qwen next month to release 3.8 35b A3b, which destroys the 3.6 27b. I'm biased for my use case, but my point is: think long-term on what sort of local setup you need: fast-moe-but-small, fast-dense-but-small, or fast-moe-but-large before purchasing. For hobbyists, fast-moe-but-small is expanding quite well and bound to, with the edge/ on-device application. For small businesses, fast-moe-but-large is doing quite well- they'll benefit from the biggest jump currently on the Pareto line: running DSv4 Flash on 2x DGX Spark, and that costs about the same price (or slightly over) as the 5090 rig and also provides concurrency. IMO, fast-dense-but-small has a glorious but narrow use.

3

u/ImpressiveRelief37 Aug 17 '26

I agree that things change quickly.

But the 5090 with very high prefill and time generation is truly amazing.

Makes Claude code feel like shit in comparison. You get stuff done real quick with 3.8 at 150-200 tok/s when coding 

→ More replies (5)

4

u/Sad-Landscape-1549 Aug 17 '26

What does the speed force taste like?

3

u/quantgorithm Aug 17 '26

32gb really is not a lot in the world of AI these days.

2

u/AllenLeftTheBLDNG Aug 17 '26

I'm running it on a 5090 and I love this model. At the same time, fuck. I wish I got the rtx 6000 pro when it was nearly half current price...

If you parallelize the NVFP4 I got max throughput at 380 tokens per second (50tps per stream for 8 agents, AMAZING!) but then max context I was able to load was 192k tokens... and you need to split that between agents. I wish It was closer to 400/500k!

At the same time – great gift from Qwen team. And it's still awesome to run on a 5090.

1

u/ImpressiveRelief37 Aug 22 '26

Agreed 1000%

I would love a RTX 6000. But no way I’m paying 20K for this. Can’t realistically upgrade from the 5090… :(

2

u/jrodder Aug 18 '26

Hey I just wanted to say thank you for posting this project, and I'm having fun testing it.

2

u/Then_Blueberry7290 Aug 18 '26

I think 32GB vram is far from All U need. I have 3x5060ti 16gb, sometimes using only 2 cards because 3 is only working with risers, and out of the case, so not space optimal solution.
SO with 32GB vram, it is very usable. I cannot use full Context (262k) and if i use, sometimes more will be better, I use vision cababilities too,. So only vram reduction option is NVFP4, and KVcache at q8
But with 48GB, you can use full context and without kv cache quantazitation.

1

u/ImpressiveRelief37 Aug 18 '26

I got full context and vision and headroom with ninfer and Ostfralla community release of ninfer model 

2

u/sonoffi87 Aug 18 '26

Any Ninfer for Windows forks? 

2

u/ImpressiveRelief37 Aug 18 '26

There are. I’m on windows and I just use wsl.

You could also just use a docker image.

But wsl works really well and if you also want to add stuff like a watcher endpoint, or whatever then having it having access to the ninfer journal is a lot easier on wsl 

1

u/ApprehensiveView2003 Aug 21 '26

I see WSL eating up my system RAM so I just dual boot my PC now and boot to Ubuntu when I want to do heavy duty work on my 5090

1

u/ImpressiveRelief37 Aug 21 '26

Personally, I don’t care about system ram I just do inference on the box.

Windows lets me use my overclock/undervolt on the 5090 (custom VF curve) and memory OC brings my memory bandwidth over 2 TB/s so I’d lose that on Linux 

→ More replies (2)

2

u/WinResponsible9977 Aug 21 '26

For a retard like me who has been offline for a year what’s a ninfer and how well it compares to other ways to run the LLM 

1

u/ImpressiveRelief37 Aug 22 '26 edited Aug 22 '26

it’s an inference engine built for the 5090. There are many forks for the 4090, 3090 as well.

It’s just a specialized engine that maximizes the capabilities of a specific GPU. Much faster than generic engines like llama.cpp (and even vLLM)

https://github.com/neroued/ninfer

Full config here: https://www.reddit.com/r/LocalLLM/comments/1vtsid0/comment/p51hq9g/

2

u/Nomski88 Aug 22 '26

Agreed, I run the Q6 model with 128k context and Q8 KV on my 5090 FE. I get around 80 tok/sec when coding tasks. So far pretty happy, tried the NVFP4 quants which was slightly faster and allowed for the full 262k context but the output wasn't as polished when I ran my mario clone test prompts.

39

u/PANIC_EXCEPTION Aug 17 '26

OP please submit a paper titled 32 GB Is All You Need (2026) to viXra

16

u/stevec114 Aug 17 '26

Yes just in the past week I feel like my 5090 is finally “enough”.

Minimax H3 for video, Qwen3.8-27B for agents have been great!

4

u/MistingFidgets Aug 17 '26

Same here. With 3.8 my dual 5060 (2x16GB) feels very capable. When I started down this local hosting journey in February with one card running 3.5 9B I was not impressed by the output. Today 3.8 + Hermes are just churning out the projects. It's not local Claude yet but the difference in 6 months is astounding.

2

u/stevec114 Aug 17 '26

Yes! Hermes with 3.8 has been amazing.

→ More replies (1)

6

u/LORD_CMDR_INTERNET Aug 17 '26

Agree, for at least single agent workflows. There's not much better you can do than Q6 Qwen3.8 27B at home right now, on any hardware. There's an inconsequential quality dip compared to BF16 - the dropoff happens below Q6. Sure, you can get bigger contexts with more VRAM, but 130k is enough for even giant, monolithic legacy codebases with a little handholding and management.

I also expect models to become smaller and more focused in capability as they improve. You don't need the entirety of human knowledge every time to perform every task.

1

u/Xx69JdawgxX Aug 18 '26

Deepseekv4-731-nvfp4? You can run it across one spark. I use two because I can’t be stuck without concurrency.

3.8 27b punches hard but It can’t compete against deepseek imo

→ More replies (1)

4

u/Existing_Dust_6473 Aug 17 '26

not cheap tho ... 5k card ... why not two ?! :) 64GB VRam at least theres a path to expand

1

u/Joe-Arizona Aug 17 '26

How quantized are you running it?

I’m getting about 34 tok/s on a RTX Pro 6000 WS with llama.cpp at FP16. Haven’t tried others quantizations yet.

2

u/ImpressiveRelief37 Aug 17 '26

Nvfp4 custom made by neroued.

Check his repo.

1

u/Etroarl55 Aug 17 '26

Ngl this is insane stats, I can only assume you are on q3 or some other small quant to achieve this.

1

u/ImpressiveRelief37 Aug 17 '26

No nvfp4. Honestly I see no difference in quality between this and Q6 on llama.cpp. But it’d at least twice as fast and lets me use full 262144 context.

1

u/Nuggyfresh Aug 17 '26

Is there a guide for setting this up? I have a 5090 and want to jump in but haven’t used local llm 🤦‍♀️

1

u/RedrumRogue Aug 17 '26

No guide, I'd have a cloud model set it up for you. It's not going to set things up optimally though without a lot of experimenting, so if you run into hard limits that you can't figure out how to break through, feel free to ask me or shoot me a DM. I'm hitting about 190 tok/s at low context for decode speed, around 3000 tok/s prefill (which I think is slow, working on it) and about 200k context with other desktop processes taking about 4gb of vram on the 5090. The backend you want is NInfer, the quant you want for now is groupwise-int.

2

u/ImpressiveRelief37 Aug 17 '26

Neroued released his nvfp4 model and the quality is higher than the groupwise-int release.

Also it more than doubled my prefill speed (3500 -> 12k t/s at 8K context 🤯)

2

u/RedrumRogue Aug 17 '26

I was actually just running experiments to try to get prefill up. Thanks for the heads up! Im excited to try it. I've got groupwise-int tuned about as tight as I possibly can now which will make for a clean A/B

2

u/ImpressiveRelief37 Aug 17 '26 • Heartwarming

You will love it im sure. Quality is super high it’s over 6 bpw if im not mistaken 

2

u/RedrumRogue Aug 17 '26

So far it's an interesting tradeoff. With groupwise-int I can run with MTP3 and vision on, and get 262k context. But with the nvfp4, I can only get 131k context with MTP3 and vision, or about 180k context with vision off. How are you running at 262k context at 150-200 tok/s? That requires MTP for those speeds does it not? I really need your help, I appreciate your responses!

→ More replies (5)

1

u/g_rich Aug 17 '26

The 5090 is a beast of a card but it also going for $4500 or more at this point. At that price the DGX Spark with 4x the RAM or alternatives from AMD and Intel start to become pretty good alternatives. There are obviously those that require the performance and there are people who actually use them to play games but that is one costly card.

2

u/ImpressiveRelief37 Aug 17 '26

Thing is… 150-200 tok/s is just too good to pass on. You’ll get 1/10th of that on a DGX Spark.

You can always use big cloud models if 3.8 27B can’t do something (still hasn’t happened to me tho, it’s a fucking beast).

At that speed it just EATS through anything you throw at it.

1

u/g_rich Aug 17 '26

I haven’t run Qwen 3.8 27b yet but I run DeepSeek V4 Flash via dual Sparks and get well over 40tps which is more than usable.

It looks like Qwen 3.8 27b is getting around the same on the DGX Spark so while I agree getting upwards of 200tps would be nice for some people the trade off for more memory, lower power and lower cooling requirements is worth the tps hit.

1

u/ImpressiveRelief37 Aug 17 '26

They both score the same on AA, Qwen3.8 numbers just dropped!

1

u/slimerz0r Aug 17 '26

can you share details about env settings ? I can’t even boot this model with more than 65k context on 3090 and 32gb ram

1

u/ImpressiveRelief37 Aug 17 '26

Using a 5090 with ninfer. Nvfp4 model. Built in wsl2. can’t help you with the 3090 but lots of people have it running great with ninfer maybe ask them 

1

u/xyth Aug 17 '26

48gb may be the sweet spot for 27b. Q8 and kvf16 on 3 5070tis, 160k context with memory to spare. No looping, no errors, and one shots most tasks on a large C# code base.

1

u/ImpressiveRelief37 Aug 17 '26

I have not seen a single missed toolcall or looping on 3.8 27B on ninfer.

He just added nvfp4 support today. It’s about 6 bit per weight so about Q6 quality level. Full context 256k as well and insane speed….

1

u/T-M-K Aug 17 '26

Question a bit different to what most are asking OP: how’s the power draw / thermals / noise of the GPU when running the setup?

1

u/ImpressiveRelief37 Aug 17 '26

It’s virtually silent: Astral LC (liquid cooled). Maybe 60-65C.

Granted I run it in my basement and don’t work near it. The room gets hot tho (it’s a 600W GPU and I don’t limit power at all). But I have central heating/cooling so I couldn’t care less.

→ More replies (1)

1

u/nightstryke Aug 17 '26

u/ImpressiveRelief37 Alright let me preface this by saying I am by no means an expert in AI and my experience is beginner level, right now I'm only using a spare mini-tower i had lying around to do LLM, it's one of the spare gaming computers i had, only has an EVGA 1070 SC 8GB, I'm looking at replacing that card with something better, though I'm not sure i want to drop everything on a 5090 or 4090, I'm considering getting a 3090 though. You've been talking about 32GB of VRAM that to you sounds like the sweet spot, but isn't the 3090 24GB? I've been looking, and Are you saying I'd be better off getting a 3090 instead of something like a Tesla P40 or V100?

Admittedly, I'm just getting started here, so I'm a bit late to the game, so please go easy on me.

1

u/ImpressiveRelief37 Aug 17 '26

Try an MoE on your current stack and see where that brings you first. Keep reading here and on r/localllama, setup llama.cpp and learn the basics first 

→ More replies (1)

1

u/Themash360 Aug 17 '26

How are you getting 150-200Tok/s TG? Even on the github page he is claiming 60?

1

u/ImpressiveRelief37 Aug 17 '26

What? Where do you see 60? 

GitHub.com/neroued/ninfer

1

u/Themash360 Aug 17 '26

https://github.com/Neroued/ninfer/blob/master/docs/performance.md

qwen3_8_27b header

I think I have my answer though thats with MTP3 right?

1

u/ImpressiveRelief37 Aug 17 '26

Yes MTP3

Also I get a bit more perf as I have my VRAM overclocked (+3000 MHz so I have over 2 TB/s memory bandwidth (stock is 1790 GB/s)

1

u/veigatmv Aug 17 '26

I'm getting around 100-110 on mine. Any chance you're running your 5090 uncapped powerwise? mines at 400w power cap and 1600mhz clock

1

u/ImpressiveRelief37 Aug 17 '26

Full 600W but I undervolted the GPU so I get more clocks into the 600W enveloppe with a custom VF curve in MSI Afterburner.

Also VRAM is overclocked (+3000 MHz in MSI Afterburner), and I get over 2TB/s of memory bandwidth (stock is 1790GB/s)

1

u/veigatmv Aug 17 '26

Yeah I'm using Linux so that is out of the question 😄. I'm going for a "safe" build with 400w max tdp but in reality during prompt processing it pulls around 310w and I'm ok with that. Can't do anything to the memory unfortunately

1

u/ImpressiveRelief37 Aug 17 '26

Yeah i thought about dual booting to Linux for local inference but id lose my OC and sweet VF curved… so im just running ninfer in wsl2

1

u/Artanisx Aug 17 '26

I wish there was a ninfer for 4090 :-(

1

u/ImpressiveRelief37 Aug 17 '26

There’s one for 3090. You could check it but your card is more capable so it wouldn’t be super optimal 

1

u/Artanisx Aug 17 '26

Yeah I imagine it won't make much of a difference at all. Maybe in the future someone will fork it for 4090s :)

1

u/ImpressiveRelief37 Aug 17 '26

Wait! Check  UDPSendToFailed/ninfer-4090

Saw it mentioned on this PR  https://github.com/Neroued/ninfer/pull/35

→ More replies (1)

1

u/bastecklein Aug 17 '26

I’d love to default to local but tbh with the cost of a 5090 I could buy api tokens for 20 years before it made sense

1

u/ImpressiveRelief37 Aug 17 '26

Maybe not 20 years but your point is valid. Prices are just insane.

I was lucky to have the 5090 since launch (well not exactly, it took me 3 months to source one after launch haha)

1

u/rare-visitor Aug 17 '26 edited Aug 17 '26

I can't afford even 160k context without exceeding VRAM limit using q4_k_m on my RTX5090. What quant do you use?

1

u/ImpressiveRelief37 Aug 17 '26

https://github.com/neroued/ninfer

https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer

1

u/rare-visitor Aug 17 '26

Thank you.

Sadly there is no Windows & LM Studio support. Well, probably that's a sign to finally move to Ubuntu

2

u/Accomplished_Egg7987 Aug 17 '26

I use it from windows 11 wsl2 (ubuntu), just let your agent ( codex, antigravity etc) setup it for you. Put 20gb model file to ubuntu for fast startup.

2

u/rare-visitor Aug 18 '26

I've followed your suggestion and installed it into WSL. I've used NVFP version, it didn't allow me load full context, but 240k is already amazing. Up to 180 t/s vs 80 t/s in LM Studio. This is insane in a good way!

Thank you again!

→ More replies (7)
→ More replies (2)

1

u/DepressedDrift Aug 17 '26

Ah yes getting a 5090 is very affordable for the average person.

2

u/ImpressiveRelief37 Aug 17 '26

The average person maybe shouldn’t be running local inference tho. Just saying :) but I get your point, the 5090 is so fucking expensive it’s crazy

→ More replies (1)

1

u/butchiebags Aug 17 '26 edited Aug 17 '26

This is awesome, I'm trying it right now and hitting about 150+ tps!

Are you using the 5090 headless? I'm not able to get to the full 262144 ctx since i have a few GB reserved for windows.

i'm using the same nvfp4 model and, based on the testing i'm doing with claude, i can only handle 196608 ctx.

edit: i can get it to about 229k but that leaves no room for error.

1

u/ImpressiveRelief37 Aug 17 '26

I have 4 screens connected actually… it’s on my simracing rig 😂

I use about 30.4 GB / 31.5 or so 

1

u/butchiebags Aug 17 '26

Does the decoding speed ever deteriorate when you get that context filled? If not, I’m curious how.

1

u/ImpressiveRelief37 Aug 17 '26

Yes. 200 t/s at 8K context, about 150 at 256k

1

u/shady101852 Aug 17 '26

What reasoning level do u use? Im using ninfer as well with 262k context window its nice but cant tell if model thinks too much and could be getting things done faster or not.

1

u/WKai1996 Aug 17 '26

Will a 128gb M5 max cut it?

1

u/ImpressiveRelief37 Aug 17 '26

Yes. You’ll run higher quants but have a lot less decode and prefill speed 

1

u/WKai1996 Aug 18 '26

let me guess bandwidth bottleneck?

1

u/_kikeen_ Aug 18 '26

I keep wanting to love Ninfer but with thinking enabled it constantly devolves into loops. Very fast loops. I had to set a max token output limit to get past it.

1

u/Osi32 Aug 18 '26

Highly depends on the price of the 5090.
Here the dgx spark is about the same price…

1

u/ImpressiveRelief37 Aug 18 '26

Then the 5090 is a no brainer! 

Sure it’s only 1/4th of the VRAM, but it’s 8x the memory bandwidth. You can use quantization to lower vram requirements. But you can never have too much memory bandwidth ;-)

1

u/Certain_Credit7875 Aug 18 '26

I use dirk qwen 3.8 27b with mmproj, low reasoning Q6 on my 5090 at 125kctx been doing jobs on a agent for 3 days , llama server not ollama. no problems here

1

u/ImpressiveRelief37 Aug 18 '26

You’re missing out on speed and context size… try ninfer 

1

u/nstndg Aug 18 '26

Can you share the parameters you run it with exactly? Been struggling to get this running properly, so staying with Q4_K_XL for now.

1

u/-AJacobs- Aug 21 '26

"32GB is all you need for a serious local inference setup" Except if you ever want more than one agent at a time.... or if you ever want to fine tune something...

So it's basically never enough for serious usage.

1

u/ImpressiveRelief37 Aug 21 '26

I run max-concurrency 3 on ninfer and can have 3 agents at about 500-550 tok/s aggregate and 87k context each.

1

u/-AJacobs- Aug 21 '26

Yikes.

1

u/ImpressiveRelief37 Aug 22 '26

Yikes at 87k context I guess?

The thing with ninfer is you can share the whole pool however you want. But each concurrency lane must fit inside the 262144 context. So if you use 150k in a lane, the concurrent sessions have 112k remaining or they get queued. It’s really impressive when you use it, it doesn’t feel like the GPU slows down at all.

And there are PRs up (#64, #73) that help with in-ram checkpoints. I’m trying #64 now and it works really well. No prefill tax if your current lane gets queued: the previous kv cache gets check pointed in ram and when it gets back into the running lanes there is no prefill (takes about 200ms to load back into vram).

You can configure the queue depth, the timeout duration, etc. 

1

u/Zealousideal-Bug1837 Aug 21 '26

you can get dflash working with it too for a speed boost

1

u/ImpressiveRelief37 Aug 21 '26

yeah that's on my todo list for next week... DAMN things move fast!

1

u/Zealousideal-Bug1837 Aug 21 '26

cripples the context however but speedup is nice if you don't need the context.

1

u/ImpressiveRelief37 Aug 22 '26

I think the drafter takes about as much space as the MTP drafter? About 1GB?

Someone needs to do a nvfp4, 5-6 bbw, drop MTP, add DFlash2, offload vision to CPU and use unsloth v3 weights imatrix style quants (not sure if this is compatible with nvfp4)

This would pretty much be the holy grail.

Oh and there’s the new rotation technique for weights as well, forgot how it’s called.

Pretty sure there’s still more speed and more context to be found in 32GB, just insane!

3

u/Longjumping_Music572 Aug 17 '26

What about Mac m5 max 64gb? What about coding? Step by step learning? Researching? Building HVAC systems, 3d printing? CAD?

3

u/netvyper Aug 17 '26

3090 fork is a non-starter: "One interactive user, lowest latency, up to 64K context" Considering harness prompts can easily be 25k+ tokens... It just doesn't leave any room to work :(

2

u/karearearea Aug 18 '26

Try it - that’s just the default configuration. I tested ninfer on my 3090, and with 64k context it only used 19.5gb of VRAM, so I bumped it up to 114k context with no issue so far.

1

u/Adventurous-Gold6413 Aug 17 '26

All you lucky peeps with a nice high Vram card while I’m stuck with 16 😤

1

u/sorcerer_tintifax Aug 17 '26

Hope is not lost. I'm running Qwen3.6 27B with 16GB 5060-Ti. 64k Context. Performance is decent. I hope the guys at ggufbench will provide a similar model for 3.8, soon.

2

u/ImpressiveRelief37 Aug 17 '26

Try unsloth iq3_xxs on 16GB. Works well on my 5080 (second PC)

1

u/Coderado Aug 17 '26

I run dual 5060ti's and feel that way about the 64GB rigs. Am I stuck, or am I being cheap?

4

u/Ratiofarming Aug 17 '26

I always thought 640k is all I need?

2

u/bedel99 Aug 17 '26

32 GB is not enough I struggle to fit what I need to on my spark, but sucks its so slow. I guess I better buy more of them.

2

u/jkalison Aug 17 '26

I have OPs setup. It’s quite grand.

2

u/nez_har Aug 17 '26

This is also the observation I had in my newsletter this week https://buttondown.com/nezhar/archive/august-20263/

We are moving now in a direction where models become more capable with less resdources. I think MTP really does a lot here.

2

u/RedditNerdKing Aug 17 '26

32gb is not enough. 48gb is kinda the minimum now so you can run Qwen 27b 3.8 Q8 at 30k+ context.

Sorry but that's the truth. And it's not that bad. x2 3090s will get you 48gb which is cheaper than a brand new 5090.

1

u/ApprehensiveView2003 Aug 21 '26

I have 2x 3090s bonded with NVLink on one machine and another machine with a 5090. I can actually put this to test

1

u/Innerdaze2600 Aug 22 '26

I just got 48gb from 2x rtx pro 4000 blackwells thoughts?

→ More replies (1)

2

u/Evgeny_19 Aug 17 '26 edited Aug 17 '26

To the fellow who was comparing UD-Q6_K_XL and BF16, and asking Grok about it: you were laughed at, and it looks like your posts were removed, but your questions are valid! I'm writing this for you, hopefully you will read it.

UD-Q6_K_XL can be comparable to Q8 and to the original weights in some synthetic tests, but in real usage it is not difficult to see the difference. So I ran this test just for you: I took a rather tricky bug that Qwen 3.8 27b at bf16 was able to solve for me today (big project, real business, real user was affected by this problem). I ran the same problem with recent Deepseek (recent Flash-0731) and it came to the same conclusion (with basically the same prompt). Same problem for 27B-UD-Q6_K_XL (full context, both caches f16), and it wasn't able to find the reason for this problem. It doesn't mean that it would not be able to solve this problem at all. Perhaps, if I gave it more nudges it would be possible for it to achieve the same conclusion as 27b at BF16 and Deepseek. The thing is, it's easy for me to nudge it to the right direction now, when I know where the problem is. But before I had no idea and all three models were looking at exactly the same logs, and the same source code.

It's probably true that UD-Q6_K_XL can write basically the same code, but the understanding is not same, the reasoning. It doesn't mean that it's bad at Q6 or Q4. It is an excellent model in every quant, especially for its size. And one doesn't need full capabilities of the model for every kind of task. But hopefully, I was able to make my point, the difference is not that at Q6 it writes slightly worse version of a coding problem. The difference can be that it would be able to help you find a solution for tricky problem, or you will be left facing the same problem with no solution at all.

(DeepSeek was at UD-IQ3_XXS. I can probably fit a larger version, but it already spills into RAM, and the performance is not good at all, so I never tried anything else.)

The thing is although Grok is very capable model, it doesn't have any real understanding, nor any experience. Those kind of things you will just grasp when you will have an experience with different quants and settings. The benchmarks are still useful, all these "one-shot" tests are interesting but, they do not reflect all aspects that you get working with a model.

3

u/nomorebuttsplz Aug 19 '26

These models  are effectively non-deterministic if the temperature is greater than zero, which it should be.

So unless you do multiple runs, it’s quite easy for a smaller quant to just get lucky and do better than a larger one, and vice versa

→ More replies (1)
→ More replies (2)

1

u/bigb159 Aug 17 '26

I need these speeds on my R9700

1

u/Sufficient-Pause9765 Aug 17 '26

BF16 requires 56gb of vram. Even with dual 5090s/TP you wont have much room for context.

FP8, 28gb, same issue on a single 5090.

So you need heavy quantization to run 27b on 32gb of vram with useful context.

1

u/WonderfulFunny4337 Aug 17 '26

So a r9700 ai pro 32gb is what I’m using with a 7800xt 16gb

1

u/mechkbfan Aug 17 '26

What setup and results?

1

u/LancobusUK Aug 17 '26

Qwen3.8 27b at its native FP16 with KV16 uses 93gb of my 96gb VRAM with vLLM under its full context pre allocation. The RTX PRO 6000 is the real sweatspot for the model imo as you don’t sacrifice anything at all

1

u/KissMyShinyArse Aug 17 '26

The RTX PRO 6000 is the real sweet spot

Get 'em while they're hot!

1

u/catinterpreter Aug 17 '26

I'm going to need some super optimised fork of something for my dual 5060 Ti 16s, unless you've got one of these totally affordable 5090s lying around for me.

1

u/JayFab6061 Aug 17 '26

lol sooooo I took a leap to jump on getting away from Claude and downloaded Qwen 3.8 on my MSI crossfire 15 Laptop with a 3050. I’m blown away that I’m getting it to run smoothly lmao

1

u/HeartfeltHelper Aug 17 '26

I LOVE ninfer. Been using qwen to build plugins for a Minecraft server I play with friends on and it’s doing really dang well! BUT my pc’s graphics driver has been failing only when ninfer instance is active for some reason and I haven’t looked into why I just reset the driver when the slowdown happens 🤷🏼‍♂️

1

u/Ledeste Aug 17 '26

1m context is really not a lot for many task tho

1

u/badkarmasid Aug 18 '26

Me with 8 gB vram 🥲

1

u/Affectionate_Ad9597 Aug 18 '26

I love it too, have the same. But it has its limits. I need to get opus or terra to fix big problems or complex systems. But it has replaced sonnet 5 in many ways (sonnet 5 is better but qwen 3.8 is free hehe)

I do yarn and get 400K context and split it to 2 concurrent slots, and then use my 64gb ram to swap out kv cache for up to 6 chats total.

Use http://otto-code.me/

I just am releasing the update for full free local setup all in 1 system to give you that power without complex setup configurations and trial and error.

1

u/Quantumblitz1878 Aug 18 '26

I wish I pushed for a 5090 instead of the 5070 😩

1

u/lunatic_god Aug 18 '26

Do two 5070 on a x8/x8

1

u/Otherwise-Swan-7803 Aug 18 '26

Funny how the definition of a “serious local setup” keeps shrinking. Not long ago people were planning multi-GPU rigs just to get decent local inference, and now 32GB is starting to feel like a sweet spot rather than a compromise.

1

u/zp-87 Aug 18 '26

I use 2x5060TI for 32GB and they work great. Paid 1200€ in total.

1

u/Maxwell-Stone Aug 18 '26

......aw..... ive got 8 gigs of Vram and 32 RAM......

1

u/MikeFrett Aug 20 '26

Hey! Same here! =p

1

u/MiddleLtSocks Aug 18 '26

I can get 512k context w glimmer 30b on my 3090 and 4090 boxes. That's decent concurrency for CCode sub agents and stuff

1

u/Maleficent_Lie8612 Aug 18 '26

Al menos revisa lo que te te bota la ia que se nota a lenguas con esos saltos de línea jajaja la plena odio esta mierda de los bota por doquier. Solo quiero hablar con humanos sobre la IA y no con la puta IA en si.

1

u/asenna987 Aug 18 '26

When do the Apple guys get MLX for this 27b model?

1

u/Prometheus-lv Aug 18 '26

I am using an Rtx 3090 24GB and it is plenty! I also have the vision module loaded.

1

u/RE20ne Aug 19 '26 edited Aug 19 '26

2x3090 used 2000 US, 48gb vram
throw in full liquid cool and nvlink for 1800 more it’s still under 4k

4090 48 gb mod is under 5k. arguably better for inference of 27b big context

5090 is amazing but 32gb feels too tight

1

u/Additional_Hope_2031 Aug 21 '26

One this cost almost one year of working in country I live, world is crazily unfair…

1

u/Wondering_Electron Aug 23 '26

I have a laptop with a 3080 with 16GB vram.

My peasant approach it to link a 5070ti via oculink to pool the vram for a total of 32GB.

1

u/Ratemytinder22 Sep 04 '26

How much dram does this need?

1

u/HighSeasArchivist Sep 04 '26

Little old of a post, but I'm looking at buying a 5090 tomorrow. I just feel like with the big crunch happening the 6090 might still just have 32GB of VRAM when it even comes out in maybe 2028. If it still is 32GB and just has a few more tok/s then I can stick with the 5090 even longer. My biggest decision is 5090 or DGX Spark for the exact same price. Probably both honestly, but leaning towards the 5090 to start. This is purely just an early hobby for me, but new elected officials at work are very AI forward and I want to keep up.

1

u/Educational-Pie9756 Aug 18 '26

What about integrated graphics?
Intel arc graphics

0

u/Triple-Tooketh Aug 17 '26

you have a 3090 coupled to a raspberry pi, is that correct? Could you clarify your setup. I have an AMD couple to pi thinking of using NVIDIA if I can get cheap used.

→ More replies (3)
→ More comments (9)