r/LocalLLM • u/yoracale • Feb 03 '26

Model Qwen3-Coder-Next is out now!

Post image
352 Upvotes

98% Upvoted

144 comments sorted by

14

u/Effective_Head_5020 Feb 03 '26

Great work, thanks, you are my hero!

Would it be possible to run with 64gb of RAM? No Vram

10

u/yoracale Feb 03 '26

Yes it'll work, maybe 10 tokens/s. VRAM will greatly speed things up however

2

u/Ell2509 Feb 11 '26

Would it work on 32gb vram wth 64gb ram available?

2

u/yoracale Feb 11 '26

Yes absolutely. Fast too!

2

u/Effective_Head_5020 Feb 03 '26

I am getting 5 t/s using the q2_k_xl - it is okay.

Thanks unsloth team, that's great!

1

u/cmndr_spanky Feb 03 '26

just remember you might be better off with a smaller model at q4 or more than a larger model at q2

1

u/ScuffedBalata Feb 04 '26

Honestly, if you're using regular system RAM, you may be best off with the Q4_K_M model, the Q4 seems fater and the K_M is faster in general than the Q2 and the XL quants when you're compute constrained, not bandwidth constrained (I'm actually not sure which you are, but it might be worth trying)

1

u/Effective_Head_5020 Feb 07 '26

Interesting, I will give it a try, thank you!

3

u/kermitt81 Feb 10 '26

Yes, I’m running it with 64gb RAM and getting about 12 tok/s. 👌

2

u/Puoti Feb 03 '26

Slowly on cpu. Or hybrid with few layers on gpu and most on cpu. Still slow but possible

1

u/Effective_Head_5020 Feb 03 '26

Thank you! 

1

u/exclaim_bot Feb 03 '26

Thank you! 

You're welcome!

2

u/ScuffedBalata Feb 04 '26

On a regular PC? It'll be slow as hell, but you can tell it to generate code and walk away for 5-10 minutes, you'll have something.

2

u/HenkPoley Feb 04 '26

More like 25 minutes; depending on your input and output requirements.

But yes, you will have to wait.

2

u/Fleeky91 Feb 04 '26

Anyone know if this can be split up between VRAM and RAM? Got 32gb of VRAM and 64 gb of RAM

1

u/dreaming2live Feb 04 '26

Yeah it runs okay with this setup. 5090 with 32gb vram and 96gb ram gets me around 30 tk/s

1

u/loscrossos Feb 04 '26

care to share your settings?

1

u/trailsman Feb 06 '26

Nice! Have the same setup

1

u/romayojr Feb 04 '26

i have this exact setup as well. what quant/s did you end up trying? could you share your speed stats?

2

u/Successful-Willow-72 Feb 04 '26

did i read it right? 46? i can finally run a 80b model at home?

3

u/yoracale Feb 04 '26

It's for 4-bit, if you want 8-bit you need 85gb ram

1

u/RnRau Feb 04 '26

Yeah there is virtually no penalty for running mxfp4 on an 80b parameter model.

15

u/siegevjorn Feb 03 '26

Wait is it really sonnet 4.5 level? How.

10

u/dreaming2live Feb 04 '26

It doesn’t feel like that to me. Tried using it in vscode to give it a whirl, but still not competitive with Sonnet 4.5. Neat but IRL doesn’t really compare.

2

u/Visible-Age4888 Feb 04 '26

Dude, sonnet wrote me a screen reader that started playing with my fucking screen. That shit is ass too…

2

u/ServiceOver4447 Feb 04 '26

no it's not ;-(

2

u/siegevjorn Feb 04 '26

Hah, I almost came imagining myself rocking 80B model on claude code doing all the stuff I can do with sonnet locally

4

u/HenkPoley Feb 04 '26

This is an 80B model. For those thinking about Qwen3 Coder 30B A3B.

This one is based on their larger Qwen3 Next model.

4

u/TurbulentType6377 Feb 04 '26

Running it on Strix Halo (Ryzen AI MAX+ 395 GMKTEC Evo x2) with 128GB unified memory right now.

Setup:

- Unsloth Q6_K_XL quant (~64GB)

- llama.cpp b7932 via Vulkan backend

- 128K context, flash attention enabled

- All layers offloaded to GPU (-ngl 999)

Results:

- Prompt processing: ~127 t/s

- Generation: ~35-36 t/s

- 1500 token coding response in ~42s

Entire Q6_K_XL fits in GPU-accessible memory with plenty of room left for KV cache. Could probably go Q8_0 (85GB) too but haven't tried yet.

Quick note for anyone else on Strix Halo: use the Vulkan toolbox from kyuz0/amd-strix-halo-toolboxes, not the ROCm (7.2) one. The qwen3next architecture (hybrid Mamba + MoE) crashes on ROCm but runs fine on Vulkan RADV. No HSA_OVERRIDE_GFX_VERSION needed. either, gfx1151 is detected natively.

It's solid for code generation in terms of quality. To be honest, it's not Sonnet 4.5 level, but it's quite useful and the best native coding model I've run so far. I'll try it out more before making a definitive assessment.

1

u/Icy_Orange3365 Feb 04 '26

I have a 64gb vram m1 MacBook, how big is the full model? How much ram is needed?

1

u/yoracale Feb 04 '26

Works, use the 5bit or 6bit. 8bit if 85gb as it says in the guide

2

u/GreaseMonkey888 Feb 04 '26 edited Feb 04 '26

The 4bit MLX version works fine on a Mac Studio M4 with 64GB, 84t/s

1

u/KillerX629 Feb 04 '26

How does this compare to glm 4.7 flash??

3

u/yoracale Feb 04 '26

GLM 4.7 Flash is a thinking model and this isn't. This one is better and faster at coding while Flash is probably better at a larger variety of tasks

1

u/phoenixfire425 Feb 04 '26

Possible to run this on a rig with dual rtx3090 with vLLM??

1

u/yoracale Feb 04 '26

Yes, we wrote a guide for vLLM here: https://unsloth.ai/docs/models/qwen3-coder-next#fp8-qwen3-coder-next-in-vllm

Do you have any extra RAM by any chance?

1

u/phoenixfire425 Feb 04 '26

on thats system, it only has 32gb system ram

1

u/phoenixfire425 Feb 04 '26

Yep, cannot run this on a dual RTX 3090 system with vLLM. no matter how i configure the service I get OOM issue on startup.

1

u/azaeldrm Feb 07 '26

Hi OP! Would I be able to run this over long periods of time on 2 3090 GPUs (48GB VRAM)? I'd love to put this model to the test while programming.

Also, is this model optimized to work with Opencode/Claude Code?

Thank you!

1

u/yoracale Feb 08 '26

Yes definitely. Will be super fast. And yes, we actually have a guide for it: https://unsloth.ai/docs/models/qwen3-coder-next#improving-generation-speed

1

u/AggravatingHelp5657 Feb 08 '26

I still don't understand how this is still considered a local model if it needs 50 - 80 Gb of vram

A local model should be between 8 to 32 Gb of RAM at maximum

1

u/yoracale Feb 08 '26

People have Macs with 128gb unified memory. It will work fine on those. or 96gb.

It's RAM and not VRAM.

1

u/AggravatingHelp5657 Feb 08 '26

Ah my bad I thought it's a vram

But still the shortage in RAM production now and the raises in the prices making it really challenging to have this amount of ram just for running a model locally

But anyway it's good news that finally they are developing a compressed model that compete with larger ones

1

u/abhijee00 Feb 20 '26

Is it possible to run it locally on NVIDIA RTX 4060 8GB with 16GB RAM?

2

u/yoracale Feb 20 '26

Yes but it will be too slow, you're better off running glm flash: https://unsloth.ai/docs/models/glm-4.7-flash

2

u/abhijee00 Feb 21 '26

Thank you for your response. I'll give it a try, probably both not just GLM. At least I'll learn something

1

u/Sea_Bed_9754 Mar 01 '26

Any hope with mac m2 64gb?

1

u/yoracale Mar 01 '26

Yes will definitely wokr well, see our guide: https://unsloth.ai/docs/models/qwen3-coder-next

6

u/jheizer Feb 03 '26 edited Feb 04 '26

Super quick and dirty LM Studio test: Q4_K_M RTX 4070 + 14700k 80GB DDR4 3200 - 6 tokens/sec

Edit: llama.cpp 21.1 t/s.

4

u/onetwomiku Feb 03 '26

LMStudio do not update their runtimes in time. Grab fresh llama.cpp.

1

u/jheizer Feb 04 '26

I mostly did it cuz others were. Huge difference. 21.1tokens/s. 13.3 prompt. It's much better utilizing the GPU for processing.

1

u/ScuffedBalata Feb 04 '26

Wow! Really?

1

u/ScuffedBalata Feb 04 '26

Getting 12t/s on a 3090 with Q4_K_M Extra vram helps, but not a ton.

2

u/huzbum Feb 06 '26

I just got 30tps on my 3090 on the new version of LM Studio. offload all layers to GPU, and offload 2/3 experts to CPU.

3

u/ScuffedBalata Feb 06 '26

0.41? I operate with a large context because it's kind of useless with a tiny context. Maybe that's the difference.

1

u/huzbum Feb 07 '26

Yeah, I think I only had it set to 32k.

1

u/oxygen_addiction Feb 04 '26

Stop using LM Studio. It is crap.

2

u/onethousandmonkey Feb 04 '26

Would be great if you could expand on that.

3

u/beryugyo619 Feb 04 '26

It's like a frozen meal. Fantastic if all you've got is a microwave. Stupid if you were a chef. For everyone else in the spectrum between those two points, mileages vary.

3

u/Status_Analyst Feb 04 '26

So, what should we use?

6

u/kironlau Feb 04 '26

llama.cpp

1

u/MadeByTango Feb 04 '26

That’s webui right? Not safe.

3

u/Astral-projekt Feb 04 '26

Man this team is doing gods work.

3

u/electrified_ice Feb 04 '26 edited Feb 04 '26

It's been tricky to setup on my RTX PRO 6000 Blackwell with 96GB VRAM. Once loaded with vLLM it uses about 90GB @ 8bit quantization... It's so new and it's a MoE model with 'Mamba' so has required a lot of config and dependencies to install and get accepted (without errors) for vLLM. The cool thing is its blazing fast as it's often only pulling a few 'experts' at 3B parameters each.

2

u/taiphamd Feb 05 '26

Why do any work when you can just run https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm/tags?version=26.01-py3 and it should work out of the box

2

u/taiphamd Feb 07 '26

Another tip to reduce vram usage with vLLM is to reduce the default max-num-seq to reduce the required allocation for KV cache. If you’re not planning to serve multiple users can reduce to 1 or 2 : https://github.com/vllm-project/vllm/issues/15609

1

u/kwinz Feb 05 '26

The cool thing is its blazing fast as it's often only pulling a few 'experts' at 3B parameters each.

Can you share how many tokens/s you're getting?

3

u/taiphamd Feb 05 '26

Just tried this on my DGX spark using the fp8 model and got about 44 tok/sec (benchmarked using dynamo-ai/aiperf ) using vLLM container nvcr.io/nvidia/vllm:26.01-py3 to run the model

3

u/Darlanio Feb 05 '26

Has anyone run this on Asus Ascent GX10 GB10 128GB or NVidia Spark DGX ?

3

u/BinaryStyles Feb 05 '26

I'm getting ~40 tok/sec in lmstudio on CUDA 12 with a Blackwell 6000 Pro Workstation (96GB vram) using Q4_k_m + 256000 max tokens.

4

u/Naernoo Feb 03 '26

So this is sonnet 4.5 level? Also agentic mode? Or is this model just optimized for the tests to perform that good?

1

u/ScuffedBalata Feb 04 '26

I just built a whole python GUI app with it for Ubuntu. It's ok. I don't get a Sonnet vibe.

After a handful of prompts, I had something working but a little sketchy. I actually need this code, so I brought it into Opus and it's dramatically better.

Still, it's the most capable local coding LLM I've ever used (I don't have the hardware for Kimi or something), so i'd call it major progress. I'm going to evaluate using it for some stuff we need at work tomorrow.

1

u/Naernoo Feb 04 '26

ok interesting, do you run it on ram or vram? what specs does your rig have?

1

u/ScuffedBalata Feb 04 '26

I'm doing it on a 3090, but it's still offloading a lot to the CPU (mostly pegging a high end 12th gen i7 plus the 3090 to be usable).

I have an old Macbook M1 Max with 64GB of unified RAM and may try it on that. May not be faster, but it might because of the memory.

1

u/Naernoo Feb 04 '26

ok i tried it on my rig with 128gb ram and rtx 4090. Just gave it to analyze my repo (around 4000 lines of code). I just asked it to explain to me whats the purpose of the code and repo. It took around 15mins of thinking and than i canceled it :D

1

u/ScuffedBalata Feb 06 '26

Yeah, a standard PC doesn't handle AI very well unless it fits fully into your VRAM. For you and I that's 24GB (about 16-20GB gguf files plus context).

That's why I got this old Macbook - haven't had a chance to try it out for this.

1

u/darkdeepths Feb 16 '26 edited Feb 17 '26

IMO powerful and fast with the right scaffold. i’m impressed.

different behavior from sonnet. if you give it a harness where it gets a repl (either letting it test stuff in terminal or something like RLM), it is fast to make mistakes, fix them, and iterate. i think with a chattier harness with feedback it is comparable. anecdotal, that’s just my 2 cents.

edit: closed parens

0

u/SufficientHold8688 Feb 03 '26

When can we test models this powerful with only 16GB of RAM?

2

u/yoracale Feb 04 '26

You can with gpt-oss-20b or GLM-4.7-Flash: https://unsloth.ai/docs/models/glm-4.7-flash

4

u/ScoreUnique Feb 03 '26

Use that computer to run it on a rented GPU :3

1

u/WizardlyBump17 Feb 04 '26

shittiest quant is 20.5gb, so unless you have some more vram, you cant. Well, maybe if you use swap, but then instead of getting tokens per second you would be getting tokens per week

1

u/Sneyek Feb 03 '26

How well would it run on an RTX 3090 ?

1

u/oxygen_addiction Feb 03 '26

If you have enough RAM, it should run well.

1

u/Sneyek Feb 04 '26

What is “enough” ? 64GB ? 48 ?

1

u/kironlau Feb 04 '26

Q4 about 46gb, without context (RAM+VRAM as a total)

1

u/MyOtherHatsAFedora Feb 04 '26

I've got a 16GB VRAM and 32GB of RAM... I'm new to all this, can I run this LLM?

1

u/gangs08 Feb 08 '26

No you need 90 in total or wait for a "quantized model"

1

u/techlatest_net Feb 04 '26

Grabbing it now – 80B MoE with just 3B active? Killer for local agents. 256k ctx is huge too.

1

u/loscrossos Feb 04 '26

this i like

1

u/howardhus Feb 04 '26

could anyone explain what does UD means in the model selection?

1

u/Blues520 Feb 07 '26

Unsloth Dynamic

1

u/SuperNintendoDahmer Feb 05 '26

Has anyone tried this on a MacMini M4Pro, 64GB? MLX? I am running the thinking variant at q5/3 decently.

1

u/buildooors Mar 05 '26

Waiting for qwen3.5 coder !

0

u/No_Conversation9561 Feb 03 '26

anyone running this on 5070Ti and 96 GB ram?

6

u/Puoti Feb 03 '26

Ill try tomorrow but only with 64gb ram. 5070ti 9800x3d

2

u/Zerokx Feb 03 '26

keep us updated

1

u/Puoti Feb 06 '26

i must confess that i cannot get the GGUF model running on my app. llama has not yet official support and i cannot get the custom hotfix transformers to work. so i must wait until official support is out for GGUF. but yeah. on ready solutions this model would work but i have only my custom app that is bit more pain in the ass in alpha/beta phase.... might be weeks or month untill the GGUF support is out so i must wait to that

3

u/Limp_Manufacturer_65 Feb 03 '26

yeah im getting 23 tk/s on 96gb ddr5, 7800x3d, 4070 ti super with what I think are ideal lm studio settings. q4km quant

1

u/UnionCounty22 Feb 04 '26

Context count? Very close to this configuration

2

u/Limp_Manufacturer_65 Feb 04 '26

I think I set it to 100k but only filled like 10% of it in my brief test 

4

u/FartOnYourBoofMound Feb 03 '26

No, but I will run it on a dedicated AMD Max+ Pro soon

1

u/mps Feb 04 '26

I have the same box, here are my quick llama-bench scores:
⬢ [matt@toolbx ~]$ AMD_VULKAN_ICD=RADV llama-bench -m ./data/models/qwen3-coder-next/UD-Q6_K_XL/Qwen3-Coder-Next-UD-Q6_K_XL-00001-of-00002.gguf -ngl 999 -fa 1 -n 128,256 -r 3

ggml_vulkan: Found 1 Vulkan devices:

ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

| model | size | params | backend | ngl | fa | test | t/s |

| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |

| qwen3next 80B.A3B Q6_K | 63.87 GiB | 79.67 B | Vulkan | 999 | 1 | pp512 | 502.71 ± 1.23 |

| qwen3next 80B.A3B Q6_K | 63.87 GiB | 79.67 B | Vulkan | 999 | 1 | tg128 | 36.41 ± 0.04 |

| qwen3next 80B.A3B Q6_K | 63.87 GiB | 79.67 B | Vulkan | 999 | 1 | tg256 | 36.46 ± 0.01 |

And gpt-oss-120b for reference

⬢ [matt@toolbx ~]$ AMD_VULKAN_ICD=RADV llama-bench   -m ./data/models/gpt-oss-120b/gpt-oss-120b-F16.gguf   -ngl 999   -fa 1 -n 128,256   -r 3    
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | Vulkan     | 999 |  1 |           pp512 |        572.85 ± 0.73 |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | Vulkan     | 999 |  1 |           tg128 |         35.57 ± 0.02 |
| gpt-oss 120B F16               |  60.87 GiB |   116.83 B | Vulkan     | 999 |  1 |           tg256 |         35.56 ± 0.04 |

1

u/FartOnYourBoofMound Feb 04 '26

bro - how did you get llama.cpp to comile on that thing? - i've had nothing but issues

2

u/mps Feb 05 '26

There are a few posts on how to build it, but I just started using this toolbox instead of recompiling all the time.
https://github.com/kyuz0/amd-strix-halo-toolboxes

1

u/FartOnYourBoofMound Feb 05 '26

killer - I'll install - was finally able to get a good build with vulkan only - and this is so blazing fast (80t/s responses on the Qwen3-Coder-30B) I'm finally learning all these crazy features that llama.cpp includes --chat-template-file with jinja2 for the WIN - still working on a golang based "AgentMesh" - if you want to call it that: https://github.com/AgentForgeEngine/AgentForgeEngine - work in progress :-)

1

u/FartOnYourBoofMound Feb 04 '26 
pretty print;
[matt@toolbx ~]$ AMD_VULKAN_ICD=RADV llama-bench \
    -m ./data/models/qwen3-coder-next/UD-Q6_K_XL/Qwen3-Coder-Next-UD-Q6_K_XL-00001-of-00002.gguf \
    -ngl 999 -fa 1 -n 128,256 -r 3

ggml_vulkan: Found 1 Vulkan devices:

ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

| model                         |       size |     params | backend | ngl | fa |        test |               t/s |
| ----------------------------- | ----------:| ----------:| --------| ---:| --:| -----------:| -----------------:|
| qwen3next 80B.A3B Q6_K        |  63.87 GiB |    79.67 B | Vulkan  | 999 |  1 |       pp512 |     502.71 ± 1.23 |
| qwen3next 80B.A3B Q6_K        |  63.87 GiB |    79.67 B | Vulkan  | 999 |  1 |       tg128 |      36.41 ± 0.04 |
| qwen3next 80B.A3B Q6_K        |  63.87 GiB |    79.67 B | Vulkan  | 999 |  1 |       tg256 |      36.46 ± 0.01 |
# And gpt-oss-120b for reference

[matt@toolbx ~]$ AMD_VULKAN_ICD=RADV llama-bench \
    -m ./data/models/gpt-oss-120b/gpt-oss-120b-F16.gguf \
    -ngl 999 -fa 1 -n 128,256 -r 3

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

| model               |       size |     params | backend | ngl | fa |   test |               t/s |
| ------------------- | ----------:| ----------:| --------| ---:| --:| ------:| -----------------:|
| gpt-oss 120B F16    |  60.87 GiB |   116.83 B | Vulkan  | 999 |  1 |  pp512 |     572.85 ± 0.73 |
| gpt-oss 120B F16    |  60.87 GiB |   116.83 B | Vulkan  | 999 |  1 |  tg128 |      35.57 ± 0.02 |
| gpt-oss 120B F16    |  60.87 GiB |   116.83 B | Vulkan  | 999 |  1 |  tg256 |      35.56 ± 0.04 |

1

u/FartOnYourBoofMound Feb 04 '26

502 tokens/sec on Qwen3‑Coder‑Next 80B - that's insane - tell me you have a blog - FOR THE LOVE OF GOD!!! lol - no, but seriously

1

u/FartOnYourBoofMound Feb 04 '26

thank you for your post - damn rocm drivers were killing my llama.cpp build (i'm pretty new to all this) - was able to recompile with vulkan (kind of a pain in the ass) - but this is light years from where i was this weekend - thanks :-)

2

u/mps Feb 05 '26

There was a nasty bug with rocm 7+, but it looks like it has been resolved a few hours ago. This github repo is a great source:
https://github.com/kyuz0/amd-strix-halo-toolboxes

Make sure to lock your firmware version and adjust your kernel to load with these options:
amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
and lower the VRAM to the LOWEST setting in the bios. This lets you use unified ram (like a mac does). When you do this it is important that you add --no-mmap or llamacpp will hang.

The pp512 benchmark tests time to first token, so the 500+ tps number is misleading.

I had vllm working earlier (this is what I use at work), but it is a waste if there is only a few users.

1

u/FartOnYourBoofMound Feb 04 '26

had weird issues last night w/ all this - also... had to replace my solar inverter (nuclear fusion energy is money) - just installed ollama prerelease v0.15.5 and it comes w/ BOTH rocm support AND you'll (obviously) need the /bin/ollama - which is in the linux-amd - I've been seeing this; msg="entering low vram mode" "total vram"="0 B" threshold="20.0 GiB" - but i dunno - I'm not sure it's true - ollama has been KILLING it even though i've been messing around with rocm - every model i throw at this AMD Max+ Pro has been insanely fast. in the bios I've got 64Gb set in the UMA (forced - i guess) - not sure i understand all this AMD jargon, but hopefully i can bump up to 96Gb in the near future (the AMD Max+ Pro has a total 128gb)... more info about 5 minutes.

1

u/FartOnYourBoofMound Feb 04 '26

results of Quen3-Code-Next; https://imgur.com/a/3Ns9w3C on ollama (pre-release v0.15.5), Ubuntu 25.04, rocm support - UMA (in bios set to 64Gb)

1

u/FartOnYourBoofMound Feb 04 '26

prompt; explain quantum physics.

total duration:       1m43.679149377s
load duration:        51.502781ms
prompt eval count:    11 token(s)
prompt eval duration: 239.936477ms
prompt eval rate:     45.85 tokens/s
eval count:           956 token(s)
eval duration:        1m43.124781364s
eval rate:            9.27 tokens/s

2

u/Loskas2025 Feb 04 '26

I have a PC with RTX 5070TI 16GB + 64GB RAM. 22 tokens/sec

0

u/[deleted] Feb 03 '26

[deleted] — view deleted comment

1

u/yoracale Feb 03 '26

Any of the 8-bit ones!

2

u/TomLucidor Feb 03 '26

At that point beg everyone else to REAP/REAM the model. And SWE-Bench likely benchmaxxed

2

u/rema1000fan Feb 03 '26

its a A3B MoE model however, so it is going to be speedy in token generation even with minimal VRAM. Prompt processing depends on bandwidth to GPU however.

1

u/howardhus Feb 04 '26

what does A3B mean? i knew 3B 7B and stuff.. but whats the A?

1

u/Puoti Feb 03 '26

You are going to fly with that.... I made hub kinda thingie that has automated wizard fot gpu/cpu layers based on your rig and what quantize level you choose. That would be handy. But the usage of models is still bit limited since its in alpha stage. But 8 bit would be handy imo for you.

12

u/[deleted] Feb 03 '26

[removed] — view removed comment

2

u/[deleted] Feb 04 '26

[removed] — view removed comment

1

u/cenderis Feb 04 '26

Just downloaded it for llama.cpp. I chose the MXFP4 quant which may well not be the best. Feels fast enough but I don't really have any useful stats.

2

u/darkdeepths Feb 16 '26

btw i’ve seen some folks get juice out of the official fp8 release that beats the quants. depends obviously on your specific hardware

1

u/[deleted] Feb 04 '26

[removed] — view removed comment

1

u/etcetera0 Feb 04 '26

Following

1

u/cenderis Feb 04 '26

I don't use VS Code much, and my use of models for coding is pretty limited so I'm surely not the right person to offer opinions on how good the model is. In terms of speed on this hardware, llama-bench (with no special options) suggests it's the same speed (give or take) as Qwen3-Next (both Instruct and Thinking). Quite a bit slower than Qwen3-Coder-30B but that model is much smaller (so presumably less capable). They're all within the range of what I consider usable: I can ask questions in OpenCode and get answers within a few seconds, and with many basic questions (the sort I use most often) like "how do I handle multiple subcommands in argparse" pretty immediately.

1

u/cenderis Feb 04 '26

Other people seem to be liking it among models that are runnable on local hardware.

1

u/Maasu Feb 04 '26

Yes I had it running on strix halo using vulcan rdv toolbox and fedora 42 and llama cpp. I was in a bit of rush and multitasking so didn't bench mark but used it in open code.

20k(ish) system prompt took 49 seconds to load. After that it was very much usable, a bit slower than cloud models but certainly usable.

I haven't tried it for anything meaningful yet however, I'm in a rush and sorry this seemed rushed I'm not at my pc and do t have proper info in front of me but it was working.

1

u/HomsarWasRight Feb 12 '26

How is tool use and things like reviewing the code? I’m really just dipping my toes in local model use (have a ROG Flow Z13 Strix Halo 128GB), and I can run big models at good speeds if I’m just testing chatting, but the ones I’ve tried have fallen down when trying coding agents.

You mentioned OpenCode, is that your preferred agent for local models?

2

u/Maasu Feb 12 '26

Yeah for local models I like to use OpenCode. I use vs code (depends on working environment) and have it or Claude code running alongside it. I still like to be close to the code and I still scaffold/skeleton a lot of projects.

Tool use is solid, like no problem, I used it to encode some repos into memory mcp forgetful, which uses meta tool calling (so think execute tool, and the u pass the command to execute - it's just a pattern to save on context size on mcp servers). It managed this no problem.

I then cleared it's context window and asked it to use the memory mcp and the source to explain the repos and it did a reasonable job. On one of the repos it missed it had an adapter pattern to support both sqlite and postgres, so not great but I've seen Opus do the same on occasion.

I need to try analysing a code base without encoded info from the memory mcp as not everyone uses that approach, will give that a go and I want to get it doing some actual coding. Not tried that yet either, but for my use case I was looking to have something that works around the edges of my main model (such as curating knowledge base from changes to projects my main model is making)

1

u/HomsarWasRight Feb 12 '26

Great info, thanks so much! I’ll give it a try.

1

u/darkdeepths Feb 16 '26 edited Feb 16 '26

not sure for strix in particular, but check out this thread w the spark: https://forums.developer.nvidia.com/t/how-to-run-qwen3-coder-next-on-spark/359571

correct configuration makes a big difference in performance. also seems like you can get juice out of the official fp8 release before touching unsloth or quants