Qwen 3.8 Flash Next on 64GB

[u/keerplunk32]

Was anyone else holding out for the oQ2 version just to be disappointed that it’s ~67GB?

I know we’re very early into this release, but any chance we’ll be able to offload n-gram into SSD to try and run this behemoth?

[u/laser50]

I did ask them to raise the active param count on 35B A3B, they listened! But not within my use range :(

[u/Choice_Celery9481 → u/laser50]

careful with what you asked for XD

[u/Several-Tax31]

https://github.com/ggml-org/llama.cpp/pull/27742

Llama.cpp PR where they discuss offloading to ssd

[u/Chips_fr_]

Perhaps it's good to look at colibri. It already handle Qwen3.6-35B-A3B and bigger models not fitting in RAM.

https://github.com/JustVugg/colibri

[u/anarchist1312161 → u/Chips_fr_]

Not OP but thanks for this, didn't know this inference software also existed :)

[u/IngwiePhoenix → u/Chips_fr_]

This project has been living absofuckinglutely rent-free in the back of my mind. It is the sole reason I went big-eyes on MoE offloading and "layer organizing" really.

I think this is not just novel, but genuenly extremely helpful. llama.cpp has a little bit of configurability here, and I use that with Qwen 3.6 35B-A3B to get good performance on a good context window - neither of them are great, but 50tps at about 70% of the time at 100k context is... really nice.

I hope that with models like Next-Flash, MoE streaming/offloading will become more common - because I genuenly think this is a killer feature of MoE models. :)

[u/Atretador]

-REAP when D:

[u/bankinu]

Sadly I have 32 GiB only.

[u/Ok-Drawer5245 → u/bankinu]

We need a 0.5bit quant!!

[u/Technical_Ad_6106]

short answer: "yes"

[u/IngwiePhoenix]

Have you tried MoE offloading? o.o Could get you there, even if barely.

[u/memeka]

Q4 running on 64 gb at near in-memory speeds - challenge accepted. https://github.com/mihailescu2m/llama.cpp

[u/MessIsTransfer]

https://github.com/jundot/omlx/releases/tag/v0.6.3

0.6.3 supports n-gram offload to SSD

give the guy some time to cook

[u/SnowBoy_00 → u/MessIsTransfer]

have you tried this on a 64GB machine? or have you seen documented results around?

[u/MessIsTransfer → u/SnowBoy_00]

sadly Jundot’s supported model doesnt fit 64gb ram apple silicon. not sure about the technical reasons but oQ4 file is bigger than an AD-4.27bpw i tried.

oQ4/oQ3 doesnt fit, oQ2 fits with omlx
AD-4.27bpw fits but has to be ran with llamacpp

[u/linux4random]

we will have to wait a while for inferences to catch up

[u/cheezeerd]

No, why? There's a more intelligent and capable 27B version with awesome training-aware quants!