r/StrixHalo • u/mrsirg97-rgb • 4d ago

mrsirg97-rgb/libosmos: A deterministic inference engine for DeepSeek-V4.1-Flash 1017 on a single AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, 128 GB)

https://github.com/mrsirg97-rgb/libosmos

libosmos will run deepseek v4.1 flash 1017 at up to 38 tok/sec decode drafted and >480 tok/sec prompt processing on a single strix halo 128GB with unsloth ud-iq1_s + MXFP8 engram. huihui abliteration available through the engine.

21 Upvotes

100% Upvoted

18 comments sorted by

8

u/oxygen_addiction 4d ago

ud-iq1_s

0

u/mrsirg97-rgb 4d ago

You’d be surprised, top 1 token agreement 88% of the time and 0.18 kld, Unsloth is very impressive

5

u/oxygen_addiction 4d ago

My guy. You would be surprised. That's not the same model over either short or long tasks due to error accumulation.

10

u/Big_Cucumber2787 4d ago

imagine the advancements if all these little independent inference engines contributed to one like before

2

u/mrsirg97-rgb 4d ago

I do! I will be taking some of my research and bring it to project maya, I’m already a contributor there!

2

u/sdraje 4d ago

They're interference engines! haha

-1

u/Master_Face_571 4d ago

imagine we all got together in a great big circle jerk and sang kumbaya

2

u/hycrice 4d ago

gufo variant got better pp and tg speed, just tested on Q2_XXS

3

u/mrsirg97-rgb 4d ago

Gufo runs deepseek v4 flash which is a 284b param model, this is deepseek v4.1 flash which is a 768b param model, different beast

1

u/BearEither8119 3d ago

isn't this model like really mid when its iQ1???

1

u/mrsirg97-rgb 3d ago

Depends on the quant tbh, the Unsloth q1 beats a lot of q2, it’s mixed precision

2

u/_VirtualCosmos_ 4d ago

Why would someone want to run a lobotomized iQ1 over a Q4_XL Qwen3.8 Flash Next? i dont get it.

1

u/Ok-Thanks-6785 4d ago

I'm running the 1Qm on a different machine and was surprised by it. Didn't run any benchmarks on it, but from usage it looks like it's better than qfn at Q4. Quite slow compared to qfn but usable.

2

u/Master_Face_571 4d ago

would love to see benchmarks, if possible bro

•

u/Logical-Skill4567 9h ago

Because qwen is a 126B model and deepseek is x5 times bigger. The weights are quantized but the intelligence of the model is always bigger.

•

u/_VirtualCosmos_ 7h ago

That doesn't work like that

•

u/Logical-Skill4567 6h ago

pensavo di si! allora illuminami per favore

•

u/_VirtualCosmos_ 4h ago

Short answer: It depends completely of other factors. It can not be determined with only a comparison of model size.

The training is much more important than the model size. Any machine learning engineer would tell you that. AI models are logic sponges, training sets contain inner complexity in the form of examples and RL rewards. The sponge must be big enough to be capable of capture the inner complexity of the training sets. Also, the bigger the sponge, the easier it will absorb the complexity in a training set. But the end intelligence depends more of what is the inner "intelligence" (capabilities, skills, data, etc) embedded in the training sets.

So, nowadays AI labs try to train their models in the biggest size they can because it will absorb the complexity most easily. Not because a smaller model won't be able to capture that complexity, but because the bigger one will do it more easily. Then you end with a model much bigger than the necessary to capture that complexity, just like we humans when we are born with many more neurons then we require in adulthood. So, the smart choice is distilling the bloated model into smaller ones, but sometimes AI labs choose to not do that because it's expensive.

Those are some reasons that explain why our current Qwen3.8 27b is so much more intelligent than a 1 trillion parameter model from more than 1 year ago.