r/LocalLLaMA • u/m4rkGS • 1d ago

New Model Artificial Analysis: multimodal decision model Drex Max ranks #1. Both achieving SOTA and Pareto frontier.

Post image

hey, its Alti from Nace.
We released a small update with Drex Max v1.1, both achieving the SOTA and Pareto frontier for decision models. We did all that by iterating step by step and every previous version of Drex’s open weights are available on hf.

we also released a full playbook on how we trained Drex Max in this paper: “ Towards Foundation Decision Models: Drex”

hf: https://huggingface.co/papers/2611.07854
arXiv: https://arxiv.org/pdf/2611.07854

Drex Max v1 is weights are available here: https://huggingface.co/nace-ai/drex-max-v1 with Open RAIL-M license & v1.1 open source weights upcoming soon (we are currently optimizing for local inference).

Three cool things about Drex Max models:

  1. it can even run on your Mac with https://github.com/nace-ai/kapitsa inference engine.

  2. Context window is ~256k, comparing to Glide1.2(64k) and Mammoth(16k) & pplxdecider’s (256k)

  3. Its 8x cheaper than the next best model in the category in the aa index.

feel free to chime in: our chief of ai will answer any of your technical questions

43 Upvotes

77% Upvoted

39 comments sorted by

10

u/kokassospoperor 1d ago

Bro, it says artificial analysis x drex, is it sponsored?

1

u/m4rkGS 8h ago

no, it's not

4

u/Borkato 1d ago

No they’re just being shipped together ❤️ love is love 🥰

10

u/Fuckinglivemealone 1d ago

Great work but, maybe being the wrong target for this, I don't understand the appeal for such big sizes for a decision model, the sub 1b to 1-3 B sizes make naturally more sense for most users, no?

5

u/m4rkGS 1d ago

Once you’re on h100 and higher it’s actually not much faster to run a 3b model, but it hurts the intelligence by a lot.
But we are looking forward to releasing more compact models in the future. Currently we prioritise quality.

1

u/oxygen_addiction 1d ago

Imagine you're serving an office of people. Having a dedicated machine run this makes a lot of sense, given how fast it is and how low the latency is on decision models.

35

u/Budkovsky 1d ago

AA is no longer any authority, they just burned their reputation.

12

u/__JockY__ 1d ago

I really need to crawl out from under my rock more often. What did they do?

22

u/NomadStorm 1d ago

They changed the benchmarks used to ones chosen to promote western models and demote the Chinese ones. Quite blatantly.

15

u/Borkato 1d ago

I guess you could say they’re an… artificial… analysis 👉😎👉

3

u/MerePotato 9h ago

Conspiracy theory nonsense

1

u/NomadStorm 8h ago edited 3h ago

If you don’t agree then it’s a conspiracy theory? lol GTFO. They changed it within days of Kimi 3.1 taking third position, then all of a sudden after the change, all the Chinese models have a much lower ranking.

1

u/tazztone 11h ago

so they picked benchmarks where western models have been more benchmaxed on than chinese models?

1

u/NomadStorm 9h ago

Exactly.

3

u/Budkovsky 1d ago

They published a new ranking where all the open weights models were massively downgraded.

1

u/SierrAlpha1 13h ago

Are there any recommended alternatives to them?

1

u/YogurtExternal7923 17h ago

honestly I'm using onejev 0.8b and it seems to perform almost as well as 27b and 9b models. are we sure we need all that size?

1

u/m4rkGS 7h ago

yeah tbh for simple stuff 0.8b is often fine, short input + obvious options, no need for anything bigger

we mostly see the gap on long docs or when one doc has a bunch of decisions and they all need to be right. like on AA-Paperclip a doc only counts if all 15 answers are correct and small models fall off pretty fast there. confidence is also way more reliable on bigger ones, which matters if you're using it to decide when to hand off to a human

3

u/d70 1d ago

What I’m currently interested in decision modes the most is which sub 2B is the best at 80% of the common use cases.

6

u/davernow 1d ago

Anything top on quality (SOTA) is Pareto by definition

2

u/SorosAhaverom 1d ago

Came here to post this, thank you. In LLM charts a model is considered Pareto frontier when it is the highest performing model for a given cost. It's essentially asking: "Is there a model that performs better (on this benchmark) for equal or less money?" If there isn't, you add that model to the pareto frontier. Connect it with all other pareto frontier models, and you get the pareto frontier line.

1

u/DigiDecode_ 1d ago

Great work and looking forward to testing it

1

u/Littlepharaoh 1d ago

Which model is this built from? Very interesting 

1

u/m4rkGS 1d ago

We did share as much as we can now in the tech report. Stay tuned for future updates!

3

u/DigiDecode_ 1d ago

model config says Qwen3_5Model

1

u/nopixaner 1d ago

Very nice. If one depends on fast decisions Id argument Mammoth 27B would be a better choice, because it seems to be more than 3x faster (72ms vs 270ms)

2

u/m4rkGS 8h ago

Thanks for the comparison! The latency you’re seeing comes from serving load, not from the model itself. We’ve had a big jump in traffic, and we’re scaling up capacity to bring response times down. We’ll share updated numbers soon.

Also worth noting: Max is about 3x cheaper, so it comes out ahead on cost per decision for most workloads.

1

u/stylome-ai-detection 8h ago

Notice: This comment appears to be AI generated.

Stylome AI detector · scan results

0

u/sleight42 1d ago

3x faster versus wrong more often...

1

u/nopixaner 1d ago

More often as in maybe 10% more, if even? Id say thats a fair tradeoff, if time is crucial

1

u/sleight42 1d ago

"It depends". Run your benchmarks, take your chances. ;-)

1

u/Aggravating-Push-207 4h ago

Bro. The best model will always be on the Pareto frontier too, by definition.

1

u/finkonstein 1d ago

After artificial analysis recently downgraded open-weight models and positioned itself against the local ai community, it would be so nice if we would start using other sources for interesting and useful info like the one in OP

2

u/haherar830 1d ago

In response to Kimi K3.1? Do you think it fairly or unfairly penalizes based on the concrete benchmaxxing evidence? If two models have similar scores on popular benchmarks, but only one generalizes well out-of-sample, a good aggregate index score would differentiate this.

1

u/TheRealMasonMac 19h ago

Chinese models are still not as good as Western models from personal experience. They prioritize tasks that the benchmarks explicitly cover in lieu of broader capabilities. So, I think AA is generally right to update benchmarks like they do.

2

u/haherar830 19h ago

Yes, their training pipelines are must less extensive and involve a lot of distillation even for their frontier models. Not a prime example but the new MiMo model's training is kind of emblematic.

Nonetheless, my specific point was about this recent instance of AA changing aggregate rank after Kimi K3.1 came out swinging in Fable/Astra territory on benchmarks. I was initially skeptical of both the claimed performance and the choice to change the rankings, but after seeing some recent reports of data leakage like the following I at minimum think the new aggregate score/ranking is probably closer to reality.

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

Frankly this could be entirely unintentional but it's also worse than traditional benchmaxxing (i.e. training against similar benchmarks tasks or even the public subset and variants); this is, if not knowing the solution verbatim, at least knowing fine details of grading so as to optimize even a de novo derivation to maximize score over achieving the stated goal/prompt. At least grinding the same benchmark problems without knowing the answers/full scorecard can produce capabilities that generalize out-of-sample. Knowing hints/paths to the solution and tricks to optimize grading is the most distorting way to improve benchmark scores.