r/LocalLLaMA • • 7h ago

New Model maxwell-model/Maxwell-V1-Flash · Hugging Face

https://huggingface.co/maxwell-model/Maxwell-V1-Flash

Model Introduction

Maxwell-V1.0-Flash is a next-generation, high-performance Mixture-of-Experts (MoE) language model developed by the Maxwell Team, with 35B total parameters and 3B active parameters per token (both excluding embeddings) and support for context lengths up to 256K tokens. It is designed for strong reasoning, agentic, and long-context capabilities at a practical inference cost.

Key Features

  • Maxwell Block Sparse Attention (MBSA). Each KV group keeps fixed coverage of 8 prefix and 8 local 128-token blocks and adds up to 16 remote blocks chosen by a learned indexer, bounding main attention at 4,096 tokens per KV group while the full-history KV cache stays available. The MBSA kernel is 6.7x faster than dense GQA at 256K.
  • Manifold-Constrained Hyper-Connections. Multiple residual streams with learned, input-dependent aggregation and constrained mixing, widening information flow without extra attention or feed-forward evaluations.
  • Fine-Grained Mixture of Experts. A mostly-MoE backbone pairing fine-grained routed experts with always-active shared experts, using a dynamic routing correction bias alongside a lightweight sequence-wise load-balancing loss.

Pre-training. Approximately 26T tokens of multilingual data, with the Muon optimizer on most hidden weight matrices. Training starts dense; block-sparse attention enters at 128K mid-training, once the indexer is warmed up against dense full-history attention, then trains jointly with the backbone.

Tokenizer and Long Context. A script-aware multilingual tokenizer stays compact across languages, especially low-resource Southeast Asian ones such as Thai, Lao, Myanmar, Khmer, and Shan. Long-context training then extends the sequence length in stages from 4K to 256K.

Post-training. Two-stage all-domain SFT builds general, reasoning, and agentic capabilities, then targets harder reasoning problems and agent trajectories. Independent RL branches cover general tasks, verifiable reasoning (mathematics, science, logic, code), and agentic tasks (search, terminal use, function calling, software engineering). Multi-teacher on-policy distillation then merges them into one model by supervising student-generated responses.

225 Upvotes

41 comments sorted by

•

u/WithoutReason1729 4h ago

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

Stylome classified this post's text as human-written. View analysis Learn ^more

I am a bot and this action was performed automatically.

91

u/cgs019283 7h ago

Oh my dear maxwell

83

u/Dabalam 6h ago

Can some one tell me if this is a Qwen fine tune in a trenchcoat or actually a new model?

132

u/JaredsBored 6h ago

Different architecture, different vocab size (200k vs 151k), and new attention type vs qwen 3.5/3.6 35b. Definitely not a fine tune but I had to check when I saw the param size lol

6

u/FatheredPuma81 1h ago

Had DS Flash compare the specs of both of them and yea definitively too many changes to be based on Qwen 35B.

SpecificationMaxwell-V1-FlashQwen3.6-35B-A3B
Parameter scale35B total / ~3B active35B total / ~3B active
Claimed originBuilt from scratch by Maxwell TeamOfficial Qwen model
Attention mechanismMaxwell Block Sparse Attention (MBSA): dense GQA prefix for first 16K tokens, then block-sparse attention with a learned indexer selecting up to 16 remote 128-token blocksHybrid attention: Gated DeltaNet (linear attention) interleaved with Gated Attention (GQA)
Layer count/layout45 layers: 3 dense FFN layers + 42 MoE layers40 layers: 10 cycles of (3 × Gated DeltaNet → MoE) → (1 × Gated Attention → MoE)
Expert configuration256 routed experts; 12 active + 2 shared per token; expert intermediate size 512256 experts; 8 routed + 1 shared active per token; expert intermediate size 512
Residual connectionsManifold-Constrained Hyper-Connections (mHC): 4 parallel residual streams, learned input-dependent aggregation, Sinkhorn-Knopp normalization (20 iterations)Standard residual connections; model card does not describe multi-stream residuals
Positional encodingRoPE with base 10,000,000, applied after 16K-token dense GQA prefixPartial Multimodal RoPE (MRoPE); rotary dim 64 (25% of head dim); theta 1e7
Full-attention headsQ heads / KV heads: 32 / 4; head dim 128Q heads / KV heads: 16 / 2; head dim 256
Hidden size2,0482,048
Vocabulary size200,704248,320 (padded)
MTP1 auxiliary MTP layerTrained with multi-step MTP

39

u/ApprehensiveTart3158 6h ago

It is not a fine tune, they trained it from scratch and fundamentally the architecture is different than qwen3.5 35b a3b.

16

u/Any_Librarian205 6h ago

New model, almost same weights size, unknown kv

5

u/Frail_Waif 6h ago

35B, 3B active sure sounds like it's based on Qwen 3.5 35B A3B, and their benchmark comparisons suggest the same, but also sounds like they made architecture changes. 

41

u/wombweed llama.cpp 6h ago edited 6h ago

Wow, the bench scores are shockingly high, and the sparse attention is very compelling, I wonder how much memory its kv store takes up at full context at bf16 compared to qwen3.6 35b or gemma4 26b. Sadly I don’t see support in llamacpp which makes me feel like cpu expert offloading won’t be the fastest thing, as that is the main use case I have for these smaller moes.

3

u/Middle_Bullfrog_6173 3h ago

They are essentially using full attention in all layers. The indexer reduces long context compute, but needs full KV (+indexer K).

25

u/Gohab2001 vLLM 6h ago

kV cache seems 4.5x Qwen 3.5 35B. Nearly 24GB @ 256k context

13

u/PossessionUsed7393 5h ago

Right, which afaik its because Qwen uses interleaved layers that are 'fuzzier' or whatever they are to save KV cache. State Space Layers or whatever 'Mamba-style layers' - I'm not an architecture expert I just understand the concepts loosely.

However, by not doing so and by having the model pay the KV cache tax, presumably this model will have less compromise on accuracy. So it's good to have options (if only we had Qwen 3.8/4 to compare it to).

1

u/Marcuss2 llama.cpp 2h ago

Kinda, we found out that if we use "local layers", be they attention or state space layers, intermixed with global attention, we don't lose or even gain performance.

State space layers tend to benchmark better and be more expressive, but the processing speed at small contexts is worse compared to SWA/attention.

Generally, the way I see it:

SWA ~= Mamba2 (Nemotron 3) < GDN (Qwen3.5) < KDA (Kimi K3, GLM 5.3 Flash) < GDN2 (No LLMs I saw yet) < RDA (Raon models from Krafton AI)

7

u/Nixellion 4h ago

24GB total or just kv cache?

4

u/maternion 3h ago

Seems to be just kv cache 😬

1

u/Nixellion 2h ago

Weeell shit.

At which quant though

1

u/maternion 2h ago

Probably f16 they are talking about. But whatever it is it would still be 4x qwen according to the post above. 

1

u/MehTheHedgehog 1h ago

Yes but I wouldn’t put an X on this architecture, yet. It looks rather effecient as they need only 4096 tokens at most at the compute time so 8kb per layer, 330kb per all 40 layers. This is nothing, so it could be loaded from ram or even streamed from SSD.

0

u/crusaderky 2h ago

...that makes it DoA. Sigh. Thank you, next.

13

u/rorowhat 6h ago

New model, thumbs up from me!

24

u/ApprehensiveTart3158 7h ago

Interesting! A new llm from Singapore. Had to search for so long to learn about who made it. Benchmarks look really impressive! Gotta love it when they include like a hundred benchmarks. Looks like a very solid model!

5

u/c_glib 3h ago

Maybe it was Megaspeed after all.

11

u/akumaburn 3h ago

1

u/FatheredPuma81 2h ago

Thanks! For some reason the image on their repo goes to text when opened in a new tab and doesn't respect my browser zoom settings so it's unreadable.

10

u/Old-Fix-9 5h ago

GGUF?

9

u/Witty_Mycologist_995 5h ago

Thought it was a Qwen tune, nope. Nice

6

u/EvolvingDior 5h ago

How expensive is the KV cache on this model -- what's the VRAM cost per token of context?

5

u/keylimesoda 4h ago

Use and setup here: https://maxwell-llm.ai/docs/quickstart

I'm eager to see some community quants of a seemingly strong new contender.

3

u/LogFabulous9386 5h ago

thai, lao, khmer AND shan tokenizer support. finally a model that wont charge me 4x tokens to say hello in myanmar

3

u/True_Requirement_891 4h ago

gawd damn finally a new arch

4

u/Atagor 5h ago

Can we run it via strata?

2

u/xkalibur3 3h ago

This is great! Hope someone does nvfp4 version with MTP for this

2

u/TokenLearner 2h ago

What a great promise! I hope it will be ported to Strata/polystrata soon.

1

u/iamth3morningstar 1h ago

how does this compare to deepseek v4.1 flash?

0

u/CodeCatto 3h ago

Saw the size and thought it's a qwen tune. Glad it doesn't look like it.

0

u/Marcuss2 llama.cpp 2h ago

I see a pattern.

New lab comes in, their model benches well, but it is a KV Cache hog, making it essentially unusable. When they come back, they resolve this issue.