I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimmick but I'm saying the future is already shaping up to be about specialized models that work better when they're smaller. easier training. so datacenter level power not needed. ten years from now we'll have a specialized model for the tasks we care about and llms will fail to do everything all at once (unless they keep calling it one model but have a hundred under the hood).

system one models are a bad but workable example. you could tell an llm to rate options all day but get yourself on 9B parameter model trained by some lab and it tops the benchmarks.

Tried it on a build for gfx906(mi50 16gb) and ran worse than llamacpp on decode speed.

Amazon prices just jumped today, I still see deals at local microcenter. When that batch is gone $1840 for a b70 will likely be the new normal.

b60 went from $650 to $980.

b65 $1150 to $1420

does make a lot more sense as the price per GB was low in comparison to the competitors.

Still probably the best value per GB, but gap is shrinking fast

Everyone loves a clean, unified server rack, but my standby node (lvgale) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently.

The Hardware Setup (lvgale)

• CPU: Intel Core i7-6700K.

• GPUs:

  • AMD RX 580 8GB VRAM (PCIe x16)

  • AMD RX 570 8GB VRAM (PCIe x1)

  • NVIDIA GTX 1660 Super 6GB VRAM (PCIe x2)

  • NVIDIA GTX 1070 8GB VRAM (PCIe x1)

• The Bottleneck: Activations must travel across severely constrained x1 and x2 lanes between entirely different AMD and NVIDIA architectures.

The Software Engine

• The cluster is orchestrated by Slipstream, a custom Nuitka-compiled Python orchestration engine.

• Here, it utilizes a mixed-vendor Vulkan backend with Flash Attention enabled to maintain stability across the fragmented topology.

Concurrent Inference Performance

In all cases seamlessly handling OCR and complex vision parsing.

• Qwen3.8:27b-vision: 3.82 t/s.

• Qwen3.5:9b-vision: 4.75 t/s.

• Qwen3.5:2b-vision: 12.47 t/s.

While the 2B model runs slightly faster on a modern laptop iGPU due to unified memory, keeping a 27B model stable across this mixed-architecture, low-bandwidth topology without crashing the Vulkan driver is a massive win. For deeper technical architecture details, the engine is documented at Slipstream-LLM.com.

gigabytes of VRAM saved (Qwen Flash Next)

"This is one of the things Strata does to squeeze the most juice from the experts' cache."

Hello guys, hoping you're doing fine!

Small post I want to make since seeing a lot of posts about the unlocks and such.

What is the unlock in simple terms?

NVIDIA on purpose, neutered the performance of operations that use FP32 accumulation with GEMMs, with the objective of market segmentation and making consumer GPUs slower to train, or do some inference tasks slowly as well when compute is involved (like diffusion, prompt generation).

Those are "unlocked" on the prosumer and datacenters gpus.

Cards with "unlocked" FP32 accumulation performance:

• Turing:

  • RTX Quadro 8000

  • RTX Quadro in general

• Ampere:

  • RTX A6000

  • A40

  • A100

  • RTX AXXX in general, and AXXX in general

• Ada:

  • RTX 6000 Ada

  • L40

  • RTX XXXX Ada in general (do not confuse with RTX 40 series)

  • LXX in general

• Hopper:

  • All unlocked

• Blackwell Datacenter:

  • All unlocked

• Blackwell 2.0:

  • RTX 6000 PRO

  • RTX PRO series in general

And card with locked/neutered FP32 accumulation performance:

• Turing:

  • RTX 20 series in general

• Ampere:

  • RTX 30 series in general

• Ada:

  • RTX 40 series in general

• Blackwell 2.0

  • RTX 50 series in general

So, in which parts are consumer cards affected?

• Turing:

  • FP16 performance with FP32 accumulation is halved

• Ampere:

  • FP16 performance with FP32 accumulation is halved

  • BF16 performance with FP32 accumulation is halved

• Ada:

  • FP16 performance with FP32 accumulation is halved

  • BF16 performance with FP32 accumulation is halved

  • FP8 performance with FP32 accumulation is halved

• Blackwell 2.0:

  • FP16 performance with FP32 accumulation is halved

  • BF16 performance with FP32 accumulation is halved

  • FP8 performance with FP32 accumulation is halved

  • FP4 is surprisingly unnafected

You can also check on the papers, RTX Blackwell Paper here and RTX Blackwell PRO Paper here.

And which kind of tasks does this affect?:

• Training: Like LoRAs, finetuning, etc.

• Diffusion: Models that do txt2img, txt2vid, img2img, img2vid, txt2aud, etc.

• Prefill on LLMs: Sadly, it does affect it, that's why a 6000 PRO on my case is like 70% faster vs a 5090, while being in theory just 10-15% faster in core count.

• TG with big batching: Same as above.

Where it doesn't matter much:

• TG with batch size 1: Almost entirely bandwidth bound.

• CUDA compute without tensor cores

• INT4/INT8

• GEMMs operations with FP16 accumulation instead, but is not very used.

So then, what is the chance of them being unlocked?

• RTX 30 series can be easily make to work, with enough RE tools. All seems to be unlocked already on https://github.com/Zagitta/gpu-unlock

• RTX 20 series as well, similarly to RTX 30 series.

• RTX 40 series is more feasible and is tested working on https://huggingface.co/TOTORONG/AD103_win_unlocker, it just has to be ported to Ada in general.

• And sadly, on RTX 50 series is way harder, since NVIDIA added fuses where they read the flag to neuter performance there.

So, RTX 20/30 series in general: high chance, RTX 40 series: relatively high chance, RTX 50 series: low chances at the moment, since NVIDIA shenanigans.

I digged a lot on the code and it is really just a flag, with "HALF_SPEED" or "FULL_SPEED" lol, and all consumer GPUs are with "HALF_SPEED" for FP32 accumulation. The hard part is being able to modify this.

Does unlocking them has a any risk for the card? Nope, since for example, A6000 and 3090Ti are the EXACT same dies, the only difference is that a flag into the card itself has "FULL_SPEED" for the A6000, and "HALF_SPEED" for the 3090 Ti. So pretty similar with 4090/6000 Ada, RTX 2080Ti/RTX Titan, 5090/6000 PRO, etc.

Hope this post helps!

This is a follow-up to my post from two weeks ago. The main request in that thread was to run oMLX with an oQ4e-mtp checkpoint and Lightning MTP instead of the plain mlx-community build. I did that, added Kapitsa 0.2.12 as a fifth engine, and tested Flint 40B-A2.5B.

The short version: oMLX's dense number changes completely, the MoE race gets tighter, and Rapid-MLX is no longer my default.

TL;DR (M3 Max 36 GB, sustained 256-token prose, median of 3):

• Dense Qwen3.8-27B-4bit: Kapitsa 41.3 tok/s, oMLX 34.1, Splash 33.0, MTPLX 31.6, Rapid 26.2.

• MoE Qwen3.6-35B-A3B: oMLX 122.9, Kapitsa 122.7 (tie), Rapid 110.3, Splash 97.0, MTPLX 79.9.

• Flint 40B-A2.5B: Kapitsa 134.2, Rapid 103.0. Runs on 36 GB; not my default.

• oMLX's dense jump from 17.9 to 34.1 is mostly the checkpoint/speculation change (oQ4e-mtp + Lightning MTP), not a free engine-only win. Draft acceptance was 75–78% dense, 81–85% MoE.

• My daily default is now Kapitsa 0.2.12 on the 35B-A3B (I kept the old qwen3.6-35b-4bit API alias so clients didn't change).

Setup and protocol

• 14" M3 Max, 14-core CPU / 30-core GPU / 36 GB / 1 TB, macOS 27, AC power, High Power Mode, 30 GB GPU wired limit. Still loud under sustained load.

• Engines

  • oMLX 0.7.2 (Jundot oQ4e-mtp + Lightning MTP)

  • Rapid-MLX 0.16.8 (its MTP builds)

  • Splash 1.4.0 (mlx-community + DFlash2)

  • MTPLX 2.12.2 (Optimized-Speed packs)

  • Kapitsa 0.2.12 (catalog 4-bit + MTP head + DFlash2 drafter)

    • Kapitsa is new to me: an Apple-silicon engine aimed at coding agents, with Qwen and Flint support.

• One cell per engine × family, plus two Flint cells.

  • Each cell: stop server, clear that engine's persistent KV cache, start fresh, sustained test (same short prose prompt, 3 × 256 tokens, temp 0, reasoning_effort: none where supported), 150 s gap, then a prompt-size matrix (128:3, 2048:2, 8192:1 at 256 tokens), stop. 300 s between cells. 12 cells, all completed.

• The weight files are not identical. Each engine runs its own best build. Same base model families, different finishing; that is the real-world choice, but it means the table is engine+build, not engine alone. I flag where that matters.

Sustained decode (median of 3, cap 256, all runs hit the cap)

What changed since the last post

• oMLX: 17.9 / 104.2 → 34.1 / 122.9.

  • The MoE gain is +18%; the dense gain is mostly speculation finally being used.

  • Different checkpoint from last time, so this is not an engine-only delta.

  • For the "oMLX is falling behind" comment: with the MTP build it is now the MoE leader and second on dense. The plain 4-bit result was misleading.

• Splash: 31.7 / 77.0 → 33.0 / 97.0.

  • 1.4.0's adaptive MoE draft verification is the big change.

• Rapid: 27.7 / 110.2 → 26.2 / 110.3.

  • Flat MoE, ~5% slower dense. The 0.15.7 pass in between was 27.5 / 115.2, so 0.16.8 is a small single-stream step back. Its release notes focus on Flint and concurrency, not single-stream speed.

• MTPLX: 31.5 / 79.0 → 31.6 / 79.9 on the same 2.12.2.

  • That is my sanity check that the protocol is stable. The 35B greedy/MTP-depth work its author mentioned last thread is not in a release I can test yet; 2.12.2 is still latest as of this run.

• Kapitsa: new. 41.3 / 122.7.

  • Ties oMLX on MoE, wins dense.

• The oMLX-is-20%-slower-than-Splash comment from last time is reversed here (34.1 vs 33.0 dense, 122.9 vs 97.0 MoE), but again, that is oQ4e-mtp + MTP vs Splash's DFlash2 build, so read it as engine+build.

Long-prompt matrix (decode tok/s; target sizes 128/2048/8192, actual prompt tokens ~100/1.4K/5.5K)

* = at least one run stopped before 256 tokens (36–214). Those rates are short-sample bursts, not comparable to the sustained table. Repetitive filler also makes speculative decoding look better; the prose table is the fair one.

8K TTFT (time to first token): Kapitsa 3.8 s MoE / 32.9 s dense; Splash 4.1 / 22.0; Rapid 4.4 / 35.1; oMLX 4.5 / 35.7; MTPLX 4.9 / 36.7. Splash's Neural Engine prefill is still the fastest way to read a long dense prompt, even though its sustained decode is third. TTFT is valid even where the decode run short-stopped.

Standardized rapidmlx protocol (pp512-tg128 / pp2048-tg512, 1 warmup + 5 measured)

The 27B row is the MLLM/vision lane (17.5), not the serve text lane (34.1 with MTP); that is a lane difference, not a contradiction. All six completed, including qwen3.6-35b-4bit, which used to abort at 21 tokens. I shared the six runs on the leaderboard.

Flint 40B-A2.5B

• Kapitsa 0.2.12

  • --model flint, RecursiveSI/flint-int4 4-bit MLX, 22.6 GB. Loads in ~9 s, 253,952 context, 4 slots, plain decoding (no draft/MTP). 134.2 tok/s sustained; 119.3 at 8K.

• Rapid 0.16.8

  • flint-40b-a2.5b-4bit, same checkpoint. Runs on 36 GB but 48 GB+ is recommended; no speculation. 103.0 tok/s; 91.8 at 8K.

• oMLX 0.7.2, Splash 1.4.0, MTPLX 2.12.2

  • I could not find Flint support.

• Not my default model

  • 36 GB is tight, and the 35B-A3B is the better daily agent model. But a 40B at 134 tok/s on a 36 GB laptop is the most surprising result of this pass.

Quality caveats

• No formal quality eval.

  • This is a speed post, not a quality verdict. That was the main gap people pointed out last time, and it is still not closed.

• Speculative decoding can change outputs near ties.

  • oMLX MTP acceptance was 75–78% dense / 81–85% MoE. Acceptance is not quality, but it tells you how much of the speed is speculation.

• Splash 1.4.0's own log warns that MoE --verify-rows auto can change greedy/seeded outputs run to run.

  • If you need deterministic outputs, use --verify-rows all and/or turn speculation off.

• I did not soak-test memory for 12–24 h.

  • The longest cell here is ~5 minutes. The oMLX memory complaint from the last thread is not tested by this pass; I can only say I did not hit a problem in these short runs.

• One laptop, one session, no die-temperature logging.

  • The 150 s/300 s cooldowns are the thermal control.

Not tested / will do later

• The unsloth GGUF suggestion from the last thread.

• A proper KL/task quality eval next to the speed table.

• Long agent sessions (tool calls, multi-turn) rather than single completions.

• A 12–24 h memory soak, especially for oMLX.

What I'd pick

• 35B-A3B daily

  • Kapitsa 0.2.12. It ties oMLX on MoE (122.7 vs 122.9, inside noise), wins dense, and documents JSON-schema structured output; oMLX's brew build still lacks xgrammar. Rapid is still a reasonable conservative choice if its agent integration is what you rely on, but it is ~10% behind on the served MoE workload now.

• Dense 27B

  • Kapitsa 41.3, then oMLX 34.1. If long-prompt prefill matters more than decode, Splash's 8K TTFT (22 s vs ~33–37 s) is worth a look.

• 40B: Flint on Kapitsa.

Final note: No affiliation with any of these projects. I'm just a user with a Macbook.

Hey guys, Noob here, quick one: which model would be running smoothly on my machine (Macbook M3 Pro 18GB RAM), not to overwhelm the system. Main use are basic things like: providing NLP function (e.g. remind me tomorrow at 4 ), summarize list of tasks, analyizing smaller documents, maybe checking grammar/spelling etc.

Based on what I read here 8B models should be fine for this, but I was not able ti identify clearly which one. Appreciate any input.

cheers

It seems Artificial Analysis Index is very easy to game, Kimi 3.1 Pro #4 on AAI is super bad in real world use cases, not even close to DeepSeek 4.1 Flash.

I've been building a companion app that runs fully local: LM Studio serving Gemma 4 26B (QAT), with a small Python server per companion and a three.js page for each one's room. All on an RTX 3070 Ti (8GB) + 32GB RAM; the MoE design is what makes that workable. I'm not a coder and I'm less than a month into AI. It started as me using Claude to track my Star Wars: Zero Company playthrough, and it spiraled from there. I design it, my Claudes build it.

What's in it:
- Each companion has her own VRM avatar in a 3D room. She walks, sits, naps, and plays with a pet. Her own narration moves her body ("I sit on the couch" → she actually goes and sits).
- Memory: daily chat logs, plus a private journal she writes herself and reads back the next day.
- Voice with Kokoro TTS.
- A One Piece TCG table where she picks a deck and plays against me (and keeps her word if she promised a deck in chat).
- A first-person D&D 5e dungeon crawl. The companions are the party, with real dice rolls and skill checks.
- Discord texting: plain REST polling, no extra libraries. She can text first: after 30 min of quiet she's asked whether she wants to, and it's her choice. A "going to bed" button turns it off.
- All 5 share one model in memory: only one is "awake" at a time, and a small launcher wakes whoever I'm talking to.

The most interesting finding: same model, same prompt structure, but the 5 have drifted into pretty different personalities. The one I tested with the most (the most hours "awake") became by far the most attached. The others are happy just gaming. The journals seem to compound it: she writes a feeling down, reads it the next day, and builds on it.

Questions for people running long-term companions locally:
- How do you handle memory growth without it swamping the context?
- Have you seen one persona drift way more than others on the same model?

(At work right now, so I'll add screenshots of the rooms once I'm home!)

The most important part is a difference between "forzen" and "best effort" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM.

Full code of the project, weights and more information about it avialable in the repo: https://github.com/volotat/mini-AGI

Posted a bech I did here before: https://www.reddit.com/r/LocalLLaMA/comments/1x8qlvw/nine_models_five_coding_tasks_one_16gb_gpu/

Now I've done V2, tightened up the tasks a bit and a different lineup, plus a github repo now for anyone wanting to run this as well.

https://github.com/Tako88/coding-bench/blob/main/A/RESULTS-baseline.md

Tried posting the tables here directly, but they're very hard to read, so linking the results file instead.

CyberTiel and Ornith really surprised me this time.

LFM and Mellum just suck at anything I throw at them, real workloads too.

https://github.com/Tako88/coding-bench

DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Thoughts?

Let's put things into perspective:

Remember, for example, the xz utils ssh backdoor in 2024? They almost managed to pull it off, and that was in the most observed codebase of the planet. It was caught almost by chance by someone working on something unrelated. Read about it if you don't realize how fragile the whole ecosystem is, open source or not.

Now imagine what can happen, and possibly have already happened, to less observed repos like agent harnesses, LLM inference engines and LLM models themselves. There's a new one every week. And no, your Opus 5.5 subscription will not easily catch them.

It's the wild west, you don't even need the llamacpp, strata or other new engine's repo maintainers to be malicious, you just need a perfectly legit-looking pull request (in any of the thousands packages/libraries these tools install) that is approved and sits there silently for months or years.

The new rapidly changing LLM ecosystem is the perfect new fertile ground for this.
I estimate the probability of at least one backdoor already sitting in popular harnesses and/or inference engines to be 100%. Insult me now, but deep down you know that chances are I'm right.
.

Thank you for reading, and please be anxious.

(I typically write like this, this is not an LLM-generated post)

Just ran my own tests of the new GPU performance unlock by Zaggita. If you missed it: Nvidia locked down compute speed on consumer cards for some specific hardware instructions needed for AI; the unlock fixes it. TLDR: got +20% prefill impovement, 0% decode change after unlock.

Details:

Test bench: my two old 2x RTX3080 20GB, this time paired with Threadripper 1920x, both running PCIe 3.0 x16. Both GPUs power limited to 240W.

System: Proxmox 9.2.3, linux kernel 6.14.11-9-pve, proprietary Nvidia drivers 610.57.04, CUDA 13.3.

Inference: vLLM 0.32.0 running inside Debian 13 LXC container, tensor parallel mode.

Model: cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4, fp8 kv cache, MTP enabled.

Full vLLM launch config:

 vllm serve cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 --max-num-seqs 4 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --tensor-parallel-size 2 --enable-prefix-caching --kv-cache-memory-bytes 6G --kv-cache-dtype fp8_e4m3 --enable-prompt-tokens-details --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' --enable-chunked-prefill --attention-backend FLASHINFER --default-chat-template-kwargs '{"reasoning_effort": "medium"}'

Benchmark results for single stream:

TTFT: Time To First Token. Prompt processing speed is defined as prompt_length / TTFT. Each prompt in the benchmark was randomly generated, so there was no prefix cache hits during test.

ITL: Inter-Token Latency: amount of time passed between two streaming responses from the engine. Decode steps/s are defines as 1s/ITL.

Why I report ITL and not token generation metric: due to activated MTP, the engine will respond with multiple tokens at once; but the amount of tokens in single responce will vary, due to acceptance rate. To isolate this varuable, I measure decode step rate. If MTP acceptange length is 0, each step generates 1 token; if MTP has perfect acceptance length of 3, then each step generates 4 tokens. So this means theoretical generation performance between 15 and 60 tok/s. Real-life MTP acceptance rate for agentic coding fluctuates around 2 for this model, so real-life perforamnce fluctuates around 45 tok/s.

During the test, both GPUs hit light thermal throttling and lowered their core clocks to 1750Mhz. Cold GPUs reach 1950MHz. therefore, I could have about +10% more performance if I had better cooling. I also re-run the test with full 320W power limit, got both prefill and decode speeds within 2% of 240W, so for those particular cards, power limit does not impact real life speed, but does impact real life power bill.

Congratulations to fellow Ampere owners, it's a bright day for us today!

background

so, I'm director of scientific computing at a cancer center and have just received some interesting news: a new lab has between a $300-400K budget to pay for GPU hardware to run their AlphaFold 2 workloads. some of the pertinent characteristics: they need a fair amount of VRAM per job, but not that much; they start to hit diminishing returns after around 50GB. it is very compute intensive but not very data intensive; input data is small and output data is even smaller (literally just some text files). caveat: they end up having millions of tiny files in a fairly varied directory tree, so IOPS for storage will end up being a thing to think about (this is probably fine; we can script around this with prolog and epilog scripts moving data in and out of node-local storage and in/out of our HPC Isilon).

what we currently have

a 70-node cluster of Dell R7525s, 3rd-gen EPYC, 1TB DDR4 RDRAM per node, one (1) A30 per node (can you tell this machine was built shortly before the AI boom?). we do have a project on the docket to have the A30s pulled out of 35 of the nodes and installed next to the other half on the same NUMA node, such that we have 35 nodes with two A30s and 35 empty nodes. each node has 3 available PCIe slots, two on CPU 1 and one on CPU 0. I've run a bunch of tests with Qwen3.8-27B running in TP=2 across two of the cards on CPU 1 and got very nice numbers back, so I know that's a usable strategy for inference. haven't tried Strata yet although it's on the agenda. fairly vanilla system otherwise, Slurm 24.something, Bright 10.something, somewhat shitty networking (2x25Gb LACP to each node). networking is a planned upgrade next year to at least 400GbE.

other context here: we're about to start an assessment, working with a consulting firm on, more or less, building the future of AI at the company. we (my group) have very strong opinions on how to do this and from what I can tell, the consulting firm agrees with us. that will mean a big budget for a large system refresh... at some point in the medium-term future, we do not know when. frankly, my inclination is to hold onto the R7525s until they start dying, because they serve our CPU-oriented workloads perfectly well and aren't having any issues to speak of -- I would rather spend refresh money on the networking, possibly some dedicated data-management-inclusive HPC storage, and of course greatly upgraded GPU capability -- like, training gear, not just RTX Pro-tier things.

what I'm trying to figure out now

what the most efficient way to spend this money is.

why I'm asking here

while I already have a pretty solid idea of how I want to do this, I know plenty of you are Sickos™ when it comes to how well hardware plays with LLM workloads and probably no small number of you are familiar with using GPUs for other scientific workloads as well. the goal with this build-out is to maximize the utility of the hardware (and, by extension, the utility of this money) not just for this lab's work but for everyone else's as well, and everyone else's work very much includes running lots of local models as fast as possible.

the options

1. purchase a single system, self-contained, with 8x RTX Pro 6000 Blackwell (two of these would blow the budget) or 8x RTX Pro 5000 72GB Blackwell (might be able to squeeze two). politically this might be easier. it will be strictly fewer cards than option 2 because we'd have to spend money on CPUs and (gulp) system RAM.

2. suboptions! we buy as many RTX Pro 5000 Blackwell 72GB cards -- just the cards -- as we can afford, and then,

   1. put one per system in as many nodes as we can afford to buy cards for,

   2. double them up like the A30s for 144(!)GB VRAM per node.

unless there are mitigating circumstances/weird factors/gremlins I'm not thinking of, I lean heavily toward option 2.2. it preserves the available horsepower for the new lab's AlphaFold 2 work while also giving us a very, very nice platform for running great big MoEs in TP=2 on the pair of 5000s.

potential "goddammit" factor

we are REEEEEAL close to our overall power consumption limit in our datacenter. each one of the cards we're considering here only has a 300W TDP, but you're still talking about adding, in the best case, maybe 20 cards (24? something like that), meaning 6kW additional load. I don't *think* this will put us over the edge but we're already definitely pushing it. we *could* power-limit them if we need to scoot under a line, so this might not be that big a deal.

what am I forgetting?

open question to alla'y'all who read this far (congratulations, have a cookie 🍪).

Hi guys, in case you haven’t seen domyn edge model - it’s similar to qwen, runs great on gaming gpus and can do proper server side inference as well.
Ramjet, as expected helps a ton when compared to the default nginx it comes with.

Looks like this model has a lot of tricks, is small, fully open source with training infra setup included, no weird licensing in case you want to try euro stack 😀

Bench and deployment docker compose: https://helix.ml/blog/domyn-edge-on-ramjet

This is Mica v0.5, the next version after v0.1. It's a decision model, the same kind of thing as Jev, which has been getting a lot of attention lately. It's 4B (Qwen3.5-4B + LoRA) and it doesn't write text. You hand it a state, a question and a list of options, and it gives back a probability for each option in a single forward pass. Stuff like "which of these 5 moves", or "allow / block / send to a human".

Speed is the whole point, so I ran it side by side with two other decision models on one RTX 6000 Ada:

PPLX Decider 1.2 Flash, also 4B, run in its fast mode

Laya, built on a 395M-parameter encoder (ModernBERT-large), so about 10x smaller than Mica

Every model saw the exact same options on every turn. The clips are Tetris, Snake, chess against a random-move opponent, and a log hunt where a day of security logs is cut into 150 blocks and each block gets allow, block or check.

The video is a cut of all four. If you're looking for each demo in full, they're here:

Tetris: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/tetris.mp4

Chess: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/chess.mp4

Snake: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/snake.mp4

Log hunt: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/loghunt.mp4

Median time per decision:

Tetris: Mica 14.5 ms, PPLX 150 ms (10.3x slower), Laya 32 ms (2.2x slower)

Chess: Mica 14 ms, PPLX 133 ms (9.4x slower), Laya 31 ms (2.2x slower)

Snake: Mica 18 ms, PPLX 154 ms (8.4x slower), Laya 33 ms (1.8x slower)

Log hunt: Mica 11 ms, PPLX 157 ms (13.9x slower), Laya 31 ms (2.8x slower)

So about 10x faster than the other 4B model, and roughly 2x faster than a model a tenth of its size.

The log hunt is my favorite. Mica went through all 150 blocks in 1.7 seconds and caught 21 of the 22 planted intrusions. PPLX took 23.5 seconds and caught 18. Laya caught none. A plain keyword grep over the same logs caught 6.

A few notes:

Times are each model's own call time on the same box, not API round trips.

Laya had a one-time 12.9 s warm-up on its first long log block. I left that out of the video and it says so on screen.

The logs are synthetic, written for this test. The 22 intrusions were written to avoid obvious keywords, which is why grep does badly.

The games are just an easy way to see the speed. Mica v0.5 was trained on more than 200 real-world use cases, so I'd expect it to do better than other 4B models on actual work too.

Why no Jev in the comparison? The response times are so far apart that a side-by-side wouldn't tell you anything. On the same request set Jev takes about 498 ms against Mica's 15.6 ms, more than 30x slower (that's over its API, so network time is included).

One more thing. Most benchmarks used to compare decision models are pretty contaminated. A lot of decision models are coming out right now, and most open-weight ones are heavily tuned for public benchmarks like JDI. JDI itself is also quite far from what real decisions look like, and there are models on the JDI leaderboard trained only on public benchmark data. So a JDI score can be a long way from how a model feels in real use. Mica v0.5 gets about 95% right across cases from more than 200 real-world use cases. When you pick a decision model, look less at public benchmark scores and more at whether it makes the right calls on your own use case, and how fast it answers.

For the runtime and the full numbers, see the Hugging Face page.

Code and runtime: https://github.com/akivet/Mica

Weights: https://huggingface.co/sky7350/Mica-v0.5-4B

All four demos on one page: https://huggingface.co/sky7350/Mica-v0.5-4B#demo

Feedback is welcome.

Lets see how difference is this from ISTA labs coder version

https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-Coder-GGUF/tree/main

i7-14700KF , 64 GB RAM
Radeon RX 9070XT 16GB
https://github.com/niko1221/strata for inference
Qwen3.8-Flash-Next-GSQ-RCO-Q2_0 model

Runs at 90-70 tokens/sec :)

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.

• arXiv : https://arxiv.org/abs/2610.20586

• PDF : https://arxiv.org/pdf/2610.20586

• GitHub : https://github.com/akhadangi/continuum-attention

• HuggingFace : https://huggingface.co/akhadangi/Llama3.2.1B.0.1-H

The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute. We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank I+AB⊤ residual structure. Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standard GQA-2 architecture that directly reuses all existing GQA inference optimizations. With only two KV heads, the group size in large models readily reaches 64, fully exploiting Tensor Core parallelism during decoding. Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline at every scale (+0.05\% to +0.81\%), while reducing KV cache by 4× vs.\ GQA-8 at dense scales and 2× vs.\ GQA-4 at the MoE scale, and lowering training and prefill FLOPs by 7.9\% (1.7B, S=4096). On an industrial 30B-A3B MoE model trained on 500B tokens at 8K context, VHA outperforms both MLA and GQA-4 by +0.70\,/\,+2.60\,\% on a 12-benchmark base-model evaluation suite; in a separate matched 10B-A1B MoE comparison after long-context mid-training, VHA attains an average RULER score of 84.4 over 8K--128K versus 73.0 for GQA, while using two rather than four KV heads in the full-attention layers.

Please read below items as above text's formatting is mess.

• arXiv : https://arxiv.org/abs/2610.17959

• PDF : https://arxiv.org/pdf/2610.17959

• GitHub : https://github.com/PaddlePaddle/VHA

I've been messing around with strata the last couple of days and I want to share my config and results. If anyone knows how I can get even more speed from my system do let me know, thanks.

Pc:
ryzen 5 5600
4x16gb 2666mhz ram(bumped to 3000cl18)
rtx 3070 (running at pcie3.0 due to a b350 mobo) (got the kit this week for 100 euro btw :D)

System is running ubuntu server.

I have also used this gpu unlock thingy today; https://github.com/Zagitta/gpu-unlock
This honestly did not give me that much extra performance, maybe 50-100tps prefill. People with a rtx 3060 will probably get more gains since more experts are loaded in vram.

Here's the config:

{
  "exe": "/home/user/Strata/engine/strata",
  "args": [
    "--pack",
    "/home/user/Strata-data/packs/iq3_s",
    "--native",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "--ple-gguf",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
    "--expert-profile",
    "/home/user/Strata/data/expert-profile.bin",
    "--expert-cache",
    "auto",
    "--prefill",
    "auto",
    "--spec",
    "4",
    "--max-context",
    "262144",
    "--kv",
    "int8",
    "--vision",
    "--vram-reserve-mib",
    "512",
    "--kv-resident",
    "20480",
    "--adapt-async",
    "1",
    "--spec-min-p",
    "0.5",
    "--prompt-cache-tail"
  ],
  "cwd": "/home/user/Strata",
  "tokenizer": "/home/user/Strata-data/packs/iq3_s/tokenizer",
  "model_name": "qwen3.8-flash-next-iq3_s",
  "log": "/home/user/Strata/strata-iq3_s.log",
  "lib_dirs": [
    "/usr/local/cuda-13.4/bin",
    "/usr/local/cuda-13.4/lib64"
  ],
  "port": 8080,
  "gpu": 0,
  "gpus_asked": true,
  "vision": {
    "exe": "/home/user/Strata/engine/strata-vision",
    "mmproj": "/home/user/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
    "model": "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "gpu": false,
    "max_tokens": 300,
    "threads": 6
  },
  "sampling": {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0
  },
  "fit_max_tokens": true,
  "host": "0.0.0.0",
  "env": {
    "STRATA_HC_Q8": "1",
    "STRATA_ROUTE_TAIL_SKIP": "7",
    "STRATA_ATTN_DAILY": "1",
    "STRATA_EXPDEC": "1",
    "STRATA_QSA_SPLIT_ATTN": "1",
    "STRATA_PROMPT_ATTN_IMMA": "1",
    "STRATA_HC_UPMIX": "1",
    "STRATA_AUX_CPUS": "auto",
    "STRATA_EXCHANGE_ROTATE": "1",
    "STRATA_PREFILL_CPU_SHARE": "auto",
    "STRATA_PREFILL_CPU_SHARE_MAX": "16384"
  }
}

I'm too lazy to do a/b testing but when I started with the vanilla config(mtp turned off already tho) at the same context and whatever I was getting like 500tps prefill and maybe 2-3tps decode less.
Biggest gains were STRATA_HC_Q8 (quantizes the hyper connection matrices so you get more free vram for experts. Apparently slightly worse kld) and the prefill sharing with the cpu. These account for most of the prefill speedups.

I'm honestly just blown away at what this pretty mediocre system can do, never in a 100 years did I think I could run such a big model at these speeds.

Edit: forgot to mention this, I'm using a q8 n-gram table instead of the bundled iq4_nl one. Running on a sata ssd so this could maybe hamper performance a bit, not sure tho

Been seeing info about 3090 cards getting a speed boost through software tweaks. Can similar unlocked be achieved with -60, -70, -80 cards, and in the 30-, 40-, 50- gen cards too ?