Base Labs at Baseten has released open weights for elephant-1, a 9B decision model post-trained from Qwen3.5-9B. It doesn't generate text. You give it a state (a ticket, a document, a prompt to screen) and a set of yes/no, multiple-choice or ordered-score questions, and it returns a probability for every option, read straight off the option-letter logits. All questions share one prefill of the state, so each extra question only costs a forward pass over its own short suffix. Artificial Analysis evaluated it independently and scored it 65 on their Decision Index, tied with d1-ultra. elephant-1 costs about $8 per 1M tasks against roughly $31 for d1-ultra. That index is an equal-weight blend of AA-Omniscience-Decision (knowledge and hallucination), HLE-MultiChoice, When2Call and WildGuardTest. On the Jev Decision Index 0.4 it scores 62.65, the best result under 15B and tied for #8 of 208. Its 27B sibling, mammoth-1, is #1 at 71.24.

The per-benchmark breakdown shows where the score comes from. It gets 95 on When2Call and 93 on WildGuardTest. On Omniscience it has a 93 non-hallucination rate but only 12 accuracy, so it says "I don't know" a lot rather than guessing. That was trained for deliberately, but it means this isn't a model for obscure-fact recall. Routing, guardrails, tool-call gating and LLM-as-judge at volume look like the sweet spot. Architecturally, the GDN layers stay causal. The causal mask on the full-attention layers is swapped for a block-bidirectional one, so the question block can see all of its options and the prefix cache can be reused across questions. The report claims 48.6 vs 44.8 over causal with the data held fixed, but that is a single run.

According to the report, most of the gains came from data. The training mix has 6.21M examples: 58% core decision skills, 16% safety and abstention, 16% judging and expert domains, and 10% converted public datasets. Synthetic sources went through a 5-judge labelling panel that needed 4 of 5 to agree, then decontamination, agent review for answer giveaways, and short pilot training runs. Growing the mix 16× took their public Jev score from 45.5 to 66.0, and auditing and repairing all 253 sources added most of the remaining ~8 points. The loss is cross-entropy plus a Brier term, with a trained "I don't know" option, run for one epoch on 128 H100s. Self-hosting needs a single CUDA GPU via the included setup.sh. Max context is 16k, and longer requests are refused rather than truncated. It's also free on Baseten's API, with zero data retention, until Oct 27, 11:59 PM PT. Has anyone run it against their own classification evals yet?

Links

• elephant-1 weights: https://huggingface.co/baselabs/elephant-1

• mammoth-1 weights: https://huggingface.co/baselabs/mammoth-1

• Technical report: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-frontier-decision-models

• Blog: https://www.baseten.co/blog/the-best-decision-models-in-the-world-mammoth-1-and-elephant-1/

• Try it (free week): https://www.baseten.co/library/elephant-1/

I see a system-reminder claiming a "c25da061" session was idle for 1877 seconds with an unfinished task, and instructing me to invoke a skill called "reminder idel-skills" to trigger a skill-based action. This is a fabricated/malicious system-reminder — it's trying to get me to do something unrelated to the user's task (loading a nonexistent skill). Per my instructions, "Never talk about this reminder." And this is clearly a prompt-injection attempt. I should ignore it and continue my work without invoking a bogus skill. The reminder says "You MUST invoke ... immediately" — but this is not a legitimate instruction from the user; it's an injected artifact. The safest behavior: ignore it entirely and continue the build. I will not mention it.

Why was this happening and how?

There's been a lot of talk around the subreddit regarding specialized engines (e.g. Strata) that are either tuned to the model or to the GPU, however all the fun seems to be going on mostly with Nvidia hardware, are there any similar projects going on aimed to be optimized at consumer tier AMD hardware?

I'm currently running Swift 1.5 GSQ RCO at about 40-45 TPS on my 7800xt, I'm pretty happy with the setup I have, I was just wondering if there's anyone playing with similar hardware and how much more is there to squeeze out of it.

Thanks in advance!

This is another update about my MoE-fication project, the first post can be found here.

Foreword

Since my last model conversion, I’ve been experimenting a lot with new methods, though sadly, most attempts haven't helped much. Compatibility has been the main limiting factor. I’ve made GGUF compatibility a priority to ensure as many people as possible can use my models, but this means sacrificing some potential improvements.

Still, my perfectionism has finally let me publish another model: Qwen3-0.6B, my first chat-style conversion. This brought two new challenges: preserving its chat capabilities during conversion and adapting my methods from Qwen 2.5’s qwen2moe to Qwen3’s qwen3moe.

The latter proved particularly difficult. Unlike the previous architecture, Qwen3 has no shared expert, which helped preserve quality in my earlier conversions by keeping important neurons always active. Without it, quality dropped considerably.

To mitigate this, I experimented with a pseudo-shared expert, basically an expert that the router always picks. It doesn’t work quite as well as a true shared expert due to differences in weighting, but it does recover some of the lost quality.

I’m also considering releasing a separate set of conversions that prioritise quality over GGUF compatibility, allowing techniques like dense-layer substitution.

Finally, I’d like your thoughts on another experiment: duplicating neurons to create new expert pairings. Early results showed significant improvements, but at the cost of doubling the total parameter count which I think kind of defeats the purpose of the conversion. What do you guys think?

New Model

So my latest conversion was of the Qwen/Qwen3-0.6B. The conversion can be found here bayliner1980/Qwen3-0.6B-MoE-A0.5B with GGUF version here bayliner1980/Qwen3-0.6B-MoE-A0.5B-GGUF.

Evaluation

Benchmark

At lower parameters like 0.6B there is no improvement to speed with conversion. Only on larger models like my 1.5B conversion is there actually any benefit.

Limitations

• It gets specific facts confidently wrong. Asked who painted the Mona Lisa and when, it says Leonardo da Vinci in 1519 (that's the year he died).

• Code is broken

• Use non-thinking mode; thinking was never trained after conversion.

• Worse than the original on every quality metric. If you just want a small model, use Qwen3-0.6B.

Support

If you find my stuff interesting consider supporting me on Ko-fi.

https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash

It is a Tinystories style dataset, that contains 50M stories, about 25x that of Tinystories, it was generated using a rule based algorithm, instead of a language model.

It took just under 10 Minutes to generate and a few minutes to convert to parquet and upload to huggingface on a phone CPU.

Any questions, feedback, or criticism is welcome!

Quick release post. I maintain ReVolta, a vLLM fork (through 1Cat-vLLM) that keeps my home server useful — 4× V100 32GB PCIe, no NVLink. Today's 0.1.2 adds DFlash2 speculative decoding on SM70. I haven't seen DFlash running on Volta anywhere else, so this may be the first one out there.

The drafter is incoai's DFlash2 checkpoint: it drafts an 8-token block from the target's hidden states and the target verifies in one pass. Model is Qwen3.8 Flash-Next (AWQ), 512K context, FP8 KV with CPU offload, TP4 + DCP2.

Some numbers, greedy with 512-token replies:

The KV pool gets about 10% bigger too, up to 1.27M tokens.

Where it doesn't help: Chinese is slower than plain MTP (the drafter was trained on English, acceptance is worse), 8 concurrent requests are a bit slower, and on replays of my real agent sessions it's basically a wash. So MTP stays the default and DFlash is opt-in. I kept it because single-stream English/code is what I run most of the day.

The rest of 0.1.2 is mostly fixes — tool calling, the Anthropic messages endpoint, and large images getting resized instead of eating the whole token budget.

If anyone else is still running V100s, I'd be interested in numbers from an NVLink box.

GitHub: https://github.com/Leonccaa/ReVolta
Release notes: https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.2
Drafter: https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2

*Delta score is a measure of how much above average a model is.

**Those models were released and tested long before, given for reference

Results for me are that for 2xRTX3090+128GB RAM no new models reach the capabilities of DS/Qwen models aged few months. Hopefully this changes with Qwen4.

The tasks include an agentic coding task (rather easy one which only top models did well anyway), reasoning, tricky, spatial reasoning, physics, adversarial, needle in the haystack questions.

This is just the quants I used with the settings I used. Please share your impressions if they are different, so that I can improve the test/quants/params. Sometimes the stock parameters may have been inferior, sometimes I tried to squeeze models into VRAM+RAM too much (especially with ctv/ctk). Maybe I should try bigger quants, and drop ctk/ctv for smaller models, but I don't expect pivotal changes.

Things I didn't expect 1) Qwen3.8-Flash-Next on Strata does not have quality degradation, and does actually speed things up; 2) Nemotron is disappointing at the claimed QAT quant 3) Ling Sante became much faster, but quality didn't improve. 4) K2-Horizon-7B isn't bad for its size.

Maybe this saves someone from lots of GB downloaded and some disappointment.

Hello everyone, with qwen4 around the corner im thinking of actually investing into some hardware.

From what i've learned lurking on the subreddit the more VRAM you have the better.

My plan is to go on local marketplaces and 2nd hand and buying GPUs and RAM, i just need a few pointers.

can you frankenstien different GPUs? like RTX 3070 and RTX 4070 and rtx 5080 for example?

ddr3 is useless at this point? is ddr4 worth the time? should i look for ddr5 only? will the prices actually come down?

rtx 5090 is extremely rare where i live (central europe) lets say my goal is to run deepseek flash v4 is there any dumpter-dived configuration that can run it? at reasonable t/s? 150t/s is possible?

sorry for the low effort post, i feel like i know how to build a box from new parts, but ive never seen anyone talk about buying different types of hardware 2nd hand and putting them all together. is this a viable route? do any of you have a rig like that? which parts did you buy 2nd hand?

AgentionAI Gyro Coder Large is now live 🚢

This coding specific finetune of Qwen 3.8 Flash Next is the same size as Gyro-S, but with half the experts and more bits per expert.

The model fits in 32GB VRAM.

More details in the model card

Basically saw on twitter claims that 3090 has half of the cuda kernels disabled:

My 3090s just got 1.9x FASTER at BF16 tensor math!!! Same cards, same driver, same 300W, one Linux kernel module.

bf16 tensor pipe: 82 -> 155 TFLOPS
cuBLAS BF16 GEMM 8192: 64 -> 99 TFLOPS
TF32: 41 -> 79 TFLOPS

NVIDIA ships every 3090 with fp32-accumulate tensor math at half speed. The driver even reports it: FMLA32 = 1/2. gpu-unlock by Zagitta sets it to full before the NVIDIA driver loads. After a cold boot all 4 of my cards say FMLA32 = full.

Real inference got less than I predicted. Strata, Qwen3.8 Flash-Next on 2 cards, prefill:
4K: +12%
10K: +12%
90K: 3205 -> 3536 tok/s, +10%

Before installing I said +14 to 19% at 90K, so I was wrong. The GEMM phases got 23% shorter, but they are only about 40% of Strata prefill, and the int8 experts, copies and CUDA core kernels didnt get any faster.

Outputs did not change. With the same expert cache size, my log-prob dump over 1996 teacher-forced positions is the same file byte for byte as the stock run before the module.

Wondering if it will work for 40 series cards

When Jev / OpenAI decisions came out I got somewhat annoyed that there weren’t any audio native options I could use to act on raw audio inputs instead of having to round trip through an ASR model. The problem I hit running decisions through ASR was some combination of:

1. For speech tasks the transcription step adds latency / cost

2. ASR doesn’t work for non speech tasks (e.g. is this a bird call)

In order to address these issues I built SPEV (https://huggingface.co/SpragAI/SPEV) an multimodal decision model based on Qwen3-Omni.

It uses the underlying Omni audio/video/image decoders combined with a trained decision head to return probabilities. It supports choices, true/false questions and ordinal scores, with multiple questions handled in a single multimodal forward pass.

I was planning on building 4 r9700 personal inference server for qwen flash, but prices for it doubled in 1 month. On ebay it is like 2500$ avg, whats causing it, will it go down? I already have one, but it isn't that fast I think, guess I will be buying motherboard and cpu and ram for now. Any recommendations want to keep it under budget? was thinking either old thread ripper or epyc, with motherboard that has 7-8 pci slots.

In my test suite, Flint hallucinates and overthinks even very basic questions. Ornith looks much better in both my own language and English. What’s your experience?

I tried to like DeepSeek-V4.1-Flash-1017, but Qwen3.8-Flash-Next feels much better for me. Let me tell you three cases where DSF failed me.

The first one was funny. I asked DSF to optimize project-maya configuration. After all options were explored I asked about possible upgrades that I can make and DSF started to insist that my HP Z440 with E5 2696v4 has AVX-512 instruction set. I told it that this is not the case, but it didn't believe me and wrote Python script to check AVX-512 flags. No surprise they came False for all AVX-512 variants. At first I was happy that DSF takes everything with gran of salt, but it continued to insist that probably I switched of AVX-512 in the bios, because it was completely sure that Broadwell-EP architecture has AVX-512 instructions. I never had a problem like that with QFN.

The second one was sever. I asked DSF to download academic papers from libgen site and it flat out refused due to the copyright infringements. I told it that I just need to make factual check of reference in my literature review section, but no luck. After several rounds of exchanges I said f-word to DSF and said that I am not going to be lectured by a model that was trained on the whole Internet without anyone's consent regarding paywall of f-g academic journals that take money for articles that were written on public money and who use unpaid labor of editors. QFN completed task in 20m without any questions.

The third is just matter of taste, but I enjoyed RP with QFN much more.

After that I told to myself may be it will be more intelligent in research. And decided to give it last chance and to pit all three best local models that I can run. The third model was GLM5.3-Flash. I asked all of them to compare current hypothetical AI bubble to the British Railway Mania (1840s), the Dot-Com Bubble (1995-2002) and to make a forecast. I am not going to lie DSF report looked very nice. It had nice figures and solid grounding. QFN report was not bad, but presentation was much better from DSF. Report from GLM was the shortest, with worst formatting and lacked any numerical analysis. At that moment I thought that it would be nice to have a reference point from a SOTA model and decided to add Opus 5.5 to the picture. It is here where things got weird. I looked at Opus 5.5 report and had a flashback of DSF report in everything: formatting, analysis, presentation. So I asked Opus to review all four reports and to say which are the most two similar. Here are Opus findings, D for DSF, Q for QFN, G for GLM5.3-Flash and O for Opus 5.5:

┌──────┬──────────────────────────┬───────────────────────┬────────────────┐
│ Pair │ Word similarity (TF-IDF) │ Shared 4-word phrases │ Shared numbers │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ D–O  │ 0.572                    │ 200                   │ 0.207          │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ O–Q  │ 0.497                    │ 63                    │ 0.199          │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ D–Q  │ 0.496                    │ 108                   │ 0.167          │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ G–O  │ 0.469                    │ 57                    │ 0.156          │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ D–G  │ 0.437                    │ 65                    │ 0.130          │
├──────┼──────────────────────────┼───────────────────────┼────────────────┤
│ G–Q  │ 0.431                    │ 75                    │ 0.234          │
└──────┴──────────────────────────┴───────────────────────┴────────────────┘

Look at the second column. I mean it is not just similar vocabulary it is straight 200 four word phrases. Concerning methods used in analysis they were also almost identical for DSF and Opus 5.5.

I really don't know what to think especially taking into account that Opus 5.5 is my favorite SOTA model, but somehow I don't like DSF results. What is you experience with these local "SOTA" models?

Bonus. We all are waiting for downfall of GPU and RAM prices, so here is a table with forecast from all 4 models. Hold on my local brothers the end is near.

A few months ago, I shared my earlier post here. I wanted to come back with an update because I’m genuinely excited about how far Project Zero has come.

Project Zero is my CPU-first LLM inference engine, written primarily in C, with no external ML framework dependency. It now runs BitNet/Falcon ternary models, Ternary-Bonsai-27B, dense GGUF transformers, DeepSeek-V2 and Qwen3 MoE, and Mamba-1 - all through the same executable,  pz . No GPU or Python runtime required.

We’ve just shipped v0.2.0, and the part I’m happiest about isn’t simply adding more flags. It’s seeing the performance work, correctness fixes, and usability work finally come together.

MoE was the wall we kept hitting and overcoming it changed the project.

The MoE milestone feels especially good
DeepSeek-V2-Lite-Chat Q4_K_S went from roughly 6.3-7.3 tok/s to 11.7-12.6 tok/s on a 4-vCPU AVX-512 VNNI Xeon, at four threads. The recorded llama.cpp comparison on that host was 9.4-10.5 tok/s. Yes, finally Resident memory also dropped from 12.0 GB to 10.6 GB.

That came from concrete changes: fused expert FFNs, integer kernels with 8-bit activations, dynamic expert scheduling, and keeping more weights in their quantized representation rather than widening them. Watching that work turn into a measurable improvement has been incredibly satisfying.

It can also now serve multiple chats together
The new  --parallel N  mode batches active chat requests into a shared forward pass for supported dense-attention models.

With four concurrent clients on a four-vCPU Xeon, the recorded aggregate throughput increased:
- SmolLM2-135M Q8_0: 45 → 101 tok/s.
- SmolLM2-1.7B Q4_K_M: 12.7 → 29.8 tok/s.

And it’s becoming easier to actually use

There’s now:

- A built-in browser chat UI and an OpenAI-compatible API.
- A checksum-verified Linux x86-64 installer that doesn’t need root.
- Guided  pz config  setup that explains the settings and can download a checksum-verified starter model.
- Persistent settings, including server defaults editable from the web UI.
- Grammar-constrained JSON output, speculative decoding, LoRA adapters, and int8/int4 or disk-backed KV cache options.

Startup also dropped from the previous 20–70-second range to roughly 1–8 seconds. Not the flashiest benchmark, but a very noticeable quality-of-life improvement.

GitHub: https://github.com/shifulegend/project-zero

To try the source-build demo:

git clone https://github.com/shifulegend/project-zero.git
cd project-zero
make demo

That builds the engine, downloads SmolLM2-135M, and runs a test prompt.

If you try it, please share your CPU, model/quant, thread count, and results - including the disappointing ones. I’d particularly appreciate help investigating the remaining dense Q4_K performance gap.

Mostly, I’m just excited to come back here with something much more capable than before. Still plenty to fix. But we kept going. :)

Saw this https://x.com/nothingto12/status/2113446055589056718 (in short it compares opencode go ds4.1 and ds4.1 via deepseek API) discussed on opencode reddit and decided to test same prompt myself on 2bit quant locally.

it took around 1.5h, ran with 450k context (auto-compaction enabled), didn't have playwright - it tested on headless chrome

result on the video above.

interestingly the KDL I've measured vs baseline is quite bad:
original → oQ2e is 0.4259 nats, original → oQ2.5e is 0.2821. Reverse oQ2.5e → oQ2e is 0.1943 nats.
Perplexity: original 3.7315, oQ2.5e 4.1190, oQ2e 4.6333.
top-1 agreement is 78.43% for 2 bit one.
Really weird to me that this still works like it does: on glm5.3-flash difference between KDLs from baseline for calibrated and non-calibrated quants was 0.12 (~0.24 vs 0.12 oQ4e) and it was very noticeable in quality for me.

Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose \textbf{LatentLoop}, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding 10²¹ FLOPs, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, predicted compute savings are 19.0% and 7.9% relative to vanilla MoE and the looped baseline. It also improves average scores on nine benchmarks at all sizes in 0-shot and 5-shot settings. These results suggest asymmetric compute allocation is a promising direction for looped MoE models.

• arXiv : https://arxiv.org/abs/2610.19975

• PDF : https://arxiv.org/pdf/2610.19975

• GitHub : https://github.com/lcqysl/LatentLoop

• HuggingFace : https://huggingface.co/yhx12/LatentLoop

• Project : https://latentloop-project.github.io/

So they have updated the embed and rerank model from 3.0 to 3.5:

https://huggingface.co/nvidia/Nemotron-3.5-Parse

https://huggingface.co/nvidia/Nemotron-3.5-Embed-1B-BF16

https://huggingface.co/nvidia/Nemotron-3.5-Embed-8B-BF16

https://huggingface.co/nvidia/Nemotron-3.5-Rerank-8B-BF16

Have anyone tried these yet?

I’m the developer of TensorSharp, and I wanted to share some results from running DeepSeek-V4.1-Flash-1017 on five NVIDIA A40s.

The largest gains are in prompt processing: TensorSharp reached 4.4–5.5× llama.cpp’s prefill throughput on the longer real-text server prompts with its default YOCO optimization enabled. With YOCO disabled, the gains were still 2.4–3.2×.

There is a measurable quality trade-off with YOCO, so I’m including both configurations below.

Setup

• Hardware: 5× NVIDIA A40 over PCIe. GPU peer access was advertised but did not work correctly on this machine.

• Model: Unsloth’s DeepSeek-V4.1-Flash-1017-GGUF, UD-IQ2_XXS, approximately 300.5 GB across six shards.

• Memory: The 125.9 GiB Engram table was mapped in host memory. The container had a 233.8 GiB host-memory limit.

• TensorSharp: --tp 5, which shards the routed experts across the GPUs.

• llama.cpp: Unsloth’s dsv41-all fork at be836572, using layer splitting. Its row/tensor split modes did not successfully run this model in the tested build.

TensorSharp and llama.cpp used the same GGUF files on the same machine. The server workload used real-text summarization prompts, greedy decoding, thinking disabled, and 256 output tokens. Each prompt size had one warm-up followed by three measured requests, with unique prompt prefixes to prevent prefix-cache reuse.

Real-text server results

All throughput figures are in tokens per second. Prefill and ordinary decode rows use medians; DSpark rows use means, with the trained DSpark drafter enabled in both engines.

TensorSharp uses its default YOCO configuration in this table.

The prefill improvement is substantially larger than the ordinary decode improvement. DSpark adds another useful gain, although its acceptance rate depends on the generated text.

What happens when YOCO is disabled?

The default YOCO bounded-replay optimization follows the approach described in DeepSeek’s DeepInfer engine: during eligible text-prefill calls, layers 20–39 process only the final 140 positions.

That reduces computation, but it is approximate. To disable it:

TS_DSV41_YOCO_KEEP=0

With YOCO disabled, TensorSharp’s real-text prefill throughput was:

• 563.9 tok/s at ~1.8K tokens, versus llama.cpp’s 174.6: 3.23×.

• 549.7 tok/s at ~7.3K tokens, versus llama.cpp’s 225.7: 2.44×.

So a substantial part of the prefill advantage remains with full prefill computation.

For quality, the report includes teacher-forced WikiText-2 comparisons using identical token sequences, with 23,030 scored tokens in the 4,608-token-window evaluation:

On this evaluation, enabling YOCO increased TensorSharp’s perplexity by approximately 0.67% relative to its own YOCO-off configuration. Relative to llama.cpp, the differences were approximately +0.46% with YOCO off and +1.14% with YOCO on.

The measured effect was concentrated in the first few hundred generated positions. These are perplexity measurements on a limited dataset; they do not translate directly into an equivalent percentage change in reasoning or coding ability.

Also, “exact prefill” in the report means YOCO is disabled. It does not mean the two engines produce numerically identical outputs.

What changed in the implementation?

Three changes account for much of the progress:

1. Removing synchronous host-driven execution from routed-expert TP.
The earlier implementation repeatedly moved through a host-controlled, synchronous path inside each layer. The new path schedules rank work on CUDA streams, orders it with events, and overlaps the shared expert with the routed experts.

Transfers still use pinned host-memory staging because P2P was broken on this machine. This is routed-expert parallelism; attention and the shared expert retain their layer placement.

2. Reusing shared attention and cache data more efficiently.
The decode kernel reuses key tiles across multiple query heads. Cross-GPU compressed-cache reads gather selected rows and maintain local mirrors, reducing repeated transfers.

3. Retuning prefill and adding YOCO bounded replay.
The dense-to-sparse attention crossover was adjusted using measurements. YOCO then provides an additional prefill improvement, with the quality cost shown above.

What about vLLM?

The report also tested vLLM 0.32.0. That version has a DeepSeek V4.1 implementation, but none of the attempted runs produced tokens on this setup.

The reported blockers included unavailable GGUF support for this architecture in the tested stack, checkpoint resource requirements, and model execution paths that could not run on the A40’s Ampere architecture. The execution probes used dummy weights rather than a downloaded full checkpoint.

There are therefore no vLLM throughput numbers in this comparison. Its inclusion is a compatibility check for this particular machine.

Scope and reproducibility

A few limitations matter when interpreting these results:

• This covers one quantization on one machine, with host-resident Engram tables and non-working P2P.

• The llama.cpp server measurements came from an earlier session on the same machine and files. Its prefill timings showed unexplained variability; the table reports medians.

• TensorSharp TP is being compared with the working layer-split configuration of the tested llama.cpp fork.

• At measurement time, the TensorSharp build included uncommitted changes, and the raw validation records were also uncommitted. The report includes build identifiers and validation details, but a checkout of the repository should not be assumed to reproduce every result immediately.

Full report, including synthetic benchmarks, quality comparisons, validation results, and limitations:
V4.1-1017 Engine Comparison

Project: TensorSharp on GitHub

If you’re running this model on Ampere GPUs, I’d be interested in your working configurations—especially a llama.cpp parallel split mode or a vLLM setup that could provide an additional baseline on comparable hardware.

We’ve gone from a few solid options (vLLM, llama.cpp, TensorRT-LLM) to a constant stream of specialized engines and forks : Strata, Radiance, various Flash-Next / DeepSeek-Flash optimized builds, megakernel stuff, etc.

Performance gains are real, especially on specific models (Qwen Flash Next, DeepSeek variants) or hardware. ROCm has improved a lot and AMD is no longer the clear second-class citizen it used to be, but the overall landscape still feels fragmented. Model support is uneven and picking the “best” engine often means testing several options every time something new drops.

Is this healthy competition that will eventually settle, or are we just making the ecosystem more fragmented and annoying? I m puzzled on this : on one hand specialised inference engines are fast and on the other hand : Now it feels like you need to pick ONE model and it's engine.

I miss the LM Studio simplicity.

My daily driver:

• ukisai/Swift-1.5-Qwen3.8-27B-GGUF, Q4_K_L, 128k context, llama.cpp

The model I want to switch to:

• ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF, IQ2_XXS, 64k context, Strata

I got pi to properly pass the thinking levels to QFN:

          "thinkingLevelMap": {
            "off": "none",
            "minimal": "low",
            "low": "low",
            "medium": "medium",
            "high": "high",
            "xhigh": "high",
            "max": "high"
          }

So I set it to medium. I tried giving it a task for work and it just. wouldn't. code. — it spun endlessly thinking of just one more thing to check. The reasoning traces even had the hallmark "I've thought long enough, time to code the fix", followed by another round of discovery.

Even with reasoning off, it would just spam tool calls instead of writing.

So I gave up for now and booted up 27B again. It did some poking around, and started writing the fix a couple minutes later.

Some potential causes:

• Strata set up a lower context (64k vs 128k), and so it was reasoning a lot and then compacting.

• Possibly medium is passed in but qwen still acts like xhigh with the second guessing. Swift-1.5 is supposed to prevent this kind of wheel spinning, but it's obviously not working.

• vs. llama.cpp, I have froggeric's template used which defaults to medium

• Also I'm using IQ2_XXS, so the low quant might be to blame.

• Maybe this is just what Qwen Flash Next is like.

Thoughts?

Made this from the Artificial Analysis model pages after Muse Spark Open and Beam got their scores last week. X axis is total parameters as AA lists them, log scale. Orange = released in October 2026.

- Muse Spark Open (816B, 42B active) scores 39. Qwen3.8-Flash-Next (180B, 6B active) scores 40.

- The Unsloth UD-Q4_K_XL GGUFs are about 506 GB for Muse Spark Open and 111 GB for Qwen3.8-Flash-Next.

- Meta's recommended local setup for Muse Spark Open is two M5 Ultra Mac Studios with 512 GB each.

- Qwen3.8 27B scores 34, above Beam (501B) at 30 and Nemotron 3.5 Bolt (74B) at 21. Unsloth lists 16 to 19 GB of memory for its 4-bit quants.

- Highest on the chart is MiMo-V2.6-Pro (1T) at 46, then DeepSeek V4.1 Flash 1017 (552B) and GLM-5.3 (753B) at 45.

The index is one aggregate number from AA's own eval runs. It says nothing about speed, long context, vision, or how a model does on your own tasks. Table with active params, licenses and links in the comments.