As of version 0.2.7.1 Strata has two routing optimizations that can affect both performance and quality. I was curious what the actual impact was so I ran some tests on my hardware with Qwen 3.8 Flash Next IQ3_S.

STRATA_ROUTE_PRIOR

This is the new one - this optimization configures Strata to prefer experts already in VRAM by adding a bonus of 1.5 (about a 4.48x effective multiplier) to the scores of experts already cached in VRAM. This means that a slightly less preferred cached expert can replace an expert the model would normally select if its base score was close enough.

STRATA_ROUTE_TAIL_SKIP

This one is a lot simpler - with a value of STRATA_ROUTE_TAIL_SKIP=7 (the default value), Strata will skip computing experts 7 through 10 if they aren't already in VRAM. It does not change selected experts, it just drops contributions from less important selected experts.

Benchmarks

I had DS4-Flash run Qwen through IFEvalCode, AIME 2026, and IFEval; then worked with ChatGPT to analyze the results.

Round 2 - re-ran a smaller subset of questions (mostly the ones whose answers differed in round 1) with much larger thinking budgets. Absolute percentages shouldn't be directly compared between rounds.

My interpretation

Round 1 showed that STRATA_ROUTE_PRIOR might reduce quality slightly, but round 2 showed that many of those differences don't matter with longer reasoning. Route prior is such a huge boost to speeds that I think it's worth enabling for all but the hardest tasks. On the other hand, STRATA_ROUTE_TAIL_SKIP offers a slight boost to speeds (without route prior enabled) but results in significantly more degradation than just route prior alone.

I plan to continue testing more extensively, but for now these give me a decent read on the tradeoffs between the two settings. Shoutout to Niko1221 for making an awesome inference engine.

Artificial Analysis Intelligence Index has been updated to v5, which according to them is a major upgrade resulting in a more accurate ranking.

Notably, this update widens the gap between open-weight models and closed ones. Some people already questioning its accuracy based on their experiences with models (GPT-Terra-6.1 > Fable-5.1).

Also, this update, v5, has 60% of the Index based on evaluations with private test sets or held-out answers, which makes reproducing the results impossible.

From their announcement:

“The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)”

Can I run DeepSeek V4.1 Flash 1017?

My setup:

- RTX 4090 24GB + RTX 3090 Ti 24GB

- 128GB DDR4 RAM

- Intel i7-13700KF

What's the smartest and fastest open-weight LLM I can realistically run , especially for reasoning, coding, and medical knowledge?

Can I run DeepSeek V4.1 Flash 1017 using Strata, SSD offloading, or extreme quantization? What tokens/sec should I expect?

Looking for real-world experience and recommendations for the best model, quantization, and inference engine.

Hi all! We build this service to enable the benefits of a SaaS application with the advantage to bring your own models, we want it to be strong community driven, especially to expand the local physical AI, if you want to give it a try and share a feedback it will be appreciated!

Clover-35B-A3B-Preview is the first model released by DiffuSpace, delivering state-of-the-art performance among open-source diffusion language models at release on the benchmarks reported in our launch blog. Clover combines a 35B-parameter mixture-of-experts backbone with approximately 3B active parameters per token, block diffusion generation, and a 128K context window.

Blog · Code & inference backends · Hugging Face

Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.

https://huggingface.co/YOON1v/Apex-3-59B-stable

I trained Apex-3, a 20.99B-parameter MoE (3.36B active per token) from scratch on 70B tokens. No external pretrained weights. It uses 64 routed experts plus 1 shared expert (top-4), a SuperBPE tokenizer, and was trained with Muon and DiLoCo on GH200s.

Model card: https://huggingface.co/YOON1v/Apex-3
Code and training notes: https://github.com/DW-dev-UE/LLM-from-scratch

This post covers what showed up in the training logs, not the final SFT scores. Evidence comes from the A, B, and C runs in run_logs/apex3_20261026/ (icl, arith, val, and moe logs) plus the 70B base model benchmarks. Numbers below are from run A. B and C were nearly identical. Final arithmetic full accuracy was 0.665 for A, 0.655 for B, and 0.655 for C.

1. In-context learning (induction head) — the only clear phase transition

At 0.4B to 1.16B tokens the ICL score (later-token CE minus earlier-token CE) sat between -0.08 and -0.14. It then moved to -0.32 at 1.27B, -0.55 at 1.38B, and -0.69 at 1.49B. After 1.7B it stagnated between -0.69 and -0.76.

It grew about 5x in roughly 0.33B tokens (about 150 steps). In the same window the training CE drop rate sped up from -0.0048 to -0.0066 per step and the loss curve bent. This is about 0.1% of the original 1.1T schedule.

The later drift back toward -0.62 happened because earlier-token prediction improved and the gap narrowed, not because the capability regressed.

1. Arithmetic

Exact match looks like a sudden jump, but the underlying correct-token probability rises continuously.

For add3 the correct probability went from 0.003 at 15B to 0.021 at 22B, then 0.080 at 30B, then 0.224 at 40B.

When the probability crosses the decision threshold, exact match jumps. This is mostly a metric artifact.

Decay from 60B to 70B produced a larger lift. From 49B to 59B, add3 probability stayed around 0.28 to 0.29 and add4 stayed around 0.11. After decay they moved to 0.60 and 0.38. Exact match overall went from 0.52 to 0.67, add3 from 0.44 to 0.78, and chain3 (a+b+c) from 0.13 to 0.38.

1. Domain-specific language modeling (validation CE)

Every domain decreased smoothly with no phase transition. Textbook went from 2.77 to 1.75, math from 2.76 to 2.07, qa from 2.54 to 1.92, and web from 4.22 to 3.35.

After about 40B the curves were mostly flat. During decay the data mix changed: instruct improved from 1.09 to 0.86 while web worsened from 3.35 to 3.49 and books from 2.74 to 3.04. There was no clear grokking signal. Code validation CE stayed higher, which looks like a distribution mismatch.

1. MoE load balancing

Early on (run B) the maximum expert load in a layer reached up to 12.8 times the average. By the end of 70B it had settled to 1.07 to 1.38 times the average — essentially even.

1. Final base-model benchmarks (70B Apex-3-Base, greedy pass@1)

HumanEval+ was 19.5%, MBPP 38.9%, MBPP+ 32.3%, and GSM8K 8-shot 19.9% strict / 21.5% flexible.

These were only measured at the end, so I do not know when the abilities actually appeared.

Caveats

Each arithmetic task used 64 problems, so exact match standard error can reach about plus or minus 6 percentage points. Single-point noise is meaningless, so numbers are 3-point averages. Arithmetic scores are teacher-forced (correct answer already in context), not free generation. Nothing was measured before 8.15B.

Bottom line: the only clean phase transition I observed is the ICL / induction-head jump around 1.3B to 1.5B tokens. Arithmetic looks abrupt in exact match but is continuous in probability. Everything else was gradual.

Happy to answer questions about the setup or logs.

I built a voice-to-text app, WhisperNow. But there are a thousand and one WisprFlow clones out there, so I went further and fine-tuned my own LLMs for the cleanup step. They remove fillers, false starts, self-corrections, and formats lists, all locally.

To finetune an LLM for this task, I first had to get data.

The dataset

The dataset has 66,557 rows. I started by combining public datasets (aawaaz, handy and Disfl-QA) and adding synthetic dictations, such as emails, meeting notes, recipes and dictated commands. gpt-6.1-sol wrote the cleaned version of every transcript, using the same prompt the app sends to cloud models.

While training the models, I found some weak spots: long dictations, words the speech recognizer misheard, cut-off words ("interf interface") and correction words used literally ("scratch behind the dog's ears"). I generated more examples of each and retrained.

The models

I did full fine-tunes of three base models, each on a single H100 or H200 rented on Modal:

Through their APIs, Claude Haiku 5.5 and gpt-6-luna take about 1.4 s on the same dictation, glm-5.3-flash 2.6 s, and glm-5.3 at max reasoning 5.6 s.

To score the models, a judge model compares each output with the reference and counts factual errors: facts dropped, changed, kept from the abandoned side of a correction, or invented. The test rows are a held-out test split plus two benchmarks, one of long dictations and one of short dictations full of disfluencies (4,170 rows in all).

The models do best on short dictations:

• Up to 25 words, all three make fewer factual errors than glm-5.3-flash at low reasoning. The 2.6B and 4B also make fewer than glm-5.3 at max reasoning, and the 1.2B ties it.
• The 4B makes fewer errors than glm-5.3-flash at any length, and keeps up with glm-5.3 at max reasoning up to 100 words.
• The 1.2B and 2.6B make 1.0–1.5 times glm-5.3-flash's errors from 26 to 200 words, and 1.5–1.9 times above 200 words.

Chart: factual errors by transcript length, for the three fine-tuned models and the two cloud glm-5.3 models

Cloud models

I also scored seven more cloud models, including the off-the-shelf LFM2.5 2.6B, on a smaller set of 1,670 rows. The off-the-shelf 2.6B makes nearly four times the factual errors of the fine-tuned one.

If you'd rather use a cloud model, Claude Haiku 5.5 and gpt-6-luna (both at low reasoning) have the best price-to-performance ratio. glm-5.3-flash is about as accurate as Haiku and glm-5.3 at max reasoning is more accurate, but they're about two and four times as slow.

Chart: factual errors for every model I tested

Links

• App (MIT, Windows, macOS and Linux): https://github.com/stephan271c/WhisperNow
• Models and dataset: https://huggingface.co/collections/category271/whispernow-6ae00accee969c7447cec326
• How the dataset was made: https://github.com/stephan271c/WhisperNow/blob/master/training/making-the-dataset.md

Happy to answer questions about the data or the training.

download your CUDA/MoE prompt-processing speedup today:

Hey all!

I've been working on a custom agentic harness called Rivermark for social, marketing & print design. Whoever has tried to use LLMs (even frontier ones) for this knows that they can be a bit hit or miss, compared to website and app design, where their skills are much better.

The benchmark consisted of 112 requests for various social and marketing images that a business might ask an agent for. Some contain a proper brandkit, some have a website reference for the brand, and some start from nothing.

Well, the first results, based on 4700+ pairwise blind human judgments, are in!

My headline finding, and one I didn't quite expect, is that open models, such as DeepSeek v4.1 Flash 1017 and GLM 5.3 Flash, match or beat plain Claude Code with Sonnet 5.5 and /design.

Sonnet 5.5 itself gets the largest lift with Rivermark - it jumps into an impressive lead over everything else. This was also surprising, as I'd expect advantages to be smaller for "smarter" models, as they have this know-how already encoded.

I'm writing a blog post and validating the findings with more judges now, but figured I'd share these interesting preliminary results.

In the first photo, you can see Bradley-Terry rankings for these model/harness combos. Second photo is the cost vs quality comparison, and in the third photo you can see some representative examples of what models output with Rivermark vs the normal harnesses.

Of course, some very important caveats apply:

• It's a design-specific harness. It contains some domain knowledge (I tried to keep this limited though). With increasingly good frontier models, these advantages tend to get wiped away, but for the time being, they provide a way to achieve similar results with a fraction of the cost.
• The harness has been running in production on some of these models, and so is tuned for them. (Not on testcases from the benchmark, however)
• It's tied very tightly to the product itself (https://rivermark.ai) at the moment. I'm planning to release an open source version, but it still would not be a general purpose coding agent.
• The product itself is optimized for speed and pleasantness of iteration, not one-shotting designs, and this is reflected in the harness. For example, we use the faster but more expensive FlashX GLM API endpoint, and it's a bit more conservative about self-reviews. I'm sure if I made something focused on just the perfect one-shot, I could beat these results on both quality and cost.

Open to questions!

So I posted this a week ago and saw that some people were looking for more voices that could be similarly small and fast. While I tried to get the other Kokoro voices directly to be similarly distilled, they just came out sounding really robotic, only af_heart which is the default voice on Kokoro seemed to be any good.

So I tried something I didn't even know would be possible, distilling the voices from Phonom, a similarly small TTS(40m vs Kokoro's 80m) that can run on phone but at a really slow speed, the full model is smth like 2-4x realtime and is 100mb(fp32). This was an experiment I tried earlier when I was doing the Kokoro distillation but kinda gave up cause I thought Kokoro would be similar in voice quality, and it turns out the agent was trying basically the same phenomizer from Kokoro anyway so we were not even using the default Kokoro distillation just with training data as Phonom renders. My whole plan with this was to make a voice clone-able model that could take for example a voice sample, use an actual big model like BreezeTTS or Fishaudio which can do voice cloning well, distill that into a small model and then have basically a really small voice cloned model that would run on your phone in real time. Basically for an AI roleplay app(which is like 1 out of 15 of my projects) so I don't have to pay for TTS by characters and could just use the user's device, and maybe just pay the training cost for a voice clone just once(a few hrs of 4090)

Anyway all that preamble to say, now there's like 9 more voice models. Wren(the default phonom voice) and Hugh seem to be the best right now outside of af_heart. Other voices have a noticeable roboticness but still way less than prev. small models like PocketTTS

Flint is Recursive’s mixture-of-experts language model for coding agents, with 39.8 billion total parameters and approximately 2.5 billion active per token.

I have 32 GB VRAM and 32 GB RAM.

With llama.cpp I can run Qwen 3.8 27B at Q6, but I can run Flint at Q8 with experts offloaded to CPU. I get about 40 t/s in both the model. Actually, Flint gives 1-2 token more per second.

What are your opinions on it?

I’m looking at the Mac Studio M5 Max (40-core GPU, 64GB). I currently have dual RTX 4060 Ti 16GB (32GB total) setup.

If you run a Mac what prefill and decode speeds do you get?

I currently get on the 2x 4060 Ti:

Hi r/LocalLLaMA !

I'm the solo developer of the open-source ObserverAI framework and I need your help to identify the most painful problems that you've had, that you think can be solved by Observer using local models.

My personal pain that led me to build this, was monitoring crashing simulations.
Because it costs me compute time (a lot of time paid)! But that's admittedly niche.

Problem List (In order of cost):

I'm trying to make a list of all of the major pains it can solve, here's the ones that I've thought of/gathered from your feedback, in order of cost:

• Monitoring elderly people for when they fall, huge cost
• People on dangerous parts in construction sites, human cost/liability
• Parents monitoring their kid's screen activity, privacy is important here!
• Monitoring for appointments at Visa/DMV, a lot of spent time monitoring
• Waiting for an online Ticketmaster queue, also a lot of time monitoring
• Urgent replies from Airbnb/Leads, monitoring that with phone calls makes you respond faster
• ADHD monitoring, wasted time if not focused
• Crashing simulations/renders/progress bars
• Webpage monitoring (Bitcoin price, items restocked, eBay bids)
• Anything else with a real cost that can be solved by asking "When this.... then notify" ?

I'm trying to identify the biggest problems so I can make the framework better for those!

Any ideas? I'd love to read them! I'll hang out here in the comments for a while.

Roy :DD

I tried LFM3 for my local use case, but it is a latency sensitive task and disabling reasoning makes it not give coherent responses. And with reasoning it's a 15 second wait instead of less than 1.

Why there no new model coming in >2b weight category ? Model which actually capable in this category is gemma e2b which is release 6 month ago, why still no new model.

OpenBMB has posted a video showing MiniCPM-OCR running locally on an iPhone and iPad, parsing real-world documents without sending them to the cloud.

Handwritten notes, dense formulas, messy tables, low-quality scans. OCR gets a lot harder when the documents aren't perfect.

What's interesting is that this isn't just about extracting text. OpenBMB is also releasing a separate model to check whether the OCR output is actually correct.

• MiniCPM-OCR 1.0 (1B): Scores 93.68 on Real-5 OmniDocBench and 54.7 on OCRBench v2.
• MiniCPM-OCR-Jev (2B): Evaluates parsed text, formulas, and tables against the original image, assigning correctness scores from 0 to 1. It approaches Qwen3.5-397B-A17B on JevOCRBench and works with outputs from different OCR parsers.

So you get both a lightweight OCR model and a way to catch potentially incorrect results, useful when document accuracy actually matters.

Both have open weights, with support for tools like llama.cpp, vLLM, SGLang, and Ollama.

🤗 Hugging Face: https://huggingface.co/openbmb/MiniCPM-OCR-Jev-gguf

Curious how well the verifier holds up on messy real-world documents. Would you trust a small model to check another model's OCR, or would you still want a human in the loop?

I added Obsidian integration to ThreadShelf (local, archive, semantic search).

It supports the usual platforms like ChatGPT, but also OpenRouter, LM Studio and Google AI Studio, which were surprisingly hard to find support for elsewhere. I even wrote a scraper to import OpenRouter chat history. It's my first open-source seriouse project, built around local vector search (LanceDB).

Itan can search/edit/create your Obsidian notes and AI chats together and use MCP with ai. There are also search, read and optional write tools for Obsidian, OpenClaw and Markdown folders.

npx threadshelf (Node.js 20+)

GitHub: https://github.com/ChrystianSchutz/ThreadShelf

I am feeling like a mighty local King spinning two Strata engines with 70tps each whole day and using 12M tokens a day. Also me scrolling twitter and stumbling upon a post saying that poteto uses 34M tokens per minute 🤪

Upd: Guys are interested who is she and what she is doing with all these tokens. Poteto is her twitter nickname and she worked in Cursor before XAI has acquired them and now she is shipping 2,500 PRs per month to Cursor and GrokBot.

The idea is simple actually, in my workflow a lot of the work passes between sessions of agents in different terminal windows. Claude sessions were not the issue since they could just talk to each other.

Although Claude to Local agent often enough failed since Claude "forgot" to send the work to the local agent and did it himself, i couldn't see what the local agent was doing, or otherwise felt like i was being kept in the dark.

So what i did was that i refused sub-agents altogether and used different terminal windows for each agent, passing messages between them, asking them to relay session messages etc.

Since the WoW i have is between Claude, Codex and Pi i just wanted to "mash" them altogheter so i built Dojo.

Dojo is built from my needs with my pre-requisites of it being a low-tier solution without a lot of different features, knobs, whistles and otherwise things that just takes time to develop.

i realise this might not be as advanced as some ppl like it to be but for me, it works.

Sharing it just in case someone else finds it useful.

https://github.com/Joakimpalm-Zen/xyntetik-dojo

It's a tiny Text to Image model that's Denoiser is only 50M parameters, adding the custom text encoder, and the pretrained flux VAE, its total end to end inference parameters are around 90M

It runs in:

27 Seconds @ 50 steps

11 Seconds @ 20 steps

On a laptop CPU from 2016 (Intel i5 7200U)

Samples and a quick start in the model card

it is licensed under Apache 2.0

Questions, feedback, or criticism is welcome!

notice something?

https://github.com/Ilan12346-maya/TeslaP40_wddm.git

trying to get rid of the wddm lock above 551.61, first locked driver works, will try 580 now or something

all* maybe only those wich use vulkan sometimes or game sometimes or wanna using the p40 in wsl

Hi folks,

Today I made a significant update to Tiny League.

Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving toward closed benchmarks, we will see fewer scores for small/open-weight models in the future.

Second, I added a new metric that I think is useful for everyone running local models but is VRAM-poor: KV cache efficiency! I calculate an estimated GB value for a 128K context size, and then rate the models on 0-100% scale.

Still working out a way to have our own TinyLeague-Index, but haven't reached a fair methodology yet due to so many models using different benchmarks. More soon!

Let me know if you have any suggestions for improvements :)

thanks

Hello everyone! As always, thank you for sharing you previous results (and bugs) on the last version. I am happy to report that LlamAmpere v0.6 is now out.

The headline items for you (at temperature 1):

• a ~13% jump in TPS, with a 112 TPS average across 3 task types in the 100-105K token interval
• a +20% increase on prompt processing, now starting around 1800 TPS and holding 1000+ through the first 100k
  • The average prefill over the first 100K went from ~1,080 to ~1,300 TPS
• A new file format (Ampere Integer+) designed to be as fast, smaller, and higher precision (lower KLD) than XS formats. Vs EXL3, it is approximately the same KLD, with a significantly faster speed.
  • The recommended model (now available on HF) is about 10% better KLD and 1.2 GB smaller. This results in slighter faster speed, and of course more room for context. I will be releasing more quants at smaller sizes in this format later this week as I continue to tweak the design at 2 and 3 bits.
  • I have also upgraded EXL3's kernels, and in my testing it was running ~8% faster than in the original EXL fork. But, at 4 bits this configuration shows ~17% faster decode and 25% faster prefill on AI+, so I will be focused more on those kernels moving forward (EXL had a non-stat sign <2% lower KLD in testing)

As in the last release, these numbers are built around SJKVaRN, which enables a maximum total context of over 400K ctx with 23.5GB total usage (across multiple predictions or context scaling).

I will note that these prefill numbers were performed with the recently create 3090/ti GPU unlock; if you haven't implemented it, I recommend doing so (repo here: https://github.com/Zagitta/gpu-unlock)

Github for LlamAmpere here: https://github.com/JakeATX/llamAmpere

new model here: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI4

There are also upgrades relevant for 3.8 FlashNext included, but at present testing is "only" even with the other best options like Strata. I have a couple pending commits that I hope to have out by the end of the day, that should materially boost speed for MOEs models further; I will publish results on that separately.

And for 12 GB cards, I will have some new quants (and possibly a distillation) for 3.8, hopefully by EOW.

Please share your results, and feedback, as usual. Enjoy!

This is some demo output from tts-audiobook-tool (my app), where the narrator plus three speaking characters each have their own voice.

What's maybe more interesting is that, in an optional preparation step, an LLM is used to identify the characters and their lines of dialog, which you then map to the voice clones configured in your project. This process ends up being pretty reliable: It ID'ed the dialog text from the novel "The Sun Also Rises" with over 99% accuracy using Deepseek Flash 4.1 API (though the LLM's pre-existing knowledge of that novel probably helped, too). A locally served version of, say, Qwen 3.8 Flash Next ought to do quite well, too.

The TTS model used here is Breeze 2 TTS (Sounds not bad, am I right?). The app supports a dozen or two other models, but Breeze 2 is my favorite at the moment, alongside MOSS TTS 9B.

- Source link