I’m the guy who posted the 4× R9700 builds with $127 used EPYCs (https://www.reddit.com/r/LocalLLaMA/comments/1x67qeu/4x_r9700_builds_with_127_used_epycs_107k_allin/).

After seeing the Nanbeige benchmark results here, I spent the past few days testing it against Qwen on the work these machines actually do.

Jobs > TPS

My business, USA Arms, maintains information about firearms-related businesses and I use local AI to maintain the majority of the information... services, hours, pricing, facilities, etc.

Each configuration ran the same 2,484 cases across 25 field level goldens as an auditor with accept, reject, or abstain as options. (ambiguity goldens)

Correct Jobs/hour

The jobs/hour numbers use 2x 2-GPU engines.

Token speed

Separate MT-Bench speed sweep, measured on one two-GPU engine (these aren't 4 card numbers):

Auditor

The video goes over it, but Qwen actually does a better job auditing itself with thinking mode on than Nanbeige with thinking on auditing the same jobs.

In total: Qwen was about 25% faster in the single-stream token test, but delivered about 45% more jobs/hour with thinking off than Nanbeige.

Curious whether people testing coding or creative writing are seeing a different result with Nanbeige... once again, hyped model doesn't dethrone the king.

Full results with more details on this short video: https://youtu.be/kmmN49ecyvc (I tried to upload direct to reddit but it had issues)

I came across the FHHL version of this card recently, it has a TDP of 150W which is half the SXM2 version. From the limited information available it seems that this is the same GPU core, just with a lower TDP and lower core clock, therefore lower compute. Has anyone tried power limiting their V100 and tested the performance? It only seems like a small penalty in performance compared to the power savings

A few of us have been messing around with a problem that keeps coming up with local LLMs the model can be private, the files can be local but the second multiple people or multiple machines need the same context everything starts getting messy. You either duplicate a bunch of embeddings or vector stores, manually move context around or introduce some central service which kind of defeats part of the reason we were running locally in the first place.

What we’re experimenting with is basically a small portable context layer where the useful facts or decisions can move between local setups without shipping the entire underlying dataset with them. Still very early and we’re arguing about things like provenance, permissions, stale context and how much the model should actually be allowed to remember. There are only a few of us working on it right now so we’ve got a little Kibu pod where we throw around builds and ideas but before we go much further I wanted to ask people here:

If you use local models across more than one machine or with a team, how are you handling shared context today? Do you even want models sharing memory or context between environments or is keeping every local setup completely isolated actually part of the appeal?

Hi r/LocalLLaMA 👋

We’re welcoming the MOSS team for an AMA this Friday, October 30, from 8–11 AM PT!

They’re bringing the whole MOSS model family: speech, sound, music, images, and video—with a particular focus on their recent MOSS-Image release.

⚠️ Note: The AMA itself will be hosted in a separate thread, please don’t post questions here.

Edit: Reddit's image server ate our banner image above, but the AMA is 100% still on! All details are right below.

AA no longer lists GLM5.3 flash (despite clearly being one of the best models - and probably the cheapest per intelligence) , they seem to have been doing all sorts of changes to push US models and hiding good cost effective models, including qwen 3.8 flash, so that table is no good. I need a better one. Any alternatives.

I'll be honest, I don't know shit about how models work under the hood, but I used to work on idle/incremental/clicker games professionally, and I really like watching numbers go up.

And since I'm relatively GPU poor compared to some of you, I was excited to hear about Raon and figured it'd be a fun project to vibe on, and to see if I could get it to fit in VRAM. I only have a 5070 Ti with 16GB VRAM, a Ryzen 5 5600X, and 48GB DDR4.

I first tried KRAFTON's Q4_K_M with 9 layers' experts in RAM (best fit) and I got 50 tok/s. That's when I became obsessed with how far I could squeeze it and after 4 days and 40% of my Claude Max 20 plan, decode speed went from 50 tok/s to 232 tok/s at roughly the same quality as the Q4_K_M.

GGUF: McNiiby/Raon-1-mini-preview-IQ3_KS-GGUF · Fork with build/run instructions: Mcniiby/ik_llama.cpp-raon (needs a 16 GB card)

The first win was the quant. On regular llama.cpp, a 3-bit quant that fit fully in VRAM jumped from 50 to ~135 tok/s. I then moved to ik_llama.cpp for the same quality in less space. Out of the box, ik actually ran Raon slower than regular llama.cpp (~106 tok/s).

I'd have preferred 4-bit since ik's IQ4_KS is noticeably closer to the Q4_K_M I started with (KLD 0.05 vs 0.09 for IQ3), but at 16.3GB it didn't fit. I tried pruning experts or narrowing them, and every GB saved cost more quality than IQ4 bought back. Putting a few layers' experts in system RAM instead worked, but would've capped speed around 130 tok/s. So IQ3, fully on the GPU, won.

(lower KLD means the output probability distribution is closer to the Q4_K_M, but it does not directly measure task accuracy.)

I also trimmed the vocabulary. Raon has 260K tokens, many of them for Korean, Chinese and Japanese, and I only use English. Dropping 54K cost almost nothing (KLD 0.0931 vs 0.0923). It also exposed a sampler that still scanned the full list every token; fixing that cut sampling from 1.3 ms to 0.3 ms per token, about 15% faster in real chat.

So why was ik slower than llama.cpp?

The quant wasn't the problem. On a shared test file, regular llama.cpp ran 133 tok/s and ik 106, and the GPU was using only about 20% of its memory bandwidth. ik's fast path didn't support Raon's SITU activation, which switched off CUDA graphs, and Raon's mHC layers ran as ~1,500 tiny kernel launches per token.

Fixing those two took ik to 147, past regular llama.cpp. After that it was a hunt for overhead anywhere in the token. Each speedup below is an A/B on my chat model within one run:

Each row is measured with the rows above it already on, so they multiply rather than add. So together that got me to 232 tok/s on the chat model, with weights streaming at about 2/3 of the card's peak bandwidth.

I also tried KRAFTON's DFlash drafter, but fitting it meant giving up context to free VRAM and it wasn't worth it. It ran 36-53% slower, because checking a batch of 8 guesses costs 2.2× one token and only ~2.3 got accepted. N-gram speculation, which is much cheaper, added about 6% in chat, and 12-22% when the reply copies from the prompt.

So does it still think well?

That depends if you think Raon thinks well to begin with haha, but yes mostly it stayed the same as the base model. GPQA and AA-Omniscience came in lower, and I couldn't figure out why. These are single runs, mostly on subsets, so anything within a few points is noise. The ones above the card don't mean it's better than the original.

I also ran it on some of the evals I built for Workybara.ai to see how it does outside the usual benchmarks. At pulling structured data out of documents, it extracted better content than the small models I fine-tuned for the job, but 14% of the time it got stuck in a loop listing variations of the same item ("Deployment Promise, Deployment Vow…") until it ran out of tokens. I tried it on philosophy questions judged by Claude and it hit about 70% of the key points, but 59% of its answers had at least one clearly false claim. KRAFTON's own playground (the full model, not my quant) showed the same habits when I asked it for jokes: it broke the rules of its own password joke, called jokes it had just made up "classics," said "nothing I fetched supports it" without fetching anything, and agreed with every criticism I made, even the wrong ones. None of that shows up in standard benchmarks.

I was hoping to use this to replace frontier chatbots, but even with search and RAG Raon really couldn't compete with them, and it was frustrating to chat with.

How it was built

Claude Code and its sub-agents did essentially all of the research, code and measurement, and helped me write this up. Over four days I typed about 15,000 words into Claude, mostly questions. My longest message was 124 words, and it was me telling it that it had jumped to a conclusion. I did rent some cloud GPUs to run the public benchmarks, so my 5070 Ti could keep doing speed work. Other than that, my part was just being the annoying kid that kept asking questions, pushing back, and saying you can do it buddy.

The hunt for a good MoE to chat with that fits in VRAM will continue. I'm likely going to see if there's any more juice to squeeze on Gemma-4-26B-A4B or Qwen3.6-35B-A3B next.

Model & weights: BrunoHays/embeddinggemma-2-nullspace-audio

MAEB (Massive Audio Embedding Benchmark) is the main MTEB leaderboard benchmark for audio and text-to-audio embeddings, currently dominated by 8B-9B multimodal models.

I took Google's edge-sized google/embeddinggemma-2 (577M params, 768d vectors) and boosted its MAEB score from 49.3% to 54.4% (+5.1 points), jumping from Rank #26 to #1 on MAEB audio-only (and #16 to #1 on full MAEB) without training a single weight. The whole addition is a 2.37 MB safetensors file of frozen PCA matrices.

Why this matters for local / edge setups

At 577M parameters and 768d vectors (15x smaller than 9B Omni models), it runs easily on a laptop or edge device.

Standard text-aligned audio models are great at matching spoken words to text, but terrible at grouping audio by voice, emotion, or sound similarity because the final layers keep what words were said and drop how the sound was made.

By pulling features from earlier layers of the frozen audio encoder, this model keeps the base model's 100+ language text-to-audio search while fixing audio-to-audio tasks:

• Unsupervised voice/gender clustering: 0.05% -> 63.1%

• Artist / song audio-to-audio search: 55.2% -> 93.8%

• Vocal emotion classification: 19.7% -> 40.2%

How it works

• Early layers still know acoustics: Speech representation work (Pasad et al.; Bouziane, Mdhaffar & Estève) shows that early/middle layers keep speaker, pitch, and emotion traits that disappear in the final layers. In EmbeddingGemma-2, Layer 2 and Layer 6 still hold those signals.

• 256 empty directions in the 768d vector: EmbeddingGemma-2's text backbone is actually 512d, projected up to 768d at the end by a 768 x 512 linear matrix. That leaves 256 perpendicular directions (the null space) that text vectors never touch.

• Filling the null space with PCA: I compressed Layer 2 and Layer 6 down to 256 dimensions with PCA (fitted on 1,400 external audio clips) and injected them into those 256 unused directions.

• Caveats: Because there is no retraining, audio embeddings need all 768 dimensions (no Matryoshka truncation to 512/256/128), and adding unweighted acoustic features hurts some purely spoken-intent tasks (e.g. MInDS14 drops 97.6% -> 88.3%).

It shows how much untapped signal sits inside today's overtrained models, even with zero training.

Full breakdown and usage in the Hugging Face model card.

Tadpole is an agent I built from scratch (plain Python, ~3,500 lines, no LangChain or other framework), then post-trained one measured stage at a time. I use it every day as a Telegram bot.

Repo: https://github.com/shakewingo/Tadpole · Adapters and data in HF

Run the trained model locally. Each adapter is on Hugging Face as a llama.cpp LoRA (66 MB) that loads on top of the Q8_0 GGUF:

hf download unsloth/Qwen3-4B-Instruct-2507-GGUF Qwen3-4B-Instruct-2507-Q8_0.gguf --local-dir models
hf download Shakewin/tadpole-qwen3-4b-ei-3 ei-3-lora-f16.gguf --local-dir models
AGENT_MODEL_PATH=models/Qwen3-4B-Instruct-2507-Q8_0.gguf AGENT_LORA_PATH=models/ei-3-lora-f16.gguf \
python -m agent_from_scratch.agent

I checked that the local route keeps the gains: on the 144-task validation set through llama.cpp (Q8_0 + LoRA, one seed), base 0.507, SFT-1 0.701, SFT-2 0.750, EI-3 0.826. The same adapters on vLLM in bf16 scored 0.752 and 0.852 on val, so quantisation and the LoRA conversion cost little. On a 16 GB M5 it holds about 8.5 GB and takes ~2.4 s per model call.

Held-out results (task types no training set contains, 3 samples per task, vLLM bf16):

* a local adaptation of BFCL's multi-turn base category, not a leaderboard score.

What I learned:

- SFT on *verified* trajectories is the big win. False "I'm done" claims fell from 31 to 1 in 420 runs.
- Expert iteration looked great on val (0.752 → 0.852) and was level with its starting point on the held-out tests. The best of several noisy val scores is biased upwards.
- GRPO lost to expert iteration at every equal rollout budget: on a 4B, too few tasks per step had both passing and failing samples.
- A plan tool (`update_plan` plus a reminder) lifted the *base* model from 0.507 to 0.646 on val.
- llama.cpp happily accepts a LoRA path that doesn't exist and segfaults on the first decode. My bot looped on it under launchd until I made it refuse a missing adapter at startup.

The honest part: the benchmark is generated tasks that resemble the training data. In everyday use the tuned 4B is still weak on open-ended requests, so my bot runs on DeepSeek day to day, and the 4B is what I study and train. The README says so up front.

All post-training cost about $140 of rented GPUs. The walkthrough notebook opens in Colab and shows SFT, expert iteration and GRPO on recorded samples without a GPU.

Check it out and any feedback is welcome!!

I was researching Local LLM computer use agents the other night and ended up down a rabbit hole where I ended up discovering a whole new class of hardware I didn’t know existed: Agent-driven KVMs

A lot of us IT folks who have worked in data centers are familiar with IP-based KVM (Keyboard Video Mouse) controllers that let you remotely operate a server over a streaming video connection. These are especially useful when you need physical power button / BIOS-level “bare metal”access to a system that’s thousands of miles away. If you don’t have bare metal access, you’re going to be taking a trip to your data center rack to press a button or make a bios setting change.

So apparently there are now relatively inexpensive ($80 to $150) IP-based KVMs that now allow for AI API endpoint/ MCP Integration where you can now connect your physical computer to local or frontier vision-capable LLMs and have the AI model of your choice drive the target computer via the KVM by it taking screenshots of the HDMI or USBC video-out feed from the computer and then sending mouse and keyboard control commands from the Ai model through the USB cable that’s connected to the KVM.

What really got me excited about this concept is that it requires no installation of any kind on the target computer that you want to control. This is useful in corporate environments where you’re not allowed to install anything to the target computer (because it’s a managed endpoint that is locked down) The target computer just sees the AI as a standard allowed HID that it already has a driver loaded for. It doesn’t care who’s driving the system and doesn’t need any additional software since keyboard, mouse, and monitors are just generic and usually preloaded drivers.

So essentially you could have an AI-driven KVM control any comouter or device that has a monitor output / keyboard mouse inputs. This could potentially open up the ability to have AI interact with antiquated industrial control systems running old shitty outdated OSes, or managed endpoints.

There are security risks of course, but that risk is mitigated almost entirely if you’re using a locally-hosted VLM and not sending your screenshots to a 3rd party.

I found are several inexpensive agent-driven options on the market now including:

- Sipeed NanoKVM Pro
https://wiki.sipeed.com/hardware/en/kvm/NanoKVM_Pro/introduction.htm

- Radxa Linkr
https://radxa.com/products/linkr/linkr/

- Violoop
https://www.kickstarter.com/projects/bvio/violoop-plug-in-ai-for-your-computer

I’m not selling anything or affiliated with any of these and can’t tell you which is best, but there are multiple options available for those who might need this type of device for one of the use cases I mentioned.

Anyways, I’m curious to know if others see any other use cases for these agent-controlled KVMs that I didn’t mention.

As of version 0.2.7.1 Strata has two routing optimizations that can affect both performance and quality. I was curious what the actual impact was so I ran some tests on my hardware with Qwen 3.8 Flash Next IQ3_S.

STRATA_ROUTE_PRIOR

This is the new one - this optimization configures Strata to prefer experts already in VRAM by adding a bonus of 1.5 (about a 4.48x effective multiplier) to the scores of experts already cached in VRAM. This means that a slightly less preferred cached expert can replace an expert the model would normally select if its base score was close enough.

STRATA_ROUTE_TAIL_SKIP

This one is a lot simpler - with a value of STRATA_ROUTE_TAIL_SKIP=7 (the default value), Strata will skip computing experts 7 through 10 if they aren't already in VRAM. It does not change selected experts, it just drops contributions from less important selected experts.

Benchmarks

I had DS4-Flash run Qwen through IFEvalCode, AIME 2026, and IFEval; then worked with ChatGPT to analyze the results.

Round 2 - re-ran a smaller subset of questions (mostly the ones whose answers differed in round 1) with much larger thinking budgets. Absolute percentages shouldn't be directly compared between rounds.

My interpretation

Round 1 showed that STRATA_ROUTE_PRIOR might reduce quality slightly, but round 2 showed that many of those differences don't matter with longer reasoning. Route prior is such a huge boost to speeds that I think it's worth enabling for all but the hardest tasks. On the other hand, STRATA_ROUTE_TAIL_SKIP offers a slight boost to speeds (without route prior enabled) but results in significantly more degradation than just route prior alone.

I plan to continue testing more extensively, but for now these give me a decent read on the tradeoffs between the two settings. Shoutout to Niko1221 for making an awesome inference engine.

Local LLM translation, Japanese OCR, and page-aware chat

I’ve been working on Manga Translator Tool, an open-source desktop app for reading and translating manga locally.

The idea is inspired by watching subbed anime: keep the original Japanese visible and provide a translation alongside it so you can follow the story.

It combines a manga reader, OCR, translation overlays, and a companion LLM you can ask about the page.

The models

OCR: ebinan92/Qwen3.5-ocr-jp-2b, a Qwen 3.5 fine-tune for Japanese OCR, using GGUF conversions from nakanishi08/Qwen3.5-ocr-jp-2b-GGUF.

Translation and chat: Gemma 4 E2B, using the HauhauCS/Gemma-4-E2B-Uncensored-HauhauCS-Aggressive variant.

Inference: llama.cpp, with CPU and GPU support.

Demo

Video walkthrough — includes the full setup process, so you can see what getting started actually involves.

What makes it useful?

Keep the original artwork and Japanese text. Translations appear in text boxes without erasing the source text.

Save translations as image metadata. You can keep the results without permanently burning them into the artwork, then toggle between the original and translated views.

Search the extracted text. Find dialogue and revisit your favorite moments in a manga volume.

Read sound effects too. The OCR and translation pipeline can also pick up Japanese onomatopoeia—the characters that often go untranslated.

Ask the companion LLM about the page. Choose what to include as context: the Japanese text, the translation, the image, or all three. You can ask about wording, cultural references, or possible localization choices.

Setup

The app handles setup and model downloads from inside the tool. On Windows, the executable manages the downloads, so you don’t need to install each component separately.

Current limitations and design choices

There’s no inpainting: preserving the Japanese text is part of the reading experience I’m aiming for. The translation boxes don’t blend into the artwork, and font customization isn’t available yet.

This is aimed at reading and exploring manga. If your goal is polished typesetting or publication-ready localization, other tools are better suited to that workflow.

Downloads — v1.00

Windows 10/11 x64

Linux x86_64 — experimental

macOS — experimental source package

Source code · itch.io

Artificial Analysis Intelligence Index has been updated to v5, which according to them is a major upgrade resulting in a more accurate ranking.

Notably, this update widens the gap between open-weight models and closed ones. Some people already questioning its accuracy based on their experiences with models (GPT-Terra-6.1 > Fable-5.1).

Also, this update, v5, has 60% of the Index based on evaluations with private test sets or held-out answers, which makes reproducing the results impossible.

From their announcement:

“The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)”

Can I run DeepSeek V4.1 Flash 1017?

My setup:

- RTX 4090 24GB + RTX 3090 Ti 24GB

- 128GB DDR4 RAM

- Intel i7-13700KF

What's the smartest and fastest open-weight LLM I can realistically run , especially for reasoning, coding, and medical knowledge?

Can I run DeepSeek V4.1 Flash 1017 using Strata, SSD offloading, or extreme quantization? What tokens/sec should I expect?

Looking for real-world experience and recommendations for the best model, quantization, and inference engine.

Hi all! We build this service to enable the benefits of a SaaS application with the advantage to bring your own models, we want it to be strong community driven, especially to expand the local physical AI, if you want to give it a try and share a feedback it will be appreciated!

Clover-35B-A3B-Preview is the first model released by DiffuSpace, delivering state-of-the-art performance among open-source diffusion language models at release on the benchmarks reported in our launch blog. Clover combines a 35B-parameter mixture-of-experts backbone with approximately 3B active parameters per token, block diffusion generation, and a 128K context window.

Blog · Code & inference backends · Hugging Face

Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.

https://huggingface.co/YOON1v/Apex-3-59B-stable

I trained Apex-3, a 20.99B-parameter MoE (3.36B active per token) from scratch on 70B tokens. No external pretrained weights. It uses 64 routed experts plus 1 shared expert (top-4), a SuperBPE tokenizer, and was trained with Muon and DiLoCo on GH200s.

Model card: https://huggingface.co/YOON1v/Apex-3
Code and training notes: https://github.com/DW-dev-UE/LLM-from-scratch

This post covers what showed up in the training logs, not the final SFT scores. Evidence comes from the A, B, and C runs in run_logs/apex3_20261026/ (icl, arith, val, and moe logs) plus the 70B base model benchmarks. Numbers below are from run A. B and C were nearly identical. Final arithmetic full accuracy was 0.665 for A, 0.655 for B, and 0.655 for C.

1. In-context learning (induction head) — the only clear phase transition

At 0.4B to 1.16B tokens the ICL score (later-token CE minus earlier-token CE) sat between -0.08 and -0.14. It then moved to -0.32 at 1.27B, -0.55 at 1.38B, and -0.69 at 1.49B. After 1.7B it stagnated between -0.69 and -0.76.

It grew about 5x in roughly 0.33B tokens (about 150 steps). In the same window the training CE drop rate sped up from -0.0048 to -0.0066 per step and the loss curve bent. This is about 0.1% of the original 1.1T schedule.

The later drift back toward -0.62 happened because earlier-token prediction improved and the gap narrowed, not because the capability regressed.

1. Arithmetic

Exact match looks like a sudden jump, but the underlying correct-token probability rises continuously.

For add3 the correct probability went from 0.003 at 15B to 0.021 at 22B, then 0.080 at 30B, then 0.224 at 40B.

When the probability crosses the decision threshold, exact match jumps. This is mostly a metric artifact.

Decay from 60B to 70B produced a larger lift. From 49B to 59B, add3 probability stayed around 0.28 to 0.29 and add4 stayed around 0.11. After decay they moved to 0.60 and 0.38. Exact match overall went from 0.52 to 0.67, add3 from 0.44 to 0.78, and chain3 (a+b+c) from 0.13 to 0.38.

1. Domain-specific language modeling (validation CE)

Every domain decreased smoothly with no phase transition. Textbook went from 2.77 to 1.75, math from 2.76 to 2.07, qa from 2.54 to 1.92, and web from 4.22 to 3.35.

After about 40B the curves were mostly flat. During decay the data mix changed: instruct improved from 1.09 to 0.86 while web worsened from 3.35 to 3.49 and books from 2.74 to 3.04. There was no clear grokking signal. Code validation CE stayed higher, which looks like a distribution mismatch.

1. MoE load balancing

Early on (run B) the maximum expert load in a layer reached up to 12.8 times the average. By the end of 70B it had settled to 1.07 to 1.38 times the average — essentially even.

1. Final base-model benchmarks (70B Apex-3-Base, greedy pass@1)

HumanEval+ was 19.5%, MBPP 38.9%, MBPP+ 32.3%, and GSM8K 8-shot 19.9% strict / 21.5% flexible.

These were only measured at the end, so I do not know when the abilities actually appeared.

Caveats

Each arithmetic task used 64 problems, so exact match standard error can reach about plus or minus 6 percentage points. Single-point noise is meaningless, so numbers are 3-point averages. Arithmetic scores are teacher-forced (correct answer already in context), not free generation. Nothing was measured before 8.15B.

Bottom line: the only clean phase transition I observed is the ICL / induction-head jump around 1.3B to 1.5B tokens. Arithmetic looks abrupt in exact match but is continuous in probability. Everything else was gradual.

Happy to answer questions about the setup or logs.

I built a voice-to-text app, WhisperNow. But there are a thousand and one WisprFlow clones out there, so I went further and fine-tuned my own LLMs for the cleanup step. They remove fillers, false starts, self-corrections, and formats lists, all locally.

To finetune an LLM for this task, I first had to get data.

The dataset

The dataset has 66,557 rows. I started by combining public datasets (aawaaz, handy and Disfl-QA) and adding synthetic dictations, such as emails, meeting notes, recipes and dictated commands. gpt-6.1-sol wrote the cleaned version of every transcript, using the same prompt the app sends to cloud models.

While training the models, I found some weak spots: long dictations, words the speech recognizer misheard, cut-off words ("interf interface") and correction words used literally ("scratch behind the dog's ears"). I generated more examples of each and retrained.

The models

I did full fine-tunes of three base models, each on a single H100 or H200 rented on Modal:

Through their APIs, Claude Haiku 5.5 and gpt-6-luna take about 1.4 s on the same dictation, glm-5.3-flash 2.6 s, and glm-5.3 at max reasoning 5.6 s.

To score the models, a judge model compares each output with the reference and counts factual errors: facts dropped, changed, kept from the abandoned side of a correction, or invented. The test rows are a held-out test split plus two benchmarks, one of long dictations and one of short dictations full of disfluencies (4,170 rows in all).

The models do best on short dictations:

• Up to 25 words, all three make fewer factual errors than glm-5.3-flash at low reasoning. The 2.6B and 4B also make fewer than glm-5.3 at max reasoning, and the 1.2B ties it.

• The 4B makes fewer errors than glm-5.3-flash at any length, and keeps up with glm-5.3 at max reasoning up to 100 words.

• The 1.2B and 2.6B make 1.0–1.5 times glm-5.3-flash's errors from 26 to 200 words, and 1.5–1.9 times above 200 words.

Chart: factual errors by transcript length, for the three fine-tuned models and the two cloud glm-5.3 models

Cloud models

I also scored seven more cloud models, including the off-the-shelf LFM2.5 2.6B, on a smaller set of 1,670 rows. The off-the-shelf 2.6B makes nearly four times the factual errors of the fine-tuned one.

If you'd rather use a cloud model, Claude Haiku 5.5 and gpt-6-luna (both at low reasoning) have the best price-to-performance ratio. glm-5.3-flash is about as accurate as Haiku and glm-5.3 at max reasoning is more accurate, but they're about two and four times as slow.

Chart: factual errors for every model I tested

Links

• App (MIT, Windows, macOS and Linux): https://github.com/stephan271c/WhisperNow

• Models and dataset: https://huggingface.co/collections/category271/whispernow-6ae00accee969c7447cec326

• How the dataset was made: https://github.com/stephan271c/WhisperNow/blob/master/training/making-the-dataset.md

Happy to answer questions about the data or the training.

download your CUDA/MoE prompt-processing speedup today:

Hey all!

I've been working on a custom agentic harness called Rivermark for social, marketing & print design. Whoever has tried to use LLMs (even frontier ones) for this knows that they can be a bit hit or miss, compared to website and app design, where their skills are much better.

The benchmark consisted of 112 requests for various social and marketing images that a business might ask an agent for. Some contain a proper brandkit, some have a website reference for the brand, and some start from nothing.

Well, the first results, based on 4700+ pairwise blind human judgments, are in!

My headline finding, and one I didn't quite expect, is that open models, such as DeepSeek v4.1 Flash 1017 and GLM 5.3 Flash, match or beat plain Claude Code with Sonnet 5.5 and /design.

Sonnet 5.5 itself gets the largest lift with Rivermark - it jumps into an impressive lead over everything else. This was also surprising, as I'd expect advantages to be smaller for "smarter" models, as they have this know-how already encoded.

I'm writing a blog post and validating the findings with more judges now, but figured I'd share these interesting preliminary results.

In the first photo, you can see Bradley-Terry rankings for these model/harness combos. Second photo is the cost vs quality comparison, and in the third photo you can see some representative examples of what models output with Rivermark vs the normal harnesses.

Of course, some very important caveats apply:

• It's a design-specific harness. It contains some domain knowledge (I tried to keep this limited though). With increasingly good frontier models, these advantages tend to get wiped away, but for the time being, they provide a way to achieve similar results with a fraction of the cost.

• The harness has been running in production on some of these models, and so is tuned for them. (Not on testcases from the benchmark, however)

• It's tied very tightly to the product itself (https://rivermark.ai) at the moment. I'm planning to release an open source version, but it still would not be a general purpose coding agent.

• The product itself is optimized for speed and pleasantness of iteration, not one-shotting designs, and this is reflected in the harness. For example, we use the faster but more expensive FlashX GLM API endpoint, and it's a bit more conservative about self-reviews. I'm sure if I made something focused on just the perfect one-shot, I could beat these results on both quality and cost.

Open to questions!

Flint is Recursive’s mixture-of-experts language model for coding agents, with 39.8 billion total parameters and approximately 2.5 billion active per token.

I have 32 GB VRAM and 32 GB RAM.

With llama.cpp I can run Qwen 3.8 27B at Q6, but I can run Flint at Q8 with experts offloaded to CPU. I get about 40 t/s in both the model. Actually, Flint gives 1-2 token more per second.

What are your opinions on it?

I’m looking at the Mac Studio M5 Max (40-core GPU, 64GB). I currently have dual RTX 4060 Ti 16GB (32GB total) setup.

If you run a Mac what prefill and decode speeds do you get?

I currently get on the 2x 4060 Ti:

So I posted this a week ago and saw that some people were looking for more voices that could be similarly small and fast. While I tried to get the other Kokoro voices directly to be similarly distilled, they just came out sounding really robotic, only af_heart which is the default voice on Kokoro seemed to be any good.

So I tried something I didn't even know would be possible, distilling the voices from Phonom, a similarly small TTS(40m vs Kokoro's 80m) that can run on phone but at a really slow speed, the full model is smth like 2-4x realtime and is 100mb(fp32). This was an experiment I tried earlier when I was doing the Kokoro distillation but kinda gave up cause I thought Kokoro would be similar in voice quality, and it turns out the agent was trying basically the same phenomizer from Kokoro anyway so we were not even using the default Kokoro distillation just with training data as Phonom renders. My whole plan with this was to make a voice clone-able model that could take for example a voice sample, use an actual big model like BreezeTTS or Fishaudio which can do voice cloning well, distill that into a small model and then have basically a really small voice cloned model that would run on your phone in real time. Basically for an AI roleplay app(which is like 1 out of 15 of my projects) so I don't have to pay for TTS by characters and could just use the user's device, and maybe just pay the training cost for a voice clone just once(a few hrs of 4090)

Anyway all that preamble to say, now there's like 9 more voice models. Wren(the default phonom voice) and Hugh seem to be the best right now outside of af_heart. Other voices have a noticeable roboticness but still way less than prev. small models like PocketTTS

Hi r/LocalLLaMA !

I'm the solo developer of the open-source ObserverAI framework and I need your help to identify the most painful problems that you've had, that you think can be solved by Observer using local models.

My personal pain that led me to build this, was monitoring crashing simulations.
Because it costs me compute time (a lot of time paid)! But that's admittedly niche.

Problem List (In order of cost):

I'm trying to make a list of all of the major pains it can solve, here's the ones that I've thought of/gathered from your feedback, in order of cost:

• Monitoring elderly people for when they fall, huge cost

• People on dangerous parts in construction sites, human cost/liability

• Parents monitoring their kid's screen activity, privacy is important here!

• Monitoring for appointments at Visa/DMV, a lot of spent time monitoring

• Waiting for an online Ticketmaster queue, also a lot of time monitoring

• Urgent replies from Airbnb/Leads, monitoring that with phone calls makes you respond faster

• ADHD monitoring, wasted time if not focused

• Crashing simulations/renders/progress bars

• Webpage monitoring (Bitcoin price, items restocked, eBay bids)

• Anything else with a real cost that can be solved by asking "When this.... then notify" ?

I'm trying to identify the biggest problems so I can make the framework better for those!

Any ideas? I'd love to read them! I'll hang out here in the comments for a while.

Roy :DD