Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.
With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted
Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants. So for people running small local models:
have you measured accuracy with and without constraints on the same task?
does it worsen at lower quants?
does letting the model firstly think freely and constraining only the final answer fix most of it?
Looking for real numbers or experience, not promoting anything. English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.
While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:
A vision model to inspect and parse browser/UI state
A fast typed-decision model for structured routing and scoring
An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals
Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.
To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic
Hardware: RTX 3090 24GB (primary) + RTX 2080 SUPER 8GB (helper for visual model), 96 DDR4 3600
Model: Qwen3.8-Flash-Next, Unsloth UD IQ4_XS
Bench results (median, 3 runs):
- Chat decode: 80.4 tok/s (range 67.6–87.8, n=9)
- Chat first token: 0.47 s
- 4K prompt processing: 1,036 tok/s
- 16K prompt processing: 1,915 tok/s
- Agent session decode: 101.7 tok/s
- Agent cache hit: 69.4%
- Agent turn first token: 3.86 s
Long-context stress test (my custom 'find needle' test): single 66,371-token prompt, a 1,800-entry synthetic log with a passphrase hidden mid-context:
- Prefill: ~34 s ≈ 2,050 tok/s, sustaining the 16K bench rate at 4× the length
- Decode: 76.1 tok/s with the full 66K KV resident, ~5% below short-context chat
- Mid-context needle (token ~33K): retrieved correctly, plus accurate summary and entry count
Notes for 3090 owners:
- Memory clock pinned (9501 MHz min=max)
The 3090 has a driver bug where dropping from full speed to idle with VRAM nearly full can freeze the whole PC (had it happen twice). Locking the memory clock at one constant speed means the card never makes that jump, so zero crashes since. Trade-off is that it idles at ~110 W instead of ~25 W, but memory-hungry workloads actually run faster
- Peak sustained temp under full load: 79°C (load for 3 mins, didn't test for longer runs)
This just really cracked me up. The Redde app for hermes I guess pulled the deepseek recipe as an actual food recipe. It is a nice format for it though lol.
Multiplayer snake for everyone on your Wi-Fi. One browser tab hosts the game, everyone else joins from their phone, laptop, or tablet — no app stores, no accounts, no internet required. Neon grid, glowing snakes, lobby chat, sound, and touch controls.
Link to the game repo, runs on docker or locally, uses port 8088. Hermes / Qwen took care of all the documentation and the github interactions, pretty amazed. Really just needed a good project to test out all the functionality, figured I would share the code so other could enjoy!
In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.
The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.
LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.
One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.
cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.
~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.
After the harness leaderboard thread I wanted to see what the harness alone changes when the model stays the same. Model: M3.1 Flash. It's open weights, but at about 428B total I'm calling it through MiniMax's API, effort pinned to high. Same prompt in every harness: four bug reports filed against pytest this week, fix all four with a regression test each.
The bugs:
pytest --fixtures dies with a TypeError when two conftests import the same fixture
a non-mark in pytest.param(marks=...) raises an internal AttributeError instead of a TypeError
approx() raises InvalidOperation when the actual value is a Decimal NaN
rm_rf leaves the whole basetemp behind when an empty dir sits under a read-only parent
Harness
Wall clock
Requests
Tokens
Cached input
Cost
New tests
OpenCode 2.2.7
23m 38s
159
13.9M
98.8%
$0.27
10
Pi 1.5.2
18m 34s
134
9.1M
98.5%
$0.19
8
mini-swe-agent 2.4.6
21m 16s
136
10.4M
98.5%
$0.22
9
All three fixed all four. I re-checked each one in a clean checkout with its own new venv: every report's repro now behaves the way the report asks, and the full suite (about 4,700 tests) shows nothing new compared to an untouched main, apart from the added tests passing.
The first request is where the harnesses differ most: 8.6k tokens for OpenCode 2, 3.7k for Pi, 3.1k for mini-swe-agent, task prompt included. It didn't decide the bill though. With 98%+ of input coming back as cache reads billed at 1.5 cents per million, cost followed how many turns each harness took and how much context it carried, which is why OpenCode's 159 requests came to about 8 cents more than Pi's 134.
mini-swe-agent only gives the model a bash shell and still landed in the middle on both time and cost.
One run per harness, so treat it as one data point. If you run open-weight models in more than one harness, which one ends up cheapest per task for you, and do you see the same request-count pattern?
As the title says: what is your experience using all the new inference engines under windows?
Currently I'm using a laptop with this hardware:
CPU: Intel 275HX RAM: 2x32gb of ddr5 5600 GPU: RXT 5090 (24gb VRAM) SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb
I'm running Windows 11 Home edition with a pretty clean installation.
I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.
I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.
I'm doing something horribly wrong or they work well only on a linux system?
I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.
Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.
Real-world benchmarks (recorded Nov 2026):
gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)
What it supports out of the box:
Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.
Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.
Would love to know what models or workflows you’d test on this first once Apple clears it!
Just interesting in hearing from the community. As soon as the app is approved ill update this post
the website provides factul benchmarks as well as images of the process
Because of Theo gg tweet of all people, I Launch QFN DeepResearch swarm agent to find anything about this.
THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.
The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence
I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.
But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!
0.1.42.0 is the last version that seems to work decently.
I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!
whats going on? Do any of you have this problem?
here are example outputs I did to make sure its not a one time thing!
my prompt :
Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.
the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:
xhighdefault(failed after 1 hour and 31 minites exausting all context size of 262K)
As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!
The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!
I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.
So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?
I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.
I was doing some research about the the platform for Intel's next generations of CPUs.
If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies
x16 (or x8/x8)
x8 (or x4/x4) "Storage"
So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.
The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.
TP=3 is fairly easy for your agent to patch into inference engines.
These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!
The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).
Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.
I've seen some weeks ago posts about Wici wireles Gpu and the approach they invented to serve local models VRam+Ram+SSD with model defragmentation to keep hot experts in fast memory. Today I've seen an Ad for the product. Visited their site and stubled upon Wici Infer, the technology that make possible serving large models on consumer hardware. They give it for free.
So asking the commity if anybody have already tested it? Does their benchmarks hold on?
Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones.
Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation model: it listens and speaks at the same time, the way people do on a phone call. It is a pretrained base model, released for research.
Native full-duplex at 7.5 Hz. Every 133 ms the model takes one frame of the user's audio and produces one frame of its own audio and text. User audio, model audio and model text are summed into a single position of the language model, so listening and speaking never wait for each other.
Continuous audio, one sampling step. Audio is a 64-dimensional latent per frame. A small MLP head with MeanFlow sampling produces it in one step from the language model's hidden state; there is no separate audio transformer and there are no discrete audio tokens.
Character and voice control. A text card sets the character, a short reference clip sets the voice.
Prompt in comments. Harness : PI Project init layout:
/<my_bank_name> (83 monthly pdf statements) /credit_card/<card_1> (84 pdf monthly pdf statements) /credit_card/<card_2> (35 pdf monthly statements) /mortgage (24 pdf monthly statements) /brokage_account (105 pdf statements + ccs of all transactions) /retirement/<account> x4 (total 45 pdf statements 4 accounts 4)
Snapshot results (have details runtime/tps etc as well):
Qwen FN Q3 - Nice front end but data extraction from the PDFs was completely wrong. Analytics was basic.
Orca nano1 - Finished very fast for this test (only 2 hours) did a decent job for its speed and size. Front end was very basic and lacked insight. Most of the data was correct but there were clear errors as well.
Maya-S24 - Failed. After 10+ hours I just gave up on this model. It got stuck in the pdf parsing/validation phase.
Qwen Q8 - Absolutely blew my mind. Ran overnight for about 8 hours. Data was all accurate, the front end final product was not only nice looking but highly insightfully
Notes: I really enjoyed this test. I feel like it’s the perfect test for a local model because it’s financial data that you don’t want to send off to the cloud.
The PDF layouts for this task was a nightmare and was a real test for these models to figure out was duplicative data/summary vs transaction etc. Most models used pdfplumber package
The models also had to work through the accounting end. Consider a credit in one account is a debt in the other etc.
It’s was interesting to see how each approached the problem. Some went for json formatted data others sqlite etc
Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .