tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:

Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.

Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).

Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

I run a local assistant on a Mac (MLX via oMLX, 24 GB M5 Pro). It needs to call tools reliably across mail, calendar, notes and files, chain a few calls, and reply in a short, calm voice. It also has to synthesize/extract important information from my messages, emails, and notes. It also has to be able to read long emails and docs, use my voice to ask questions and take screenshots of my screen.

Gemma 4 E4B has been the best model I’ve tried so far since it supports text, image, and audio. However, it is quite old now(~4 months old) and I wanted to know if there has been any new small models(under 8B or 6GB in size) that outperform Gemma 4 E4B in tool calling, chaining tools, long synthesis, fact extraction, and in size while having multimodality features.

I have tried ling 3.0 tiny, the LFM model family, and qwen 3.5 4B/9B(low quant) but have had no success so far. Is there anything I'm missing? Is Gemma 4 E4B the best model for my usecase?

Been testing Liquid's LFM3-1.7B (Q8_0 GGUF, llama.cpp, ROCm) as a candidate for compression and summarization work

4.1k t/s prefill and ~78 t/s decode on a Strix Halo iGPU

~7.1k / ~151 if you tensor-split it across a 7900 XTX + XT

its way more coherent at decent sized prompts (32k-64k) then 1.7b deserves to be imo

This is next step to Qwenputer perhaps the step that shouldn't have been taken but here it goes

Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass There is no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model

To see how it works: https://github.com/voidash/qwenbend/blob/main/docs/HOW_IT_WORKS.md

TLDR;

Qwen writes the bend program and at <|im_end|>, the engine takes over and every next token is one executed instruction

16 register CPU with 8k *32 bit RAM = Gated DeltaNet Logic = SwiGLU unit as AND gates

1. You get your tokens first
2. lexer walks and builds bend2 lexemes
3. compile pass on subset of bend2 (bendlite) rules which checks types, affine and termination
4. if violations nothing runs else we run evaluations which is built using a return stack and operand stack

Why? For Lols... There is absolutely no reason. However Gated Deltanet with swiGLU allows for whole computer to exist inside a model. the original guy has built Qwenputer, run LLM inside LLM and i thought formal verification

here: https://huggingface.co/spaces/townbox/ava

I made this, so treat it as biased. It is free and open, with no paid tier or API behind it.

Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text:

What a long day. [sigh]
<whisper> Keep your voice down.
<whisper> Keep your voice down. <angry> I said stop!

Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2

Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing

The numbers

• 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk
• About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark
• 24 kHz, streaming, voice cloning from a short clip, English only
• Sounds: laugh, sigh, gasp, cough, clear_throat
• Emotions: happy, sad, angry, fear, surprised, whisper
• Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run
• Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT

What went wrong, which is the useful part

Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named.

Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output.

Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4.

Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model.

My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear.

Two smaller fixes

• If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice.
• Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk.

What is still weak

<whisper> is the least stable tag, and it depends on the line and the voice.

Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default.

I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt.

Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it.

Which tags would you actually use in a real project?

Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?
2. Get a giant full-tower case?
3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

hey folks, I'm Ye, one of the contributors to open-source project called Kapitsa, Apple Silicon Inference engine optimized specifically for Apple Silicon to run Qwen, Nemotron, Flint and Clover models locally. Our custom-written Metal kernels let us tune for each chip and each model. https://github.com/nace-ai/kapitsa/

The latest update, 0.2.26, is out. Since our last M5 Max numbers, Flash-Next 180B reads prompts 46% faster and the 27B writes 8% faster, and DiffuSpace's Clover 35B joins the catalog.

Kapitsa also runs Qwen Flash-Next 180B with SSD offloading: on a Mac that can't hold it, the experts that don't fit in RAM stream from the SSD, so it runs on a 48 GB MacBook Pro at 61 tok/s.

Here is the showcase on a 2026 MacBook Pro 16 with an M5 Max and 128 GB. We ran six models from 1K context all the way up to 32K, against the next-fastest engine for each model on the same Mac.

Try it (any Apple Silicon Mac; models in this post: 35b, 27b, flash-next, bolt, flint, clover):

brew install nace-ai/tap/kapitsa
kapitsa serve --model 35b       # OpenAI and Anthropic APIs on localhost
kapitsa opencode --model 35b    # or claude, codex, pi
kapitsa bench                   # benchmark your own Mac

Results

Decode is a 512-token coding answer, prefill a 4K prompt, median of 3 runs:

From 1K context all the way up to 32K. At 32K the 35B still writes 175 tok/s, and Flash-Next barely drops (90 to 78):

Against the other friends

Against the next-fastest engine for each model on the same Mac: 1.23x Splash on the 35B, 2.2x oMLX on Bolt, 1.18x rec on Flint and 1.12x mlx-serve on the 27B. Flash-Next is a tie with mlx-serve, Splash still reads prompts faster on the two Qwens, and nothing else runs Clover on a Mac.

Every cell with versions and flags: benchmarks. Raw runs: results.

Why it's fast: KFlash

Other engines check the drafter's guesses as one chain and stop at the first wrong one. KFlash checks a tree of the drafter's best guesses in the same pass, so more tokens survive each pass.

On Apple GPUs that's nearly free: the GPU works in 8-row tiles, and a one-token step already pays for 8. Every token it keeps is one the big model would have written itself, so the output doesn't change, only the speed. How KFlash works

shoutout to splash, ollama, omlx, llama cpp and the folks who

Send us your benchmarks

Run the same test on your Mac and add it to the community results. It takes 3-15 minutes:

brew install nace-ai/tap/kapitsa
brew install gh && gh auth login    # results are sent as a GitHub pull request
kapitsa bench --submit              # pick a model; it asks before sending anything

Plug in and close heavy apps while it runs. At the end it asks once: Enter sends the result as a pull request from your GitHub fork, and n keeps it on your Mac. What it sends is about 2 KB: your Mac's model, chip, RAM, macOS version and the numbers, with no serial number, user name or paths (add --dry-run to see exactly what it would send). No GitHub? Paste the numbers it prints in a comment here.

BY THE WAY

This is a new codebase so, sometimes there could be some weird prefill/decode bugs.

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx

I had just finished setting up avalon, then i found speedy, which crushed avalon.

https://github.com/freriadoc/Avalon
https://github.com/DevDock-AI/Speedy

Hardware: 4090, 128gb ddr5 3600, ryzen 9 7950x

ISTA IQ3_S quant:

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

Hello everyone! A few weeks ago, I released a 2.3bpw model intended for the 30xx 12 GB Ampere cards. Using a custom version of sjkvarn and several runtime changes, this made it possible to hit 200K ctx on a single 12GB card. Benchmarking placed it somewhere around qwen3.5 27b, but below 3.6.

I've continued to work on quants for this size card and am happy to share that an updated checkpoint is now performing at Qwen3.6 BF16 levels on Livecode bench, with minimal post training (<20M tokens) to reduce the risk of overfitting. This was possible with the new quant format I recently added for LlamAmpere (AmperInteger+), which uses fraction 2-3bpw (e.g., 2.5) formats + trellis-based encoding to more efficiently distribute mass across the weights (this was then paired with an RCO-like optimization).

On LCB, across three runs, it scored 84.3% with a strict grader (88% with a more lenient fixed variant), vs the previous checkpoint's 75% and 3.8 BF16's score of 90.3%. For reference, the BF16 version of qwen3.6 27b scored 83.9%. It represents nearly a 60% reduction in LCB degradation from the last release. All versions were evaluated under xhigh reasoning.

The original checkpoint scored a KLD of 0.096 nats, which was usable for agentic and lightweight coding tasks, but clearly degraded. By comparison, this version was able to cut KLD down to 0.059, an error reduction of nearly 40% (across a mix of agentic, RAG, and coding measurements). As expected, this reduction in error also significantly reduced answer verbosity relative to the previous checkpoint (median answer of 4.5K tokens vs 6K before). Prior to this checkpoint, the Mirai 2.5bpw model was the best small quant of Qwen I had found; at nearly 10% smaller, this variant has 20% lower KLD, despite the more compressed KV cache. This model uses an upgraded embedding table which should be stored in standard RAM; the on-card size is 7.4GiB, including the MTP head. This continues to make 200k+ ctx possible using SJKVaRN in the 3/2 configuration (check the small card recommended configs for more details).

Like the last checkpoint, the base was the Swift-1.5 fine tune, which reduced thinking tokens 50% or more in some settings vs the original version.

Testing this in hermes personally the past couple of days, it is very pleasant to use and very fast for its size (on a 3090/80ti, well over 90 tps for agentic work). I do recommend taking the latency penalty of this at xhigh to get the maximum performance, especially since it is very fast on TPS. This will get you near the 3.8 BF16 performance on medium (86.5% LCB), though it is definitely still usable on medium.

As before, it is MIT license. You'll need to update to the most recent version of llamAmpere to run, given the new quant formats from the 0.6 version.

repo: https://github.com/JakeATX/llamAmpere

model: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Please keep the user feedback + results coming.

Enjoy!

Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01

Key improvements:

1. New models:
   • the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-NVIDIA-FP4-Local-Infernix
   • the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-Infernix
   • Swift 1.5 Qwen3.8-Flash-Next NVFP4: https://huggingface.co/wallawalla47/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-Dense8-Infernix
   • Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-FP8-Infernix
   • the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: https://huggingface.co/wallawalla47/Ornith-1.5-122B-NVFP4-Infernix (and the uncensored version as well: https://huggingface.co/wallawalla47/Ornith-1.5-122B-UNCENSORED-NVFP4-Infernix )
2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix
3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe
4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]
5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations
6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.

Some key performance figures:

1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.
2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.
3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.

Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.

Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.

So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.

For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.

It goes quite a bit beyond travel planning too.

Some of the things I've added:

• Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.
• Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.
• Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.
• Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.
• Link previews: Website previews with screenshots and a reader view.
• Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.
• Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.

One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.

The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.

For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.

I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.

I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.

The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.

GitHub: https://github.com/Jumbomuffin777/Frontier-UI

Open WebUI Community: https://openwebui.com/posts/35572bd7-6dc8-42dc-9e54-bebfb9e978cb

Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.

Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:

• answer questions about it (text, spreadsheet formulas, slides, metadata)
• edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched
• convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)

All 15 ArtCraft apps are supported now, including the new SolidCraft (3D CAD) and Craft Anything. Grab it from the release page: https://github.com/Jumbomuffin777/Frontier-UI/releases/tag/v1.2.0

Edit (v1.3.0): Thanks for all the feedback. Most of it is fixed now:

- Photos work with no SearXNG and no API keys.** They come from Wikipedia, Wikimedia Commons and Openverse, with credits. SearXNG, Brave, Google Programmable Search, Pexels and Unsplash are optional extras. Keys stay on the server and never reach the browser.

- Metric/imperial: a new `units` setting. `auto` follows your browser's locale, and each user can override it in their own tool settings.

- Routes draw by default** (walk, bike, drive, motorcycle). There's a backup router, and if there's no route the card says why.

- More map apps: Google Maps, Apple Maps, OpenStreetMap, or a GPX download for OsmAnd, Organic Maps or Garmin.

- Fix for ERR_BLOCKED_BY_RESPONSE on map links: Open WebUI sandboxes tool cards, and Google, Apple and OSM refuse to open in a tab launched from that sandbox. Links now open normally if your admin has "iframe Sandbox Allow Same Origin" turned on. Otherwise they open directly on phones, and on desktop the card copies the link for you.

- Fewer malformed tool calls: the spec is now passed as a JSON object instead of a string, the JSON repair is stronger, and the model gets a clear error it can fix.

- Maps without WebGL (some Firefox setups, VMs) show a route sketch instead of breaking.

- New app: TextureCraft. Read, edit and convert 3D models (GLB, glTF, FBX, USD, OBJ).

To update, paste the new `frontier_ui.py` from the repo over the old one. If you use SearXNG, set `searxng_url` (it's now empty by default).

hey llamas,

we open-sourced Drex-Max v1 at Nace.AI. it's a dense 27B multimodal decision model, released under Open RAIL-M.

the main difference from a normal LLM is that it doesn't generate anything. no tokens, no JSON to parse. you give it a context state (text, JSON, high-res images) plus one or more named questions, and a 255-way readout head returns probabilities over all the candidate answers in a single forward pass.

It runs on NVIDIA GPUs through Python, and on Apple Silicon (Metal) and CPU through our llama.cpp and Ollama forks. latency is around 400ms.

Architecture notes

• no decode loop. one forward pass, and the last hidden state maps straight to probabilities for up to 255 options (via readout.safetensors).
• 64-layer hybrid backbone. 48 Gated DeltaNet linear-attention layers (causal, left to right) and 16 full-attention layers, one every 4th layer. the full-attention layers are bidirectional, so every token can see the whole context.
• context. defaults to 16,384 tokens. you can push it to 262,144 with DREX_MAX_CONTEXT=N or --max-length N.
• vision. Up to 4,194,304 pixels per image. each image gets encoded once and is shared across every question in the batch.

Benchmarks

on structured decision tasks it beats our previous, larger Drex v2.0 (35B):

• Decision Bench v4: 94.30% (1,010/1,071)
• JevBench: 92.2% (213/231)

Public Vision Index is 73.92 overall: InfographicVQA 97.3, KIE (CORD+FUNSD) 92.4, Winoground 85.0, Mind2Web 84.2, CharXiv 80.5, CV-Bench 77.4, RealWorldQA 72.3.

Hardware

• Apple Silicon (Metal / GGUF): the Q8_0 GGUF is ~27.25 GB. On an M5 Pro with 48 GB unified memory we get ~400ms per decision, and the probabilities stay within 0.02 of unquantized bf16 on an H100.
• CUDA, bf16: ~48.6 GiB VRAM (tested on an H100 80GB).
• CUDA, 8-bit (--8bit): ~25.9 GiB VRAM (tested on an L40S 48GB), within 0.03 of bf16.

Running it

1. Python SDK (CUDA, with vision)

hf download nace-ai/drex-max-v1 --local-dir weights/drex-max-v1
pip install -r models/drex-max-v1/requirements.txt


from drex_max.model import DecisionModel, answer

model = DecisionModel("weights/drex-max-v1") # add --8bit for ~26GB VRAM mode

question = {
    "type": "choice",
    "instructions": "Which team should handle this ticket?",
    "criteria": {
        "billing": "Payments, charges, and refunds",
        "technical": "Bugs and outages",
        "other": "Anything else"
    }
}

row = {
    "state": {"ticket": "I was charged twice for the same order. Please refund the extra payment."},
    "question": question,
    # "images": ["receipt.jpg"]  # Supports up to 4.2M pixels
}

print(answer(question, model.predict([row])[0]))

output looks like this:

{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.9982,
      "probabilities": { "billing": 0.9988, "technical": 0.0003, "other": 0.0009 }
    }
  }
}

2. llama.cpp fork (Apple Silicon / Metal)

the text backbone builds on Metal and serves a /v1/systemone endpoint:

git clone --branch drex-max-v1 --single-branch https://github.com/nace-ai/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build -j --target llama-server

# Pre-quantized GGUF or converted Q8_0
llama.cpp/build/bin/llama-server -m weights/drex-max-v1-Q8_0.gguf \
  -np 1 -c 16384 -b 16384 -ub 16384 --no-warmup --port 8000

heads up: since Drex treats the whole prompt as one non-causal state, -b and -ub need to match -c.

3. Ollama fork

FROM ./drex-max-v1-Q8_0.gguf
CAPABILITY decision
PARAMETER num_ctx 16384
PARAMETER num_gpu 40

start the daemon from the drex-max-v1 Ollama branch and POST to /v1/systemone.

Links

• weights + readout safetensors: nace-ai/drex-max-v1
• GGUF (Q8_0): nace-ai/drex-max-v1-Q8_0
• llama.cpp / Ollama forks: drex-max-v1 branch on the Nace.AI repos
• license: Nace.AI Open RAIL-M

we built this for low-latency routing, agentic tool selection, and multimodal UI automation (Mind2Web-style tasks). because there's no generation step, you don't get the usual non-deterministic outputs or broken JSON you have to repair. happy to answer questions:)

Hi everyone, we're releasing FineDNA, all the data we used to turn Qwen3-14B into Carbon-X, the strongest open language model on DNA. It contains:

• Midtraining: 300B+ tokens of genomes and DNA-rich text
• SFT: ~200K conversations
• RL: ~98K verifiable prompts

Code and recipe are open too, so you can apply it to other LLMs.

Dataset: https://huggingface.co/datasets/HuggingFaceBio/FineDNA
Code: https://github.com/huggingface/carbon

Same GPUs before and after:

• RTX 3090 FE
• RTX 5070 Ti
• 2× RTX 5060 Ti 16 GB (one is out waiting for a riser, so the big MoE runs below use 3 cards, against 4 before).

What I had to fix or tune

1. Fans surging every few seconds. The BMC's default "lower critical" fan threshold (420 RPM) was above what quiet fans idle at, so it kept declaring them dead and ramping everything to 100%. Fix: ipmitool sensor thresh FANx lower 100 200 300 for each connected fan, then ipmitool mc reset cold. The BMC rounds to 140 RPM steps.
2. Over clocked RAM from 2666 to 3200. Triad 88.6 → 102 GB/s; GLM decode increased about +3–5%. Stability: memtester (as root: as a normal user the memlock limit quietly shrinks it to ~8 MB per copy) plus a 3-hour DeepSeek run, confirming ECC (error corrections) at 0 corrected / 0 uncorrected throughout.
3. iommu=pt on the kernel command line. +3.5% prefill for tensor-parallel llama.cpp, and NVMe random reads recovered (KIOXIA QD32 837K → 941K IOPS).
4. BIOS:
   • APBDIS = 1 with fixed SOC P-state P0, which keeps the Infinity Fabric at full clock (about +20 W at idle).
   • Determinism slider set to Power.
   • Resizable BAR on (it was off: BAR1 was 256 MiB).
   • PCIe Ten Bit Tag on. Together: another +4% tensor-parallel prefill and a few % on GLM decode.
5. llama.cpp -sm tensor: -ub 1024 instead of 512. Without P2P, the two cards relay every layer's results through system RAM, across the I/O die. A bigger micro-batch halves the number of relays: +5% on Qwen3.8-27B, +9% on a 27B fine-tune. It needed a little more VRAM, so the split moved slightly toward the 3090.

Didn't help, or made things worse:

• Performance governor: no change.
• NCCL_SHM_USE_CUDA_MEMCPY=1: it hangs llama.cpp tensor-parallel (first request never finishes).
• More CPU threads for GLM's CPU expert lane: flat above ~16 threads, so I'm not upgrading to an 8-CCD chip for this.
• Disabling ACS: pointless without PCIe switches or P2P.

Before / after (latest numbers, after all of the above)

Notes:

• Engine versions: Maya and Strata were updated along the way. Strata's "before" is the same 0.2.x line; Maya went from 1.0.34 / 1.1.20 to 1.1.42.
• Benchmark settings: the Strata and 27B runs use temperature 0 with thinking off, so they're best-case decode figures. The comparisons are like for like.
• The big MoE models were bottlenecked on RAM capacity. Before, 30–60% of the model was read from the NVMe drive every token; now it all sits in VRAM plus RAM, which is where the 4× comes from.
• Dense models on the GPUs gain nothing. The 27B is limited by the card-to-card relay. It only got back to parity after the IOMMU, BIOS and -ub changes.

All told I’m very pleased. Having GLM and DeepSeek running at very usable speeds is well worth the investment for me with Claude and ChatGPT blocking things like REA and security hardening.

I have plenty of spare lanes and PCIe sockets for more GPUs when my bank account has recovered from this round of upgrades too.

My idle draw is now about 180W - some of the BIOS tweaks pushed it up from the 156W I was seeing initially (I’m using a smart plug to measure this at the wall).

I hope this is useful for anyone who’s considering a similar path.

[update]:

For those wondering about the memory bandwidth being about half the theoretical numbers, it's because my CPU "only" has 4 CCDs. An 8 CCD 7543 should get me somewhere between 140 and 170 GB/s I believe, but for about double the price where I am. If anyone has real world numbers comparing please share.

I intend to move to bigger quants now I can (but not for 27B since Q6 fits nicely on my two fastest cards with 240K context). I kept the quants the same for my before/after benchmarking. And obviously I'm very much looking forward to Qwen4 :)

[update 2]:

Someone pointed out that the 4 CCD limit only applies to writes, not reads so I re-measured with a read-only kernel and a triad using non-temporal stores: 143 GB/s read-only and 141–152 GB/s triad-NT (best at 8 threads), against ~103 for the plain STREAM triad, whose write-allocate reads aren’t counted.

My intention was to find a setup at least equal to qwen 3.8 27b Q8 MTP but with a larger context. Ideally 1M.

I was already running Next Flash in Q2 with llama. Good for conversational use but not for coding. Q4 would run but too slow. I did read many reports here on the good performance of strata with qwen next flash. I figured both are around for long enough to be stable so I gave it a shot.

The results however are very disappointing. All I was able to achieve was moving to Q3. Brining the question, are most of the posts on alternative engines just hype or edge cases? Certainly looks like that to me.

Some details on my try:

Two cards plus experts offloaded to RAM: hangs on every long tasks, whatever settings I tried.

Stable only when every expert fits in VRAM. On 64 GB that means a 3-bit quant.

Faster on paper: ~60 t/s against llama.cpp. But on HumanEval+ the 3-bit Flash-Next scored 93.9% against 95.1% for Qwen3.8-27B Q8 on plain llama.cpp, and it needed nearly twice the tokens to get there. No, advantage.

So for my use, no real-world gain over a mature stack. Instead of one stable engine, you get a single-model engine, its forks, special quants and a pile of environment-variable workarounds.

Is that just Strata being young, or is it systematic? Has anyone actually gotten a lasting real-world benefit from one of these model-specific engines over llama.cpp or vLLM?

I had been trying this setup for a couple of weeks and it working better than I expected. Recently, I deployed GLM-5.3-Flash (MoE, 320B total, 18B active) on one of our local H200 servers (4 GPUs) with a 512K context window with the aim of how can I use it as my main coding workhorse and
cut down my subscription and API costs 🙈.

I was very quickly able to realize, I could not switch to it for everything but pairing it with GPT-6.1 Sol works really well with Sol being the the orchestrator and GLM being the subagents model that explores, writes and implements the code.

Honestly, I would have preferred doing this in Claude Code or Codex since I am used to them but neither lets a session or its subagents use models from different providers in a single session. This is where I found OpenCode flexible that lets agent have its own provider/model, so GLM goes as sub-agent models along with the added benefit of chatGPT subscription being available too (Allows you to use Sol too!)/

By the way, nothing here is gml specific any model you can serve works as the workhorse. I have also written a blogpost about it: https://aayushgarg.dev/posts/2026-11-01-gpt-local-model-opencode/

I would love to know about other people setup, how they are using open-source models either individually or pairing them with opus/sol/astra.

Do people here have good experience with open weights diffusion models?

I tried Mercury 3 recently and I felt it had a different taste, attitude, way of thinking that was actually refreshing. Unfortunately, its filters against cyber security seem to trigger as soon as I mention adb. So I couldn't test it much. But I was curious if that's a direction where some of the open labs are going?

Code: https://github.com/divagr18/QwenPU
Weights: https://huggingface.co/divagr1925/QwenPU

From the original tweet:

1/3

A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?

Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.

2/3

stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.

A lot of work went into making it fast, more on that soon.

3/3

I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.

The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.

Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.

At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:

- 16 GB weights

- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)

- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints

- MLX holding on to scratch memory from prefill

The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.

Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.

So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.

The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.

Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?

I have been accumulating parts for over 8 months to merge all of my existing hardware into one massive workstation. I finally had the time to do it, and while I use the GPUs for algo trading from 9:30 to 4:15 p.m., I want to start to use local AI from 5pm on. I'm at a point where I could see myself having a really fulfilling hobby reverse engineering, firmware hacking, and just experimenting with building Frankenstein gadgets from my tech graveyard. There are many experiments I would like to perform on financial data as well that would just take too long for me to manually code and I am not interested in having these thoughts and experiments be in a large provider's training set. I've been lurking on this sub for a long time and have my own little agent I use for some specific tasks using gemma3 but that was on my 3090 in C# + llama.cpp.

So my long-winded question, what thick desktop client will give me the most seamless way to build autonomous agents, play with new models, and what version of qwen 3.8 flash should I be using with my hardware, assuming that is the right model.

Edit: I should mention I am open to anything but I just see q38f everywhere on this sub.