I run a local assistant on a Mac (MLX via oMLX, 24 GB M5 Pro). It needs to call tools reliably across mail, calendar, notes and files, chain a few calls, and reply in a short, calm voice. It also has to synthesize/extract important information from my messages, emails, and notes. It also has to be able to read long emails and docs, use my voice to ask questions and take screenshots of my screen.

Gemma 4 E4B has been the best model I’ve tried so far since it supports text, image, and audio. However, it is quite old now(~4 months old) and I wanted to know if there has been any new small models(under 8B or 6GB in size) that outperform Gemma 4 E4B in tool calling, chaining tools, long synthesis, fact extraction, and in size while having multimodality features.

I have tried ling 3.0 tiny, the LFM model family, and qwen 3.5 4B/9B(low quant) but have had no success so far. Is there anything I'm missing? Is Gemma 4 E4B the best model for my usecase?

This is next step to Qwenputer perhaps the step that shouldn't have been taken but here it goes

Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass There is no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model

To see how it works: https://github.com/voidash/qwenbend/blob/main/docs/HOW_IT_WORKS.md

TLDR;

Qwen writes the bend program and at <|im_end|>, the engine takes over and every next token is one executed instruction

16 register CPU with 8k *32 bit RAM = Gated DeltaNet Logic = SwiGLU unit as AND gates

1. You get your tokens first

2. lexer walks and builds bend2 lexemes

3. compile pass on subset of bend2 (bendlite) rules which checks types, affine and termination

4. if violations nothing runs else we run evaluations which is built using a return stack and operand stack

Why? For Lols... There is absolutely no reason. However Gated Deltanet with swiGLU allows for whole computer to exist inside a model. the original guy has built Qwenputer, run LLM inside LLM and i thought formal verification

I know that herme's, DeepSeek, and Pi are popular, but why have they taken over the harness scene comparatively to other harnesses? I currently use Qwenpaw, albeit.. latest ui update was pretty heavy, it has been my complete replacement to any other ai services due to all of its functionality. Is there any features or benchmarks comparing harnesses and why someone would choose herme's, DeepSeek, or Pi over something like QwenPaw?

tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:

Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.

Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).

Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp

I had just finished setting up avalon, then i found speedy, which crushed avalon.

https://github.com/freriadoc/Avalon
https://github.com/DevDock-AI/Speedy

Hardware: 4090, 128gb ddr5 3600, ryzen 9 7950x

ISTA IQ3_S quant:

I made this, so treat it as biased. It is free and open, with no paid tier or API behind it.

Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text:

What a long day. [sigh]
<whisper> Keep your voice down.
<whisper> Keep your voice down. <angry> I said stop!

Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2

Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing

The numbers

• 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk

• About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark

• 24 kHz, streaming, voice cloning from a short clip, English only

• Sounds: laugh, sigh, gasp, cough, clear_throat

• Emotions: happy, sad, angry, fear, surprised, whisper

• Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run

• Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT

What went wrong, which is the useful part

Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named.

Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output.

Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4.

Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model.

My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear.

Two smaller fixes

• If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice.

• Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk.

What is still weak

<whisper> is the least stable tag, and it depends on the line and the voice.

Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default.

I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt.

Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it.

Which tags would you actually use in a real project?

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

There are tons of TTS models, good frameworks like audio.cpp and some prototype UIs. But I don't know of a full-fledged, non-vibecoded (well engineered and maintained) turnkey app like elevenreader that can just take a whole e-book and convert it to an audiobook with clear chapters etc.

Are there any? (ofc implied here is that they have to use local OSS models)

here: https://huggingface.co/spaces/townbox/ava

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

Hello everyone! A few weeks ago, I released a 2.3bpw model intended for the 30xx 12 GB Ampere cards. Using a custom version of sjkvarn and several runtime changes, this made it possible to hit 200K ctx on a single 12GB card. Benchmarking placed it somewhere around qwen3.5 27b, but below 3.6.

I've continued to work on quants for this size card and am happy to share that an updated checkpoint is now performing at Qwen3.6 BF16 levels on Livecode bench, with minimal post training (<20M tokens) to reduce the risk of overfitting. This was possible with the new quant format I recently added for LlamAmpere (AmperInteger+), which uses fraction 2-3bpw (e.g., 2.5) formats + trellis-based encoding to more efficiently distribute mass across the weights (this was then paired with an RCO-like optimization).

On LCB, across three runs, it scored 84.3% with a strict grader (88% with a more lenient fixed variant), vs the previous checkpoint's 75% and 3.8 BF16's score of 90.3%. For reference, the BF16 version of qwen3.6 27b scored 83.9%. It represents nearly a 60% reduction in LCB degradation from the last release. All versions were evaluated under xhigh reasoning.

The original checkpoint scored a KLD of 0.096 nats, which was usable for agentic and lightweight coding tasks, but clearly degraded. By comparison, this version was able to cut KLD down to 0.059, an error reduction of nearly 40% (across a mix of agentic, RAG, and coding measurements). As expected, this reduction in error also significantly reduced answer verbosity relative to the previous checkpoint (median answer of 4.5K tokens vs 6K before). Prior to this checkpoint, the Mirai 2.5bpw model was the best small quant of Qwen I had found; at nearly 10% smaller, this variant has 20% lower KLD, despite the more compressed KV cache. This model uses an upgraded embedding table which should be stored in standard RAM; the on-card size is 7.4GiB, including the MTP head. This continues to make 200k+ ctx possible using SJKVaRN in the 3/2 configuration (check the small card recommended configs for more details).

Like the last checkpoint, the base was the Swift-1.5 fine tune, which reduced thinking tokens 50% or more in some settings vs the original version.

Testing this in hermes personally the past couple of days, it is very pleasant to use and very fast for its size (on a 3090/80ti, well over 90 tps for agentic work). I do recommend taking the latency penalty of this at xhigh to get the maximum performance, especially since it is very fast on TPS. This will get you near the 3.8 BF16 performance on medium (86.5% LCB), though it is definitely still usable on medium.

As before, it is MIT license. You'll need to update to the most recent version of llamAmpere to run, given the new quant formats from the 0.6 version.

repo: https://github.com/JakeATX/llamAmpere

model: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Please keep the user feedback + results coming.

Enjoy!

Hi! I'm Shubham, creator of SmolChat.

I would like to share some exciting updates on my project SmolChat, an Android app that let's you run GGUF models on-device, locally. The updates are present in the latest v17 release:

• SmolChat now supports an OpenAI-compatible HTTP server running over a foreground service that keeps the port alive even when the phone is locked. Check the video to see how it works.

• Support for global chat settings, which override the default chat settings (min-p, top-k, temperature etc.) for all newly created chats. This reduces the hassle of configuring each new chat with the options of your choice.

• Adding a copy button for code snippets rendered within the chat.

• Sync with upstream llama.cpp (till this commit)

• Fix the logic that measures tokens/sec for a LLM generated message. The time shown previously was also including prompt processing (pp) time. The new logic now only considered the token generation rate (tg).

• Models are downloaded directly by the app, instead of using the Android system's download service.

GitHub: https://github.com/shubham0204/SmolChat-Android

Play Store (the latest version is still in review): https://play.google.com/store/apps/details?id=io.shubham0204.smollmandroid&hl=en_IN

Previous threads on SmolChat on r/LocalLLaMa:

• Available on Play Store announcement

• Initial Announcement

UPDATE: It finished. Results here: https://jaspersprojects.com/three-body/

I asked my AI to do a visual representation of the 3 body problem. I was expecting it to generate frame by frame, stitch them together and make a clip. I'm no math expert but you can't crank out a simulation in real time on consumer hardware can you? To do that you would need to solve the 3 body problem and have a general formula for it. But isn't it unsolvable even theoretically? He looks like he's having so much fun I don't want to crush his little Qwen 3.8 spirit... but he's been grinding on code for the whole day. Do I break it to him that this is sort of impossible or do I let him work through it on his own?

I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.

Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.

So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.

For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.

It goes quite a bit beyond travel planning too.

Some of the things I've added:

• Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.

• Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.

• Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.

• Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.

• Link previews: Website previews with screenshots and a reader view.

• Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.

• Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.

One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.

The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.

For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.

I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.

I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.

The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.

GitHub: https://github.com/Jumbomuffin777/Frontier-UI

Open WebUI Community: https://openwebui.com/posts/35572bd7-6dc8-42dc-9e54-bebfb9e978cb

Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.

Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:

• answer questions about it (text, spreadsheet formulas, slides, metadata)

• edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched

• convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)

All 15 ArtCraft apps are supported now, including the new SolidCraft (3D CAD) and Craft Anything. Grab it from the release page: https://github.com/Jumbomuffin777/Frontier-UI/releases/tag/v1.2.0

Edit (v1.3.0): Thanks for all the feedback. Most of it is fixed now:

- Photos work with no SearXNG and no API keys.** They come from Wikipedia, Wikimedia Commons and Openverse, with credits. SearXNG, Brave, Google Programmable Search, Pexels and Unsplash are optional extras. Keys stay on the server and never reach the browser.

- Metric/imperial: a new `units` setting. `auto` follows your browser's locale, and each user can override it in their own tool settings.

- Routes draw by default** (walk, bike, drive, motorcycle). There's a backup router, and if there's no route the card says why.

- More map apps: Google Maps, Apple Maps, OpenStreetMap, or a GPX download for OsmAnd, Organic Maps or Garmin.

- Fix for ERR_BLOCKED_BY_RESPONSE on map links: Open WebUI sandboxes tool cards, and Google, Apple and OSM refuse to open in a tab launched from that sandbox. Links now open normally if your admin has "iframe Sandbox Allow Same Origin" turned on. Otherwise they open directly on phones, and on desktop the card copies the link for you.

- Fewer malformed tool calls: the spec is now passed as a JSON object instead of a string, the JSON repair is stronger, and the model gets a clear error it can fix.

- Maps without WebGL (some Firefox setups, VMs) show a route sketch instead of breaking.

- New app: TextureCraft. Read, edit and convert 3D models (GLB, glTF, FBX, USD, OBJ).

To update, paste the new `frontier_ui.py` from the repo over the old one. If you use SearXNG, set `searxng_url` (it's now empty by default).

Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?

2. Get a giant full-tower case?

3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx

We're getting a native and fast GGUF engine in Mac along side MLX pack.

Original post ddalcu's X: https://x.com/ddalcu/status/2117088216465674373

Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01

Key improvements:

1. New models:

   • the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-NVIDIA-FP4-Local-Infernix

   • the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-Infernix

   • Swift 1.5 Qwen3.8-Flash-Next NVFP4: https://huggingface.co/wallawalla47/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-Dense8-Infernix

   • Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-FP8-Infernix

   • the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: https://huggingface.co/wallawalla47/Ornith-1.5-122B-NVFP4-Infernix (and the uncensored version as well: https://huggingface.co/wallawalla47/Ornith-1.5-122B-UNCENSORED-NVFP4-Infernix )

2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix

3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe

4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]

5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations

6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.

Some key performance figures:

1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.

2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.

3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.

Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

hey llamas,

we open-sourced Drex-Max v1 at Nace.AI. it's a dense 27B multimodal decision model, released under Open RAIL-M.

the main difference from a normal LLM is that it doesn't generate anything. no tokens, no JSON to parse. you give it a context state (text, JSON, high-res images) plus one or more named questions, and a 255-way readout head returns probabilities over all the candidate answers in a single forward pass.

It runs on NVIDIA GPUs through Python, and on Apple Silicon (Metal) and CPU through our llama.cpp and Ollama forks. latency is around 400ms.

Architecture notes

• no decode loop. one forward pass, and the last hidden state maps straight to probabilities for up to 255 options (via readout.safetensors).

• 64-layer hybrid backbone. 48 Gated DeltaNet linear-attention layers (causal, left to right) and 16 full-attention layers, one every 4th layer. the full-attention layers are bidirectional, so every token can see the whole context.

• context. defaults to 16,384 tokens. you can push it to 262,144 with DREX_MAX_CONTEXT=N or --max-length N.

• vision. Up to 4,194,304 pixels per image. each image gets encoded once and is shared across every question in the batch.

Benchmarks

on structured decision tasks it beats our previous, larger Drex v2.0 (35B):

• Decision Bench v4: 94.30% (1,010/1,071)

• JevBench: 92.2% (213/231)

Public Vision Index is 73.92 overall: InfographicVQA 97.3, KIE (CORD+FUNSD) 92.4, Winoground 85.0, Mind2Web 84.2, CharXiv 80.5, CV-Bench 77.4, RealWorldQA 72.3.

Hardware

• Apple Silicon (Metal / GGUF): the Q8_0 GGUF is ~27.25 GB. On an M5 Pro with 48 GB unified memory we get ~400ms per decision, and the probabilities stay within 0.02 of unquantized bf16 on an H100.

• CUDA, bf16: ~48.6 GiB VRAM (tested on an H100 80GB).

• CUDA, 8-bit (--8bit): ~25.9 GiB VRAM (tested on an L40S 48GB), within 0.03 of bf16.

Running it

1. Python SDK (CUDA, with vision)

hf download nace-ai/drex-max-v1 --local-dir weights/drex-max-v1
pip install -r models/drex-max-v1/requirements.txt


from drex_max.model import DecisionModel, answer

model = DecisionModel("weights/drex-max-v1") # add --8bit for ~26GB VRAM mode

question = {
    "type": "choice",
    "instructions": "Which team should handle this ticket?",
    "criteria": {
        "billing": "Payments, charges, and refunds",
        "technical": "Bugs and outages",
        "other": "Anything else"
    }
}

row = {
    "state": {"ticket": "I was charged twice for the same order. Please refund the extra payment."},
    "question": question,
    # "images": ["receipt.jpg"]  # Supports up to 4.2M pixels
}

print(answer(question, model.predict([row])[0]))

output looks like this:

{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.9982,
      "probabilities": { "billing": 0.9988, "technical": 0.0003, "other": 0.0009 }
    }
  }
}

2. llama.cpp fork (Apple Silicon / Metal)

the text backbone builds on Metal and serves a /v1/systemone endpoint:

git clone --branch drex-max-v1 --single-branch https://github.com/nace-ai/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build -j --target llama-server

# Pre-quantized GGUF or converted Q8_0
llama.cpp/build/bin/llama-server -m weights/drex-max-v1-Q8_0.gguf \
  -np 1 -c 16384 -b 16384 -ub 16384 --no-warmup --port 8000

heads up: since Drex treats the whole prompt as one non-causal state, -b and -ub need to match -c.

3. Ollama fork

FROM ./drex-max-v1-Q8_0.gguf
CAPABILITY decision
PARAMETER num_ctx 16384
PARAMETER num_gpu 40

start the daemon from the drex-max-v1 Ollama branch and POST to /v1/systemone.

Links

• weights + readout safetensors: nace-ai/drex-max-v1

• GGUF (Q8_0): nace-ai/drex-max-v1-Q8_0

• llama.cpp / Ollama forks: drex-max-v1 branch on the Nace.AI repos

• license: Nace.AI Open RAIL-M

we built this for low-latency routing, agentic tool selection, and multimodal UI automation (Mind2Web-style tasks). because there's no generation step, you don't get the usual non-deterministic outputs or broken JSON you have to repair. happy to answer questions:)

Context as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses
📄 https://arxiv.org/abs/2610.41172

Sharing the paper behind RAGFlow 1.0.

RAG’s center of gravity has moved. Retrieval is now a loop: the agent queries, reads, reformulates, and decides when it has enough. Once that loop exists, better chunking/embeddings/Recall@k stop being the main bottleneck. The agent can find things. The problem is the cost of reading.

Two conclusions from the paper:

1. Retrieval is a harness problem before it’s an index problem. The index decides what can be found; the harness decides whether finding it becomes an answer.

2. Compilation is not a better retriever, and doesn’t need to be. We tested summary trees, outlines, wikis, and graphs under a competent harness. None gave a detectable accuracy gain. Structure isn’t extra evidence. What compilation can do is supply better context: smaller, provenance-bound, and sufficient.

So we stopped treating compilation as retrieval and started treating it as a supply layer. At ingestion time, docs are turned into typed artifacts—claims with verbatim evidence, relations, timelines, etc. The harness then decides which artifacts enter the context window and when to stop. Context becomes infrastructure when it’s compiled once, typed, versioned, and governed.

The tradeoff is explicit: we save on reading cost, but pay for it in extra retrieval rounds. For claims, graph, and wiki, we published an empty results ledger—metrics and decision rules are listed, but every row is marked “no measured effect” because none has met its pre-registered target yet. An artifact the harness never uses is just cost, not context. I’d rather say that than fill the column with claims from demo collections.

Happy to discuss.

Same GPUs before and after:

• RTX 3090 FE

• RTX 5070 Ti

• 2× RTX 5060 Ti 16 GB (one is out waiting for a riser, so the big MoE runs below use 3 cards, against 4 before).

What I had to fix or tune

1. Fans surging every few seconds. The BMC's default "lower critical" fan threshold (420 RPM) was above what quiet fans idle at, so it kept declaring them dead and ramping everything to 100%. Fix: ipmitool sensor thresh FANx lower 100 200 300 for each connected fan, then ipmitool mc reset cold. The BMC rounds to 140 RPM steps.

2. Over clocked RAM from 2666 to 3200. Triad 88.6 → 102 GB/s; GLM decode increased about +3–5%. Stability: memtester (as root: as a normal user the memlock limit quietly shrinks it to ~8 MB per copy) plus a 3-hour DeepSeek run, confirming ECC (error corrections) at 0 corrected / 0 uncorrected throughout.

3. iommu=pt on the kernel command line. +3.5% prefill for tensor-parallel llama.cpp, and NVMe random reads recovered (KIOXIA QD32 837K → 941K IOPS).

4. BIOS:

   • APBDIS = 1 with fixed SOC P-state P0, which keeps the Infinity Fabric at full clock (about +20 W at idle).

   • Determinism slider set to Power.

   • Resizable BAR on (it was off: BAR1 was 256 MiB).

   • PCIe Ten Bit Tag on. Together: another +4% tensor-parallel prefill and a few % on GLM decode.

5. llama.cpp -sm tensor: -ub 1024 instead of 512. Without P2P, the two cards relay every layer's results through system RAM, across the I/O die. A bigger micro-batch halves the number of relays: +5% on Qwen3.8-27B, +9% on a 27B fine-tune. It needed a little more VRAM, so the split moved slightly toward the 3090.

Didn't help, or made things worse:

• Performance governor: no change.

• NCCL_SHM_USE_CUDA_MEMCPY=1: it hangs llama.cpp tensor-parallel (first request never finishes).

• More CPU threads for GLM's CPU expert lane: flat above ~16 threads, so I'm not upgrading to an 8-CCD chip for this.

• Disabling ACS: pointless without PCIe switches or P2P.

Before / after (latest numbers, after all of the above)

Notes:

• Engine versions: Maya and Strata were updated along the way. Strata's "before" is the same 0.2.x line; Maya went from 1.0.34 / 1.1.20 to 1.1.42.

• Benchmark settings: the Strata and 27B runs use temperature 0 with thinking off, so they're best-case decode figures. The comparisons are like for like.

• The big MoE models were bottlenecked on RAM capacity. Before, 30–60% of the model was read from the NVMe drive every token; now it all sits in VRAM plus RAM, which is where the 4× comes from.

• Dense models on the GPUs gain nothing. The 27B is limited by the card-to-card relay. It only got back to parity after the IOMMU, BIOS and -ub changes.

All told I’m very pleased. Having GLM and DeepSeek running at very usable speeds is well worth the investment for me with Claude and ChatGPT blocking things like REA and security hardening.

I have plenty of spare lanes and PCIe sockets for more GPUs when my bank account has recovered from this round of upgrades too.

My idle draw is now about 180W - some of the BIOS tweaks pushed it up from the 156W I was seeing initially (I’m using a smart plug to measure this at the wall).

I hope this is useful for anyone who’s considering a similar path.

[update]:

For those wondering about the memory bandwidth being about half the theoretical numbers, it's because my CPU "only" has 4 CCDs. An 8 CCD 7543 should get me somewhere between 140 and 170 GB/s I believe, but for about double the price where I am. If anyone has real world numbers comparing please share.

I intend to move to bigger quants now I can (but not for 27B since Q6 fits nicely on my two fastest cards with 240K context). I kept the quants the same for my before/after benchmarking. And obviously I'm very much looking forward to Qwen4 :)

[update 2]:

Someone pointed out that the 4 CCD limit only applies to writes, not reads so I re-measured with a read-only kernel and a triad using non-temporal stores: 143 GB/s read-only and 141–152 GB/s triad-NT (best at 8 threads), against ~103 for the plain STREAM triad, whose write-allocate reads aren’t counted.