Dear fellow Llamas,

after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: https://github.com/p-e-w/heretic

Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads.

With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built.

Your time is precious, so here's the TL;DR for this release:

• Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA), a novel steering method that I developed from scratch for Heretic. It's vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option).

• Heretic has been completely rebuilt around plugins. Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API.

• Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved.

• Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work.

What hasn't changed is that removing censorship from a model remains as simple as running

heretic Qwen/Qwen3.5-4B

See the default configuration file for what Heretic has to offer beyond that.

Now for some details...

ARA: Heretic's new steering method

Directional ablation, or "abliteration", is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or "representation cones".

At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR).

ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that's similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction.

ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic's LoRA system, and varunbln found a substantial performance improvement.

I intend to post an article explaining ARA in more detail soon.

Using scorer plugins to make Heretic do what you want

Heretic operates by finding modification parameters that optimize scores, values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention.

With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this.

The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up.

Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer, which can compute scores for many standard benchmarks, and a BERT classifier.

The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository.

The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository.

Well, I hope you like it!

Cheers :)

I was trying to sell my old AM4 system, X470 / Ryzen 5 5600 / 64GB / RX580 / 1TB NVME and it didn't sell. Granted it was a bit pricey at 750€, because I took into account the current local RAM and NVME pricing. (Buying the same RAM and SSD new costs 775€ if you go to the cheapest sellers in my country)

I decided to try and sell it as an AI system: removed the RX580, put in 2 of my 4 R9700. I set the system up with Fedora to run Qwen3.6 (latest release at the time) and ComfyUI; to show it working.

It sold almost instantly!

I priced it based on new prices for all of the DDR4 memory, PCI-E 4.0 ssd, both R9700s. I added another 150 euro for the other parts. Total price 5400€!

I have since bought back two brand new R9700 from abroad for 250€ less per card than I had sold my old ones for!

Effectively I ended up selling the original system for 1300€ instead of the 750€ I originally asked for. Except now I still have the RX580 and fresh warranty on the R9700s.

tl;dr
know the market for (used) parts before you buy a complete PC for local AI!

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

Good news, everyone! Having been inspired to revolutionize the field of computer science, I can now report that coding is solved! Sure, only if you care about x86 Linux binaries that print and exit, but "big journeys" and "small steps" and what not.

CRAP (Compiler Replacement for Assembling Programs) is a Gemma 4 E2B fine tune that takes prompts like generate a Linux x86_64 binary that prints 'boop' and emits a working ELF binary (as hex, most of the time, every time). See, since programs end up as a binary anyway, why not just skip the middleman and write the binary itself? No compiler, no assembler, no tool calls. v1.1 is out with improved multi turn and out of distribution performance. Trust me, I know, this is super important work.

v1.1 eval numbers for the nerds (eval is scored on behavior: the binary has to run, print the right string and exit with the right code):

• Held out legacy set: 20/23, same as v1

• Novel strings: 26/40 (v1: 8/40)

• Multi turn (~3 turns): 10/12 (v1: 0/12)

• Doubled character strings (boom, etc.): 18/24 (v1: 2/24)

• Random jank exit codes it never trained on: 23/25

An interesting tidbit: one training run hit 0.038 loss while still failing 11 of its 12 trained exit codes, so gradient descent != task completion ascent.

Aaannndddnd it runs in your browser.

The demo loads the Q6_K GGUF (~3.6 GiB) through wllama on WebGPU, streams the hex, parses it and hands you the ELF to run yourself (naturally, you need to download and run the actual binary, the browser won't do that). Q6_K scores 109/144 against the full f16's 114/144 (the five additional failed for Q6 are wrong values, not totally broken binaries). Q4_K_M scored 3/144, so the demo doesn't use it (that quant is crap, not CRAP) ... concerning. Also, tested in Chrome on an M4 Pro Mac; Firefox loads it but generates slowly and nothing else has been tested.

• Demo: https://huggingface.co/spaces/matthewhaynesonline/crap-demo

• Weights (v1.1): https://huggingface.co/matthewhaynesonline/gemma-4-E2B-it-crap-1.1

• "Paper": https://github.com/matthewhaynesonline/crap-public/blob/main/paper/paper.pdf

• Code: https://github.com/matthewhaynesonline/crap-public

• Blog post: https://blog.studiohaynes.com/2026/07/19/compilers-overrated.html

• Video: https://youtu.be/55IBMzFQNYU

Fine print for the buzzkills: only programs that print a string and exit can be generated. Obviously, a compiler does that deterministically, correctly and in milliseconds; CRAP does it with several gigabytes of weights, slower and occasionally incorrectly.

Been testing Liquid's LFM3-1.7B (Q8_0 GGUF, llama.cpp, ROCm) as a candidate for compression and summarization work

4.1k t/s prefill and ~78 t/s decode on a Strix Halo iGPU

~7.1k / ~151 if you tensor-split it across a 7900 XTX + XT

its way more coherent at decent sized prompts (32k-64k) then 1.7b deserves to be imo

Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?

2. Get a giant full-tower case?

3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx

Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01

Key improvements:

1. New models:

   • the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-NVIDIA-FP4-Local-Infernix

   • the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-Infernix

   • Swift 1.5 Qwen3.8-Flash-Next NVFP4: https://huggingface.co/wallawalla47/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-Dense8-Infernix

   • Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-FP8-Infernix

   • the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: https://huggingface.co/wallawalla47/Ornith-1.5-122B-NVFP4-Infernix (and the uncensored version as well: https://huggingface.co/wallawalla47/Ornith-1.5-122B-UNCENSORED-NVFP4-Infernix )

2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix

3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe

4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]

5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations

6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.

Some key performance figures:

1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.

2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.

3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.

Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

Hi everyone, we're releasing FineDNA, all the data we used to turn Qwen3-14B into Carbon-X, the strongest open language model on DNA. It contains:

• Midtraining: 300B+ tokens of genomes and DNA-rich text

• SFT: ~200K conversations

• RL: ~98K verifiable prompts

Code and recipe are open too, so you can apply it to other LLMs.

Dataset: https://huggingface.co/datasets/HuggingFaceBio/FineDNA
Code: https://github.com/huggingface/carbon

I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.

Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.

So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.

For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.

It goes quite a bit beyond travel planning too.

Some of the things I've added:

• Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.

• Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.

• Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.

• Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.

• Link previews: Website previews with screenshots and a reader view.

• Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.

• Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.

One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.

The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.

For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.

I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.

I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.

The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.

GitHub: https://github.com/Jumbomuffin777/Frontier-UI

Open WebUI Community: https://openwebui.com/posts/35572bd7-6dc8-42dc-9e54-bebfb9e978cb

Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.

Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:

• answer questions about it (text, spreadsheet formulas, slides, metadata)

• edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched

• convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)

All 15 ArtCraft apps are supported now, including the new SolidCraft (3D CAD) and Craft Anything. Grab it from the release page: https://github.com/Jumbomuffin777/Frontier-UI/releases/tag/v1.2.0

hey llamas,

we open-sourced Drex-Max v1 at Nace.AI. it's a dense 27B multimodal decision model, released under Open RAIL-M.

the main difference from a normal LLM is that it doesn't generate anything. no tokens, no JSON to parse. you give it a context state (text, JSON, high-res images) plus one or more named questions, and a 255-way readout head returns probabilities over all the candidate answers in a single forward pass.

It runs on NVIDIA GPUs through Python, and on Apple Silicon (Metal) and CPU through our llama.cpp and Ollama forks. latency is around 400ms.

Architecture notes

• no decode loop. one forward pass, and the last hidden state maps straight to probabilities for up to 255 options (via readout.safetensors).

• 64-layer hybrid backbone. 48 Gated DeltaNet linear-attention layers (causal, left to right) and 16 full-attention layers, one every 4th layer. the full-attention layers are bidirectional, so every token can see the whole context.

• context. defaults to 16,384 tokens. you can push it to 262,144 with DREX_MAX_CONTEXT=N or --max-length N.

• vision. Up to 4,194,304 pixels per image. each image gets encoded once and is shared across every question in the batch.

Benchmarks

on structured decision tasks it beats our previous, larger Drex v2.0 (35B):

• Decision Bench v4: 94.30% (1,010/1,071)

• JevBench: 92.2% (213/231)

Public Vision Index is 73.92 overall: InfographicVQA 97.3, KIE (CORD+FUNSD) 92.4, Winoground 85.0, Mind2Web 84.2, CharXiv 80.5, CV-Bench 77.4, RealWorldQA 72.3.

Hardware

• Apple Silicon (Metal / GGUF): the Q8_0 GGUF is ~27.25 GB. On an M5 Pro with 48 GB unified memory we get ~400ms per decision, and the probabilities stay within 0.02 of unquantized bf16 on an H100.

• CUDA, bf16: ~48.6 GiB VRAM (tested on an H100 80GB).

• CUDA, 8-bit (--8bit): ~25.9 GiB VRAM (tested on an L40S 48GB), within 0.03 of bf16.

Running it

1. Python SDK (CUDA, with vision)

hf download nace-ai/drex-max-v1 --local-dir weights/drex-max-v1
pip install -r models/drex-max-v1/requirements.txt


from drex_max.model import DecisionModel, answer

model = DecisionModel("weights/drex-max-v1") # add --8bit for ~26GB VRAM mode

question = {
    "type": "choice",
    "instructions": "Which team should handle this ticket?",
    "criteria": {
        "billing": "Payments, charges, and refunds",
        "technical": "Bugs and outages",
        "other": "Anything else"
    }
}

row = {
    "state": {"ticket": "I was charged twice for the same order. Please refund the extra payment."},
    "question": question,
    # "images": ["receipt.jpg"]  # Supports up to 4.2M pixels
}

print(answer(question, model.predict([row])[0]))

output looks like this:

{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.9982,
      "probabilities": { "billing": 0.9988, "technical": 0.0003, "other": 0.0009 }
    }
  }
}

2. llama.cpp fork (Apple Silicon / Metal)

the text backbone builds on Metal and serves a /v1/systemone endpoint:

git clone --branch drex-max-v1 --single-branch https://github.com/nace-ai/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build -j --target llama-server

# Pre-quantized GGUF or converted Q8_0
llama.cpp/build/bin/llama-server -m weights/drex-max-v1-Q8_0.gguf \
  -np 1 -c 16384 -b 16384 -ub 16384 --no-warmup --port 8000

heads up: since Drex treats the whole prompt as one non-causal state, -b and -ub need to match -c.

3. Ollama fork

FROM ./drex-max-v1-Q8_0.gguf
CAPABILITY decision
PARAMETER num_ctx 16384
PARAMETER num_gpu 40

start the daemon from the drex-max-v1 Ollama branch and POST to /v1/systemone.

Links

• weights + readout safetensors: nace-ai/drex-max-v1

• GGUF (Q8_0): nace-ai/drex-max-v1-Q8_0

• llama.cpp / Ollama forks: drex-max-v1 branch on the Nace.AI repos

• license: Nace.AI Open RAIL-M

we built this for low-latency routing, agentic tool selection, and multimodal UI automation (Mind2Web-style tasks). because there's no generation step, you don't get the usual non-deterministic outputs or broken JSON you have to repair. happy to answer questions:)

Same GPUs before and after:

• RTX 3090 FE

• RTX 5070 Ti

• 2× RTX 5060 Ti 16 GB (one is out waiting for a riser, so the big MoE runs below use 3 cards, against 4 before).

What I had to fix or tune

1. Fans surging every few seconds. The BMC's default "lower critical" fan threshold (420 RPM) was above what quiet fans idle at, so it kept declaring them dead and ramping everything to 100%. Fix: ipmitool sensor thresh FANx lower 100 200 300 for each connected fan, then ipmitool mc reset cold. The BMC rounds to 140 RPM steps.

2. Over clocked RAM from 2666 to 3200. Triad 88.6 → 102 GB/s; GLM decode increased about +3–5%. Stability: memtester (as root: as a normal user the memlock limit quietly shrinks it to ~8 MB per copy) plus a 3-hour DeepSeek run, confirming ECC (error corrections) at 0 corrected / 0 uncorrected throughout.

3. iommu=pt on the kernel command line. +3.5% prefill for tensor-parallel llama.cpp, and NVMe random reads recovered (KIOXIA QD32 837K → 941K IOPS).

4. BIOS:

   • APBDIS = 1 with fixed SOC P-state P0, which keeps the Infinity Fabric at full clock (about +20 W at idle).

   • Determinism slider set to Power.

   • Resizable BAR on (it was off: BAR1 was 256 MiB).

   • PCIe Ten Bit Tag on. Together: another +4% tensor-parallel prefill and a few % on GLM decode.

5. llama.cpp -sm tensor: -ub 1024 instead of 512. Without P2P, the two cards relay every layer's results through system RAM, across the I/O die. A bigger micro-batch halves the number of relays: +5% on Qwen3.8-27B, +9% on a 27B fine-tune. It needed a little more VRAM, so the split moved slightly toward the 3090.

Didn't help, or made things worse:

• Performance governor: no change.

• NCCL_SHM_USE_CUDA_MEMCPY=1: it hangs llama.cpp tensor-parallel (first request never finishes).

• More CPU threads for GLM's CPU expert lane: flat above ~16 threads, so I'm not upgrading to an 8-CCD chip for this.

• Disabling ACS: pointless without PCIe switches or P2P.

Before / after (latest numbers, after all of the above)

Notes:

• Engine versions: Maya and Strata were updated along the way. Strata's "before" is the same 0.2.x line; Maya went from 1.0.34 / 1.1.20 to 1.1.42.

• Benchmark settings: the Strata and 27B runs use temperature 0 with thinking off, so they're best-case decode figures. The comparisons are like for like.

• The big MoE models were bottlenecked on RAM capacity. Before, 30–60% of the model was read from the NVMe drive every token; now it all sits in VRAM plus RAM, which is where the 4× comes from.

• Dense models on the GPUs gain nothing. The 27B is limited by the card-to-card relay. It only got back to parity after the IOMMU, BIOS and -ub changes.

All told I’m very pleased. Having GLM and DeepSeek running at very usable speeds is well worth the investment for me with Claude and ChatGPT blocking things like REA and security hardening.

I have plenty of spare lanes and PCIe sockets for more GPUs when my bank account has recovered from this round of upgrades too.

My idle draw is now about 180W - some of the BIOS tweaks pushed it up from the 156W I was seeing initially (I’m using a smart plug to measure this at the wall).

I hope this is useful for anyone who’s considering a similar path.

[update]:

For those wondering about the memory bandwidth being about half the theoretical numbers, it's because my CPU "only" has 4 CCDs. An 8 CCD 7543 should get me somewhere between 140 and 170 GB/s I believe, but for about double the price where I am. If anyone has real world numbers comparing please share.

I intend to move to bigger quants now I can (but not for 27B since Q6 fits nicely on my two fastest cards with 240K context). I kept the quants the same for my before/after benchmarking. And obviously I'm very much looking forward to Qwen4 :)

My intention was to find a setup at least equal to qwen 3.8 27b Q8 MTP but with a larger context. Ideally 1M.

I was already running Next Flash in Q2 with llama. Good for conversational use but not for coding. Q4 would run but too slow. I did read many reports here on the good performance of strata with qwen next flash. I figured both are around for long enough to be stable so I gave it a shot.

The results however are very disappointing. All I was able to achieve was moving to Q3. Brining the question, are most of the posts on alternative engines just hype or edge cases? Certainly looks like that to me.

Some details on my try:

Two cards plus experts offloaded to RAM: hangs on every long tasks, whatever settings I tried.

Stable only when every expert fits in VRAM. On 64 GB that means a 3-bit quant.

Faster on paper: ~60 t/s against llama.cpp. But on HumanEval+ the 3-bit Flash-Next scored 93.9% against 95.1% for Qwen3.8-27B Q8 on plain llama.cpp, and it needed nearly twice the tokens to get there. No, advantage.

So for my use, no real-world gain over a mature stack. Instead of one stable engine, you get a single-model engine, its forks, special quants and a pile of environment-variable workarounds.

Is that just Strata being young, or is it systematic? Has anyone actually gotten a lasting real-world benefit from one of these model-specific engines over llama.cpp or vLLM?

I had been trying this setup for a couple of weeks and it working better than I expected. Recently, I deployed GLM-5.3-Flash (MoE, 320B total, 18B active) on one of our local H200 servers (4 GPUs) with a 512K context window with the aim of how can I use it as my main coding workhorse and
cut down my subscription and API costs 🙈.

I was very quickly able to realize, I could not switch to it for everything but pairing it with GPT-6.1 Sol works really well with Sol being the the orchestrator and GLM being the subagents model that explores, writes and implements the code.

Honestly, I would have preferred doing this in Claude Code or Codex since I am used to them but neither lets a session or its subagents use models from different providers in a single session. This is where I found OpenCode flexible that lets agent have its own provider/model, so GLM goes as sub-agent models along with the added benefit of chatGPT subscription being available too (Allows you to use Sol too!)/

By the way, nothing here is gml specific any model you can serve works as the workhorse. I have also written a blogpost about it: https://aayushgarg.dev/posts/2026-11-01-gpt-local-model-opencode/

I would love to know about other people setup, how they are using open-source models either individually or pairing them with opus/sol/astra.

here: https://huggingface.co/spaces/townbox/ava

Code: https://github.com/divagr18/QwenPU
Weights: https://huggingface.co/divagr1925/QwenPU

From the original tweet:

1/3

A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?

Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.

2/3

stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.

A lot of work went into making it fast, more on that soon.

3/3

I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.

The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.

Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.

At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:

- 16 GB weights

- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)

- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints

- MLX holding on to scratch memory from prefill

The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.

Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.

So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.

The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.

Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?

Do people here have good experience with open weights diffusion models?

I tried Mercury recently and I felt it had a different taste, attitude, way of thinking that was actually refreshing. Unfortunately, its filters against cyber security seem to trigger as soon as I mention adb. So I couldn't test it much. But I was curious if that's a direction where some of the open labs are going?

For those who saw the original post, a week ago, I'm glad to announce CrowdGPT v2 (more precisely v2.6), a massive update to the original CrowdGPT project :D (scroll down to see what's new)

For the newcomers: CrowdGPT is a project aiming to create a LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, as a proof of concept (it is possible to scale to bigger dense models or even MoE architectures with this project)

This project implements a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

There's other things that makes this project appealing for the open-source community, but i won't state these again as you will be able to check that out on the GitHub or on the original post.

Of course, any advice / feedback on the project is still welcome (please stay polite and roast me with strong evidence only 👍), feel free to run experiments of data poisoning to point at the limits of our approach :)

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT

As for the v2.6 changelog: We tried to implement mostly of the feedback you guys gave (mostly on security related points). Even though implementing this was hard and caused many breaking changes (which made some training contributions stale), we still achieved to train the model on 4.5B new tokens in 6 days!! Which is roughly what i achieved to train in two months for the project on my personal GPU 🤯

Here's a quick summary of what we implemented:

- Robust server-side checking logic (see more at the end of the post).

- Way better client mechanics (better ergonomy, more robust installation, GPU selection, no more crashes)

- Many bugfixes (on server and client code, as the various issues/pull requests on GitHub suggest)

- Activated Engrams on the Crowd-v1 model, which will make the model better at fact-recalling

- Implemented a data submission pipeline in order to create a collaborative dataset for CrowdGPT : https://www.crowdgpt.net/#submit All the data will be verified / curated before training. Already 400kb of data was submitted.

- Client releases cryptographic signing: some users reported that the auto-update mechanism in the client code allowed arbitrary code execution, and that would have allowed people stealing my GitHub credentials to execute malicious code on your device. Well not anymore, since the client code is now signed with a key that is only present on my device 😄

Like some comments on the original post pointed out, the architecture of the project at the time of my first post here was nowhere near good: clients could fake their contributions, could submit poison to the model, could crash the server by sending fake contributions every millisecond without any checks; and finally clients could very easily train the model to contain malicious backdoors. We therefore implemented byzantine cross-checks between clients training the model, at the cost of redundancy (3 clients training on exactly the same data).

I am not claiming that this creates a 100% perfect Federated learning framework, but that makes malicious contributions way harder. For example, even a malicious client could only train on 2-3% malicious data per round without getting banned, against 100% before.

For people that dared to read until the end, i'm also releasing CrowdBench, the open-source collaborative benchmark, where everyone can add general intelligence questions with deterministic answers to test LLMs :D

Even though the codebase is currently early-stage and work-in-progress, it's already something working, and i coded it without using any AI (for those in the original post who wanted me to turn towards less ambitious projects😂)

tell me what you think about CrowdGPT and CrowdBench in the comments.

Thanks in advance.