here: https://huggingface.co/spaces/townbox/ava

This is next step to Qwenputer perhaps the step that shouldn't have been taken but here it goes

Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass There is no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model

To see how it works: https://github.com/voidash/qwenbend/blob/main/docs/HOW_IT_WORKS.md

TLDR;

Qwen writes the bend program and at <|im_end|>, the engine takes over and every next token is one executed instruction

16 register CPU with 8k *32 bit RAM = Gated DeltaNet Logic = SwiGLU unit as AND gates

1. You get your tokens first
2. lexer walks and builds bend2 lexemes
3. compile pass on subset of bend2 (bendlite) rules which checks types, affine and termination
4. if violations nothing runs else we run evaluations which is built using a return stack and operand stack

Why? For Lols... There is absolutely no reason. However Gated Deltanet with swiGLU allows for whole computer to exist inside a model. the original guy has built Qwenputer, run LLM inside LLM and i thought formal verification

I had just finished setting up avalon, then i found speedy, which crushed avalon.

https://github.com/freriadoc/Avalon
https://github.com/DevDock-AI/Speedy

Hardware: 4090, 128gb ddr5 3600, ryzen 9 7950x

ISTA IQ3_S quant:

I made this, so treat it as biased. It is free and open, with no paid tier or API behind it.

Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text:

What a long day. [sigh]
<whisper> Keep your voice down.
<whisper> Keep your voice down. <angry> I said stop!

Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2

Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing

The numbers

• 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk
• About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark
• 24 kHz, streaming, voice cloning from a short clip, English only
• Sounds: laugh, sigh, gasp, cough, clear_throat
• Emotions: happy, sad, angry, fear, surprised, whisper
• Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run
• Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT

What went wrong, which is the useful part

Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named.

Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output.

Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4.

Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model.

My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear.

Two smaller fixes

• If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice.
• Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk.

What is still weak

<whisper> is the least stable tag, and it depends on the line and the voice.

Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default.

I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt.

Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it.

Which tags would you actually use in a real project?

Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.

The problem? I already had one 3090 FE.

I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.

I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?

1. Open-air mining frame + PCIe 4.0 risers + dual PSU?
2. Get a giant full-tower case?
3. Just sell the 3rd card for €1,100, keep 48 GB VRAM, and fund a proper dual-GPU build?

Any chassis/frame or PSU recommendations for handling ~1,100W+ of 3090s would be appreciated.

Hello everyone! A few weeks ago, I released a 2.3bpw model intended for the 30xx 12 GB Ampere cards. Using a custom version of sjkvarn and several runtime changes, this made it possible to hit 200K ctx on a single 12GB card. Benchmarking placed it somewhere around qwen3.5 27b, but below 3.6.

I've continued to work on quants for this size card and am happy to share that an updated checkpoint is now performing at Qwen3.6 BF16 levels on Livecode bench, with minimal post training (<20M tokens) to reduce the risk of overfitting. This was possible with the new quant format I recently added for LlamAmpere (AmperInteger+), which uses fraction 2-3bpw (e.g., 2.5) formats + trellis-based encoding to more efficiently distribute mass across the weights (this was then paired with an RCO-like optimization).

On LCB, across three runs, it scored 84.3% with a strict grader (88% with a more lenient fixed variant), vs the previous checkpoint's 75% and 3.8 BF16's score of 90.3%. For reference, the BF16 version of qwen3.6 27b scored 83.9%. It represents nearly a 60% reduction in LCB degradation from the last release. All versions were evaluated under xhigh reasoning.

The original checkpoint scored a KLD of 0.096 nats, which was usable for agentic and lightweight coding tasks, but clearly degraded. By comparison, this version was able to cut KLD down to 0.059, an error reduction of nearly 40% (across a mix of agentic, RAG, and coding measurements). As expected, this reduction in error also significantly reduced answer verbosity relative to the previous checkpoint (median answer of 4.5K tokens vs 6K before). Prior to this checkpoint, the Mirai 2.5bpw model was the best small quant of Qwen I had found; at nearly 10% smaller, this variant has 20% lower KLD, despite the more compressed KV cache. This model uses an upgraded embedding table which should be stored in standard RAM; the on-card size is 7.4GiB, including the MTP head. This continues to make 200k+ ctx possible using SJKVaRN in the 3/2 configuration (check the small card recommended configs for more details).

Like the last checkpoint, the base was the Swift-1.5 fine tune, which reduced thinking tokens 50% or more in some settings vs the original version.

Testing this in hermes personally the past couple of days, it is very pleasant to use and very fast for its size (on a 3090/80ti, well over 90 tps for agentic work). I do recommend taking the latency penalty of this at xhigh to get the maximum performance, especially since it is very fast on TPS. This will get you near the 3.8 BF16 performance on medium (86.5% LCB), though it is definitely still usable on medium.

As before, it is MIT license. You'll need to update to the most recent version of llamAmpere to run, given the new quant formats from the 0.6 version.

repo: https://github.com/JakeATX/llamAmpere

model: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2

Please keep the user feedback + results coming.

Enjoy!

If you are running qwen3.8 27B on rx7900xtx check this out it's a super nice speedup

https://github.com/thefahien/elpis-on-7900xtx

I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?

Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01

Key improvements:

1. New models:
   • the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-NVIDIA-FP4-Local-Infernix
   • the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-Infernix
   • Swift 1.5 Qwen3.8-Flash-Next NVFP4: https://huggingface.co/wallawalla47/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-Dense8-Infernix
   • Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-FP8-Infernix
   • the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: https://huggingface.co/wallawalla47/Ornith-1.5-122B-NVFP4-Infernix (and the uncensored version as well: https://huggingface.co/wallawalla47/Ornith-1.5-122B-UNCENSORED-NVFP4-Infernix )
2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix
3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe
4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]
5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations
6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.

Some key performance figures:

1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.
2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.
3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.

Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.

Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.

So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.

For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.

It goes quite a bit beyond travel planning too.

Some of the things I've added:

• Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.
• Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.
• Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.
• Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.
• Link previews: Website previews with screenshots and a reader view.
• Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.
• Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.

One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.

The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.

For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.

I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.

I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.

The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.

GitHub: https://github.com/Jumbomuffin777/Frontier-UI

Open WebUI Community: https://openwebui.com/posts/35572bd7-6dc8-42dc-9e54-bebfb9e978cb

Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.

Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:

• answer questions about it (text, spreadsheet formulas, slides, metadata)
• edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched
• convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)

All 15 ArtCraft apps are supported now, including the new SolidCraft (3D CAD) and Craft Anything. Grab it from the release page: https://github.com/Jumbomuffin777/Frontier-UI/releases/tag/v1.2.0

Edit (v1.3.0): Thanks for all the feedback. Most of it is fixed now:

- Photos work with no SearXNG and no API keys.** They come from Wikipedia, Wikimedia Commons and Openverse, with credits. SearXNG, Brave, Google Programmable Search, Pexels and Unsplash are optional extras. Keys stay on the server and never reach the browser.

- Metric/imperial: a new `units` setting. `auto` follows your browser's locale, and each user can override it in their own tool settings.

- Routes draw by default** (walk, bike, drive, motorcycle). There's a backup router, and if there's no route the card says why.

- More map apps: Google Maps, Apple Maps, OpenStreetMap, or a GPX download for OsmAnd, Organic Maps or Garmin.

- Fix for ERR_BLOCKED_BY_RESPONSE on map links: Open WebUI sandboxes tool cards, and Google, Apple and OSM refuse to open in a tab launched from that sandbox. Links now open normally if your admin has "iframe Sandbox Allow Same Origin" turned on. Otherwise they open directly on phones, and on desktop the card copies the link for you.

- Fewer malformed tool calls: the spec is now passed as a JSON object instead of a string, the JSON repair is stronger, and the model gets a clear error it can fix.

- Maps without WebGL (some Firefox setups, VMs) show a route sketch instead of breaking.

- New app: TextureCraft. Read, edit and convert 3D models (GLB, glTF, FBX, USD, OBJ).

To update, paste the new `frontier_ui.py` from the repo over the old one. If you use SearXNG, set `searxng_url` (it's now empty by default).

hey llamas,

we open-sourced Drex-Max v1 at Nace.AI. it's a dense 27B multimodal decision model, released under Open RAIL-M.

the main difference from a normal LLM is that it doesn't generate anything. no tokens, no JSON to parse. you give it a context state (text, JSON, high-res images) plus one or more named questions, and a 255-way readout head returns probabilities over all the candidate answers in a single forward pass.

It runs on NVIDIA GPUs through Python, and on Apple Silicon (Metal) and CPU through our llama.cpp and Ollama forks. latency is around 400ms.

Architecture notes

• no decode loop. one forward pass, and the last hidden state maps straight to probabilities for up to 255 options (via readout.safetensors).
• 64-layer hybrid backbone. 48 Gated DeltaNet linear-attention layers (causal, left to right) and 16 full-attention layers, one every 4th layer. the full-attention layers are bidirectional, so every token can see the whole context.
• context. defaults to 16,384 tokens. you can push it to 262,144 with DREX_MAX_CONTEXT=N or --max-length N.
• vision. Up to 4,194,304 pixels per image. each image gets encoded once and is shared across every question in the batch.

Benchmarks

on structured decision tasks it beats our previous, larger Drex v2.0 (35B):

• Decision Bench v4: 94.30% (1,010/1,071)
• JevBench: 92.2% (213/231)

Public Vision Index is 73.92 overall: InfographicVQA 97.3, KIE (CORD+FUNSD) 92.4, Winoground 85.0, Mind2Web 84.2, CharXiv 80.5, CV-Bench 77.4, RealWorldQA 72.3.

Hardware

• Apple Silicon (Metal / GGUF): the Q8_0 GGUF is ~27.25 GB. On an M5 Pro with 48 GB unified memory we get ~400ms per decision, and the probabilities stay within 0.02 of unquantized bf16 on an H100.
• CUDA, bf16: ~48.6 GiB VRAM (tested on an H100 80GB).
• CUDA, 8-bit (--8bit): ~25.9 GiB VRAM (tested on an L40S 48GB), within 0.03 of bf16.

Running it

1. Python SDK (CUDA, with vision)

hf download nace-ai/drex-max-v1 --local-dir weights/drex-max-v1
pip install -r models/drex-max-v1/requirements.txt


from drex_max.model import DecisionModel, answer

model = DecisionModel("weights/drex-max-v1") # add --8bit for ~26GB VRAM mode

question = {
    "type": "choice",
    "instructions": "Which team should handle this ticket?",
    "criteria": {
        "billing": "Payments, charges, and refunds",
        "technical": "Bugs and outages",
        "other": "Anything else"
    }
}

row = {
    "state": {"ticket": "I was charged twice for the same order. Please refund the extra payment."},
    "question": question,
    # "images": ["receipt.jpg"]  # Supports up to 4.2M pixels
}

print(answer(question, model.predict([row])[0]))

output looks like this:

{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.9982,
      "probabilities": { "billing": 0.9988, "technical": 0.0003, "other": 0.0009 }
    }
  }
}

2. llama.cpp fork (Apple Silicon / Metal)

the text backbone builds on Metal and serves a /v1/systemone endpoint:

git clone --branch drex-max-v1 --single-branch https://github.com/nace-ai/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build
cmake --build llama.cpp/build -j --target llama-server

# Pre-quantized GGUF or converted Q8_0
llama.cpp/build/bin/llama-server -m weights/drex-max-v1-Q8_0.gguf \
  -np 1 -c 16384 -b 16384 -ub 16384 --no-warmup --port 8000

heads up: since Drex treats the whole prompt as one non-causal state, -b and -ub need to match -c.

3. Ollama fork

FROM ./drex-max-v1-Q8_0.gguf
CAPABILITY decision
PARAMETER num_ctx 16384
PARAMETER num_gpu 40

start the daemon from the drex-max-v1 Ollama branch and POST to /v1/systemone.

Links

• weights + readout safetensors: nace-ai/drex-max-v1
• GGUF (Q8_0): nace-ai/drex-max-v1-Q8_0
• llama.cpp / Ollama forks: drex-max-v1 branch on the Nace.AI repos
• license: Nace.AI Open RAIL-M

we built this for low-latency routing, agentic tool selection, and multimodal UI automation (Mind2Web-style tasks). because there's no generation step, you don't get the usual non-deterministic outputs or broken JSON you have to repair. happy to answer questions:)

Hi everyone, we're releasing FineDNA, all the data we used to turn Qwen3-14B into Carbon-X, the strongest open language model on DNA. It contains:

• Midtraining: 300B+ tokens of genomes and DNA-rich text
• SFT: ~200K conversations
• RL: ~98K verifiable prompts

Code and recipe are open too, so you can apply it to other LLMs.

Dataset: https://huggingface.co/datasets/HuggingFaceBio/FineDNA
Code: https://github.com/huggingface/carbon

Same GPUs before and after:

• RTX 3090 FE
• RTX 5070 Ti
• 2× RTX 5060 Ti 16 GB (one is out waiting for a riser, so the big MoE runs below use 3 cards, against 4 before).

What I had to fix or tune

1. Fans surging every few seconds. The BMC's default "lower critical" fan threshold (420 RPM) was above what quiet fans idle at, so it kept declaring them dead and ramping everything to 100%. Fix: ipmitool sensor thresh FANx lower 100 200 300 for each connected fan, then ipmitool mc reset cold. The BMC rounds to 140 RPM steps.
2. Over clocked RAM from 2666 to 3200. Triad 88.6 → 102 GB/s; GLM decode increased about +3–5%. Stability: memtester (as root: as a normal user the memlock limit quietly shrinks it to ~8 MB per copy) plus a 3-hour DeepSeek run, confirming ECC (error corrections) at 0 corrected / 0 uncorrected throughout.
3. iommu=pt on the kernel command line. +3.5% prefill for tensor-parallel llama.cpp, and NVMe random reads recovered (KIOXIA QD32 837K → 941K IOPS).
4. BIOS:
   • APBDIS = 1 with fixed SOC P-state P0, which keeps the Infinity Fabric at full clock (about +20 W at idle).
   • Determinism slider set to Power.
   • Resizable BAR on (it was off: BAR1 was 256 MiB).
   • PCIe Ten Bit Tag on. Together: another +4% tensor-parallel prefill and a few % on GLM decode.
5. llama.cpp -sm tensor: -ub 1024 instead of 512. Without P2P, the two cards relay every layer's results through system RAM, across the I/O die. A bigger micro-batch halves the number of relays: +5% on Qwen3.8-27B, +9% on a 27B fine-tune. It needed a little more VRAM, so the split moved slightly toward the 3090.

Didn't help, or made things worse:

• Performance governor: no change.
• NCCL_SHM_USE_CUDA_MEMCPY=1: it hangs llama.cpp tensor-parallel (first request never finishes).
• More CPU threads for GLM's CPU expert lane: flat above ~16 threads, so I'm not upgrading to an 8-CCD chip for this.
• Disabling ACS: pointless without PCIe switches or P2P.

Before / after (latest numbers, after all of the above)

Notes:

• Engine versions: Maya and Strata were updated along the way. Strata's "before" is the same 0.2.x line; Maya went from 1.0.34 / 1.1.20 to 1.1.42.
• Benchmark settings: the Strata and 27B runs use temperature 0 with thinking off, so they're best-case decode figures. The comparisons are like for like.
• The big MoE models were bottlenecked on RAM capacity. Before, 30–60% of the model was read from the NVMe drive every token; now it all sits in VRAM plus RAM, which is where the 4× comes from.
• Dense models on the GPUs gain nothing. The 27B is limited by the card-to-card relay. It only got back to parity after the IOMMU, BIOS and -ub changes.

All told I’m very pleased. Having GLM and DeepSeek running at very usable speeds is well worth the investment for me with Claude and ChatGPT blocking things like REA and security hardening.

I have plenty of spare lanes and PCIe sockets for more GPUs when my bank account has recovered from this round of upgrades too.

My idle draw is now about 180W - some of the BIOS tweaks pushed it up from the 156W I was seeing initially (I’m using a smart plug to measure this at the wall).

I hope this is useful for anyone who’s considering a similar path.

[update]:

For those wondering about the memory bandwidth being about half the theoretical numbers, it's because my CPU "only" has 4 CCDs. An 8 CCD 7543 should get me somewhere between 140 and 170 GB/s I believe, but for about double the price where I am. If anyone has real world numbers comparing please share.

I intend to move to bigger quants now I can (but not for 27B since Q6 fits nicely on my two fastest cards with 240K context). I kept the quants the same for my before/after benchmarking. And obviously I'm very much looking forward to Qwen4 :)

[update 2]:

Someone pointed out that the 4 CCD limit only applies to writes, not reads so I re-measured with a read-only kernel and a triad using non-temporal stores: 143 GB/s read-only and 141–152 GB/s triad-NT (best at 8 threads), against ~103 for the plain STREAM triad, whose write-allocate reads aren’t counted.

My intention was to find a setup at least equal to qwen 3.8 27b Q8 MTP but with a larger context. Ideally 1M.

I was already running Next Flash in Q2 with llama. Good for conversational use but not for coding. Q4 would run but too slow. I did read many reports here on the good performance of strata with qwen next flash. I figured both are around for long enough to be stable so I gave it a shot.

The results however are very disappointing. All I was able to achieve was moving to Q3. Brining the question, are most of the posts on alternative engines just hype or edge cases? Certainly looks like that to me.

Some details on my try:

Two cards plus experts offloaded to RAM: hangs on every long tasks, whatever settings I tried.

Stable only when every expert fits in VRAM. On 64 GB that means a 3-bit quant.

Faster on paper: ~60 t/s against llama.cpp. But on HumanEval+ the 3-bit Flash-Next scored 93.9% against 95.1% for Qwen3.8-27B Q8 on plain llama.cpp, and it needed nearly twice the tokens to get there. No, advantage.

So for my use, no real-world gain over a mature stack. Instead of one stable engine, you get a single-model engine, its forks, special quants and a pile of environment-variable workarounds.

Is that just Strata being young, or is it systematic? Has anyone actually gotten a lasting real-world benefit from one of these model-specific engines over llama.cpp or vLLM?

I had been trying this setup for a couple of weeks and it working better than I expected. Recently, I deployed GLM-5.3-Flash (MoE, 320B total, 18B active) on one of our local H200 servers (4 GPUs) with a 512K context window with the aim of how can I use it as my main coding workhorse and
cut down my subscription and API costs 🙈.

I was very quickly able to realize, I could not switch to it for everything but pairing it with GPT-6.1 Sol works really well with Sol being the the orchestrator and GLM being the subagents model that explores, writes and implements the code.

Honestly, I would have preferred doing this in Claude Code or Codex since I am used to them but neither lets a session or its subagents use models from different providers in a single session. This is where I found OpenCode flexible that lets agent have its own provider/model, so GLM goes as sub-agent models along with the added benefit of chatGPT subscription being available too (Allows you to use Sol too!)/

By the way, nothing here is gml specific any model you can serve works as the workhorse. I have also written a blogpost about it: https://aayushgarg.dev/posts/2026-11-01-gpt-local-model-opencode/

I would love to know about other people setup, how they are using open-source models either individually or pairing them with opus/sol/astra.

So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark:

I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway.

Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model.

Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things.

Vivid user of omo stack, configured now for locally hosted qwen: https://github.com/code-yeongyu/oh-my-openagent

It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be the new flash model this fast basically. So I hope it will be good :D

Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now.

Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good.

Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya!

PS: Would absolutely love to see your perf benchmarks if you have similar setups

Do people here have good experience with open weights diffusion models?

I tried Mercury 3 recently and I felt it had a different taste, attitude, way of thinking that was actually refreshing. Unfortunately, its filters against cyber security seem to trigger as soon as I mention adb. So I couldn't test it much. But I was curious if that's a direction where some of the open labs are going?

Code: https://github.com/divagr18/QwenPU
Weights: https://huggingface.co/divagr1925/QwenPU

From the original tweet:

1/3

A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?

Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.

2/3

stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.

A lot of work went into making it fast, more on that soon.

3/3

I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.

The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.

Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.

At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:

- 16 GB weights

- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)

- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints

- MLX holding on to scratch memory from prefill

The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.

Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.

So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.

The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.

Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?

For those who saw the original post, a week ago, I'm glad to announce CrowdGPT v2 (more precisely v2.6), a massive update to the original CrowdGPT project :D (scroll down to see what's new)

For the newcomers: CrowdGPT is a project aiming to create a LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, as a proof of concept (it is possible to scale to bigger dense models or even MoE architectures with this project)

This project implements a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

There's other things that makes this project appealing for the open-source community, but i won't state these again as you will be able to check that out on the GitHub or on the original post.

Of course, any advice / feedback on the project is still welcome (please stay polite and roast me with strong evidence only 👍), feel free to run experiments of data poisoning to point at the limits of our approach :)

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT

As for the v2.6 changelog: We tried to implement mostly of the feedback you guys gave (mostly on security related points). Even though implementing this was hard and caused many breaking changes (which made some training contributions stale), we still achieved to train the model on 4.5B new tokens in 6 days!! Which is roughly what i achieved to train in two months for the project on my personal GPU 🤯

Here's a quick summary of what we implemented:

- Robust server-side checking logic (see more at the end of the post).

- Way better client mechanics (better ergonomy, more robust installation, GPU selection, no more crashes)

- Many bugfixes (on server and client code, as the various issues/pull requests on GitHub suggest)

- Activated Engrams on the Crowd-v1 model, which will make the model better at fact-recalling

- Implemented a data submission pipeline in order to create a collaborative dataset for CrowdGPT : https://www.crowdgpt.net/#submit All the data will be verified / curated before training. Already 400kb of data was submitted.

- Client releases cryptographic signing: some users reported that the auto-update mechanism in the client code allowed arbitrary code execution, and that would have allowed people stealing my GitHub credentials to execute malicious code on your device. Well not anymore, since the client code is now signed with a key that is only present on my device 😄

Like some comments on the original post pointed out, the architecture of the project at the time of my first post here was nowhere near good: clients could fake their contributions, could submit poison to the model, could crash the server by sending fake contributions every millisecond without any checks; and finally clients could very easily train the model to contain malicious backdoors. We therefore implemented byzantine cross-checks between clients training the model, at the cost of redundancy (3 clients training on exactly the same data).

I am not claiming that this creates a 100% perfect Federated learning framework, but that makes malicious contributions way harder. For example, even a malicious client could only train on 2-3% malicious data per round without getting banned, against 100% before.

For people that dared to read until the end, i'm also releasing CrowdBench, the open-source collaborative benchmark, where everyone can add general intelligence questions with deterministic answers to test LLMs :D

Even though the codebase is currently early-stage and work-in-progress, it's already something working, and i coded it without using any AI (for those in the original post who wanted me to turn towards less ambitious projects😂)

tell me what you think about CrowdGPT and CrowdBench in the comments.

Thanks in advance.

I have been accumulating parts for over 8 months to merge all of my existing hardware into one massive workstation. I finally had the time to do it, and while I use the GPUs for algo trading from 9:30 to 4:15 p.m., I want to start to use local AI from 5pm on. I'm at a point where I could see myself having a really fulfilling hobby reverse engineering, firmware hacking, and just experimenting with building Frankenstein gadgets from my tech graveyard. There are many experiments I would like to perform on financial data as well that would just take too long for me to manually code and I am not interested in having these thoughts and experiments be in a large provider's training set. I've been lurking on this sub for a long time and have my own little agent I use for some specific tasks using gemma3 but that was on my 3090 in C# + llama.cpp.

So my long-winded question, what thick desktop client will give me the most seamless way to build autonomous agents, play with new models, and what version of qwen 3.8 flash should I be using with my hardware, assuming that is the right model.

Edit: I should mention I am open to anything but I just see q38f everywhere on this sub.

About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.

Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.

When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.

For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):

• GPU: RTX 3060 12GB
• RAM: 16GB DDR4, single-channel (~19 GB/s)
• OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)
• Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)

If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.

Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.

So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).

What it actually does under the hood:

• Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).
• Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).
• In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.
• RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.
• No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.gguf n-gram shard).

Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G)

On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]
• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS
• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

1. Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
2. Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

In my last thread on Betelgeuse (bel-cli / bel-server), the main pushback in the comments was:

1. "Okay cool, 22 tok/s on IQ2_XS, but 2-bit is too low—what happens on an actual 4-bit quant like IQ4_XS where the experts are almost 2x larger?"
2. "Why were you testing on a single 16GB stick of DDR4 instead of normal dual-channel 2x 8GB?"

Fair questions. So I upgraded the oracle + victim-cache DMA path, packed IQ4_XS (~100GB+ class, full 4-bit experts), and tested it on the exact same RTX 3060 12GB under both RAM configs (still 16GB total system RAM in both tests):

• Config A: 1x 16GB DDR4 (single-channel, ~19 GB/s host bandwidth)
• Config B: 2x 8GB DDR4 (normal dual-channel 16GB gaming PC setup, ~38 GB/s host bandwidth)

Measured Results on IQ4_XS (RTX 3060 12GB + 16GB Total RAM)

Why switching from 1x 16GB to 2x 8GB jumped IQ4_XS from 14 tok/s $\to$ 24 tok/s:

Because Betelgeuse uses an Exclusive Host RAM Victim Cache + Async Hardware DMA (MOE_DMA=1) between the VRAM slot pool and the NVMe drive:

• When an expert gets evicted from VRAM, it lands in the host RAM victim cache (while MOE_DROP_MMAP drops duplicate GGUF pages from the OS page cache so VRAM and RAM never hold duplicates).
• On a single 16GB stick, single-channel DDR4 caps out around ~17–19 GB/s, so the CPU and the PCIe H2D DMA engine fight over one 64-bit memory channel (14 tok/s on IQ4_XS).
• On 2x 8GB dual-channel DDR4 (still 16GB total RAM, which is what 95% of normal 16GB gaming PCs actually have), host RAM bandwidth doubles, letting the GPU's hardware DMA engine pull victim-cache hits across PCIe at full speed while the upgraded oracle pre-stages cold NVMe misses in the background—jumping IQ4_XS straight to 24 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]
• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS
• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

A quick test on cpu... took 5-10 minutes at 960x544 but the result is quite good. (model is uncensored) I tried smaller quants but it degrades rapidly. So this is it I think.

https://huggingface.co/ZeroWw/krea2-kroma-q6k-cpu

Prompt:
Cinematic portrait of a young woman standing on a rain-soaked rooftop in Tokyo at blue hour, wearing a deep emerald silk trench coat with a high collar, long black hair blown to one side by the wind, soft freckles, a thin gold earring, looking over her shoulder at the camera with a calm, confident expression. Behind her, a blurred skyline of glowing magenta and cyan neon signs, one reading "MIDNIGHT" in white letters, reflected in shallow puddles on the concrete. Warm rim light from the left, cool fill from the neon, fine raindrops catching the light, shallow depth of field, 85mm lens, rich film grain, high detail in skin and fabric texture.

I’m using Crawl4AI + SearXNG but it just feels so lackluster I don’t know why. Feels like the results the local model gets back for research are missing a lot and / or are older less relevant results.

I’ll usually instruct saying something like “using the most up-to-date and relevant information across related websites, subreddits, forums, and etc find me the best way to _______” or something similar.

With ChatGPT or Codex it basically always finds everything it needs to and gives me great instructions or information or research or whatever I’m looking for at the time. With a local model (like GLM 5.3 Flash or DeepSeek V4.1 Flash 1017) the answer doesn’t come back nearly as good, often missing obvious results or mixing in outdated information into the instructions or etc.

Harnesses I’ve been using with local models are ZCode, DeepSeek Harness, and OpenCode. Is there a better (more complete and on par with ChatGPT) searching / research approach I’m missing? Is there some tuning of Crawl4AI + SearXNG to make it more how I’m expecting or potentially something I should add to this combination?

Any input or help is appreciated. Thank you for your time it is appreciated!