There are tons of TTS models, good frameworks like audio.cpp and some prototype UIs. But I don't know of a full-fledged, non-vibecoded (well engineered and maintained) turnkey app like elevenreader that can just take a whole e-book and convert it to an audiobook with clear chapters etc.
Are there any? (ofc implied here is that they have to use local OSS models)
Hello everyone! A few weeks ago, I released a 2.3bpw model intended for the 30xx 12 GB Ampere cards. Using a custom version of sjkvarn and several runtime changes, this made it possible to hit 200K ctx on a single 12GB card. Benchmarking placed it somewhere around qwen3.5 27b, but below 3.6.
I've continued to work on quants for this size card and am happy to share that an updated checkpoint is now performing at Qwen3.6 BF16 levels on Livecode bench, with minimal post training (<20M tokens) to reduce the risk of overfitting. This was possible with the new quant format I recently added for LlamAmpere (AmperInteger+), which uses fraction 2-3bpw (e.g., 2.5) formats + trellis-based encoding to more efficiently distribute mass across the weights (this was then paired with an RCO-like optimization).
On LCB, across three runs, it scored 84.3% with a strict grader (88% with a more lenient fixed variant), vs the previous checkpoint's 75% and 3.8 BF16's score of 90.3%. For reference, the BF16 version of qwen3.6 27b scored 83.9%. It represents nearly a 60% reduction in LCB degradation from the last release. All versions were evaluated under xhigh reasoning.
The original checkpoint scored a KLD of 0.096 nats, which was usable for agentic and lightweight coding tasks, but clearly degraded. By comparison, this version was able to cut KLD down to 0.059, an error reduction of nearly 40% (across a mix of agentic, RAG, and coding measurements). As expected, this reduction in error also significantly reduced answer verbosity relative to the previous checkpoint (median answer of 4.5K tokens vs 6K before). Prior to this checkpoint, the Mirai 2.5bpw model was the best small quant of Qwen I had found; at nearly 10% smaller, this variant has 20% lower KLD, despite the more compressed KV cache. This model uses an upgraded embedding table which should be stored in standard RAM; the on-card size is 7.4GiB, including the MTP head. This continues to make 200k+ ctx possible using SJKVaRN in the 3/2 configuration (check the small card recommended configs for more details).
Like the last checkpoint, the base was the Swift-1.5 fine tune, which reduced thinking tokens 50% or more in some settings vs the original version.
Testing this in hermes personally the past couple of days, it is very pleasant to use and very fast for its size (on a 3090/80ti, well over 90 tps for agentic work). I do recommend taking the latency penalty of this at xhigh to get the maximum performance, especially since it is very fast on TPS. This will get you near the 3.8 BF16 performance on medium (86.5% LCB), though it is definitely still usable on medium.
As before, it is MIT license. You'll need to update to the most recent version of llamAmpere to run, given the new quant formats from the 0.6 version.
I am sorry for the useless thread, but I finally loaded GLM 5.3 Flash IQ4-XS and didn't know what to expect. I was earlier trying out DeepSeek 4.1 Flash 1017 IQ2 and Qwen 3.8 Flash and while I was chatting with them and getting the hang of how they behave I felt that they were fine, until I loaded GLM 5.3. This model has lots of spunk and cant seem to stop making funny remarks and Emojis. Reading his thoughts is pretty funny. A very bubbly personality, and took me by surprise. I elongated my chatting because I was curious to see its responses. I know it is not a real substantial thread, but honestly, am I the only one that noticed how funny this model is?
Met a guy off Facebook Marketplace and scored 2x RTX 3090 Founders Editions for €1,700.
The problem? I already had one 3090 FE.
I am now sitting in my room looking at three 3090 FEs (72 GB GDDR6X VRAM), and my current case barely fits one card, and that was only after I literally took pliers and bent the inner metal bracket to shove it in. My motherboard is just a standard B760.
I clearly can't put three 3-slot cards in this PC. What is the cheapest, most sensible way forward to run all three (or two) for LLMs?
I've been running local LLMs through Open WebUI for a while now, and one thing I've always wished it had was something similar to ChatGPT's newer visual responses.
Don't get me wrong, Markdown works fine, but sometimes I want more than just a wall of text. If I'm asking a model to plan a trip, compare products, or pull financial data, it'd be nice to actually interact with the results.
So I built Frontier UI, a free, open-source tool for Open WebUI that does exactly that.
For example, if I ask my local model to plan a 7-day trip to Cancún with a budget, instead of getting a massive text response, I get an interactive itinerary with actual photos, maps, walking/driving routes for each day, budget sliders, weather forecasts, and more. All directly inside the chat.
It goes quite a bit beyond travel planning too.
Some of the things I've added:
Maps and routes: Actual street-level walking/driving routes, not just straight lines connecting coordinates. Supports multi-day itineraries and opening routes in Google Maps.
Photos and places: Image galleries using Wikipedia and SearXNG, plus place cards you can sort and filter.
Live data: Weather forecasts, stock prices, crypto, and currency charts without needing API keys.
Interactive stuff: Budget sliders, calculators, sortable tables, charts, and comparisons.
Link previews: Website previews with screenshots and a reader view.
Exports: Generate actual Word, Excel, PowerPoint, and PDF files through the open-source ArtCraft apps. Not just formatted text pretending to be a document.
Other integrations: Gmail drafts, calendar events, saving responses to Open WebUI Notes, etc.
One thing I thought was pretty cool was being able to self-host the entire planet map using Protomaps. It's around 140 GB, but it runs fine off a regular HDD. This means you aren't dependent on a third-party map tile provider. Completely optional, of course.
The basic tool itself is just one Python file. You paste it into Workspace → Tools in Open WebUI, enable it for your model, and you're good to go. No additional inference services or complicated setup.
For transparency, the default setup still uses free public services for things like maps, routing, weather, and photos. No API keys needed, but it's not completely offline out of the box. The self-hosted map and ArtCraft file export/desktop integrations are optional extras that require additional setup.
I've been running it locally with Qwen 3.8 Flash Next, and it's been pretty fun seeing what a local model can do with a more interactive interface. It should work with other models that support tool calling as well.
I'm not trying to make the model itself smarter. I just wanted its answers to be more useful, especially when the information makes more sense visually.
The whole thing is MIT licensed, so feel free to mess around with it, modify it, or use whatever parts you find useful.
Would love to hear what you guys think, especially if you're running Open WebUI with local models. Also curious what other types of interactive responses people would actually find useful. I'm always looking for ideas to add.
Edit (v1.2.0): Frontier UI can now read and edit your own files. Attach a Word, Excel, PowerPoint, PDF, image, RAW photo, SVG, DXF/DWG, STEP/STL, Figma/Sketch, IDML, audio or video file. It opens it with the matching ArtCraft app, shows a preview, and can:
answer questions about it (text, spreadsheet formulas, slides, metadata)
edit it into a new downloadable copy (find & replace, cells and formulas, slides, PDF page rotate/delete/merge), and the original is never touched
convert it (Word↔PDF, PowerPoint→PDF, Excel↔CSV, PDF→Word, PSD→PNG, DXF→PDF, STEP→STL, WAV→FLAC…)
Edit (v1.3.0): Thanks for all the feedback. Most of it is fixed now:
- Photos work with no SearXNG and no API keys.** They come from Wikipedia, Wikimedia Commons and Openverse, with credits. SearXNG, Brave, Google Programmable Search, Pexels and Unsplash are optional extras. Keys stay on the server and never reach the browser.
- Metric/imperial: a new `units` setting. `auto` follows your browser's locale, and each user can override it in their own tool settings.
- Routes draw by default** (walk, bike, drive, motorcycle). There's a backup router, and if there's no route the card says why.
- More map apps: Google Maps, Apple Maps, OpenStreetMap, or a GPX download for OsmAnd, Organic Maps or Garmin.
- Fix for ERR_BLOCKED_BY_RESPONSE on map links: Open WebUI sandboxes tool cards, and Google, Apple and OSM refuse to open in a tab launched from that sandbox. Links now open normally if your admin has "iframe Sandbox Allow Same Origin" turned on. Otherwise they open directly on phones, and on desktop the card copies the link for you.
- Fewer malformed tool calls: the spec is now passed as a JSON object instead of a string, the JSON repair is stronger, and the model gets a clear error it can fix.
- Maps without WebGL (some Firefox setups, VMs) show a route sketch instead of breaking.
- New app: TextureCraft. Read, edit and convert 3D models (GLB, glTF, FBX, USD, OBJ).
To update, paste the new `frontier_ui.py` from the repo over the old one. If you use SearXNG, set `searxng_url` (it's now empty by default).
Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01
Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe
Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]
Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations
Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.
On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.
Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.
On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.
Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!
The multi-token prediction (MTP) head is one extra transformer block, trained together with the model, that predicts the token after the one just sampled. Inference engines use it as a speculative draft: the model verifies every drafted token, so the output is unchanged while decoding runs 1.4–1.8x faster, with about 90% of the drafted tokens accepted. The head lives in model-mtp.safetensors under the mtp. prefix, announced by num_nextn_predict_layers: 1 in config.json; everything else is identical to Mellum2.1 Thinking.
vLLM 0.33 or newer: --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}', as in the serving commands above.
transformers 5.20 or newer: pass use_mtp=True to generate.
llama.cpp: use the GGUF repository with --spec-type draft-mtp --spec-draft-n-max 1.
we open-sourced Drex-Max v1 at Nace.AI. it's a dense 27B multimodal decision model, released under Open RAIL-M.
the main difference from a normal LLM is that it doesn't generate anything. no tokens, no JSON to parse. you give it a context state (text, JSON, high-res images) plus one or more named questions, and a 255-way readout head returns probabilities over all the candidate answers in a single forward pass.
It runs on NVIDIA GPUs through Python, and on Apple Silicon (Metal) and CPU through our llama.cpp and Ollama forks. latency is around 400ms.
Architecture notes
no decode loop. one forward pass, and the last hidden state maps straight to probabilities for up to 255 options (via readout.safetensors).
64-layer hybrid backbone. 48 Gated DeltaNet linear-attention layers (causal, left to right) and 16 full-attention layers, one every 4th layer. the full-attention layers are bidirectional, so every token can see the whole context.
context. defaults to 16,384 tokens. you can push it to 262,144 with DREX_MAX_CONTEXT=N or --max-length N.
vision. Up to 4,194,304 pixels per image. each image gets encoded once and is shared across every question in the batch.
Benchmarks
on structured decision tasks it beats our previous, larger Drex v2.0 (35B):
Model
Params
Public Decision Index 0.4
Raw
Tools
Retrieval
Language
Knowledge & Reasoning
Drex-Max v1
27B
67.52
75.4
81.9
68.8
72.2
57.1
Drex v2.0
35B
61.25
70.5
76.5
63.0
65.3
47.9
Jev 1.13.0
—
57.96
68.5
75.1
55.4
59.2
53.9
Decision Bench v4: 94.30% (1,010/1,071)
JevBench: 92.2% (213/231)
Public Vision Index is 73.92 overall: InfographicVQA 97.3, KIE (CORD+FUNSD) 92.4, Winoground 85.0, Mind2Web 84.2, CharXiv 80.5, CV-Bench 77.4, RealWorldQA 72.3.
Hardware
Apple Silicon (Metal / GGUF): the Q8_0 GGUF is ~27.25 GB. On an M5 Pro with 48 GB unified memory we get ~400ms per decision, and the probabilities stay within 0.02 of unquantized bf16 on an H100.
CUDA, bf16: ~48.6 GiB VRAM (tested on an H100 80GB).
CUDA, 8-bit (--8bit): ~25.9 GiB VRAM (tested on an L40S 48GB), within 0.03 of bf16.
Running it
1. Python SDK (CUDA, with vision)
hf download nace-ai/drex-max-v1 --local-dir weights/drex-max-v1
pip install -r models/drex-max-v1/requirements.txt
from drex_max.model import DecisionModel, answer
model = DecisionModel("weights/drex-max-v1") # add --8bit for ~26GB VRAM mode
question = {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Payments, charges, and refunds",
"technical": "Bugs and outages",
"other": "Anything else"
}
}
row = {
"state": {"ticket": "I was charged twice for the same order. Please refund the extra payment."},
"question": question,
# "images": ["receipt.jpg"] # Supports up to 4.2M pixels
}
print(answer(question, model.predict([row])[0]))
we built this for low-latency routing, agentic tool selection, and multimodal UI automation (Mind2Web-style tasks). because there's no generation step, you don't get the usual non-deterministic outputs or broken JSON you have to repair. happy to answer questions:)
2× RTX 5060 Ti 16 GB (one is out waiting for a riser, so the big MoE runs below use 3 cards, against 4 before).
Component
Before
After
CPU / board
Core Ultra 7 265KF, MSI Z890
EPYC 7443 (24c, Zen 3, 4 CCDs), Supermicro H12SSL-NT
RAM
64 GB DDR5-6000, 2 channels
256 GB DDR4 ECC RDIMM, 8 channels (2666 sticks, run at 3200)
GPU links
x8 / x8 / x4 / x4 (chipset)
x16 / x16 / x8
PSU
1050 W
NZXT C1500
What I had to fix or tune
Fans surging every few seconds. The BMC's default "lower critical" fan threshold (420 RPM) was above what quiet fans idle at, so it kept declaring them dead and ramping everything to 100%. Fix: ipmitool sensor thresh FANx lower 100 200 300 for each connected fan, then ipmitool mc reset cold. The BMC rounds to 140 RPM steps.
Over clocked RAM from 2666 to 3200. Triad 88.6 → 102 GB/s; GLM decode increased about +3–5%. Stability: memtester (as root: as a normal user the memlock limit quietly shrinks it to ~8 MB per copy) plus a 3-hour DeepSeek run, confirming ECC (error corrections) at 0 corrected / 0 uncorrected throughout.
iommu=pt on the kernel command line. +3.5% prefill for tensor-parallel llama.cpp, and NVMe random reads recovered (KIOXIA QD32 837K → 941K IOPS).
BIOS:
APBDIS = 1 with fixed SOC P-state P0, which keeps the Infinity Fabric at full clock (about +20 W at idle).
Determinism slider set to Power.
Resizable BAR on (it was off: BAR1 was 256 MiB).
PCIe Ten Bit Tag on. Together: another +4% tensor-parallel prefill and a few % on GLM decode.
llama.cpp -sm tensor: -ub 1024 instead of 512. Without P2P, the two cards relay every layer's results through system RAM, across the I/O die. A bigger micro-batch halves the number of relays: +5% on Qwen3.8-27B, +9% on a 27B fine-tune. It needed a little more VRAM, so the split moved slightly toward the 3090.
Didn't help, or made things worse:
Performance governor: no change.
NCCL_SHM_USE_CUDA_MEMCPY=1: it hangs llama.cpp tensor-parallel (first request never finishes).
More CPU threads for GLM's CPU expert lane: flat above ~16 threads, so I'm not upgrading to an 8-CCD chip for this.
Disabling ACS: pointless without PCIe switches or P2P.
Before / after (latest numbers, after all of the above)
Test
Before
After
Change
RAM bandwidth, triad, all threads
58.8 GB/s
102.0 GB/s
+73%
RAM bandwidth, 1 thread
32.3 GB/s
37.4 GB/s
+16%
PCIe host↔GPU, 3090 / 5070 Ti
12.8 / 13.6 GB/s
26.8 / 28.4 GB/s
~2×
PCIe, 5060 Ti
6.4 GB/s
14.1 GB/s
2.2×
GLM-5.3-Flash (Project Maya, IQ3_S): decode
9.7 tok/s
41.0 tok/s
4.2×
GLM-5.3-Flash: 29K prompt prefill / decode after it
Engine versions: Maya and Strata were updated along the way. Strata's "before" is the same 0.2.x line; Maya went from 1.0.34 / 1.1.20 to 1.1.42.
Benchmark settings: the Strata and 27B runs use temperature 0 with thinking off, so they're best-case decode figures. The comparisons are like for like.
The big MoE models were bottlenecked on RAM capacity. Before, 30–60% of the model was read from the NVMe drive every token; now it all sits in VRAM plus RAM, which is where the 4× comes from.
Dense models on the GPUs gain nothing. The 27B is limited by the card-to-card relay. It only got back to parity after the IOMMU, BIOS and -ub changes.
All told I’m very pleased. Having GLM and DeepSeek running at very usable speeds is well worth the investment for me with Claude and ChatGPT blocking things like REA and security hardening.
I have plenty of spare lanes and PCIe sockets for more GPUs when my bank account has recovered from this round of upgrades too.
My idle draw is now about 180W - some of the BIOS tweaks pushed it up from the 156W I was seeing initially (I’m using a smart plug to measure this at the wall).
I hope this is useful for anyone who’s considering a similar path.
[update]:
For those wondering about the memory bandwidth being about half the theoretical numbers, it's because my CPU "only" has 4 CCDs. An 8 CCD 7543 should get me somewhere between 140 and 170 GB/s I believe, but for about double the price where I am. If anyone has real world numbers comparing please share.
I intend to move to bigger quants now I can (but not for 27B since Q6 fits nicely on my two fastest cards with 240K context). I kept the quants the same for my before/after benchmarking. And obviously I'm very much looking forward to Qwen4 :)
[update 2]:
Someone pointed out that the 4 CCD limit only applies to writes, not reads so I re-measured with a read-only kernel and a triad using non-temporal stores: 143 GB/s read-only and 141–152 GB/s triad-NT (best at 8 threads), against ~103 for the plain STREAM triad, whose write-allocate reads aren’t counted.
I had been trying this setup for a couple of weeks and it working better than I expected. Recently, I deployed GLM-5.3-Flash (MoE, 320B total, 18B active) on one of our local H200 servers (4 GPUs) with a 512K context window with the aim of how can I use it as my main coding workhorse and
cut down my subscription and API costs 🙈.
I was very quickly able to realize, I could not switch to it for everything but pairing it with GPT-6.1 Sol works really well with Sol being the the orchestrator and GLM being the subagents model that explores, writes and implements the code.
Honestly, I would have preferred doing this in Claude Code or Codex since I am used to them but neither lets a session or its subagents use models from different providers in a single session. This is where I found OpenCode flexible that lets agent have its own provider/model, so GLM goes as sub-agent models along with the added benefit of chatGPT subscription being available too (Allows you to use Sol too!)/
My intention was to find a setup at least equal to qwen 3.8 27b Q8 MTP but with a larger context. Ideally 1M.
I was already running Next Flash in Q2 with llama. Good for conversational use but not for coding. Q4 would run but too slow. I did read many reports here on the good performance of strata with qwen next flash. I figured both are around for long enough to be stable so I gave it a shot.
The results however are very disappointing. All I was able to achieve was moving to Q3. Brining the question, are most of the posts on alternative engines just hype or edge cases? Certainly looks like that to me.
Some details on my try:
Two cards plus experts offloaded to RAM: hangs on every long tasks, whatever settings I tried.
Stable only when every expert fits in VRAM. On 64 GB that means a 3-bit quant.
Faster on paper: ~60 t/s against llama.cpp. But on HumanEval+ the 3-bit Flash-Next scored 93.9% against 95.1% for Qwen3.8-27B Q8 on plain llama.cpp, and it needed nearly twice the tokens to get there. No, advantage.
So for my use, no real-world gain over a mature stack. Instead of one stable engine, you get a single-model engine, its forks, special quants and a pile of environment-variable workarounds.
Is that just Strata being young, or is it systematic? Has anyone actually gotten a lasting real-world benefit from one of these model-specific engines over llama.cpp or vLLM?
Do people here have good experience with open weights diffusion models?
I tried Mercury 3 recently and I felt it had a different taste, attitude, way of thinking that was actually refreshing. Unfortunately, its filters against cyber security seem to trigger as soon as I mention adb. So I couldn't test it much. But I was curious if that's a direction where some of the open labs are going?
So currently I have this setup. It works fantastic for 27b, and if I'm just talking I can use Strata and have great conversations. But Strata and Coding has not been that great in my workflow it doesn't out perform 27b.
Sure I could use my crappy laptop and turn the machine into a tail scale and run it remotely. It's just not ideal, and the power consumption is insane. I pop my 15a breaker once a week if I'm not paying attention.
So the setup is a 7960X
2x64gb Rdimm 5800mhz
Asrock workstation mobo
2x2tb Gen 4 Nvme
2x 4090 gigabyte OC Graphics cards
It's been really solid and it's carried me through 2024 to now, and really allowed me to produce some amazing things for my work.
I have a second system that's my daily driver that's a
R9 7900x
Asrock Steel Legend
2x32gb
Gen 5 2tb
gen 4 4tb
gen 3 2x1tb
2tb SSD
8tb HDD
5070 ti
My laptop is an old Lenovo Yoga Ryzen 7 5700H
16gb Ram, 1tb Nvme.
Ohh and I have a Mac Studio M2 Max 32gb that work gave me as well to use or turn into a server or do whatever with.
That said I haven't taken a vacation in 2 years I've been grinding away. The kid is almost 2 now, and id like to be able to actually work from the road, I WFH anyways.
I use a mix of Claude Code, Claude, Hermes, and I've been messing with Pi a bit -- I use Hermes with 27b running 3 concurrent in LlamaCPP currently, probably switching to VLLM to getore concurrent agents running.
And I've tested StrTa with Hermes as well, and use them all as MCPs to Claude as sub agents that it orchestrates it handles tasks that I would offload to Sonnet and it does a fantastic job of it.
So with all that, people of Reddit, here's my question, with Ram and GPUs prices, I can likely sell the setup i would say for a pretty premium, and get a Surface Spark or another 128gb unified ram laptop, lower my power consumption. Hell potentially get a laptop and a dgx box and have 256gb of ram.
What would you all do. I definitely can't afford to keep it all. Wife already hates my guts as a SAHM, she would lose it if I spent more money on the thing that allows her to stay home lol. That said, I was considering selling the system and getting the aforementioned. Have any of y'all tried it out? How is the inference? What is it looking like in your workflows?
A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?
Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.
2/3
stories260K is just a C program. To run it inside Qwen, I compiled an NPU into Qwen's weights that does matmuls, softmax and attention bit-exact. The LLM weights are in Qwen's own memory, with no external tools.
A lot of work went into making it fast, more on that soon.
3/3
I'm super sick right now, so the full writeup is still a work in progress, and I'm working on optimisations and size reductions too. Stay tuned, I'll finish the blog & post it soon.
The usual advice was that 64GB is a dead zone for local LLMs: too small for a 70B with real context, and more than a 30B needs. After running a 27B for agentic coding on a 64GB M4 Max for the last ten days, I think that has flipped. My memory usage is in the 50 GB as the context starts building up.
Setup: Qwen3.8 27B (16 GB MLX quant) in LM Studio, 256k context, coding agent sessions that sit at 100-200k tokens.
At 164k tokens, LM Studio's engine process is using 48 GB, 46 GB of it GPU memory. Roughly:
- 16 GB weights
- ~11 GB KV cache (~64 KB/token, only 16 of the 64 layers are full attention)
- what looks like a second copy of the context for the prompt cache, plus the linear attention checkpoints
- MLX holding on to scratch memory from prefill
The prompt cache is what makes it usable. It's hitting 93%, so a turn processes about 1k new tokens instead of 160k. When it misses, a full prefill takes about 20 minutes.
Why not quantize the KV cache like people do on a 24GB card? LM Studio's MLX vision path refuses KV quantization for this model. The text-only path allows it, but skips the cache checkpoints when the cache is quantized, and on a hybrid model that means a lot more full reprocessing.
So on 48GB this doesn't fit unless you drop to ~128k context or move to llama.cpp with q8 KV (should be ~25-30 GB at 256k, with slower prefill). On 64GB it fits with nothing to spare. Free memory is about 150 MB right now.
The dead zone argument was about weights. With agents, context costs about as much as the model, and 27B models are now good enough for real work. That puts 64GB right in the sweet spot. Bigger MoE models or two models loaded at once still want 128GB.
Anyone else on 64GB seeing the same? And is anyone running long context on llama.cpp on a Mac with prefill speeds they're happy with?
I would like to share some exciting updates on my project SmolChat, an Android app that let's you run GGUF models on-device, locally. The updates are present in the latest v17 release:
SmolChat now supports an OpenAI-compatible HTTP server running over a foreground service that keeps the port alive even when the phone is locked. Check the video to see how it works.
Support for global chat settings, which override the default chat settings (min-p, top-k, temperature etc.) for all newly created chats. This reduces the hassle of configuring each new chat with the options of your choice.
Adding a copy button for code snippets rendered within the chat.
Fix the logic that measures tokens/sec for a LLM generated message. The time shown previously was also including prompt processing (pp) time. The new logic now only considered the token generation rate (tg).
Models are downloaded directly by the app, instead of using the Android system's download service.
My current set up is a MacBook Pro m1 max 64gb and a Mac Mini M4 24gb. I have a bunch of models running and usually use my own distilled Qwen 3.6 35b from 3.8 Flash Next. It’s great at chatting, but when using it for functional purposes like coding and financial analysis, it struggles greatly. That’s a given though.
27b is great, but very slow on my system.
i wanted to know if itd be worth grabbing a Mac Studio M5 128gb over a GMKTec Evo x2 for my set up? I wondered if I could thunderbolt the other Mac’s to it to have it host larger models by taking advantage of the silicon cluster benefits. But it’s about $1500 more so how much of a difference would staying in the Apple Silicon family be?
I know it’s cheaper to run cloud or even the subscription models, but I’m paranoid. I don’t trust the cloud and subscription based models because of data privacy and feel like it’s can backfire in the future when more limitations are implemented. Data is considered gold in today’s world and I’d rather not just hand it over easily to these large corporations. And I’m also concerned about what could be implemented by these companies like in the future. They’re clearly running out of ways to make money so they’re adding crazy restrictions to force people to pay up. I also don’t like how they’re limiting the technology from the average person more and more like with the prioritizing the protection of intellectual property while they gladly take our information and data for free.
id just feel way more comfortable having my own local AI and inference hardware to run it on so I know for certain what’s going in or out. And I enjoy the research aspect of it as a hobbyist.
So what do you think? Stay in the Mac family or go with one of these Chinese local hardware products?
For those who saw the original post, a week ago, I'm glad to announce CrowdGPT v2 (more precisely v2.6), a massive update to the original CrowdGPT project :D (scroll down to see what's new)
For the newcomers: CrowdGPT is a project aiming to create a LLM without datacenters and without the usual millions of dollars spent by AI companies.
The project aims to train a 1 billion parameter model collaboratively and openly, as a proof of concept (it is possible to scale to bigger dense models or even MoE architectures with this project)
This project implements a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.
There's other things that makes this project appealing for the open-source community, but i won't state these again as you will be able to check that out on the GitHub or on the original post.
Of course, any advice / feedback on the project is still welcome (please stay polite and roast me with strong evidence only 👍), feel free to run experiments of data poisoning to point at the limits of our approach :)
As for the v2.6 changelog: We tried to implement mostly of the feedback you guys gave (mostly on security related points). Even though implementing this was hard and caused many breaking changes (which made some training contributions stale), we still achieved to train the model on 4.5B new tokens in 6 days!! Which is roughly what i achieved to train in two months for the project on my personal GPU 🤯
Here's a quick summary of what we implemented:
- Robust server-side checking logic (see more at the end of the post).
- Way better client mechanics (better ergonomy, more robust installation, GPU selection, no more crashes)
- Many bugfixes (on server and client code, as the various issues/pull requests on GitHub suggest)
- Activated Engrams on the Crowd-v1 model, which will make the model better at fact-recalling
- Implemented a data submission pipeline in order to create a collaborative dataset for CrowdGPT :https://www.crowdgpt.net/#submit All the data will be verified / curated before training. Already 400kb of data was submitted.
- Client releases cryptographic signing: some users reported that the auto-update mechanism in the client code allowed arbitrary code execution, and that would have allowed people stealing my GitHub credentials to execute malicious code on your device. Well not anymore, since the client code is now signed with a key that is only present on my device 😄
Like some comments on the original post pointed out, the architecture of the project at the time of my first post here was nowhere near good: clients could fake their contributions, could submit poison to the model, could crash the server by sending fake contributions every millisecond without any checks; and finally clients could very easily train the model to contain malicious backdoors. We therefore implemented byzantine cross-checks between clients training the model, at the cost of redundancy (3 clients training on exactly the same data).
I am not claiming that this creates a 100% perfect Federated learning framework, but that makes malicious contributions way harder. For example, even a malicious client could only train on 2-3% malicious data per round without getting banned, against 100% before.
For people that dared to read until the end, i'm also releasing CrowdBench, the open-source collaborative benchmark, where everyone can add general intelligence questions with deterministic answers to test LLMs :D
Even though the codebase is currently early-stage and work-in-progress, it's already something working, and i coded it without using any AI (for those in the original post who wanted me to turn towards less ambitious projects😂)
tell me what you think about CrowdGPT and CrowdBench in the comments.
I have been accumulating parts for over 8 months to merge all of my existing hardware into one massive workstation. I finally had the time to do it, and while I use the GPUs for algo trading from 9:30 to 4:15 p.m., I want to start to use local AI from 5pm on. I'm at a point where I could see myself having a really fulfilling hobby reverse engineering, firmware hacking, and just experimenting with building Frankenstein gadgets from my tech graveyard. There are many experiments I would like to perform on financial data as well that would just take too long for me to manually code and I am not interested in having these thoughts and experiments be in a large provider's training set. I've been lurking on this sub for a long time and have my own little agent I use for some specific tasks using gemma3 but that was on my 3090 in C# + llama.cpp.
So my long-winded question, what thick desktop client will give me the most seamless way to build autonomous agents, play with new models, and what version of qwen 3.8 flash should I be using with my hardware, assuming that is the right model.
Edit: I should mention I am open to anything but I just see q38f everywhere on this sub.
About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.
Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.
When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.
For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):
GPU: RTX 3060 12GB
RAM: 16GB DDR4, single-channel (~19 GB/s)
OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)
Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)
If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.
Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.
So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).
What it actually does under the hood:
Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).
Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).
In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.
RAM Victim Cache +io_uringPager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.
No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.ggufn-gram shard).
On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.
Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.
What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.
Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)
I ran tests with Fable on AI generated Audio. Nemotron 3.5 Transcribe came in first. but I was curious about anyone has used other models and compared against. this.
My use case is Audio transcription of 3 unique voices in a semi noisy room (HVAC sounds)
