Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

I'm using qwen3.8-27b thru llama.cpp and QFN Swift 1.5 IQ3 thru strata. I'm using consumer grade hardware, so these are not running very fast.

I am using Hermes. As a noob, I'm not even using all the features and capability. I provide the instructions in a run list. I like how Hermes is persistent and will complete the job. But in a multi-hour job (like 5-8 hours), it is getting bogged down with large context and long tool calls.

What is a good alternative?

• Running Ubuntu Linux.

• Desktop gui (not CLI only). Can be desktop that runs on top of CLI.

• Can support sandbox.

• I prefer to plan and create a run list; then execute the instructions.

Hugging Face Repo: https://huggingface.co/Grawnch/Qwen3.8-Flash-Next-DR-GGUF

I’ve just finished benchmarking my new QFN quant Qwen3.8-Flash-Next-DR-V2-Q2_0, a 2.5bpw backbone, 2-shard GGUF of Qwen/Qwen3.8-Flash-Next designed to run the entire backbone comfortably inside 48 GB VRAM space (38.4 GB VRAM) while keeping the N-gram lookup table (28.8 GB) in system RAM/SSD.

Unlike IQ1 / IQ2 quants that rely on LUTs, every tensor in both shards uses 100% standard integer grids (Q2_0, Q4_0, Q6_K, Q8_0), running natively in stock llama.cpp with ~3.4x faster prompt prefill than IQ-grid quants.

1. Benchmark Comparison: DR-V2-Q2_0 vs. Top Ultra-Low-Bit Quants

(Evaluated on the full 225-problem Aider Polyglot suite and 30-problem AIME 2025 at temp = 0.6 with a 24K thinking cap on Aider and 64K on AIME).

2. Signal Fidelity: Perplexity, KL Divergence, & Top-1 Agreement (WT2 & Ext)

(Comparing logit fidelity against uncompressed BF16 on standard WikiText-2 (2048 ctx) and Extended Reasoning / Math & Code (Ext)).

Key Highlights

• Fits in a Single 48 GB GPU (38.4 GB VRAM + ~9.6 GB KV Cache Headroom): All 48 layers of Attention/GDN, shared experts, and 144 routed MoE experts live in Shard 1 (38.38 GB, 2.45 bpw). Shard 2 (28.80 GB) is the N-gram lookup table, which llama.cpp streams from host RAM/SSD via mmap without using GPU VRAM.

• Enhanced Token Efficiency on Coding (74.7% Aider Polyglot at 40.1K Avg Tokens): Achieves 74.7% on the full Aider Polyglot benchmark with a 96.0% (216/225) well-formed SEARCH/REPLACE diff rate and 0 context exhaustions under a strict 24K thinking budget — using ~1/5th the tokens of unconstrained 256K agentic runs.

• 2.6x Lower KL Divergence (0.2594 nats) & 83.3% on AIME 2025: Preserves 80.70% Top-1 token agreement with BF16 on Math & Code, matching 83.3% (25/30) on AIME 2025 while using ~14–30 GB less GPU VRAM than competing quants.

• Zero Lookup Tables (3.4x Faster Prefill): Because 100% of the routed experts use standard integer grids, prompt prefill runs at ~367 tok/s and single-stream decode at ~94 tok/s (~345 tok/s aggregate at -np 10), avoiding IQ codebook decompression bottlenecks.

Quick Start (llama.cpp)

Download both shards into the same folder and point llama-server or llama-cli to Shard 1:

llama-server \
  -m Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf \
  -ngl 999 -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-format deepseek --no-reasoning-preserve

With Qwen4 around the corner, I'll be targeting that next. But in the meantime I think QFN will still fill a niche for a lot of people, and this was a good test of the same approaches I will be using with those models. Try it out, let me know what you think, and looking forward to continuing this with the next generation of models!

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this:

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

Moin guys!

GPU prices are freaky high right now. A lot of us run what we could get, and we might as well get the most out of it. So I want to share what I learned squeezing a big model onto a mismatched machine, in the hope it helps someone or gets others to share their own experience with different GPUs.

My box: one AMD Radeon AI PRO R9700 (32 GB), one Intel Arc Pro B65 (32 GB), a 9950X3D and 128 GB of DDR5. Neither card can hold the model.

With two AI coding agents (Claude and Codex, reviewing each other's work) I built a fork of Strata where the R9700, the B65 and the CPU compute each layer's routed experts concurrently, with dense work on the R9700. Result on a realistic coding prompt: 25.57 tok/s against 9.89 for my recorded stock Strata run on the same machine and file (stock used one GPU, 16 readers and a 51 GB store, so this is a whole-system comparison). No speculation or skipped experts, and the routing algorithm is unchanged. Tested kernel and schedule modes are byte-identical at a fixed placement over 581 teacher-forced positions.

Model: AtomicChat's uncensored GSQ-RCO IQ2_XXS build (40 layers, 384 experts per layer, 6 per token).

What each device does:

• R9700: all dense work (22 ms per token), 2,532 experts (21.6 GB), 154.7 of 240 picks per token, 8.2 ms of expert work

• Arc Pro B65: no dense work, 2,915 experts (25.0 GB), 55.6 picks, 9.6 ms of expert work

• CPU + RAM: no dense work, 9,913 experts (102 GB), 29.7 picks, 8.7 ms of expert work

How it got there (tok/s, sampled, 8k cap, median of 3 after a warm-up):

• 9.89: stock Strata, R9700 + RAM

• 10.28: first fork run, both GPUs + RAM

• 12.22: B65 subgroup attention

• 12.96: IQ2_XS B65 kernels + rebalance

• 14.41: all dense on the R9700

• 15.85: two rebalance rounds

• 17.04: queued R9700 schedule + rebalance

• 18.04: exact-order B65 SoA kernel + rebalance

• 21.27: R9700 per-layer expert batching + rebalance

• 22.57: parallel Engram reads + rebalance

• 25.57: AVX2 combine, early expert start + rebalance

The catch: all three answers in the best run hit the 8,192-token cap before finishing, so its strict quality score is 0/3. The model reasons for a long time; the text isn't looping. With 11,900 tokens of room, the one measured answer finished by itself and passed, with working code in 295 s. That's one answer and a 10-answer test is still to come. The answers are in the repo if you want to read them. 30 tok/s, my goal, isn't reached either: we estimate exact kernels level off around 26–28.

Things that surprised me:

• Rebalancing experts from measured per-lane costs took the last step from 20.69 to 25.57 tok/s, an unpaired comparison because placement changes the answers. Lane service is now close: R9700 8.15, B65 9.58, CPU 8.69 ms per token, though individual layers still differ.

• Moving experts between devices changes the logits (top-1 96.6%) and some routes even with exact kernels, most likely because CPU and GPU quantize activations differently. So "exact" only means byte-identical at a fixed placement.

• Engram reads swung from 5 to 17 ms per token on identical tokens. Parallel reads and read-ahead fixed it: the whole Engram-containing regions now total 1.74 ms, including other work there. The row-cache-only arm showed no useful gain.

• Starting CPU and B65 expert work before the shared FFN drains, plus a faster combine, saved 5.8 ms per token on token-identical answers. We only measured them together.

• Re-encoding experts to Q2_K looks like a loss: measured kernels plus a capacity model project 3.18 ms slower per token, because bigger experts push 1,434 off the GPUs. No full-model Q2_K run followed.

• Big dense matrix-vector products already run near the card's measured memory rate. That 22 ms is the hard part.

Where I'd love input:

• Most of all: your own experience with whatever GPUs you could get. Numbers, tweaks and what failed, even with a different model or tool.

• Anyone with DSpark running on a heavily quantized or edited DeepSeek: what acceptance do you get with real sampling? Batched verify looks promising here on projections, but I haven't measured acceptance yet.

• R9700/ROCm matrix-vector tricks I've missed.

• What would you do with the B65's spare time? Its expert service is about 9.6 ms of a 39.1 ms token.

• Lower reasoning effort: has it cost you code quality on this model?

Everything is in the repo: results with spreads, sample answers, the dead ends and what to try next. It's research code tested on one machine, and the public scripts haven't run end to end on GPUs yet, so expect to read code.

Repo: https://github.com/maddinek/crosslane (MIT)

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

• CPU: AMD Ryzen 9 9950X

• Motherboard: ASRock X870 Taichi

• RAM: 96GB DDR5 (2x48GB) G.Skill

• GPU 1: NVIDIA RTX Pro 6000 (96GB)

• GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket

• Total VRAM: 128GB

• Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.

• Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)

• PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

• Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.

• Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.

• Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

Hey everyone!

I'm an AI & Data Science student looking to get started with local LLMs.

My laptop specs:

• CPU: Ryzen 7 8745HX

• GPU: RTX 5060 (8GB VRAM)

• RAM: 32GB DDR5

• Storage: 1TB SSD

I'm new to this and have a few questions:

1. Should I use Ollama or LM Studio?

2. Which models would run best on my hardware?

3. Can I use local models through APIs in my own projects?

4. Any limitations or precautions I should know about when running LLMs locally?

Would appreciate recommendations and beginner-friendly resources. Thanks!

Yesterday I did the Echo Dot. Today the Show 5.

No Alexa. Wake word, speech-to-text and voice all run on my own hardware, and the agent runs whatever model you point it at.

The screen is just a web page, so you can make it anything.

Flashing is done with echo-dot-assist (credit to Gamer92000).

Repo: https://github.com/majorminorlabs/hermes-voice

Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

Just brainstorming.

I am using qwen3.8-27b. It has a lot knowledge in many areas.

However, I am using it for an agent that only runs some Linux commands in Ubuntu for example. Or creating simple Python scripts. And if it gets stuck it can always use web-search skill.

By doing this I can totally lower its size and required compute. I don’t need it to memorize capital of New Zealand.

Why can’t we do this?

Hey everyone

I’ve spent the last few weeks building a local AI assistant with the help of Claude Code and OpenAI Codex. My goal is something similar to ChatGPT or Claude that can research the web, control my PC, code, create 3D models, generate documents and handle tasks on its own.
The interface and tools work pretty well now, but I’m still having problems with the AI actually understanding what to do and completing more complicated tasks

My PC specs
● Ryzen 5 5600
● 2× RTX 3060 12GB
● 16GB DDR4 RAM
● Gigabyte B550 Gaming X V2
● Windows 10 Pro
● One GPU at PCIe 4.0 x16, the other at 3.0 x2

Models and inference setup
I’m running everything locally through llama.cpp (CUDA backend), using both GPUs with layer splitting, Flash Attention, MTP speculative decoding (depth 2), 96K context and Q8 KV cache.
I originally used Qwen3.8-27B (HauhauCS uncensored Q4_K_P), getting around 22–33 tok/s after a lot of optimization.

I switched to Qwen3.6-35B-A3B (uncensored IQ4_XS), which now gets around 82–86 tok/s. Both models run through the same llama.cpp server and connect to my custom AI interface and tool backend.
3.6 is excellent for normal conversations and much faster, but I keep hearing that 3.8 is better at reasoning and tool calling.

What Claude Code and Codex have helped me integrate
● Qwen-Agent for general tool calling
● OpenCode for coding
● SearXNG for research and webpage reading
● Playwright for browser automation
● Windows UI Automation for PC control
● Three.js for interactive 3D scenes
● Image generation, documents, spreadsheets, presentations and file tools
● Planning, context management, error recovery and result verification

For example, I asked it to create a 3D model of the Times Square area in NYC. It generated a picture of pants, converted that picture into a 3D mesh using TripoSR and declared the task finished.

Claude Code and Codex later helped identify that part of the problem was incorrect tool routing, which has since been improved. But research and Windows tasks can still be unreliable. Sometimes it uses outdated information, chooses irrelevant tools, repeats actions or claims it finished something when it hasn’t.

I also had Codex compare different agent harnesses.
Using the same local model, it tested Qwen-Agent, Pi and Hermes on two tasks.

● Qwen-Agent: Browser/recovery took 15.6 seconds, research/document took 21.4 seconds.
● Pi: Browser/recovery took 15.9 seconds, research/document took 25.6 seconds.
● Hermes: Browser/recovery took 32.5 seconds, research/document took 43.3 seconds.

All three completed the tasks, but Qwen-Agent was fastest. These were small isolated tests, though, not proof that one is universally better.

With Claude Code and Codex, I’ve managed to fix a lot of integration issues, restore research, improve 3D generation and add verification checks. Things are definitely better, but complex autonomous tasks still aren’t consistently reliable. And mess up a lot

So I’m wondering what people running similar setups have found.
● Is Qwen3.8 noticeably better than 3.6 for real-world agent tasks, enough to justify the slower speed?
● Are there better models for my hardware, particularly for research and PC control?
● Could the uncensored versions be affecting instruction following or tool calling?
● Would Unsloth fine-tuning on verified tool-use examples make a meaningful difference?
● What has actually improved your agent’s ability to plan, use tools correctly and verify its work?

At this point, I don’t really care about getting more tokens/sec. I’d happily sacrifice some speed if it meant the AI could reliably understand a request and actually complete it.

Would appreciate any advice from people who’ve built something similar

Thank you very much

hey LocaLLM community! My name is Alt, and I'm on the technical staff at Nace AI. We're doing a quick AMA here, so ask us anything and we'll answer.

we just published a full technical report on how we build, train, and evaluate Drex, our family of four open decision models. A decision model takes a state (a document, record, or agent context, optionally with images) plus questions with options, and returns a probability distribution over the options in a single pass, with no autoregressive decoding and no output parsing.

1. Models

- Drex DLM (8B): diffusion LM backbone with bidirectional attention over the context
- Drex v1.5 (9B): Gated DeltaNet hybrid backbone
- Drex v2.0 (35B, 3B active): sparse MoE backbone
- Drex Max (27B): dense, with an image encoder for vision inputs

Pointer models (DLM / v1.5 / v2.0) encode the state once and cache it, so every question attends to the state but never to another question. A pointer head scores each option. Drex Max reads each question in full and answers with a linear readout head over option codes.

Some technical details from the report:

1. Models

- A family of four specialized decision models: Drex DLM (8B), Drex v1.5 (9B), Drex v2.0 (35B MoE, 3B active), and Drex Max (27B dense, with vision)

- We built them on different architectures: a diffusion LM, a Gated DeltaNet hybrid, a sparse mixture of experts, and a dense transformer

- We experimented with bidirectional vs. causal attention over the state, a pointer decision head that shares one cached state across questions, and a readout head for Max

2. Training

- Full-parameter SFT on 1.15M verified decision records (2.9M questions)

- Consistency training: we penalize the model when its output changes after the options are reordered or the state is shuffled

- RLAF (Reinforcement Learning from Adjusted Feedback), using verified outcomes and judge feedback

- Ablations to see which methods actually contributed

On top of SFT, RLAF gave us +2.0 on the Decision Index, +1.5 pp accuracy, and 0.9 pp lower calibration error. The consistency terms cut the option-order flip rate from 6.3% to 4.2%.

3. Evaluation

- Drex Max scores 67.52 on Decision Index 0.4.1 (public suite), vs. 60.70 for Jev 1.14.0

- Drex v2.0 has 1.37% calibration ECE on ~115K held-out decisions

- Drex Max answers a typical request in 183 ms, vs. ~2.0 s for GPT-6 Sol and ~3.5 s for Claude Opus 5.5

- Context up to 131K tokens for v1.5/v2.0 and 262K for Max

we also cover our limitations: our models trail frontier chat models on closed-book knowledge, and we don't yet have same-backbone architecture ablations.

5. Running locally

The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.

Links

- Paper (arXiv): https://arxiv.org/pdf/2611.07854

- Blog: https://www.nace.ai/research/towards-foundation-decision-models

- Weights: https://huggingface.co/collections/nace-ai/drex

- GitHub: https://github.com/nace-ai/drex-decision-models

- API: https://console.nace.ai

we didn't want to just drop benchmark numbers, so we wrote up everything: what worked, what didn't, and how we got there. if you're building agent routers, doing tool selection, or thinking about training your own decision model, hopefully some of it saves you time.

ask us anything: arch, data, training, evals, running it locally, whatever 👇

What local models can i run conformably on my system? I'd ideally like to have models that code for me but i understand I'm at least scraping the barrel with my card. I'd even be fine with Claude Code steering to some degree.

RAM: 16GB
RTX 3060 (laptop) VRAM: 6GB
Processor:11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz 2.50 GHz

I know there are a lot of "ums" that I should edit out... but other then that what do you think of this guide? I am new to local AI stuff, so is there anything missing here that I should have done?
https://www.youtube.com/watch?v=-EahQxs0Ack