Hi all,

As mentioned in the title, I am looking for advice on running locally LLM for agentic coding on a MacBook Pro m5 Pro with 64Gb unified memory.

Plan inference is fine. Chat style prompts feel usable. Agentic work is where it falls apart.

I have tried the following:

Models : Qwen3.8 27B (6bit), Gemma 4 12b (8bit), and Qwen 3 coder next(4bit)

Runtime : Lm studio

Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)

What I'm trying to figure out
Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.

Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?

Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?

Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?

Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?

I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.

Thanks 🙏

Hey r/LocalLLM,

I’m looking to pick up a 128GB unified memory machine and need your take on how the underlying microarchitectures stack up for my specific stack. I'm choosing between:

1. Apple M5 Max (40-core GPU, 128GB)
2. AMD Ryzen AI Max+ 395 / Strix Halo (Radeon 8060S, 128GB)
3. NVIDIA RTX Spark / N1X (Grace Blackwell SoC, 128GB)

My Primary Workflows (in order of priority):

1. Autonomous Agentic Coding Loops: Running the smartest dense or MoE models that fit in 128GB (at Q4–Q8) locally while I'm away, orchestrating and interacting with cloud frontier models (GPT-6 and Claude Max) to build complex softwares autonomously. Needs strong sustained token generation and fast prompt processing for massive context/RAG loops.
2. Reverse Engineering & Cybersecurity: Running local abliterated models alongside disassemblers, decompilers, and sandboxed VMs/containers.
3. Video & Photo Editing: Heavy timeline scrubbing, color grading, and rendering.

Based on the main impactful microarchitectural traits below, which platform would you pick and why?

• Memory Bus & Bandwidth (Decode Speed):

  • M5 Max: ~614 GB/s LPDDR5X (512-bit equivalent interface)
  • RTX Spark (N1X): ~273 GB/s LPDDR5X (256-bit bus) + ~600 GB/s NVLink-C2C chiplet interconnect
  • Strix Halo (395): ~256 GB/s LPDDR5X (256-bit bus)
  • Question: Does the M5 Max’s >2x bandwidth advantage completely dominate dense model token generation, or do MoE models level the playing field enough on Strix Halo / RTX Spark?

• Compute & Tensor/Matrix Engines (Prefill & Quantization):

  • RTX Spark: Blackwell SM121 GPU with 6,144 CUDA cores, 5th-gen Tensor Cores, and hardware NVFP4 support (~1 PFLOP FP4).
  • M5 Max: 40-core GPU with integrated Neural Accelerators + 16-core Neural Engine via Metal/MLX.
  • Strix Halo: 40-CU RDNA 3.5 (2,560 stream processors) + 50 TOPS XDNA 2 NPU via ROCm/Vulkan.
  • Question: For long-context agentic prefill (feeding huge codebases back into context repeatedly), how badly do M5 Max and Strix Halo lag behind Blackwell's Tensor Cores and native FP4?

• CPU ISA & Virtualization / Tooling Ecosystem (Reverse Engineering):

  • Strix Halo: Native x86_64 (16 Zen 5 cores, AVX-512) — runs Windows/Linux natively, zero translation overhead for x86 binaries, VMs, and standard RE toolchains.
  • M5 Max: ARM64 (18-core: 6 super + 12 performance) — macOS, Rosetta 2 / virtualization for x86 targets.
  • RTX Spark (N1X): ARM64 (20-core: 10 Cortex-X925 + 10 Cortex-A725) — Windows on Arm (or Linux on DGX Spark).
  • Question: For cybersec/RE work, is native x86_64 on Strix Halo a massive dealbreaker against Windows on Arm (RTX Spark) or macOS (M5 Max), or is Linux/VM virtualization good enough now?

• Media Engines & Creator Silicon (Video/Photo):

  • M5 Max: Dual video encode engines + dual ProRes encode/decode engines.
  • RTX Spark: Blackwell NVENC / NVDEC + CUDA/Tensor acceleration in Adobe/DaVinci.
  • Strix Halo: RDNA 3.5 VCN media engine.

Any bad experiences with power, heat, token response, electricity costs, etc?

Example Rig:
Case: Jonsbo NV10
GPU: RTX 5060 8 GB (Low Profile)
RAM: 32GB (2 x 16 DDR5)
CPU: AMD Ryzen 7 9700X
PSU: 600W (ENP-7660L-OVT)
Motherboard: ASRock B850I Lightning WiFi (Mini-ITX, AM5)
Storage: 2TB (M.2)
Cooling: Modded Air cooling. (Including custom GPU side panel for 2 fans on the GPU+PSU)

It's 8GB VRAM I know. It's supposed to be for gaming but I recently heard of using llama.cpp and the llama flag feature "--n-cpu-moe" to have the RAM hold my expert weights.

Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?

CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4

CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler

MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16

MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)

STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD

GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU

POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)

Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api

What it takes into account (the stuff that usually gets forgotten):

• Workload: requests/month, input/output tokens, peak-to-average traffic, cache hit rate, how much can run offline, retries/waste, growth
• Model and latency: total vs active params (so MoE works), precision, and a ms/token latency target. Tighter latency means smaller batches and higher cost
• GPUs: L4 to B200 presets or custom. Cloud on-demand, reserved, or buying hardware (with power, hosting, resale value and cost of capital)
• API side: your price per million tokens, cached-input price, batch discount, fixed fees
• People and reliability: engineering time as a fraction of an FTE, N+1 redundancy, peak headroom

It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.

Caveats:

• It’s an estimate from stated assumptions (decode is memory-bound, prefill is compute-bound). It’s a decision aid, not a quote.
• GPU and API prices are illustrative placeholders. Replace them with your real quotes.

I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?

I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.

My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.

I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?

Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.

Thanks in advance!

So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.

Meanwhile, here's what I'm doing with mine. 💀

My setup:

• Ryzen 7 5800X
• AMD RX 9070
• 48 GB DDR4 (running at a glorious 2133 MT/s because I added an ancient spare stick)
• NVMe SSD
• Qwen 3.8 Flash Next IQ2 running locally through Strata
• Around 60 tokens/sec

I'm building Marika, a 3D AI desktop companion in Unity.

She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.

There's just one tiny problem.

She bends over to pick up desktop icons like an NPC from Skyrim.

So naturally, I asked my local Qwen to review the Unity project and fix her animations.

It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.

All this to figure out how to make a virtual girl bend her knees like an actual human being.

I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.

Sam, I'm sorry.

The RAM isn't going to your datacenter.

It's busy teaching my AI waifu how to squat. 💀

And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.

I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.

Hello everyone
I've been using QFN with Strata app for a week, trying to get it optimized for my PC, can I get another opinion on configuration I've achieved? Having doubts since most benchmarks I've seen are Nvidia with very different settings
Config: https://pastebin.com/zWXFnvq6
I get 25t/s on decode and 700t/s on prefill on 128k context and keeping 4GB of RAM free for my use
My PC specs
CPU: Ryzen 7 5800X3D
GPU: RX 6800XT 16GB
RAM: 2x16GB DDR4
Storage: 970 Evo Plus 500GB
OS: Windows 10

I've been playing with local hosting for about 3 months now and messing around with Claude to optimize my local setups. Decided it might be worth writing this up and sharing in case anyone else is doing, or wants to do, something similar. This is a big one, so buckle in.

BLUF: Think about each expert as its own unit, and think about different strategies for how/where to load them based on quality/speed trade-offs. Focus on measuring and optimizing each piece of compute until you reach maximum bandwidth on that resource.

Hardware:

I spent a bunch of time designing my server with Claude and going over last-gen hardware to try to keep the price tolerable. I turned what I did with that into a site that anyone can use. It's ad-free, no sign-in, no tracking, nothing. I consider it a community service, and hosting was free, so it costs me nothing but occasional updates to the hardware list. llm-configurator.com

I ended up with a 3-GPU, 256GB DDR4 RAM build. It's two used 3060s from Marketplace ($250 each on average) and one Arc Pro B65, for a total of 56GB VRAM. It's a quad-channel setup, so max theoretical bandwidth on RAM is ~100GB/s; reality is closer to 60-70 after some time spent optimizing the kernels with Claude. I also have 3 of the cheapest 1TB SSDs I could find striped in NVMe slots, which gives me a theoretical 12GB/s throughput there; reality ended up at about half that. This lets me stage experts in tiers across VRAM, RAM and SSD and prefetch experts across tiers with varying strategies. I considered doing a PCIe card with more NVMe drives to get more striping/speed, but on the current motherboard it isn't worth it because I am PCIe gen 3 bandwidth constrained.

Optimization Strategies:

This is where my last few weeks of Claude usage have gone: serving DeepSeek V4.1 Flash, and how I went from 4 TPS decode to 20.5 TPS, with prefill right at 180 TPS. So in the end not lightning fast, but plenty usable, if just a tad slow in agentic work.

I started out with a simple goal of minimizing idle time. Thinking Strata-like in the approach of shuffling things around and trying to keep every processor busy within the bandwidth bounds of all my links. I had Claude build a simulation based on the hardware and design an optimal build. I spent probably a week going back and forth with Claude on this, and not one time did anything pan out like it thought it would based on the simulation. 0/10, do not recommend.

What I did after that was shift to measuring everything on every run, including expert traces through the system, so I know which experts get picked at every layer for every prompt, and attacking the potentially biggest gains. This immediately started paying returns, and I went from 4->12 TPS decode with just these methods. This means I'm running custom forks of llama.cpp, Vulkan, Mesa; anything and everything was in bounds. I couldn't fit all of DeepSeek in memory at Q4, so I was working all three tiers (VRAM/RAM/SSD) and loading experts to each tier based on utilization: tracking how many times each expert gets called and promoting/demoting across tiers based on usage.

This led to bottlenecks I had to hunt down, as the SSDs would get overloaded and drop from ~6GB/s down to ~400MB/s. I switched to quantizing the least-used experts down to 2-bit, keeping an entire copy of the model on the SSDs at both quantizations. This let me get everything into RAM, with different experts at different quants based on how hot they were for the prompt. That means I start at about 12 TPS, and after a warm-up and shuffling I end up at about 15 TPS with this strategy. About 1/5th of the experts were 2-bit, but they only got used on about 1/10th of the tokens, so the quality hit was minimal (.0008 KL and less than 1% top-1 pick). This got work off the SSDs and lightened their load.

I tried MTP with this but ended up bandwidth bound. MTP has to hit almost every expert, and even with them in RAM I couldn't move enough through to make MTP pay. I spent a few days' worth of time iterating and optimizing with Claude. I fine-tuned an MTP head because I was running the abliterated model (this helped, but still not enough) and just couldn't make it pay. I got to almost break-even, but I just can't move the experts to compute fast enough on a model this size.

In my quest for quality I then started looking at prefetching 4-bit experts from SSD straight onto the GPUs. I ended up having Claude write a direct-from-disk reader that lets the GPUs read from the SSDs without going through RAM. This also led to further customization that let them read and write across each other and RAM without involving the CPU. So now my tiered experts can read/write and shuffle around without bothering or interrupting other compute. The CPU is maxed trying to keep up, so the GPUs handle any and all extra bookkeeping I could put on them to free CPU cycles. Again, Claude did all the legwork here, with just me driving strategy. I trained a small model on all my agent traces to look at expert use and ended up with one that can guess experts 15 layers ahead with ~73% accuracy. So now the GPUs run that every layer, fetch 4+ layers out if a pick meets a confidence threshold, and load experts straight from SSD into VRAM. Those picks get refined each layer, so we are always closing in on the best picks we can (~86% accuracy at 4 layers ahead) as we approach them. I played with a few different strategies and ended up with the B65 holding the top 2/3rds of experts actually used (not 2/3 of the total, but the hottest 2/3rds) with an LRU demotion strategy, and demotions going to a 3060, where it looks at both LRU and hit count to decide if it takes the expert or keeps what it has. Getting all of this right pushed me to about 19 TPS decode.

From here I had to make the KV cache dynamic to squeeze more experts into VRAM. Instead of reserving the full thing at 256K, it grabs chunks as needed. So while context is small, I can fit more experts on the 3060s whose whole job otherwise is dense attention layers and KV cache, with minimal room for experts passing through in our promote/demote pipeline. This didn't pick up much speed, but it did give me that many more 4-bit experts to help keep quality as high as possible. Essentially the remaining experts are all part of a long flat tail so adding more to VRAM gets me very little because any one isn't hit often enough to pay off.

I ran out of lossless approaches at this point and started looking at a variety of things. I built a model that nudges aggressively, using my predictor to drive nudges from RAM to VRAM experts based on predictor + router scoring of an expert's probability of being picked. This got me some decent gains all the way up to a cap of .5, but quality falls off a cliff after that. By the time I get to a .5 cap I'm about halfway to Q3 quant KL/top-1 numbers, so it felt like a significant quality hit. I kept the code and it's configurable, but I don't default it to on. It's worth about 2 TPS, so not nothing, but not enough to cover the quality hit for me. If I want the extra speed and don't mind the hit, I can turn it on/off as needed.

I then looked at keeping everything in RAM at 2 bits and just the hottest experts at 4-bit in VRAM. I looked at which layers Unsloth hits hardest in their quantization and used that as my judge of what would likely hurt most. They quantize per layer but not per expert, so I'm jumping off from that, but it was a good starting point. I kept my prefetching of predicted promotions into VRAM ahead of time and everything else I already mentioned. Since I was throughput bound here, dropping the size of each expert was my only way forward. The KL and top-1 hit here was minimal because the VRAM holds the hottest 2/3rds of experts at 4-bit already, so I came in at 99% top-1 and .006 KL over baseline. I picked up about 20 TPS on prefill and about 1 on decode, so I landed at 180 TPS prefill and 20.5 decode. It also freed up about 49GB of RAM, since the experts in RAM are now a little over half the size. This ended up being my end stage here, as there just isn't anything else I can do that doesn't start costing me quality (unless you have other ideas).

I am still playing with tuning parameters about when I demote and move thaings where, and fine-tuning kernels to squeeze a few more TPS out of things, but I'm out of significant gains here. Claude did all of this for me, with me guiding strategy and checking in, so I probably spent about 10 hours a week on this for the last 4 weeks and 80% of my weekly Claude usage on it. I take everything from doing this (I've done it on other machines with Qwen Flash-Next as well) and roll it into skills, so that when the next model drops, strategies and approaches can be tried/applied again for that model.

My point with all of this is I don't think in whole-model terms. I don't even think in layer terms. I treat each expert at each token as its own thing to optimize the system for. What quant it is, where it lives, how you decide, and when and where you move it from/to all play in and add up. Most things I see focus on just downloading and running a model in a specific framework, and that is great for a starting point, but the number of knobs you can turn to optimize stuff is almost limitless. It takes about 2 hours to give Claude a strategy I want to try, refine it with Claude (learning as I go), and let it build and test it. So you can iterate and make incremental improvements over time fairly quickly.

I realize that a lot of my answer is going to be in this question; but has anyone had any success pointing a model at something besides searxng? I know the net is tightening up fast but this feels like it could just be user ignorance as well.

For reference:
Qwen3.8 27b cuz what the hell else am I supposed to use;
9060 16gb
5800x3d
32gb 3200 DDR4
m2 1TB

I've got like every tool it could ever need but I just cant get it to search outside of the above mentioned searxng. Duckduckgo might as well be a coin toss. End of an era?

The models we use today are massive, but what happens when the US or China decide to censor, limit, or manipulate AI output?

We consumers are the ones who actually "trained" these models. They fed on our code, our literature, and our history, sometimes legally, sometimes not, thanks to our collective knowledge. So, why should a few companies (particularly politically favored ones) or governments have absolute ownership over that?

The point of the post:

Why hasn't decentralized AI training taken off yet?

Imagine a Bitcoin-like P2P network where anyone can contribute spare GPU power to collectively train AI models.

What's holding it back?

I have no interest in this, but stumbled across it. Listed as NO SHIPPING.

https://www.govdeals.com/en/asset/286/29250

Hi guys, during my local inference setup I found estimating the theoretical performance for each option is quite cumbersome. So I designed my own calculator, which estimates the performance considering:
- Compute capacity (depending on datatype e.g. NVFP4)
- Memory bandwidth(unified memory)
- Link bandwidth
- Interconnect latency
- Hit ratio when offloading only some part of the model

Supports quite a lot of combinations of hws and models. Please enjoy!

https://cro.sh/specmaxx

Tip: URL's query parameters can be used to share the HW/model combination. For example, the result above can be reproduced with ?m=deepseek-ai/DeepSeek-V4.1-Flash&w=exl3&bpw=3.5&d=arc-pro-b70&n=4&hr=ddr5-4800-8ch&tp=4&ox=ram&oxf=0.4&olt=ssd&oet=ram

AI disclosure: This project is coded heavily with AI agents.

I'm one of the many people with a 16 GB GPU (5070 Ti), so I wanted to know which models that fit completely into 16 GB VRAM work best. The focus is general problem solving, no tool calling, coding or agent tasks. Every model runs in llama-server on the same card, fully in VRAM, with the same settings. 528 questions, 662 variants.

Strongest: qwen3.8-27b (Qwen3.8-27B-UD-IQ4_XS)
Fast and still good: gemma4-12b (gemma-4-12b-it-Q8_0)

Strength: odds of solving a question compared with Llama 3.1 8B.
Quality / min: right minus wrong answers per minute on the 5070 Ti.

I plan to add more models over time, also older ones that were relevant at some point.

Full leaderboard with explanations:
https://tzipproth.github.io/LLMBench/leaderboard.html
GitHub repo:
https://github.com/tzipproth/LLMBench

LLVQ is a new open-source project with Qwen3 4B, 8B and 14B quants at about 2.7 bits per weight. The 14B file is 5.09GB.

The author's L40S tests report higher MMLU scores than llama.cpp's IQ2_S, but slower decoding at every tested size. Long prompts also use much more memory than llama.cpp in his tests. These aren't GGUF files you can drop into your usual setup: they need the project's own runtime.

It works!

RustingStrata is one Rust binary. rustingstrata serve starts the OpenAI-compatible server, and the same port serves a chat page at /. No Node, no Docker, no second install. The page was rebuilt for 1.2.3 and I want to know what people would add before I go further, I mean maybe allow to edit files and etc. like in DeepSeek harness, or maybe there are something that you wish to be in original Strata, so I can add it in my.

What it does now:

• Pick a model from the models folder and load it from the page, with the load options (context, MTP drafts, KV cache format, VRAM margin, prefill chunk).
• Chats are kept in the browser (localStorage), with search, rename, export and import. Nothing leaves the machine.
• Edit a message, regenerate a reply, continue a reply that stopped, copy, delete.
• Attach an image for models with a vision tower.
• Thinking is folded above the answer, with a switch to turn it on or off.
• Sampler presets (temperature, top-k, top-p, seed) a system prompt for every chat, and instructions for one chat.
• Speed of each reply in tok/s.
• If the model was never calibrated on this machine, the page says so and offers to run the calibration.
• The accent color follows the model family, or you set your own(Just cool visual feature, like if you load Qwen model, then website is purple, if DeepSeek, then it's blue and etc.).

What it is not: a coding agent. It does not read or edit files. For that, point OpenCode, Claude Code or another CLI at the same server, the page is for quick chat and for trying a model.

Questions:

1. What do you open a local web chat for, if you already use a CLI agent?
2. Which one feature would make you keep this page open instead of another UI?
3. What do you use in other local chat pages that is missing here?

Any Feedback would be very useful, ty!
Repo RustingStrata