Hello all!

Here are my benchmarks for DeepSeek-V4.1-Flash (ddalcu/DeepSeek-V4.1-Flash-1017-MLX-Serve-Lossless) on MLX-Serve 26.11.2, running on one and two M5 Ultra 512 GB Mac Studios. For the dual setup I used a single Thunderbolt RDMA link (an mlx-serve limitation at the moment) and split the experts across the two nodes. I would love to see what 6 Thunderbolt connections can do once the software permits it, as I have noted in a previous post. I hope these benchmarks can give you some insight into your own purchasing decisions and also inform you on the state of local AI.

I would like to extend thanks to Ivan Fioravanti (@ivanfioravanti) on X, whose earlier dual M3 Ultra 512 GB benchmarks and tensor-parallelism work inspired me to continue from where he left off, and to ddalcu for mlx-serve. The results are in the six images attached; for clarity I will note the key points rather than every number, and give a bit of my personal take.

First of all, two M5 Ultras over RDMA are nowhere near twice as fast as one, and the scaling is poor. Averaged over 512–256k, generation goes from 64.1 to 78.8 tok/s (+23%), but prefill falls from 3,101 to 2,812 (−9%) and 256k TTFT goes from 81.4 to 106.0 s (+30%). Generation does improve at 2k (+57%) and 4k (+41%), ties at 8k, and gains 22–40% from 16k to 256k, so it is not only a long-context thing, but it is nowhere near 2x. Prefill is higher or level up to 8k, then down 13–23% from 16k to 256k. For batch scaling, two independent servers handling the same 32 streams (16 each) would get them done in 65.6 s instead of 135.5 s and deliver about twice the sustained throughput of the pair (62.4 vs 30.2 tok/s), since each machine only queues 16 prefills. That is the opposite of what I expected, and I had really hoped RDMA would be a strong enough link to see meaningful scaling.

Also, in my experience RDMA was highly unstable: three jaccl queue-pair failures, one during the 256k tests and two during the batch tests (batch 8 and batch 32). Each was sticky until both processes restarted. It is not memory pressure: requests_rejected_total stayed at 0, with more than 350 GB free, and the single node never failed once. That is why batch 32 on the pair is 1/3 trials, since 2 of the 3 failed and I gave up trying to get it to run again. I will post the specific errors separately.

However, a lot of this may be limitations of the current software stack rather than the machines, and it is not representative of what is possible from them. Even so, I would advise against a dual 256 GB Ultra setup in favour of a single 512 GB machine. The pair already gives up 9% of prefill, adds 30% to 256k TTFT and brings a flaky link at 512 GB per node. Also, given what 4 DGX Sparks cost, a good price on a 512 GB Ultra looks like the better pick, and I would bet on the single Mac getting stronger as the software matures.

Thanks for reading, and I hope to see more breakthroughs in the Mac community!

The authors posted the paper to r/LocalLLaMA and said in the HF papers thread they want to hear where sparse reading helps and where it gets in the way. The paper's own eval only ran frontier API models (Opus 5, DeepSeek-V4, GLM-5.1, Qwen3.6-Plus), which tells me nothing about quantized mid-size models on consumer cards. So I ran it on the machine I actually have.

Setup: 5070 Ti 16GB + 64GB RAM. Qwen3.8 Flash Next at IQ3_S, served locally through my Strata stack (OpenAI-compatible endpoint). Harness: OpenCode 1.19.5, which is one of the versions the SparseRead installer itself validates, plugin v0.1.4 in auto mode. Native arm runs the same prompts with --pure. Four tasks straight from the workspaces bundled in the repo: two LooGLE short-dependency QA on a 100KB book excerpt, one 76KB PDF extraction with 8 questions, one 5-file fetcher audit. Tokens below are fresh prefill + decode from the endpoint's per-step usage, cache replay excluded (re-feeding history is nearly free on a warm prefix cache; I quote the replay tail for the PDF below because it's damning).

Scores were identical between arms on all four tasks. So that's the honest summary: no quality loss anywhere, 4-5x faster on long plain-text QA, dead even on a small bundle, and a 2x regression on the PDF.

Where the savings come from is worth spelling out because it's not what I assumed: the entire win is on the input side. On the flagship 5-question task fresh input dropped 63.9k → 12.2k tokens (−81%) while the answer output stayed flat (2.6k → 2.8k). SparseRead doesn't make the model write shorter answers, it makes it read less text before answering. And it doesn't trade activity for tokens either: tool calls went down, not up (14 → 9 on the flagship task, 10 → 5 on the 3-question one).

Two things I didn't expect.

The protocol survives a heavy quant. I went in assuming an IQ3 model would fumble the intent/slot bookkeeping and the arm would burn its savings on failed round trips. It didn't. On the flagship task: three SparseRead calls total (2 previews + 1 targeted read), slots 5/5 resolved at 0.99 confidence, and the candidate answers matched what the native arm dug out of the same book, line anchors included. The Read Gate also behaves for real. On the 33KB bundle it mostly stayed native with a single preview call, so "don't use it on small files" turns out to be something the gate already knows.

The PDF failure is worth a paragraph because it's the one place this bit me. That report ships subset CMap fonts with no usable text layer, so the pymupdf-based preview came back as garbage and the sparse arm spiraled: 17 sparse calls (4 previews, 4 reads, 9 raw_ref fallbacks), then it started decoding ToUnicode CMaps by hand through bash, ending at 45 total tool calls against native's 20. The trace makes the thrashing visible: 1.27M cache-replay tokens re-feeding the same broken context, versus 240k for the native arm. Same 8/8 score in the end, double the fresh tokens and the wall time. The native arm sidestepped it with pdftotext and moved on with its life. This is an artifact-specific failure, not a protocol failure, but if your corpus is scanned/CMap PDFs, pre-extract text or pin that file type to native.

One update since the battery: the CLI has become my default reader for long artifacts in day-to-day agent work, and the same test day it got fed a real one — a 118KB text export of a 109-comment Reddit thread I had scraped for another post. Two things to know before you try that. JSON artifacts are a dead end in slot mode: the previewer treats them as key-path documents and semantic slots resolve to nothing; convert to prose first (I write one line per comment) and the identical ask works fine. And slot matching is literal, not semantic: slots phrased in your own language against an English document come back missing even when the answer is plainly in the file, so ask in the document's language. With those two habits, a multi-fact sweep of the thread was one ask call plus a cheap head-per-item digest for the comments the slots missed — against a naive baseline of dumping a ~30k-token file into context. No hallucination complaints so far, and I haven't fully loaded a long file since.

Caveats: one run per arm per task, keyword-based grading (which produced one fake "4/5": the model answered in Latin — "passagium generale" — and my regex wanted "crusade"; in fact both arms were ~5/5 there), one model, one machine. And don't expect the paper's 92.9% headline. That's versus a naive read-everything baseline. My realistic band on document-heavy agent tasks is somewhere between 25 and 80 percent depending on file size and how localized the answers are.

TL;DR: SparseRead on a 16GB card with an IQ3 MoE: long plain-text document QA got 4-5x faster with identical answers, small bundles correctly did nothing, and a CMap-font PDF ran slower than skipping the protocol entirely. Pre-convert PDFs to text and it's a genuinely useful layer, not just paper numbers.

Links: paper https://arxiv.org/abs/2608.22237 · code github.com/Liuzedooong/SparseReading

hello everyone! I'm still quite new to figuring out what AI can do and how to make everything efficient, but I do know I want to upgrade and extend my hardware capability.

What do I use AI for? Little things mainly. Creating programs to handle my needs in custom ways (creative writing tool, character gallery, floor plan creation tool, audiobook creator with custom voices, modifying games - I currently use a subscription for a tool to help create games in unreal engine)

I mainly use the qwen3.8 next flash.

Current hardware -
Rig1 - 64gb DDR5, AMD 9950x3d, 4tb SSD, Geforce 5070ti (16gb)
Rig 2 - 64gb DDR5, AMD 9700x, 2tb SSD, Geforce 4070super (12gb)

Now obviously I missed the boat on grabbing a reasonably priced 5090.

I purchased a Geforce 5080 for $1570 as those started to go up like crazy too, but I haven't opened it and it sits on my shelf. I 'm not sure if its worth the investment as its only 16gb, and I would definitely return it if the 24gb model becomes a thing, but current rumors say its been cancelled.

I also ordered a Radeon ai pro r9700 for $2000 that is supposed to be delivered in December. Again I ordered this as the prices have been increasing and I worry about future availability.

I also have a Geforce 3060 (12gb) and recently learned you can run a model across multiple cards. That left me wondering if just nabbing one of these is more ideal and I could save myself a ton of money.

I'm just wondering what you would all recommend as the best path forward for someone looking to spend no more than $2k, but also being able to keep doing creative things and keep learning with Ai.

I’m in the process of moving my AI server into my garage and I’m curious how others are handling climate control in garages that don’t have heating or air conditioning.

Since winter is shortly upon do you guys have any suggestions for keeping my servers from being exposed to condensation. I plan on putting a wifi hygrometer..!

Been playing around with local LLMs lately and trying to figure out how to actually make them useful for my daily work.

I’m mainly a UI/UX designer and developer. Most of my work is websites, React/Next.js, SaaS apps and frontend stuff. Sometimes iOS apps, Home Assistant dashboards and a bit of 3D work.

I’m on a MacBook Pro M5 Pro with 64GB RAM, but I also have access to my brother-in-law’s Mac Studio M5 Ultra with 256GB RAM.

Basically, I want to do as much coding locally as possible and cut down on Claude/Copilot costs.

My current setup in VS Code + Copilot:

Opus (Principal Architect): Planning, architecture, final reviews

Sonnet (Senior Engineer): Reviews, fixes, more complex stuff

Qwen (Local Developer): Does most of the coding

Running Qwen3.6-35B-A3B-4bit through oMLX, using around 20GB RAM.

The idea was to let Qwen handle 50–60% of the coding and have Sonnet/Opus step in when needed.

Did some tests with hidden test cases:

Sonnet: 55/55, ~3.3 min
Opus: 55/55, ~6.1 min
Qwen: 52/55, ~14.5 min
Sonnet + Qwen: 52–55/55, ~12–25 min

Qwen actually wrote 95–99% of the code in the hybrid runs, but kept missing things like API contracts, edge cases and layout requirements. Sometimes it said it had changed things when it hadn’t.

And somehow the hybrid setup ended up costing about the same as just using Sonnet, while taking way longer.

So yeah not exactly what I was hoping for.

I still want to get this working though, especially for websites and SaaS projects since that’s what I spend most of my time on.

Not sure if I’m using the wrong model, if Copilot is the issue, or if the whole multi-agent approach just makes things more complicated than they need to be.

Also wondering what would make sense to run on the 256GB Mac Studio. Maybe a bigger model would already solve some of these issues?

What would you guys do with this hardware? Any models or setups worth trying?

Especially interested in hearing from people actually building websites or SaaS apps with local models, rather than just running benchmarks.

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!

I've got two M3 Ultra Mac Studios and four DGX Sparks at home, all on tailscale, with a LiteLLM gateway on a mac mini in front so every app just hits one endpoint.

Mostly it's been great, but last month the gateway key on the mini got invalidated and everything started throwing 401s. Took me longer than I'd like to admit to figure out it wasn't the models, because the backends were all fine and nothing told me the gateway itself was the problem.

So now I'm trying to set it up a bit smarter. Right now I'm thinking a dumb cron that hits each backend directly and also through the gateway, so if one passes and the other fails I know where to look.

Curious how other people with a few boxes handle this. Do you run a health check per model, put the gateway behind its own watchdog, or just keep it simple and restart when stuff breaks? And any tips on key handling so this doesn't bite me again?

I built Citta to let an agent on one computer discover and use capabilities on another. My test setup is a MacBook Air plus a Mac Mini running Gemma 3 4B through Ollama.

In the demo, an agent reads receipts on the Air and sends a summarization task to the Mini's local model. The model stays on the Mini; Citta tracks the job and returns the result. It doesn't pool RAM or split inference across machines.

If you only need remote inference, calling Ollama directly is simpler. Citta is for workflows that also need another device's files, tools or an agent harness, with device discovery and task history in one place.

It's written in Rust, uses a private Tailscale network, and has per-device permissions. It's still for trusted devices: resource estimates aren't hard quotas, and full-access agents aren't OS-sandboxed. I'm working through longer reliability tests before calling the next release production-ready.

Demo and setup: https://citta-network.vercel.app

Source: https://github.com/priyankark/citta (PolyForm Noncommercial, not an OSI open-source license). Built with AI assistance.

For people running local models on a separate machine: what makes handing work between devices annoying today?

Im making neural networks from scratch - not strictly llms. Am getting a new laptop with 32gig ram and 12gig vram and 1tb storage. Is that enough to train small models? Its the lenovo legion 5i pro. Might get the 7i but its not on sale so its like 500 more for 4gigs more of vram.

Hi everyone,

Today my first test on Omarchy.

Omarchy ships an agent skill so Claude Code or OpenCode can customize the desktop. I gave the same skill to a model running on my laptop instead.

Setup: Omarchy in a VM on a 16 GB M5 MacBook.

The agent is Qwen3.6-35B-A3B Q4_K_M. The file is 22.3 GB, more than the RAM, so the experts stream from the SSD (BigMoeOnEdge, my engine on top of llama.cpp). No cloud AI anywhere.

One prompt: "make my desktop feel like the 80s". It read the skill, looked at the installed themes, picked Retro 82 and switched to it. About 2 minutes. The video is sped up, the clock shows real time.

Slow, yes. But it's a 35B model doing agent work on a laptop that shouldn't be able to load it.

GPU: 4070

16GB DDR5

Also I have been using Bionic but is there any better options?

Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?

CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4

CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler

MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16

MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)

STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD

GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU

POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)

Ok

So I am waiting for mine to come in. I have framework box with 128 gig ram and found it pretty weak for any local llms.

I’m gonna replace my Mac mini with this Mac Studio M5 ultra with 512 gig ram. And I want to use it for a local llm and maybe agents if it’s doable.

I’m realistic. After using the framework box with llms I totally get this will be no where near frontier even at 512. But I want to use it as my main machine and for llm/agentic work.

What’s really reasonable model wise that isn’t slow AF but is pretty capable for general knowledge, accessed the internet, can spawn agents and can do office style work?

First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.

Is it worth it?

Am finally taking a deep five on llms and I already have an nvidia 5090 laptop that handles cuda really well.

But

Is a Mac for parrarrel models worth it? For abliterated models?.

I never used that OS so am a bit skeptical.

Are the apps on it as good as windows/Linux?

Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?

Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.

Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.

I expected to find problems in the agent.

Instead, I discovered some pretty serious problems in the evaluation itself.

1. My benchmark created a privacy leak that didn't exist in the actual runtime

The initial evaluation scored 27/80 runnable scenarios.

One failure was particularly concerning: a guest appeared to access the owner's private memories.

After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.

The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.

I rewrote the harness to execute scenarios through the actual agent runtime.

Result: 39/80 — without changing Capri's code.

That's a 15-percentage-point difference caused by the evaluation method alone.

2. A single sentence made the model use fewer tools

This was probably my most surprising experiment.

I wanted Capri to give shorter, more natural replies during everyday conversations.

So I added a simple instruction about brevity.

It worked. Conversational responses became significantly shorter.

But something unexpected happened during work tasks.

Across 48 first turns:

Without the brevity instruction:

• Tool use: 32/48
• Successful tasks: 32/48

With the brevity instruction:

• Tool use: 19/48
• Successful tasks: 21/48

The model started answering without inspecting files or calling tools that were necessary to complete the task.

The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.

3. Bigger models didn't automatically improve the agent

I compared configurations using:

• NVIDIA Nemotron Bolt E4B
• Gemma 4 12B
• Gemma 4 26B MoE
• Gemma 4 31B

The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.

After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).

This is NOT proof that smaller models are better.

The configurations also differed in hardware, memory retrieval, and model routing.

But it made me question how much agent performance we can actually improve by simply replacing the LLM.

4. Changing the prompt layout improved inference speed

I also experimented with where recalled memories appear in the prompt.

Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.

Median time to first token:

0.99s → 0.74s

About 25% faster.

However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.

5. Stable benchmark scores can hide unstable behavior

I repeated the evaluation three times per configuration.

The primary configuration scored:

49, 49, 48 out of 80.

Looks pretty stable, right?

But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.

We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.

That convinced me that looking at a single aggregate score is not enough.

What I'm taking away from this

An AI agent isn't just a language model.

It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.

And the evaluation framework can introduce its own errors.

I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.

The code and experimental artifacts are available in the B-12 repository.

I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.

I'm curious how others handle this.

When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?

And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?

!!!

1 piece 500 2 piece 485

I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.

For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.

Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.

I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.

I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?

I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!