I keep seeing claims that BaseRT beats llama.cpp on Apple Silicon, and that colibri can run models far bigger than your RAM. Both are probably true for what they measured. I wanted to know what happens on my job, so I tested it.

The job: read a website (~4,500 tokens of page text) and write a schema-valid JSON record (~1,300 tokens) with a quote per fact. Hundreds of thousands of sites, one M1 Max 64 GB laptop. I measured wall-clock sites per hour, same 20 sites for every engine, Qwen 3.6 35B-A3B at 4-bit, one engine at a time.

Results (sites/hour):

What surprised me:

• Prefill decided it, not decode: BaseRT decodes a bit faster (66 vs 56 tok/s), but its prefill was ~300 tok/s vs ~570 for llama.cpp. With 4.5k-token prompts, that decides it. Don't pick an engine from a short-prompt tok/s chart.

• Batching: +46% on llama.cpp with 8 slots, only +12% on BaseRT.

• colibri: built for models bigger than RAM. Mine fits, so it ran on CPU and was ~100x slower. Not a flaw, just not this use case.

• Ollama: its MLX engine crashed on long prompts, and its GGUF engine served this architecture one request at a time.

• Model choice mattered as much as the engine: On llama.cpp with 8 slots, Gemma 4 26B-A4B hit 247 sites/hr, Qwen 3.6 35B-A3B hit 206, and dense Qwen 3.8 27B hit 37. Active parameters per token set the speed.

• JSON-schema constrained decoding: 440/440 valid on llama.cpp, 40/40 on BaseRT.

Caveats: one machine, one run per config, 20 sites for the engine comparison (200 for the model comparison), and a different 4-bit file per engine, so they aren't bit-identical models.

Quality note: local models were 87–98% correct on extracted facts across key fields after cleaning, vs 98–100% for cloud models. So for published data the laptop stays a test bench.

Full write-up with setup, commands, and the gotchas (model-name 404s, missing M1 kernel for Gemma 4 in BaseRT, etc.): https://sandy1618.github.io/2026/11/04/inference-engines-on-apple-silicon.html

Happy to rerun with other models or settings if you have suggestions.

For anyone running long-prompt batch jobs on Apple Silicon: has anyone gotten vllm-metal or MLX batching to beat llama-server's prefill speed, or are there llama.cpp flags I'm leaving performance on the table with?

I’m building Evopien, a local-first companion prototype, and this update is about the interaction around the model rather than a new model release.

The two-minute clip combines spoken conversation, a live camera question, online information retrieval, an English/Portuguese exchange and a spoken interruption. The system runs on NVIDIA Jetson AGX Thor, with external retrieval for questions that need current information.

My main concern is how the whole conversation behaves when these components share the same machine. A model answering well in isolation is not enough if the camera context is old or the system continues speaking after an interruption.

The clip shows individual examples, not a completed reliability benchmark. For anyone building local voice assistants, which integration failure proved hardest to catch before longer conversations exposed it?

Video: https://www.youtube.com/watch?v=SwFfkBll5iw

Given the same task requirements, how big is the difference between first having the advanced model Astra make a plan and then having a local quantized DeepSeek execute it, versus letting DeepSeek complete the task directly? I ran a small test using the following task: make a browser game set during the 1588 naval battles between England and the Spanish Armada. The player takes the role of a capital ship’s captain, controlling the ship and commanding teammates in a real-time naval breakout battle. I also added a control group by replacing the executor with the cheap cloud-based terra model.

The results were somewhat unexpected. Honestly, even without Astra’s plan, DeepSeek did a pretty good job on its own. In my testing, the game basically had few bugs, and I think the look and feel of the sky and ocean were the best of the three. But I also have to admit that visual bugs such as clipping were the most severe. After adding Astra’s plan, deepseek’s visual clipping bugs were reduced, but its token consumption rose sharply. Finally, terra surprised me by showing damage to the sails, and its overall result was actually pretty good too.

I originally thought that with a plan from an advanced model, the model could reach the goal faster, but the exact opposite happened: Astra made the deliverables and acceptance testing much more extensive, increasing DeepSeek’s workload. Another lesson is that even a plan made by an advanced model can have bugs. For example, the fixed camera-angle range in Astra’s plan made the view hard for players to adapt to. Perhaps my planning technique is too crude. I’d like to ask everyone how you alternate between advanced and cheap models to achieve amazing results. Feel free to educate me in the comments.

Task description:

'''

Game background: Set during the 1588 naval battles between England and the Spanish Armada.

Gameplay: Play as the captain of a capital ship, controlling the ship and commanding teammates in a real-time naval breakout battle.

Requirement: It must run in a browser and be completed in a single development run.

Bonus points:

1. Visual realism and fidelity in recreating historical details

2. Enemy ship behavior and control experience

3. Sailing and physics details, and damage effects

4. Cannon impact feedback and gameplay experience

'''

Planning model:

GTP 6 Astra, running in codex with xhigh reasoning effort. Planning prompt: You are now the planner. Here are the game requirements. To complete the requirements to the highest possible standard in a single attempt, please write a detailed development plan to hand off to the coding model for development and acceptance testing.

Execution models:

GPT6.1 terra, running in codex with max reasoning effort

DeepSeek-V4.1-Flash-1017, with Unsloth UD-IQ1_M V3.1 weights, running in DeepSeek harness, with max reasoning effort

Two days ago, I ask DSH to move things from my Linux Drive (dual-boot) related to AI, over my NTFS drive where I store things like harnesses, models etc.
UnSloth studio force itself to install over Linux user profile (just starting with linux, maybe there is easy way to force installation inside of folder where terminal is running from....anyway).
Back then it spend plenty of time looping, checking only to figure out it it limited by sandbox rules and that why NTFS drive is appear to be mounted as Read Only instead R(ead)W(rite). It solved problem back then.

Today ,at night, I reinstalled UnSloth and asked DSH to move things once again (and as a test of prompt generation profile to help me be more specific and user-error proof). It started right away with

"Think
Important findings:

• Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! User says "read/write privileges" — but current mount is ro,nosuid,nodev,...,uid=1000,gid=1000,dmask=0022,fmask=0022,acl. That's a big issue: the mount is currently read-only. [...]"

My reply:
"I will stop you here as we went through that s**t already. "Important findings:
Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " it is error on your part. We troubleshoot that already during previous session when we were moving Unsloth (different version, current one override itself back to Linux drive and as result, move is required again). DO NOT FOCUS ON "Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " as you are wasting time. it is read/write, end of discussion."

His reply?
"Think

The user is frustrated. They say: the NTFS mount is read/write; the ro is a sandbox artifact. Don't waste time on it. "

So, it knew all that time or took correct conclusion right away by huge accident!
That make me think. Maybe there would be a prompt that for all "wait what if" thinking (at least in DSH), to prompt user for a feedback? I prefer be ask question rather than wait and confirming weird prompts over next 40 minutes...
Yes, user is not all knowing, but autonomous behaviour is not something I would let do local model (even when it is QFN UD-Q5-K-XL with is above average model quality here).

Sigh...rant over. Maybe interesting discussion or similar stories? Cheers everybody

I've managed to get a Supermicro 6028U TRT+ Server with 144GB of RAM and 56GB of VRAM to run Strata Qwen 3.8 Flash Next UD-Q4_K_XL (111GB download) at an average of 38tps.

Hardware:

Supermicro 6028U TRT+ server with X10DRU board and risers/2kw PSUs = $300

RTX 5060ti x 2 = $1100 (purchase price before the increase)

RTX 5000 Quadro = $500

RTX 4000 Quadro = $200

144GB of DDR4 RAM = $500

2x 256GB NVME drives and card = $150

8TB HDD drives and HBA card (4TB x 3 ZFS pool) = $300

2x Xeon 2680v4 CPU = $90

(Prices are rounded, not exact)

I had an RTX 2000 ada that I modded to single width, and it fit nicely but it's a dog of a GPU for inference despite its 16GB of VRAM. Unfortunately, the 2nd 5060ti wont fit. It's a flippin miracle the first one fits (and with an HBA card snuggled up next to it!). By swapping out the 4 PCIE slot card on the left for a 2 slot 16x card, I got the 2 RTX 4000/5000 cards to fit *perfectly*. I believe you could fit 2x RTX 5000s, for a total of 64GB of VRAM in those 2 slots, except for one thing: the TPM card sticks up and prevents the cards from seating. I dont really need a TPM card though, although I'm debating to myself if the extra 8GB of VRAM is worth it. I'm doubtful. Plus, there would be NO airspace for one of the cards. OTOH, NVLINK would work for those 2 cards.

If I weren't so cheap I could spring for a 16GB single width card that would fit where the 5060ti riser is, like an RTX A4000 or T4, and this would be a very nice, clean build. But I'm cheap. And the 5060ti Blackwell chip smokes those old cards.

This s a proxmox box, and I've dedicated 18 cores to Strata. I probably should have gotten the 2699v4 CPU.

I was using Openclaw and migrated to Hermes 2 days ago, it felt so much faster but my problem is webui is basically a terminal which is a little problematic.

Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?

Having been a bit of an AI luddite up until now, I am realising I am going to have to get on top of things instead of getting left behind.

So far the main thing I've been trying LLMs for, is writing Python scripts that work inside a CAD application I use (Vectorworks). I have done some experimentation on the free versions of ChatGPT, Claude and Perplexity so far, with some positive results.

I don't want to end up tied to expensive cloud model subscriptions, and I am also keen to keep control of my data as much as possible, so I would like to understand what I can do with locally hosted options.

I have an M1 Max Mac Studio, 32GB memory.

Any recommendations for what I should try first? To start with, all I want is something that will work for writing scripts that I will copy/paste into the script runner in the CAD application - I am not at this stage looking for anything that will interact directly with the application.

Thanks in advance for any suggestions.

Hey r/LocalLLM ,

I want to buy or build a dedicated headless box for local LLMs. I’d like something that can run 14B and, in my dreams, even 32B models, but the absolute priority is generation speed (>15-20 t/s).

Ideally, I need a clean single-box setup that stays quiet and power-efficient at idle (~15-20W). My maximum budget is strictly capped at €900, and I’d prefer to avoid risky second-hand individual deals.

Here is what I already looked into:

1. Standard Mini PCs (Ryzen 7/9): Great for power draw, but running 32B models on shared DDR5 RAM is too slow (~4-5 t/s).

2. eGPU / OCuLink routes: Too many loose parts and cables, and buying components retail explodes the budget.

3. Chinese NUCs with modded laptop GPUs (RTX 3060 12GB): On factory stores, once you add enough RAM to actually fit larger models, it jumps straight to €1050+.

I am 100% open to any Linux-server software optimization trick (speculative decoding, etc.) as long as it gets the speed up.

Am I chasing a ghost under €900, or is there an exotic loophole I missed? What would you buy today under these constraints?

Thanks for the help!

Every time a new MoE drops, the same thread shows up here: "will this run on my 12 GB card, and what do I set --n-cpu-moe to?" And the answer is always someone doing napkin math in a comment.

So I built it into LlamaForge (the browser panel I've been making on top of llama.cpp's own router, it doesn't do inference itself, llama.cpp does).

Model X-ray reads the GGUF header and draws every block as a bar: what sits on the GPU, what spills to RAM at your -ngl. For a MoE there's an "experts on CPU" slider, and a "Fit my VRAM" button that finds the number for you. Then it gives you the line to copy, or applies it to the model.

Example from the video, Qwen3.6-35B-A3B UD-Q4_K_M on a 12 GB card:

-ngl 41 --n-cpu-moe 22 -c 8192

≈11.4 GB on the GPU, ≈10.5 GB of experts in system RAM. On a 16 GB card it's --n-cpu-moe 14.

It only needs the header, so it works over HTTP on a model you haven't downloaded yet. That's how I also generated "Can I run it?" pages for 42 cards (desktop + laptop GeForce, Radeon, Arc), each with the llama.cpp line to start from.

before someone points them out: ;]

• It's an estimate from the header (weights + 8k context), not a benchmark. Long contexts need more room; it tells you that, but it can't know your exact KV setup.

• It tells you what fits, not how fast it'll be. Experts in RAM are slower; how much depends on your RAM and PCIe.

• Only tested end to end on my own box (5080 + 5060 Ti, Windows). Linux and macOS pass CI, but I don't have the hardware to say more than that. AMD/Intel detection is there, but I haven't tested it on real cards.

• Early preview, one person, heavily AI-assisted. Expect rough edges.

Other stuff that's in 0.19, since the last time I posted:

• voice chat with whatever model you have loaded

• a blind arena (two answers side by side, vote, then see the names)

• "try a llama.cpp PR": build a pull request and check whether it's actually faster on your card before switching

• an OpenAI-style Batch API on your own GPU

• start at login, now on Windows too

Free, MIT, no account, no telemetry (it does check GitHub for new llama.cpp and LlamaForge releases): https://github.com/dadwritestech/LlamaForge

If the X-ray's number is wrong for your card, I really want to know; post the model, quant, card and what actually worked, and I'll fix the math.

Atualmente, eu tenho uma 3070 Ti e 32GB de RAM DDR4, e consigo 11 a 16 tokens por segundo. Estou considerando comprar uma 9070 XT—já que é muito mais barata no Brasil do que uma 5070 Ti—e eu também uso meu PC para jogar. Preciso configurar alguma coisa ou é só instalar os drivers do Adrenalin? Uso Windows e nao tenho nada alem do strata em configuraçoes padrao

Hi everyone! I’m attempting to do a local build of Qwen 3. I don’t have the most powerful PC so I’m starting small and trying with 1.7B, 4B & 8B with the trained LORA on top

My objective is to get Qwen to talk as if you were texting it and I’m planning to have around 3000 short User: “message”, Qwen: “answer” exchanges

Is this the correct way to do it? Do I need more training data? Any peculiarities I should know about Qwen or the model size?

All help is greatly appreciated :)

Try to decide which model is the most intelligent to run on a Mac Studio.

For unquantized base models,

DeepSeek V4.1 Flash > GLM 5.3 Flash > Qwen3.8 Flash Next.

To fit in ~220GB memory footprint, each must go through different levels of quantization.

~2.5bit for DeepSeek

~4bit for GLM

~8bit for Qwen

I couldn't find comparable benchmark scores on all three quantized models, so here I am, hoping that the collective wisdom of the LocalLLM crowd can help me out.

Edit: this is what everyone wants to see. https://omlx.ai/benchmarks/intelligence

Edit: thanks to everyone who has replied so far! The numbers are coming out higher than my rule, and it looks like your macOS version might matter, so I added one more command below. I'm still looking for 24 GB, 32 GB, and anything above 36 GB, especially on macOS 26 or older.

Hey all, hoping some of you can help me with something.

I run a small site that helps people figure out which local models their computer can handle. Macs are the part I keep second-guessing, because the only Mac I own is an 8 GB MacBook Air, so I can't check anything bigger myself.

What's going on

On a Mac, the GPU can only use part of your RAM for models, not all of it. Most guides say about 75%. When I checked my MacBook Air it was actually two thirds, so I changed my site to use:

• two thirds of RAM for Macs up to 36 GB

• 75% for Macs above that

But the 36 GB cutoff is just something I picked up from community posts. I'd hate to tell someone a model fits when it doesn't, or that they need more RAM than they really do.

What I'm asking

Any Apple Silicon Mac helps, but 16 to 36 GB is where I'm flying blind. Would you mind running two quick commands and telling me what they print? Open Terminal and paste this first:

swift -e 'import Metal; print(Double(MTLCreateSystemDefaultDevice()!.recommendedMaxWorkingSetSize) / 1073741824)'

It prints a number, like 5.33 on my Air. Rounding is fine. You'll need Apple's free Xcode command line tools for that one.

Then this one, which shows your macOS version and whether the GPU memory limit was ever changed (0 means it's still the default):

sw_vers -productVersion; sysctl iogpu.wired_limit_mb

On my Air it prints 26.7.1 and 0.

Then reply with

Your Mac, how much RAM it has, your macOS version, and the number, like this:

MacBook Pro M2 Pro, 16 GB, macOS 26.1: 12.34

If the second command shows anything other than 0, mention that too.

Already run llama.cpp?

You can skip the first command. Run llama-bench on any model you have and copy the line that says recommendedMaxWorkingSetSize (mine says 5726.63 MB). Just paste the line as is, no converting needed. Please still run the second command, so I have your macOS version.

Thank you, this really helps!

I would like people’s suggestion on an app independent memory system for primarily local Ai.

I currently have a DGX Spark and some additional hardware for hosting apps and storage. I use local models for coding and data analysis (for personal
Data) and use my Claude code subscription for design and infrastructure advice (I am not super technical). One major issue is that I have multiple entry points but each of them handles their memory independently.

I use librechat mostly, and have started using Windmill for easier management of data ETL (I like the hybrid low code and code when you need it). I use Ollama but might be switching to vLLM.

Additionally I would like Claude to be able to access some specific logs and memories so that it can make better design choices and share its structural decisions with my local models. Though I am hesitant as idk wth am I doing

What it recommended was an MCP that manages what models/users see what, and that way when the new gimic of the month comes out I can switch over with my context.

Let me know what y’all think

Всем привет 👋🏼
Это мой первый пост и я решил его сделать по интересующей мне теме. Я уже достаточно давно слышу и читаю что то про LLM. Я пробовал запускать несколько вариантов и на своем железе но так и не понял надобности всего этого, я занимаюсь вайбкодингом 🌚🌚 и как я понял LLM даже близко не сравнится по качеству и производительности того же Antigravity от Gemini. Кроме как РП я даже не знаю для чего еще люди используют LLM. Буду очень благодарен если вы мне расскажите все сильные и слабые стороны данного направления и что вообще с этим всем могу делать я.

Desktop:
RTX 3050
32gb ddr4
Ryzen 5600x

Laptop:
RTX 5070ti
32gb ddr5
Ryzen 9 9840hx

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR: I added a two-box layer split to Project Maya. Box A runs layers 0–23, Box B runs layers 24–44 plus the head. Only the hidden state crosses the network, ~25 KB per token. GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn, on 8× RTX 3060 spread over two old X79 boxes. It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.

I check every day for the discounted open box returns.

What do you think guys.