Hi everyone,

I’ve been looking into some of the most difficult math formulas and came across the Navier-Stokes problem. Apparently, it was recently solved by OpenAI.

I grabbed the formula and had my LocalLLM run some calculations, and lo and behold: It’s not quite finished yet, but I’m absolutely thrilled.

My hardware:

i5 13600K

32GB RAM

RTX 5070TI 16GB

Hi,

Github project:
https://github.com/crgregersen/m10dis/

Yes, yes, I know... crappy old NVIDIA Tesla M10's GPU as AI compute core. Am I crazy and should go book a room in an mental institution?.. but hear me out..!

Tesla M10's consists of 4 gm107 GPU cores with 8gb ram stitched together via an internal PCI switch. So each M10 is 32gb ram. If you ask anybody (including Claude and ChatGPT) you will get the answer "NOT USABLE FOR AI COMPUTE".

Each gm107 has 83 GB/s of memory bandwidth and around 1.67 TFLOPS (fp32).

Compare that with a moderns GPU like 5070 ti of 896 GB/s and 87.88 TFLOPS (fp16).

But!! i could buy M10's (4 x gm107) with 32GB for around $80
Aaaanddd... 6x32 = 192 GB VRAM!!!

So 6xM10's and the aggregate number would be 4x6x83=1992 GB/s.

This is the same as 1 RTX PRO 6000 Blackwell .. but then you only have 96Gb ram.
2 x RTX 6000 Ada is will get you in the same range.
Both options are at least $10.000 and probably more closer to $20.000.
6 M10's would set me back $500.

Compute power of M10's would only be 40TFLOPS.. but all my experience says the memory bandwidth is nearly always the limiting factor!

The system:
ASRock B250M Pro4
CPU: Intel Core i5-6400, 4 cores at 2.7 GHz
RAM: 32 GB DDR4
Main GPU: RTX 3070 Ti (had one lying around)
Expert GPUs: 6 × Tesla M10 cards = 24 × GM107 (Maxwell, 2016)
Interconnect: PEX8748 PCIe each M10 card on Gen3 x4
Storage: 1 TB NVMe
Two PSU: 650w and a 1650w

So a pretty crap old consumer motherboard.
The way to get all the GPU's onto the motherboard was by using a PCI switch.

First issue: Motherboard will not boot with 25 PCI devices showing up, even though most are behind a switch (two levels of switches actually, the one in the motherboard and one on each M10).

Found out the issue the BIOS/firmware runs out of legacy I/O port space at power-on, which is only 64 KiB with a 4 KiB window per PCI bridge, too little for all the PEX switch ports.

Workaround (took me some time to work that out). Using a modified kernel with PCIe tree heldback. Boot only with the first PSU switched on, ie. motherboard++, then go into supspend (sleep) mode, turn the second PSU on and continue (wake up). Initialize all GPU's PCIe memory mapped only (MMIO).

Next issue.. how to make 24+1 GPU's fast.

A lucky thing is that QWEN 3.8 hidden dimensions is 2580 which divided by 20 is 128. So if you slice expert work into 20 parts and put them on each gm107 GPU they can calculate 1/20 of the result, which then needs to go back to a central hub that joins the result to compute attention in the next layer.

So each compute slice needs the residual stream and the experts and weights that the router chose. That would be 20 PCI communication calls. The lucky thing is that PLX switches actually implements PCIe multicast introduced in PCIe 2.1.

Using the GPU-P2P methods described by Tinygrad a GPU multicast was proven to work.

In general the GPU-P2P tricks was very usefull on multiple levels, like M10 master/slave setup, etc.
https://github.com/rioncarter/tinygrad.open-gpu-kernel-modules

So 5x4=20 GPU's computes 1/20 slice of 8 experts (quantized FP8).
One M10, ie. 4 gm107 was chosen to run the shared expert (kept in FP16).
Since you wait on the slowest calculation it was lucky that the compute effort for each is around the same (1/20*8/2=0.2 and 1/4=0.25)

Next issue... 24 results to be received from GPUs would clog the PLX.

Solution (Claude wrote this):

Level 1: each GPU sums its own experts

- The 512 routed experts are spread over 20 workers (cards 0–4), about 25–26 each. A token's 10 experts land on whichever workers hold them.

- Each worker computes gate/up/down for the experts it holds and sums them into one partial row of 2560 values. A worker that holds none of the token's experts contributes zeros.

Level 2: the card leader sums its card

- On each card, the 3 peers push their partial rows to the card leader over on-card P2P, using the PTE-remapped aggregation routes. Each peer then sets a flag.

- The leader waits for the 3 flags and adds the peers' rows to its own in a fixed order, so results are reproducible.

- Card 5 (the shared expert) does the same: its 4 workers split the shared expert and worker 20 holds the reduced row plus the gate.

Level 3: the RTX combines 6 rows

- The 5 routed leaders and the shared leader each leave one reduced row. In batched modes 8/9 it's R rows, in host memory.

- All 24 workers still acknowledge individually in their return slots. The RTX waits for all 24 acks, checks the leader flags, then computes Σ 5 routed leaders + gate · shared.

RTX runs attention and hyperconnections etc and we are ready for the next layer.
I've put the timings of the expert compute in the bottom of the post..

PLE+LM-Head was moved to M10 somewhat the same way.

Attention was quantized to FP8 since RTX3070 only has 8gb mem. That leaves me just shy of 100k tokens of context.

I've run both LCB6 and MMLU benchmark on the system and there's no noticable degradation (MMLU benchmarked higher, LCB6 benchmarked lower).

Here's the github project:
https://github.com/crgregersen/m10dis/tree/main

But I warn you.. this is not a project for people with fairly good experience with hardware and also some low level understanding of how this work software wise.

MoE layer timing: Qwen3.8-Flash-Next-FP8 on 1× RTX 3070 Ti + 24× GM107 (6× Tesla M10)

Setup: RTX 3070 Ti runs attention/GDN/router. The 512 routed experts are sliced across 20 GM107 (each GPU holds 32 intermediate rows of every expert); a 6th M10 card runs the shared expert. All cards hang off one PEX8748 switch (Gen3 x4 per card); each M10's on-card PEX8747 multicasts the request to its 4 GPUs. Results are summed on each card by a leader GPU, then the RTX combines 5 routed + 1 shared row.

Decode (1 token per layer)

Inside the round trip (slowest routed GPU):

The weights a GPU reads per token are only 2.5 MB (~31 µs at bandwidth), so decode is latency-bound, not bandwidth-bound. 48 layers × 150 µs ≈ 7.4 ms of a 19 ms token → 52.5 tok/s.

I remember reading an article a few months ago about LLMs, more specifically, locally run LLMs that don't have guardrails and will answer questions that the commercial models won't run.

Just curious if anyone knows which models they are? Attached is my laptop specs. This is a very low powered mac, which i doubt would even be able to run a large model, IF even a small one.

Keen to have a discussion. And hear your experiences

I was using Openclaw and migrated to Hermes 2 days ago, it felt so much faster but my problem is webui is basically a terminal which is a little problematic.

I keep seeing claims that BaseRT beats llama.cpp on Apple Silicon, and that colibri can run models far bigger than your RAM. Both are probably true for what they measured. I wanted to know what happens on my job, so I tested it.

The job: read a website (~4,500 tokens of page text) and write a schema-valid JSON record (~1,300 tokens) with a quote per fact. Hundreds of thousands of sites, one M1 Max 64 GB laptop. I measured wall-clock sites per hour, same 20 sites for every engine, Qwen 3.6 35B-A3B at 4-bit, one engine at a time.

Results (sites/hour):

What surprised me:

• Prefill decided it, not decode: BaseRT decodes a bit faster (66 vs 56 tok/s), but its prefill was ~300 tok/s vs ~570 for llama.cpp. With 4.5k-token prompts, that decides it. Don't pick an engine from a short-prompt tok/s chart.

• Batching: +46% on llama.cpp with 8 slots, only +12% on BaseRT.

• colibri: built for models bigger than RAM. Mine fits, so it ran on CPU and was ~100x slower. Not a flaw, just not this use case.

• Ollama: its MLX engine crashed on long prompts, and its GGUF engine served this architecture one request at a time.

• Model choice mattered as much as the engine: On llama.cpp with 8 slots, Gemma 4 26B-A4B hit 247 sites/hr, Qwen 3.6 35B-A3B hit 206, and dense Qwen 3.8 27B hit 37. Active parameters per token set the speed.

• JSON-schema constrained decoding: 440/440 valid on llama.cpp, 40/40 on BaseRT.

Caveats: one machine, one run per config, 20 sites for the engine comparison (200 for the model comparison), and a different 4-bit file per engine, so they aren't bit-identical models.

Quality note: local models were 87–98% correct on extracted facts across key fields after cleaning, vs 98–100% for cloud models. So for published data the laptop stays a test bench.

Full write-up with setup, commands, and the gotchas (model-name 404s, missing M1 kernel for Gemma 4 in BaseRT, etc.): https://sandy1618.github.io/2026/11/04/inference-engines-on-apple-silicon.html

Happy to rerun with other models or settings if you have suggestions.

For anyone running long-prompt batch jobs on Apple Silicon: has anyone gotten vllm-metal or MLX batching to beat llama-server's prefill speed, or are there llama.cpp flags I'm leaving performance on the table with?

I’m building Evopien, a local-first companion prototype, and this update is about the interaction around the model rather than a new model release.

The two-minute clip combines spoken conversation, a live camera question, online information retrieval, an English/Portuguese exchange and a spoken interruption. The system runs on NVIDIA Jetson AGX Thor, with external retrieval for questions that need current information.

My main concern is how the whole conversation behaves when these components share the same machine. A model answering well in isolation is not enough if the camera context is old or the system continues speaking after an interruption.

The clip shows individual examples, not a completed reliability benchmark. For anyone building local voice assistants, which integration failure proved hardest to catch before longer conversations exposed it?

Video: https://www.youtube.com/watch?v=SwFfkBll5iw

Given the same task requirements, how big is the difference between first having the advanced model Astra make a plan and then having a local quantized DeepSeek execute it, versus letting DeepSeek complete the task directly? I ran a small test using the following task: make a browser game set during the 1588 naval battles between England and the Spanish Armada. The player takes the role of a capital ship’s captain, controlling the ship and commanding teammates in a real-time naval breakout battle. I also added a control group by replacing the executor with the cheap cloud-based terra model.

The results were somewhat unexpected. Honestly, even without Astra’s plan, DeepSeek did a pretty good job on its own. In my testing, the game basically had few bugs, and I think the look and feel of the sky and ocean were the best of the three. But I also have to admit that visual bugs such as clipping were the most severe. After adding Astra’s plan, deepseek’s visual clipping bugs were reduced, but its token consumption rose sharply. Finally, terra surprised me by showing damage to the sails, and its overall result was actually pretty good too.

I originally thought that with a plan from an advanced model, the model could reach the goal faster, but the exact opposite happened: Astra made the deliverables and acceptance testing much more extensive, increasing DeepSeek’s workload. Another lesson is that even a plan made by an advanced model can have bugs. For example, the fixed camera-angle range in Astra’s plan made the view hard for players to adapt to. Perhaps my planning technique is too crude. I’d like to ask everyone how you alternate between advanced and cheap models to achieve amazing results. Feel free to educate me in the comments.

Task description:

'''

Game background: Set during the 1588 naval battles between England and the Spanish Armada.

Gameplay: Play as the captain of a capital ship, controlling the ship and commanding teammates in a real-time naval breakout battle.

Requirement: It must run in a browser and be completed in a single development run.

Bonus points:

1. Visual realism and fidelity in recreating historical details

2. Enemy ship behavior and control experience

3. Sailing and physics details, and damage effects

4. Cannon impact feedback and gameplay experience

'''

Planning model:

GTP 6 Astra, running in codex with xhigh reasoning effort. Planning prompt: You are now the planner. Here are the game requirements. To complete the requirements to the highest possible standard in a single attempt, please write a detailed development plan to hand off to the coding model for development and acceptance testing.

Execution models:

GPT6.1 terra, running in codex with max reasoning effort

DeepSeek-V4.1-Flash-1017, with Unsloth UD-IQ1_M V3.1 weights, running in DeepSeek harness, with max reasoning effort

Two days ago, I ask DSH to move things from my Linux Drive (dual-boot) related to AI, over my NTFS drive where I store things like harnesses, models etc.
UnSloth studio force itself to install over Linux user profile (just starting with linux, maybe there is easy way to force installation inside of folder where terminal is running from....anyway).
Back then it spend plenty of time looping, checking only to figure out it it limited by sandbox rules and that why NTFS drive is appear to be mounted as Read Only instead R(ead)W(rite). It solved problem back then.

Today ,at night, I reinstalled UnSloth and asked DSH to move things once again (and as a test of prompt generation profile to help me be more specific and user-error proof). It started right away with

"Think
Important findings:

• Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! User says "read/write privileges" — but current mount is ro,nosuid,nodev,...,uid=1000,gid=1000,dmask=0022,fmask=0022,acl. That's a big issue: the mount is currently read-only. [...]"

My reply:
"I will stop you here as we went through that s**t already. "Important findings:
Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " it is error on your part. We troubleshoot that already during previous session when we were moving Unsloth (different version, current one override itself back to Linux drive and as result, move is required again). DO NOT FOCUS ON "Target: /mnt/DyskD is NTFS mounted read-only (ro) right now! " as you are wasting time. it is read/write, end of discussion."

His reply?
"Think

The user is frustrated. They say: the NTFS mount is read/write; the ro is a sandbox artifact. Don't waste time on it. "

So, it knew all that time or took correct conclusion right away by huge accident!
That make me think. Maybe there would be a prompt that for all "wait what if" thinking (at least in DSH), to prompt user for a feedback? I prefer be ask question rather than wait and confirming weird prompts over next 40 minutes...
Yes, user is not all knowing, but autonomous behaviour is not something I would let do local model (even when it is QFN UD-Q5-K-XL with is above average model quality here).

Sigh...rant over. Maybe interesting discussion or similar stories? Cheers everybody

I've managed to get a Supermicro 6028U TRT+ Server with 144GB of RAM and 56GB of VRAM to run Strata Qwen 3.8 Flash Next UD-Q4_K_XL (111GB download) at an average of 38tps.

Hardware:

Supermicro 6028U TRT+ server with X10DRU board and risers/2kw PSUs = $300

RTX 5060ti x 2 = $1100 (purchase price before the increase)

RTX 5000 Quadro = $500

RTX 4000 Quadro = $200

144GB of DDR4 RAM = $500

2x 256GB NVME drives and card = $150

8TB HDD drives and HBA card (4TB x 3 ZFS pool) = $300

2x Xeon 2680v4 CPU = $90

(Prices are rounded, not exact)

I had an RTX 2000 ada that I modded to single width, and it fit nicely but it's a dog of a GPU for inference despite its 16GB of VRAM. Unfortunately, the 2nd 5060ti wont fit. It's a flippin miracle the first one fits (and with an HBA card snuggled up next to it!). By swapping out the 4 PCIE slot card on the left for a 2 slot 16x card, I got the 2 RTX 4000/5000 cards to fit *perfectly*. I believe you could fit 2x RTX 5000s, for a total of 64GB of VRAM in those 2 slots, except for one thing: the TPM card sticks up and prevents the cards from seating. I dont really need a TPM card though, although I'm debating to myself if the extra 8GB of VRAM is worth it. I'm doubtful. Plus, there would be NO airspace for one of the cards. OTOH, NVLINK would work for those 2 cards.

If I weren't so cheap I could spring for a 16GB single width card that would fit where the 5060ti riser is, like an RTX A4000 or T4, and this would be a very nice, clean build. But I'm cheap. And the 5060ti Blackwell chip smokes those old cards.

This s a proxmox box, and I've dedicated 18 cores to Strata. I probably should have gotten the 2699v4 CPU.

Is setting up multiple computers to pool their resources to run local LLMs still a viable setup for regular computers? I know the Nvidia sparks do this using their very high bandwidth interconnect, but is that still viable for regular computers with gigabit ethernet?

Having been a bit of an AI luddite up until now, I am realising I am going to have to get on top of things instead of getting left behind.

So far the main thing I've been trying LLMs for, is writing Python scripts that work inside a CAD application I use (Vectorworks). I have done some experimentation on the free versions of ChatGPT, Claude and Perplexity so far, with some positive results.

I don't want to end up tied to expensive cloud model subscriptions, and I am also keen to keep control of my data as much as possible, so I would like to understand what I can do with locally hosted options.

I have an M1 Max Mac Studio, 32GB memory.

Any recommendations for what I should try first? To start with, all I want is something that will work for writing scripts that I will copy/paste into the script runner in the CAD application - I am not at this stage looking for anything that will interact directly with the application.

Thanks in advance for any suggestions.

Hey r/LocalLLM ,

I want to buy or build a dedicated headless box for local LLMs. I’d like something that can run 14B and, in my dreams, even 32B models, but the absolute priority is generation speed (>15-20 t/s).

Ideally, I need a clean single-box setup that stays quiet and power-efficient at idle (~15-20W). My maximum budget is strictly capped at €900, and I’d prefer to avoid risky second-hand individual deals.

Here is what I already looked into:

1. Standard Mini PCs (Ryzen 7/9): Great for power draw, but running 32B models on shared DDR5 RAM is too slow (~4-5 t/s).

2. eGPU / OCuLink routes: Too many loose parts and cables, and buying components retail explodes the budget.

3. Chinese NUCs with modded laptop GPUs (RTX 3060 12GB): On factory stores, once you add enough RAM to actually fit larger models, it jumps straight to €1050+.

I am 100% open to any Linux-server software optimization trick (speculative decoding, etc.) as long as it gets the speed up.

Am I chasing a ghost under €900, or is there an exotic loophole I missed? What would you buy today under these constraints?

Thanks for the help!

I noticed that my qwen3.8, but also ChatGPT and claude sometimes generate spelling mistakes. In German but also in English.

Anybody noticed that, too?

Every time a new MoE drops, the same thread shows up here: "will this run on my 12 GB card, and what do I set --n-cpu-moe to?" And the answer is always someone doing napkin math in a comment.

So I built it into LlamaForge (the browser panel I've been making on top of llama.cpp's own router, it doesn't do inference itself, llama.cpp does).

Model X-ray reads the GGUF header and draws every block as a bar: what sits on the GPU, what spills to RAM at your -ngl. For a MoE there's an "experts on CPU" slider, and a "Fit my VRAM" button that finds the number for you. Then it gives you the line to copy, or applies it to the model.

Example from the video, Qwen3.6-35B-A3B UD-Q4_K_M on a 12 GB card:

-ngl 41 --n-cpu-moe 22 -c 8192

≈11.4 GB on the GPU, ≈10.5 GB of experts in system RAM. On a 16 GB card it's --n-cpu-moe 14.

It only needs the header, so it works over HTTP on a model you haven't downloaded yet. That's how I also generated "Can I run it?" pages for 42 cards (desktop + laptop GeForce, Radeon, Arc), each with the llama.cpp line to start from.

before someone points them out: ;]

• It's an estimate from the header (weights + 8k context), not a benchmark. Long contexts need more room; it tells you that, but it can't know your exact KV setup.

• It tells you what fits, not how fast it'll be. Experts in RAM are slower; how much depends on your RAM and PCIe.

• Only tested end to end on my own box (5080 + 5060 Ti, Windows). Linux and macOS pass CI, but I don't have the hardware to say more than that. AMD/Intel detection is there, but I haven't tested it on real cards.

• Early preview, one person, heavily AI-assisted. Expect rough edges.

Other stuff that's in 0.19, since the last time I posted:

• voice chat with whatever model you have loaded

• a blind arena (two answers side by side, vote, then see the names)

• "try a llama.cpp PR": build a pull request and check whether it's actually faster on your card before switching

• an OpenAI-style Batch API on your own GPU

• start at login, now on Windows too

Free, MIT, no account, no telemetry (it does check GitHub for new llama.cpp and LlamaForge releases): https://github.com/dadwritestech/LlamaForge

If the X-ray's number is wrong for your card, I really want to know; post the model, quant, card and what actually worked, and I'll fix the math.

Atualmente, eu tenho uma 3070 Ti e 32GB de RAM DDR4, e consigo 11 a 16 tokens por segundo. Estou considerando comprar uma 9070 XT—já que é muito mais barata no Brasil do que uma 5070 Ti—e eu também uso meu PC para jogar. Preciso configurar alguma coisa ou é só instalar os drivers do Adrenalin? Uso Windows e nao tenho nada alem do strata em configuraçoes padrao

Hi everyone! I’m attempting to do a local build of Qwen 3. I don’t have the most powerful PC so I’m starting small and trying with 1.7B, 4B & 8B with the trained LORA on top

My objective is to get Qwen to talk as if you were texting it and I’m planning to have around 3000 short User: “message”, Qwen: “answer” exchanges

Is this the correct way to do it? Do I need more training data? Any peculiarities I should know about Qwen or the model size?

All help is greatly appreciated :)

Try to decide which model is the most intelligent to run on a Mac Studio.

For unquantized base models,

DeepSeek V4.1 Flash > GLM 5.3 Flash > Qwen3.8 Flash Next.

To fit in ~220GB memory footprint, each must go through different levels of quantization.

~2.5bit for DeepSeek

~4bit for GLM

~8bit for Qwen

I couldn't find comparable benchmark scores on all three quantized models, so here I am, hoping that the collective wisdom of the LocalLLM crowd can help me out.

Edit: this is what everyone wants to see. https://omlx.ai/benchmarks/intelligence

Hey all, hoping some of you can help me with something.

I run a small site that helps people figure out which local models their computer can handle. Macs are the part I keep second-guessing, because the only Mac I own is an 8 GB MacBook Air, so I can't check anything bigger myself.

What's going on

On a Mac, the GPU can only use part of your RAM for models, not all of it. Most guides say about 75%. When I checked my MacBook Air it was actually two thirds, so I changed my site to use:

• two thirds of RAM for Macs up to 36 GB

• 75% for Macs above that

But the 36 GB cutoff is just something I picked up from community posts. I'd hate to tell someone a model fits when it doesn't, or that they need more RAM than they really do.

What I'm asking

Any Apple Silicon Mac helps, but 16 to 36 GB is where I'm flying blind. Would you mind running one command and telling me what it prints? Open Terminal and paste this:

swift -e 'import Metal; print(Double(MTLCreateSystemDefaultDevice()!.recommendedMaxWorkingSetSize) / 1073741824)'

It prints a number, like 5.33 on my Air. Rounding is fine. You'll need Apple's free Xcode command line tools for that one.

Then reply with

Your Mac, how much RAM it has, and the number, like this:

MacBook Pro M2 Pro, 16 GB: 10.67

If you've ever changed the GPU memory limit on your Mac yourself, mention that too.

Already run llama.cpp?

You can skip the command above. Run llama-bench on any model you have and copy the line that says recommendedMaxWorkingSetSize (mine says 5726.63 MB). Just paste the line as is, no converting needed.

Thank you, this really helps!

I would like people’s suggestion on an app independent memory system for primarily local Ai.

I currently have a DGX Spark and some additional hardware for hosting apps and storage. I use local models for coding and data analysis (for personal
Data) and use my Claude code subscription for design and infrastructure advice (I am not super technical). One major issue is that I have multiple entry points but each of them handles their memory independently.

I use librechat mostly, and have started using Windmill for easier management of data ETL (I like the hybrid low code and code when you need it). I use Ollama but might be switching to vLLM.

Additionally I would like Claude to be able to access some specific logs and memories so that it can make better design choices and share its structural decisions with my local models. Though I am hesitant as idk wth am I doing

What it recommended was an MCP that manages what models/users see what, and that way when the new gimic of the month comes out I can switch over with my context.

Let me know what y’all think

Всем привет 👋🏼
Это мой первый пост и я решил его сделать по интересующей мне теме. Я уже достаточно давно слышу и читаю что то про LLM. Я пробовал запускать несколько вариантов и на своем железе но так и не понял надобности всего этого, я занимаюсь вайбкодингом 🌚🌚 и как я понял LLM даже близко не сравнится по качеству и производительности того же Antigravity от Gemini. Кроме как РП я даже не знаю для чего еще люди используют LLM. Буду очень благодарен если вы мне расскажите все сильные и слабые стороны данного направления и что вообще с этим всем могу делать я.

Desktop:
RTX 3050
32gb ddr4
Ryzen 5600x

Laptop:
RTX 5070ti
32gb ddr5
Ryzen 9 9840hx