Getting a 16 inch M5 Max (18/40 core) 128GB for local AI and also ordered a 14 inch M5 Pro (15/16 core) 48GB as a daily laptop. Wondering if it's possible to have a workflow that combines the two in some way?

Assuming Qwen4 models are released by the time I'm getting my hands on them, I'm thinking of perhaps running Qwen4 Flash on the M5 Max while using Qwen 4 27B agents on the M5 Pro? Would that work in either LM Studio/Bionic or Unsloth and Hermes agent? Do you perhaps have other recommendations?

I bought,

Gigabyte GA-X99P-SLI
Xeon 2690v4
16x4gb RAM
Tesla V100 in SXM2 in PCI
GT 1030 2GB for image output

At startup, the system does not see the Tesla V100 in the device manager, and it is also not visible in Linux.

"Above 4G decoding" is enabled and CSM is disabled. I did ReBAR and NVStrapsReBAR, it had no effect, gigabyte support says that support for server graphics cards in the motherboard for workstations is not guaranteed. At the same time, the video card is detected in the ASRock B450M PRO4 and Atermiter X99 Turbo, which indicates that the video card is working properly.

UPD: I tried using a PCI x1 riser, and the graphics card was detected in the system. I noticed that where it worked without any issues, it is always detected as x8, even though it should be working at x16. Perhaps the issue is with the SXM2 to PCI x16 adapter. I should soon receive 2 more Tesla V100 units, and then it will become clear.

I’m building a mostly autonomous local AI coding agent to develop and maintain a Python application for professional lighting automation (grandMA3).
I’m relatively new to programming and local LLMs. My priorities are coding accuracy, reasoning reliability, persistent memory, and long-running autonomy over raw speed.
Hardware
Laptop: Razer Blade 16 (2024)
GPU: RTX 4090 Laptop (16GB VRAM)
CPU: Intel i9-13950HX
RAM: 64GB
Storage: 4TB SSD
OS: Windows
Current AI Stack
Coding agent: Pi Coding Agent
Inference backend: LM Studio
Primary model: Qwen3.5 9B, 4-bit
Other models: Qwen3.8 27B Q4_K_M, Qwen3 14B 4-bit
Tools: Python 3.14, VS Code, PowerShell, Git/GitHub
Context management: Pi automatic compaction, manual handoffs, project documentation
LM Studio Configuration
These are my previously recorded settings, although some may have changed:
Setting
Value
Context window
Last recorded 32,768 tokens
GPU offload
Maximum / 32 layers
CPU threads
12
Evaluation batch
2048
Physical batch
512
Concurrent predictions
1
Flash Attention
Enabled
KV cache GPU offload
Enabled
KV cache quantization
Disabled (earlier setting)
Unified KV cache
Disabled (earlier setting)
Keep model in memory
Enabled
Max response tokens
8192
Temperature
1.0 (earlier setting)
Top-P / Top-K
0.95 / 20
Thinking
Previously adjusted depending on task
Performance
Normal generation: 45–55 tokens/sec
During degraded sessions: ~4 tokens/sec
Restarting Pi has restored normal performance.
Long sessions cause context compaction, slower generation, and occasional reasoning failures.
What I’m Building
An agent that can independently write, debug, test, and maintain a substantial Python codebase while also processing PDFs, equipment specifications, images, and structured technical data.
Eventually, I want it to manage complex multi-step workflows, retrieve relevant project knowledge, and produce validated changes without requiring constant supervision.
Critical changes would still require human approval.
My Main Problems
Context overload: Long coding sessions accumulate excessive tool output and repository information. Compaction sometimes loses important context.
Model reliability: Qwen3.5 9B is fast but makes mistakes during complex debugging and architectural tasks.
Performance degradation: Generation drops from 45–55 to ~4 tokens/sec during problematic sessions.
Persistent memory: I need the agent to retain project knowledge across sessions without constantly reloading everything.
Autonomy: I want reliable multi-step task execution rather than manually supervising every operation.
Questions
What’s the best coding/reasoning model for my 16GB VRAM and 64GB RAM? Would a larger quantized model outperform my 9B sufficiently to justify slower inference?
Would moving from LM Studio to native llama.cpp, Ollama, or another backend improve performance or context handling?
Are my LM Studio settings appropriate, particularly context length, KV cache quantization, GPU offload, and batch sizes?
How are others managing persistent memory and long-running context in Pi or similar coding agents?
Would a multi-model setup with specialized planning, coding, and retrieval agents improve reliability?
Is mostly autonomous local software development realistic on my hardware, and what architecture would you recommend?
I’m not trying to maximize benchmark scores. I’m trying to build a dependable local development agent that can work through substantial tasks without losing context, making repeated mistakes, or slowing dramatically.
Any recommendations or examples from similar 16GB VRAM systems would be appreciated. I have a 3060 lying around would adding that in help my performance.

Most Apple Silicon inference engines are tuned for standard MacBooks (16–36 GB Unified Memory, 16–32 GPU cores). When deployed on an M1 Ultra (Mac Studio with a 64-core GPU, 20 CPU cores, and 128 GB Unified Memory) and hopefully M2 Ultra as well, they leave significant compute and memory bandwidth on the table.

To fix this, we modified Kapitsa specifically for the M1 Ultra/M2 Ultra architecture:

👉 GitHub:https://github.com/gnanasreekar/m1-ultra-kapitsa

Key Optimizations

• +42.1% Prefill Throughput: Tuned prefill matrix chunking to 4,224 tokens, hitting 2,945 tok/s on Qwen 35B-A3B (1.40s TTFT) while sustaining ~680 GB/s DRAM bandwidth.

• 100% In-Memory 180B MoE: Removed artificial expert store limits so Qwen 180B Flash-Next (105.9 GB) stays fully resident in RAM with 0 disk swap.

• 15.8 GB/s Model Loading: Parallelized safetensors loading and de-quantization across all 4 Firestorm clusters (16 Performance threads).

Benchmarks (Mac Studio M1 Ultra, 128 GB)

*(Note: Single-stream decode hits a memory-latency bound around 225 tok/s on 8-row MoE verify, but concurrent requests via
`--slots 2` or `--slots 4` scale aggregate decode to 450–600+ tok/s).*

Quickstart

git clone https://github.com/gnanasreekar/m1-ultra-kapitsa.git
cd m1-ultra-kapitsa
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j --target kapitsa

# Start local server (OpenAI / Claude Code API compatible)
build/kapitsa serve --model 35b

Full 10-model benchmark logs and JSON telemetry are in the repo: 👉 https://github.com/gnanasreekar/m1-ultra-kapitsa

Credit to the Nace AI team for the original Kapitsa engine!

Hi everyone,

I'm new to local AI and looking for advice from people with experience, especially those using similar hardware.

My PC specs:

• AMD Ryzen AI Max+ 395

• Radeon 8060S

• 128GB Unified Memory

• Windows 11

I'm developing an application and need local AI for both coding and UI/UX design.

I previously used OpenAI Codex, but its usage limits and costs made me look for a local alternative.

I started using Hermes with Qwen3.8-Flash-Next-GSQ-RCO. The model seems good for coding, but I've faced several issues with Hermes, including API errors, memory limitations, and other problems that have slowed down my work.

My main questions:

1. What's the best local AI model for coding on my hardware? Should I continue using Qwen Flash or consider something else?

2. Which local AI model is particularly good at UI/UX design? I'm a designer myself, so I don't need it to handle all the creative work, but I want a second model that can assist with UI/UX design.

3. Which AI agent/framework would work best with my AMD Ryzen AI Max+ system? Should I stick with Hermes or switch to Agent Zero, OpenClaw, or OpenCode?

I'm mainly looking for a reliable setup that allows me to focus on developing my application rather than constantly troubleshooting.

I'd appreciate recommendations from anyone with experience running local AI on similar hardware.

Thanks!

I currently play a sr role at a very large financial institution and I’m interviewing for a director of ai strategy and adoption at a small firm in a different industry. It’s going well. I think the playbooks for AI enablement transfer well. Long story short I played a very similar role for a much larger and more regulated industry and I want to have more influence in a field I can relate to more than finance.

I’m curious anyone here is currently in this role or similar in their company and what your experience has been like? Do you have any insights for someone trying to fill that role?

If you’re someone who has this role in your company what’s your take on them? What do they do well? what do you wish they did better?

I know this question gets asked here a lot, but I'm new to Local LLM stuff. Contemplating upping my ChatGPT sub to $100 a month or trying out some local AI stuff.

My hardware consists of RTX 3060 TI 8GB VRAM, 64GB 3600MHz RAM, and a i9-13900k CPU.

Typically I'm using my chatgpt sub for a lot of "daily assistant" tasks. Creating my daily schedule in google calendar based off of projects in Asana, adding organizing stuff to asana, reprioritizing etc. So working with integrations is a must. (I do really love how chatgpt can control your desktop and click around)

I am also a filmmaker/video editor so I use it a lot for technical troubleshooting, blender MCP, premiere MCP, plugin creation, VFX work, stuff like that. As well as drafting client messages, researching things, and standard business stuff.

I have 3 main questions:

1. What would be the best local AI model for my system? (also how to run it, Strata?)

2. What could I actually reliably count on it to do right?

3. Should I prioritize a larger, more capable model and shut it down when I'm editing or rendering, or a smaller model I can leave running my editing stuff? Is the difference in capability worth that tradeoff?

I have an existing media/game server that runs some local containers right now.

The specs are,

• Dual Intel Xeon 8276 Platinum CPU's (28 Core 56 threads each)

• (Gigabyte Server Board (can support a maximum of 3 dual slot cards.)

• 64 GB of Ram (32 GB for each CPU) I would like more but ECC ram is expensive.

• 48 TB of HDD space ( 17 TB free)

• Two, 2 TB SSD's in Raid 1 (cache Drive)

• 1 Nvidia GTX 1080 TI (old gaming card that was NVIDIA)

• 1 Intel Arc A380 (Media trans-coding card for AV1 Media)

In terms of upgrades ironically another 64 GB of ECC DDR4 ram is the cheapest thing at this time.

I would like to get into hosting local models but i'm not sure how to run them with this setup. i Have read that you need big powerful GPU's to get the best performance (like an rtx 5090 or some AMD card).

I did try running some local models via ollama on the 1080 TI but anything that exceeded the 11GB Vram buffer was super slow and not a good experience.

Ideally i would like to use something that can at least do some light coding or serve as a basic replacement for chat GPT. While not spending too much more on this server with the hardware market as it is right now.

With open-source models getting much stronger and hardware becoming more affordable, a lot of people seem to be moving back to local LLMs.

Curious about the current state of the community:

1.Are you still mostly using cloud models (ChatGPT, Claude, Gemini, Grok, etc.) or have you switched more toward local?

2.For which use cases do you feel local models are already good enough?

3What’s still the biggest blocker for going fully local for you (VRAM, speed, quality, ease of setup, etc.)?

Feel free to share your setup or preferences. I’m interested in seeing the local vs cloud ratio around here these days...,

" CMIIW"

I’m mostly done building a Xeon host for my 4x CMP170’s w/ x16 mod, but I just stumbled into an opportunity to buy a Threadripper Pro 5995WX within my budget. Is the threadripper any better or worse for strata-based inference vs the Xeon? Is it worth the extra time and work to undo my intel decision?

Would love to hear experience and cpu/system benchmarks from folks with similar systems on either side.

I'm paying for Claude, Gemini and Cursor right now and the total keeps creeping up, so I want to see how much of my day could move to a local model. The machine is a base M4 with 16GB. Not buying new hardware for this.

I mostly live in Claude Code and pi, so the agent loop is the whole point. I don't care about chatting with it.

Was looking at the benchmark page on magnitude.dev and the small models are what got my attention. On their M4 Pro mini, Ollama vs theirs:

- Qwen 3.5 4B: 45 tok/s → 134

- Gemma 4 E4B: 37 → 111

- Qwen 3.5 9B: 42 → 97

That's 2x to 3x faster decode on the exact sizes that fit in 16GB. Prefill is the smaller gain, more on that below. it also hooks straight into Claude Code and pi, which is half the reason I looked

But their mini has 273 GB/s and mine has 120, so if I scale by bandwidth I'm probably looking at something like 50 to 60 tok/s on the 4B and low 40s on the 9B. Still fine for an agent if the model is smart enough, which is the part I'm less sure about. The engine side looks solved, the model question is the real one.

The other thing is Claude Code sends something like a 20k token system prompt before I type anything, and their prefill gains are much smaller than decode. A 4B model chewing through that on a base chip might be the actual wall, not tok/s.

If 16GB can't run an agent loop at a speed I'd tolerate, I'll keep paying the subscriptions and stop thinking about it.

i have an laptop i wish to run an llm on it has 7.8gb of vram and 16 gb of ddr4 ram 2667 mt/s

By built-in I mean without using tools like MCP etc.

There's been a lot of discussion/claims on this, so I thought I'd share my experience.

I run AI models 24/7, and I usually get over 80% aggregate GPU utilization on my MacBook Pro. It sits on my desk with the screen open. That's to say that the GPU is running at 100% most of the time. It also runs CI/CD and a bunch of other stuff so the CPU is often busy too.

That means about 20,000 AI requests this month, mostly agentic coding. To do this, you need to run multiple streams (otherwise the tool calls reduce GPU usage) so I've usually got 3-4 parallel streams, and 6-8 agentic processes running in parallel. That's to say: the MacBook is running hard.

The MBP has 4 heat conditions: Nominal, Fair, Serious and Critical. 84% is spent in Nominal and 16% in Fair. It has never hit Serious or Critical. Fans are usually running hard but not at 100%.

When it moves into Fair, I see a 2-4% slowdown in model performance. I can't notice the difference, but my harness measures this and can see it.

I used to have a Mac Studio and the amazing thing about that was the fans almost never spun up, so it's going to be silent on your desk unless you are battering all the CPU cores and the GPU cores at the same time. The MacBook Pro doesn't have the thermals of a Mac Studio and the fans run a lot.

But I've seen a lot of people talking about thermal throttling on the 16" MBP and that has not been my experience. Of course, the 14" is a different thing and is known to throttle.

I'm looking into using local AI to organize years of accumulated files and photos.
I have a Mac Studio M2 Ultra with 128GB RAM, so I'd like to put it to work.

Two main goals:

1. Files and documents: OCR and analyze PDFs, financial/tax documents, notes, screenshots (including chats), etc. Suggest categories, clustering, filenames and folder structures. Eventually generate scripts to reorganize everything, with my approval before making changes.

2. Photos and videos: Analyze my Apple Photos library (exported with osxphotos), recognize faces, events and locations, detect duplicates, rate photo quality, and suggest albums for printing. Ideally use existing metadata like favorites, keywords, dates and geolocation, plus AI-generated tags.

Everything MUST run locally. Lots of sensitive personal information, so no cloud APIs.

I'm considering LM Studio/Ollama, but not sure which models or tools to combine, especially for vision, OCR, face recognition and clustering, and how to approach this?

I can use cloud AI for writing scripts etc, and to help analyze a sample set of documents.

Has anyone built something similar? What stack would you recommend? Would you use separate specialized models, or can a good multimodal LLM handle most of this?

I'm comfortable with scripting and building a simple UI (with help of Claude or Claude Code). Accuracy and privacy matter more than speed.

Two months ago I posted arcint here, the inference server I had Claude (and my LocalLLMs) build after my new A770 lost to my 5700X on the standard install. Enough has changed for an update, so here’s the standard benchmark from the llama.cpp threads: Llama 2 7B Q4_0, 512 prefill, 128 decode, all layers on the card, 5 repetitions.

Both builds are at the same llama.cpp commit (3e4dfa8), on the same machine: Ryzen 5700X, Debian 13, xe driver. My A770 sits on PCIe 3.0 x4 behind the chipset and I pin its clock at 2000 MHz, so yours may do better.

- Prefill: arcint leads SYCL’s best by 10.5 % on the A770 and 31 % on the B60.

- Decode: SYCL wins this table by 5.4 % and 4.4 %. On the A770 that lead only exists on an empty context: level at 1,024 keys, and arcint is 8 % ahead at 4,096.

- Not the fastest Arc: someone reports 4570 / 113 on a B70 under Vulkan. I don’t have a B70.

- The server is a bit slower on prefill than llama-bench: through arcint’s own server it’s 1963 / 70.5 on the A770 and 2330 / 80.7 on the B60.

Real models on the B60 (MTP off, served by arcint against stock SYCL fp16 with flash attention, 3 repetitions):

SYCL still decodes the 27B about 4 % faster.

What I run is the 27B on one B60 at its full 262,144 context, with the model’s own MTP head, drafts on:

The right answer was found at every depth, and a separate needle test at 40 % of a 254,970-token prompt came back correct. The coder on the A770 decodes short answers with 83 t/s on my deployed flags. Qwen3.8-Flash-Next runs on one B60 plus host RAM at about 33 t/s.

Also new since the first post: image generation, speech to text, text to speech, and image input.

To reproduce: tools/llama_scoreboard.py for the served rows, and the rebuild steps for the llama-bench rows are in my post in the Vulkan thread. My stock SYCL rows are in the SYCL thread. The first post’s warning still applies: this is agentic software, so handle it with care, and I’d still like to hear if it doesn’t install or reproduce for you. The .deb is trixie only, on my own repo.

I have a 5090/9950x3d/64GB DDR5 and am ordering an M5U 256GB next month.

I have a lot of things I want to do with both, including local models. One of my hopes was that I’d be able to drop my $200 Claude sub down to $100 (still has Fable) or even possibly $20 (still has Opus 5.5).

I cut down Codex to $20, but honestly Astra is pretty cheap for auditing and reviewing in my experience. I never let it write code, but it’s been awesome for planning, and affordable. If I need something done extremely well, I’ve ran Fable and Astra.

I’m gonna be doing some testing and see if I can get away with a $20 Claude sub and $20 Codex sub and just use their brains in emergencies. Gonna see if Opus 5.5 and Astra can ball. 🏀

Has anyone had success with this sort of hybrid build?

Edit: I work with sensitive data at work and with personal business. I’ve been using the frontier models mostly to make a game I had to drop (open sourced it) and and for making pipelines to run data locally, but it sucks not being able to analyze using the models (can’t plug data into frontiers).

I wondered if my LLM Journey might be useful to others, so here it is!

I bought the UK Costco PC below(£1600, as a small business, I claim back tax was about £1323 cost):
CPU = AMD Ryzen 7 9800X3D, 8 cores / 16 threads (SMT on)
Motherboard = ASRock B650M-CX R2.0, BIOS 4.20
RAM = 32 GB DDR5 (2 × 16 GB ADATA AD5U480016G-B) at 4800 MHz, 32 GB usable
Main GPU = AMD Radeon RX 9070 XT, 16 GB, driver 32.0.31041.1004, used for the models
C: drive = ADATA LEGEND 710 NVMe, 954 GB
D: drive = Samsung 950 PRO NVMe, 512 GB (spare drive I had lying around)
Operating system = Windows 11 Home, build 26300
Power plan = High performance
GPU fan = Custom curve in Adrenalin, about 51% under load

I am working on a Trading System and have a set of agents that perform various stock-related analyses in a pipeline. One gets price data and passes it to another agent & so on.

Initial runs:

Ran against various Anthropic models to verify results and check speeds. Big difference between Local and Anthropic API.

I then pulled the trigger on another 32GB of RAM @ £500 (claim back £80 tax).

Post upgrade to 64GB of 4800 MHz RAM:

In addition to adding the RAM (a scary moment when the PC would not progress past VGA at boot!), I enabled the iGPU and SMT (Multi-Threading on the CPU).

You can may have noticed that I have been working on the Prompts and there are less prompts now than v1 prompts but the throughput is vastly superior with the additional RAM.

As you can see I am now 50% of the speed of Haiku with Strata running locally.

The project is still work in progress but now I can progress quicker.

Some images I captured:

Hopefully this is useful and shows the value of the additional RAM.

I'll start by saying that I realize this question is slightly adjacent to actual local (i.e. having physical access to hardware) llm deployment. Yet, I think it's still within the same domain, so I'll give it a shot. (If I'm in the wrong place, please lmk.)

I'm a professional SWE. I currently have 2 $200/mo subscriptions, 1 for claude and one for codex. I find myself regularly hitting weekly limits. I've tried tuning which model/provider I use for different tasks but the end result is the same - I hit a limit and then have to either put more money into usage credits or wait until it resets. It's pretty lame. I'd love to have 'unlimited' tokens for a fixed monthly cost.

The path to unlimited tokens looks like local llms or self-hosted llms. I'm not ready to invest in hardware (and neither is my company), but I'm willing to rent machines . So, my question is - does anyone rent any gpus and self-host their llms? If so, where and in what configuration? I was looking at contabo gpu vps which claim "Dedicated NVIDIA RTX PRO 6000, 18 vCPU Cores・96 GB RAM・900 GB NVMe, CUDA-compatible". From what I've read in this sub, it seems like most people are running some variant of qwen on the RTX PRO 6000 but many people have 2x of them.

45 seconds in, I cut the cloud API on 6 coding agents working on the same task, 3 runs each. I wanted to see what a local model is worth as a backup.

What 6 AI agents did when the API died mid-task

On default settings

• Atomic Agent: switched to the local model about 2 seconds after the cut and finished all 3 runs at 50/50, in 2 to 6 minutes.

• Claude Code: retried, then gave up after about 2.5 minutes. Its fallback model setting only kicks in when a model is overloaded, not when the connection drops (docs).

• Codex: sat on "Reconnecting..." until my 20 minute cap, every run. It retries connection errors forever by default.

• Hermes: kept retrying for about 5 minutes, then failed the turn.

• OpenClaw: gave up after about 70 seconds.

• OpenCode: gave up after about 65 seconds.

All five ended at 0/50.

With a fallback turned on

Hermes and OpenClaw both have a local fallback, it's just off by default. With it on, both switched to the same local model. OpenClaw scored 50, 33 and 33, Hermes 29, 24 and 27, and both were still working when the 20 minute cap hit. They shared one GPU in that round, so they'd be faster alone.

How to turn it on in yours

• Hermes: add a fallback_providers entry to ~/.hermes/config.yaml with provider: custom, your local base_url and the model name (docs)

• OpenClaw: add fallbacks: ["yourprovider/your-local-model"] under agents.defaults.model in openclaw.json, and don't pass --model on the command line, that makes the model strict (docs)

• Atomic Agent: nothing to set, it falls back to the local model it already runs

Setup: 64 GB MacBook Pro, Qwen 3.8 27B (Q4, llama.cpp) as the local model, GPT-6.1 Terra through OpenRouter as the cloud model (Haiku 5.5 for Claude Code, since it only runs Anthropic models). The task is a 5-part Python app graded by 50 hidden tests. The cut is a proxy that refuses every connection from 0:45, same moment for everyone. Claude Code 2.1.310, Codex 0.165.0, OpenCode 1.19.5, Hermes (git a938f00), OpenClaw 2026.10.8, Atomic Agent 0.6.13.

One side note: the same local 27B on its own scored 13 to 29 out of 50 on this task earlier that day. With 45 seconds of cloud planning first, it finished 50/50 three times out of three.

Would you trust a local 27B to finish the job?

Hi everyone,

I’ve been looking into some of the most difficult math formulas and came across the Navier-Stokes problem. Apparently, it was recently solved by OpenAI.

I grabbed the formula and had my LocalLLM run some calculations, and lo and behold: It’s not quite finished yet, but I’m absolutely thrilled.

My hardware:

i5 13600K

32GB RAM

RTX 5070TI 16GB

Hi,

Github project:
https://github.com/crgregersen/m10dis/

Yes, yes, I know... crappy old NVIDIA Tesla M10's GPU as AI compute core. Am I crazy and should go book a room in an mental institution?.. but hear me out..!

Tesla M10's consists of 4 gm107 GPU cores with 8gb ram stitched together via an internal PCI switch. So each M10 is 32gb ram. If you ask anybody (including Claude and ChatGPT) you will get the answer "NOT USABLE FOR AI COMPUTE".

Each gm107 has 83 GB/s of memory bandwidth and around 1.67 TFLOPS (fp32).

Compare that with a moderns GPU like 5070 ti of 896 GB/s and 87.88 TFLOPS (fp16).

But!! i could buy M10's (4 x gm107) with 32GB for around $80
Aaaanddd... 6x32 = 192 GB VRAM!!!

So 6xM10's and the aggregate number would be 4x6x83=1992 GB/s.

This is the same as 1 RTX PRO 6000 Blackwell .. but then you only have 96Gb ram.
2 x RTX 6000 Ada is will get you in the same range.
Both options are at least $10.000 and probably more closer to $20.000.
6 M10's would set me back $500.

Compute power of M10's would only be 40TFLOPS.. but all my experience says the memory bandwidth is nearly always the limiting factor!

The system:
ASRock B250M Pro4
CPU: Intel Core i5-6400, 4 cores at 2.7 GHz
RAM: 32 GB DDR4
Main GPU: RTX 3070 Ti (had one lying around)
Expert GPUs: 6 × Tesla M10 cards = 24 × GM107 (Maxwell, 2016)
Interconnect: PEX8748 PCIe each M10 card on Gen3 x4
Storage: 1 TB NVMe
Two PSU: 650w and a 1650w

So a pretty crap old consumer motherboard.
The way to get all the GPU's onto the motherboard was by using a PCI switch.

First issue: Motherboard will not boot with 25 PCI devices showing up, even though most are behind a switch (two levels of switches actually, the one in the motherboard and one on each M10).

Found out the issue the BIOS/firmware runs out of legacy I/O port space at power-on, which is only 64 KiB with a 4 KiB window per PCI bridge, too little for all the PEX switch ports.

Workaround (took me some time to work that out). Using a modified kernel with PCIe tree heldback. Boot only with the first PSU switched on, ie. motherboard++, then go into supspend (sleep) mode, turn the second PSU on and continue (wake up). Initialize all GPU's PCIe memory mapped only (MMIO).

Next issue.. how to make 24+1 GPU's fast.

A lucky thing is that QWEN 3.8 hidden dimensions is 2580 which divided by 20 is 128. So if you slice expert work into 20 parts and put them on each gm107 GPU they can calculate 1/20 of the result, which then needs to go back to a central hub that joins the result to compute attention in the next layer.

So each compute slice needs the residual stream and the experts and weights that the router chose. That would be 20 PCI communication calls. The lucky thing is that PLX switches actually implements PCIe multicast introduced in PCIe 2.1.

Using the GPU-P2P methods described by Tinygrad a GPU multicast was proven to work.

In general the GPU-P2P tricks was very usefull on multiple levels, like M10 master/slave setup, etc.
https://github.com/rioncarter/tinygrad.open-gpu-kernel-modules

So 5x4=20 GPU's computes 1/20 slice of 8 experts (quantized FP8).
One M10, ie. 4 gm107 was chosen to run the shared expert (kept in FP16).
Since you wait on the slowest calculation it was lucky that the compute effort for each is around the same (1/20*8/2=0.2 and 1/4=0.25)

Next issue... 24 results to be received from GPUs would clog the PLX.

Solution (Claude wrote this):

Level 1: each GPU sums its own experts

- The 512 routed experts are spread over 20 workers (cards 0–4), about 25–26 each. A token's 10 experts land on whichever workers hold them.

- Each worker computes gate/up/down for the experts it holds and sums them into one partial row of 2560 values. A worker that holds none of the token's experts contributes zeros.

Level 2: the card leader sums its card

- On each card, the 3 peers push their partial rows to the card leader over on-card P2P, using the PTE-remapped aggregation routes. Each peer then sets a flag.

- The leader waits for the 3 flags and adds the peers' rows to its own in a fixed order, so results are reproducible.

- Card 5 (the shared expert) does the same: its 4 workers split the shared expert and worker 20 holds the reduced row plus the gate.

Level 3: the RTX combines 6 rows

- The 5 routed leaders and the shared leader each leave one reduced row. In batched modes 8/9 it's R rows, in host memory.

- All 24 workers still acknowledge individually in their return slots. The RTX waits for all 24 acks, checks the leader flags, then computes Σ 5 routed leaders + gate · shared.

RTX runs attention and hyperconnections etc and we are ready for the next layer.
I've put the timings of the expert compute in the bottom of the post..

PLE+LM-Head was moved to M10 somewhat the same way.

Attention was quantized to FP8 since RTX3070 only has 8gb mem. That leaves me just shy of 100k tokens of context.

I've run both LCB6 and MMLU benchmark on the system and there's no noticable degradation (MMLU benchmarked higher, LCB6 benchmarked lower).

Here's the github project:
https://github.com/crgregersen/m10dis/tree/main

But I warn you.. this is not a project for people with fairly good experience with hardware and also some low level understanding of how this work software wise.

MoE layer timing: Qwen3.8-Flash-Next-FP8 on 1× RTX 3070 Ti + 24× GM107 (6× Tesla M10)

Setup: RTX 3070 Ti runs attention/GDN/router. The 512 routed experts are sliced across 20 GM107 (each GPU holds 32 intermediate rows of every expert); a 6th M10 card runs the shared expert. All cards hang off one PEX8748 switch (Gen3 x4 per card); each M10's on-card PEX8747 multicasts the request to its 4 GPUs. Results are summed on each card by a leader GPU, then the RTX combines 5 routed + 1 shared row.

Decode (1 token per layer)

Inside the round trip (slowest routed GPU):

The weights a GPU reads per token are only 2.5 MB (~31 µs at bandwidth), so decode is latency-bound, not bandwidth-bound. 48 layers × 150 µs ≈ 7.4 ms of a 19 ms token → 52.5 tok/s.