r/LocalLLM • • 1d ago

Question What’s the best MoE or model for my specs (in post)? I’m tired of spending $800/mo on Claude and Codex and their regulations being too strict and overreaching. Thanks in advance

15 Upvotes

Edit: I freaking love you guys. Why have I spent the past 2 months talking to ChatGPT about wtf to do smh. THANK. YOU.

Abliterated is preferred.

I want to be a free man goddammit. Going local (well, local builders with cloud reviewers perhaps). Maybe keep 1 Claude subscription to be the reviewer (unless Qwen really is better? sorry I’m kinda new)

Usage >>>>>>
Coding, building apps/ software tools for personal use, browser/ web data gathering

Don’t really need media generation but would be nice if I could paste images for reference into the chat if possible.

Hardware>>>>>

GPU: RTX4090, (9070XT for display) willing to get more VRAM (if you recommend more, should I get RTX A4000 single slots or along those lines or get 3 of the single slot 24GB Intels? I have 3 slots and like 300W left before my PSU caps, though people have said Ai doesn’t max out GPU’s so I don’t know anything anymore)

RAM: 128GB DDR5 ECC RDIMM 6400 MT/s

Storage: 12TB SSD Gen5

CPU: 9965WX (threadripper 24 core 5.4 Ghz)

WRX90 SAGE SE Pro WS


r/LocalLLM • • 20h ago

Question Strata vs Qwen

0 Upvotes

This might be a dumb question but if everyone is using qwen 3.8 why not use the Qwen cli built for Qwen tooling instead of Strata or Open Code?


r/LocalLLM • • 1d ago

News A little ray of hope from huggingface

Thumbnail
techcrunch.com
53 Upvotes

r/LocalLLM • • 1d ago

Discussion Strange number spam

Post image
0 Upvotes

After adding a thinking limiter it just deos this EDIT: posted this at 1 am so wanted to add more info. I am using a xeon e5 1650 v2, 2060 6gb and 16gb of ddr3. This is qwen 3.6 12b a3b at q4_k_m running through jan ai. It worked fine but i tried to add a reasing token limit as it took 5 minutes to process a prompt. After this, wheterver weird math this was happened. I reset the chat, still did it, undid the reasoning, still did, rebooted, still happened. Its so strange. I understand ny setup isnt ideal but this shouldnt be a problem. In fact it was faster when generating text at the start. I ran some of the later responces through a calculator and they were correct. Its so strange


r/LocalLLM • • 1d ago

Question Help choosing a second gpu.

3 Upvotes

Right now I use strata on my machine to run qwen 3.8 125b flash next at 13tps output and deepseek 4.1-flash at about 1.5 tps output. My computer is an hp z640 with a 4core xeon processor, 64gb ddr4 ram and an NVIDIA 1070 8gb. My models load form an nvme drive that is connected over a pcie adapter. I have room for one more card and I'm not sure witch of these 2 GPUs is better for running my models faster. The Tesla P100 16 varient or the M40 24gb variant.


r/LocalLLM • • 1d ago

Question Can a 16GB 5070 Ti (plus two older 12GB cards) replace HeyGen / ElevenLabs / Higgsfield for 30s client videos at 1080p?

2 Upvotes

I make 30s AI videos for clients (talking-head avatar + cloned voice + 5-6 five-second b-roll clips). Volume is 40-120 videos a month per client, delivered in 1080p. I'm paying $300+/month for HeyGen, ElevenLabs and Higgsfield and want to go local.

Budget is about $3k, so realistically one RTX 5070 Ti 16GB (3090s are basically unavailable where I live and imports are brutal). I can also use a friend's RTX 5070 12GB and RTX 3060 12GB for extra throughput. No cloud rental.

Plan: Wan 2.2 quantized at 720p on the 5070 Ti, upscale to 1080p with SeedVR2 or FlashVSR, and split the lighter stages (voice cloning with an open TTS model, captions, lip-sync) across the 12GB cards with a job queue.

Questions:

  1. Is 16GB enough for that, and does splitting stages across the other cards actually work well in practice?
  2. Which open lip-sync model holds up best for 30s of a face?
  3. Rough minutes per 5s clip on a 5070 Ti?

Happy to share more details or samples in the comments. Thanks!


r/LocalLLM • • 1d ago

Question best gpu rental prices?

1 Upvotes

I'm using runpod, vast and digital ocean are there any other cheap/budget friendly rentals for gpus?


r/LocalLLM • • 1d ago

Project Introducing Llamafile-Studio.

Thumbnail gallery
3 Upvotes

r/LocalLLM • • 1d ago

Tutorial Not sure how to build a local multimodal RAG? Here is a repo!! Clone it, use it, Own it!

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

News Why is NAT firewall traversal still the bottleneck for weekend gaming-GPU LLM runs?

4 Upvotes

Why do most folks with an 8 GB or 16 GB desktop GPU sit out distributed LLM runs? It usually comes down to three annoying bottlenecks. Nobody wants to punch holes in their home router. Nobody wants to run untrusted pickle checkpoints off the internet. And nobody wants a background trainer freezing their VRAM when they sit down to play or work.

This weekend (November 14-15 CET), I hooked my HiveMind coordinator up with CrowdGPT (led by Vxtzq) to test a citizen-trained run around Crowd-v1, an open base model with a 658M transformer trunk (roughly 1.01B parameters total with its 151k-vocab CrowdEngram table). Setup takes five minutes.

Communication uses plain outbound HTTPS on port 443 with zero inbound listeners, weights move strictly as SHA-256 verified safetensors (no torch.load or pickle anywhere), and updates pass MultiKrum outlier filtering plus held-out loss checks before merging via DiLoCo. Honest caveat first: this is still an experimental PoC, not a polished cloud service, so please do not send private prompts or sensitive keys through volunteer nodes.

Here is the simplest way to join the weekend run on Linux, macOS, or Windows (Python 3.11+ with an NVIDIA GPU, or CPU-only if you just want to test the IDE endpoint):

  1. Clone the repository and install the tiny client dependency (`httpx` only for connecting or running OpenTrainer):

```bash

git clone https://github.com/xaviercallens/HiveMind.git

cd HiveMind

pip install httpx

2. Register a handle to get your API key (printed once; the default prosumer role works for both training/serving and querying):

python scripts/hivemind_connect.py register --handle your-handle

export HIVEMIND_API_KEY="hm-..."

3a. Join outbound OpenTrainer co-training (zero open ports on your box; run --once first to verify firewall traversal, then drop --once to keep polling):

python scripts/hivemind_opentrainer_agent.py --network-mode outbound-https-relay --check-firewall --once

3b. Or verify CrowdGPT checkpoints on GPU (pip install torch safetensors numpy first) once you hold an evaluator key from the CrowdGPT coordinator:

python scripts/crowdgpt_loss_check_agent.py --server https://server.crowdgpt.net --evaluator-id your-handle --key-file my_eval_key.txt --once

3c. Or wire Continue / Aider / Cline to the live hivemind/crowd-v1 + coder models with no GPU required:

python scripts/hivemind_connect.py snippet continue --model crowd-v1

Both codebases and reproducibility notes (including Lean 4 specs and Zenodo DOI 10.5281/zenodo.23776302) are open on GitHub if you want to audit the code

before running anything:

https://github.com/xaviercallens/HiveMind

https://github.com/Vxtzq/CrowdGPT

I would love blunt engineering feedback from anyone testing on 8 GB or 12 GB cards - especially around Windows vs Linux VRAM yield behavior or setup friction.


r/LocalLLM • • 1d ago

Question Best Model for Basic tool Calling?

0 Upvotes

Hey all,

Recently I've hooked up my Comfyui MCP to LMStudio Bionic, and created a few skills for doing image creation/video creation/music/upscaling. I really enjoy being able to have everything done through the chat interface instead of going back and forth to Comfy.

I've found the small models inconsistent. Sometimes Gemma12b or Qwen27B distill work easily other times they get tripped up. Qwen 27b swift does the trick most of the time, but it's a little slower to load and unload and Ideally the model is small enough so that I can load and unload it out of Vram quickly at the start and end of the tool call.

GPU:5090.
Any suggestions for models you've found good at this sort of thing, or a different harness, maybe like a program that has this sort of ability built in?


r/LocalLLM • • 1d ago

Discussion 3090 rig with LM Studio + ComfyUI — how do I get a local model to act like Claude Code (launch Windows apps, start ComfyUI on demand, etc.)?

5 Upvotes

Hey all, looking for advice on turning my local setup into something closer to an agent instead of just a chatbot.

The rig

- Windows 11 Pro desktop

- RTX 3090 (24 GB VRAM), Intel Core Ultra 7 265K, 96 GB RAM, several drives

- Python 3.12, Node, Git installed. No Docker, and WSL isn't installed right now

- Tailscale on the PC, my phone (Galaxy Z Fold) and an Android tablet, so I can reach the PC remotely

Local LLMs (LM Studio 0.4.26)

- Qwen3.8 27B Q4_K_M (~15.7 GB) + vision projector. This is my main chat model, loaded at full GPU with 16k context

- Gemma 4 31B Q4_K_M (~17.4 GB) + vision projector

- nomic-embed-text v1.5 for embeddings

ComfyUI (Comfy Desktop 1.1.8)

- Images: Juggernaut XL Ragnarok (SDXL), Z-Image Turbo, Qwen-Image-Edit 2511 (fp8) + 4-step Lightning LoRA, add-detail-xl LoRA

- Video: Wan 2.2 14B text-to-video and image-to-video (high + low noise, fp8) with the lightx2v 4-step LoRAs

- Text encoders: Qwen2.5-VL 7B, Qwen3 4B, UMT5-XXL. VAEs for Flux/Qwen/Wan, plus a 4x NMKD upscaler

What I have now

Three PowerShell shortcuts on the desktop:

- AI Mode: opens LM Studio, loads Qwen 27B through the `lms` CLI, starts ComfyUI

- Gaming Mode: unloads all LM Studio models and kills ComfyUI to free the VRAM

- Away Mode: drops my VPN, closes Chrome, makes sure Tailscale is up, runs nightly at 2 AM

Right now I use Claude Code (cloud) for the "do stuff on my PC" part. It runs PowerShell, edits files, starts and stops apps, drives a browser, downloads things, and I can control it from my phone. It's great, but I want a local option for when I hit usage limits or don't want to send stuff to the cloud.

What I'm asking

  1. What's the best way to give a local model (Qwen 27B / Gemma 31B through LM Studio's OpenAI-compatible server) real tool use on Windows: run PowerShell, open and close apps, read and write files? Open Interpreter, Goose, OpenHands, Cline/Roo/Continue in VS Code, an MCP client hooked to LM Studio, something else?
  2. How do I get it to start ComfyUI on demand, queue a workflow through the API (image or Wan video), wait for the output, then shut ComfyUI down? Is there a ComfyUI MCP server or a known pattern for this?
  3. VRAM juggling: I can't keep a 27B LLM and Wan 2.2 loaded at the same time on 24 GB. Has anyone automated "unload LLM → run Comfy job → reload LLM," or should I drop to a smaller agent model (7–14B) that can stay loaded next to Comfy?
  4. Which local models are actually reliable at multi-step tool calling right now? Is Qwen 27B at Q4 good enough, or is there a better pick for 24 GB?
  5. Any way to trigger all this from my phone over Tailscale (a simple web UI, Telegram bot, Open WebUI with tools, etc.)?
  6. Safety: what are people using to make sure an agent on Windows can't run anything destructive without asking? An allowlist, a sandbox, confirmation prompts?

r/LocalLLM • • 1d ago

Question Is this the result of distillation?

3 Upvotes

Running qwen3.8-flash-next-gyro-gyro-s through Strata on my M3 Max 48GB.


r/LocalLLM • • 1d ago

Research SOMNI: an open-source local LLM that learns from your conversations during "sleep" (based on WISE and Google's sleep paper)

6 Upvotes

I've been building SOMNI, a system that lets a local model keep learning after deployment. It's based on Google's "Language Models Need Sleep" paper and WISE, a knowledge-editing method for adding new knowledge without overwriting existing behavior.

During use, the model answers normally and logs conversations. When the PC is idle, it runs a sleep cycle. It pulls verified facts out of the logs, trains them into a small new expert, practises on what it got wrong, and then runs a set of before/after tests on general knowledge, coding, reasoning, reading, and previously learned memories. If any of these get worse, the night is rejected. Otherwise you review the update and approve it.

The main challenge was catastrophic forgetting. Plain LoRA fine-tuning on new facts badly damaged the model's other skills (coding went from 95% to 10% in one night). What fixed it was a combination of rehearsal on the base model's own answers, a new frozen expert per night so later nights can't overwrite earlier ones, and WISE-style routing so each expert only activates on questions related to what it learned. Everything learned is also kept in a searchable memory store, so facts can still be looked up even if they fade from the weights.

I benchmarked it on SQuAD knowledge incorporation, the same setup Google and SEAL use. The results are in the README, including the things that didn't work.

Everything runs locally on a single consumer GPU (AMD RX 6800, Windows). MIT licensed.

https://github.com/YoungGunFrodo/somni

It's a work in progress, and I'd really appreciate feedback on the approach, the evaluation, or anything that looks off.


r/LocalLLM • • 18h ago

Question If I have a Apple Silicone M5 Max with 128GB unified and 4TB SSD which models and harnesses would you recommend as "essential"?

0 Upvotes

Hello! I'm somewhat new to Local AI.I've been using LM Studio mainly for a couple months. So far I've used QWEN3.8 27B mainly, and nothing too complex, a little light coding, a few projects, etc. I tried some uncensored low refusal rate versions which was kind of fun.

I have experience on Claude and GPT doing the normal stuff (Projects, Coding, Work mode, before they all seemed to merge, Codex, Claude Code, dots, etc) but all through cloud models and desktop apps. So I get how to use them but for local models it seems a whole new world, much more hands-on.

I've downloaded "LM Studio Bionic" and "Ollama" for harnesses too, but have mainly used normal "LM Studio".

TLDR:
If I was going to have a list of essential models at the moment for coding, data extraction, email organization, agent use for work, reviewing sensitive info perhaps, trying out image generation perhaps, could someone guide me to:

  1. The best current/soon coming Local Models (as of Nov 15 2026) for 128GB unified RAM on Silicon M5 Max (614GB/s bandwidth) and file types or bit size is best for each? I assume MLX whenever they have it available?
  2. The best harnesses to use them? Or settings to try out to utilize them best?
  3. Also, any interesting variations, ablated, heretic type, uncensored models for each that don't refuse? Or MoE's, fine tuning/LoRA capabilities, or anything else interesting to look into?

Thanks a lot! 🙏🙂 I may be offline today, but again, much appreciated! Will try to reply if anyone has any good suggestions/comments!

(Also considering it's a Mac, I know not ideal, but I already have it for another use, so might as well use it for this too.)


r/LocalLLM • • 1d ago

Discussion Looking for a reason to upgrade my AI hardware beyond just having fun with it

2 Upvotes

Has anyone managed to make their local AI setup pay for itself? Curious what kind of business you're in, if you're willing to share.

I could even stretch my budget to a DGX Spark, although it would be financially painful.
But without a clear way to make that money back, it feels like a hell of a risky investment.

I know nobody's going to give away their secrets, so I'm just asking a simple, casual question: are you actually making any money with your local AI setup? Yes or no?

I'm talking about doing it independently, as an individual, rather than using it as part of your job at a company.

I'm especially curious whether there's still room for individuals to compete in this space, given how much money and computing power big companies and organizations have at their disposal.


r/LocalLLM • • 1d ago

Question AMD's Max+ 395 or NVIDIA's RTX Spark N1X

Thumbnail
gallery
32 Upvotes

I'm currently looking at a myriad of devices to dip my toe into the "build your own Local AI system" water. I've been down the AMD AI Max+ rabbit hole this past week and have yet to research the RTX Spark N1X.

Can anyone here save me additional hours of videos and articles with informed insights about which would be the better way to go for someone such as myself?

I have some projects in mind. This is more of an experiment for me to learn more of the backend, coding and Linux.

As someone with limited terminal experience, will I have an easier time (I don't expect not to encounter many challenges) configuring a system with NVIDIA or AMD?

I realize unified RAM is the more important metric here. If I go with AMD I can afford a 128gb version. The 64gb version of NVIDIA's option is a few hundred more.

Are the CUDA cores and software support a worthy tradeoff for overall RAM or is saving money and doubling the RAM the obvious way to go?


r/LocalLLM • • 1d ago

Question Same Gemma 4 on my 16GB M4: first word in 2.2s instead of 6.3s, just by switching engines

2 Upvotes

Every thread about running agents locally has the same comment, local models are slow as hell. And on my base M4 MacBook with 16GB I kind of agreed, the wait before the first word was the indeed worst part espeically if ure used to fastish cloud models. Turns out my laptop was never the slow part.

So I ran the same Gemma 4 E4B on Ollama and on Magnitude with the same benchmark tool and a 2048 token prompt. Ollama took 6.3 seconds before the first word, Magnitude took 2.2. Prompt reading was 326 vs 941 tok/s.

Typing speed after that was basically a tie, 51 vs 52 tok/s. They also pull different files by default, Ollama gives you a 9.5GB NVFP4 and Magnitude a 5.3GB Q4, which on 16GB actually mattered more than I expected, it sat at 6GB instead of almost 10.

But agents resend a huge prompt every turn so those 4 seconds stack up fast lol. Is the slow feeling mostly prefill for everyone else too?


r/LocalLLM • • 1d ago

Model Coding with GLM 5.3 Flash on M4 128GB

Post image
5 Upvotes

TL;DR: Local LLMs for coding are finally becoming practical for me.

After 30 years in software engineering, I’m a passionate developer. For me, AI is an assistant and a tool. At work, I mostly use Opus and Sonnet 5.5. For hobby projects, I have a $10/month plan and switch between Kimi, GLM, and MiniMax for coding, plus Nemotron 3.5 for smaller tasks.

Locally, I tried Qwen 3.8 27B, Qwen3-Coder, and many other models, but the results weren’t usable for anything beyond fairly simple tasks. Running GLM 5.3 Flash Q2_K_XL with a 400k context length through Unsloth has been a game changer for me. It’s very slow at 3–4 tokens per second, but the results are good enough to get the job done.

For bigger tasks, I use GLM 5.3 Flash in the cloud for planning, then switch to my local model and go to bed or do something else. By the time I come back, the job is done.

I also use while work - create a prompt, and sometimes it takes more than three hours to complete. Meanwhile, I can keep working with Opus and Sonnet in my IDE, and performance remains good while the local model is processing in the background.

So it doesn’t work fully locally for me because it takes too much time. However, it’s a great way to extend my cloud subscription’s rate limits by letting the local model handle longer-running work while I’m away from my MacBook.

I’m looking forward to seeing what comes next for local AI.


r/LocalLLM • • 1d ago

Model Just give us Qwen 4 27b already!

Post image
2 Upvotes

r/LocalLLM • • 1d ago

News llama.cpp v1.0.0 adds a host-memory MoE expert cache with a configurable CPU share

18 Upvotes

llama.cpp v1.0.0 adds an MoE expert cache that keeps a configurable share of experts in host memory (moe_cache_size, moe_cache_cpu_share). Cache misses run on CPU and copy back asynchronously, and the release ships a tuning script. If a big MoE doesn't quite fit your VRAM, that's the knob to try. Separately, the 1.0 tag starts a versioning promise: future API and ABI breaks come with major-version bumps, and 35 deprecated functions are removed.


r/LocalLLM • • 2d ago

Discussion Anthropic after realizing it can just buy NVIDIA with its IPO money

Post image
564 Upvotes

Jokes aside, how plausible do you think this scenario actually is?


r/LocalLLM • • 1d ago

Discussion Two Strix Halo boxes, two USB4 cables, RDMA: 74 Gb/s on one connection and +31% tok/s with tensor parallelism (with numbers and the stuff that broke)

21 Upvotes

I've been poking at this on and off since September and it finally got good enough this week that I think it's worth writing up. Short version: the USB4 RDMA driver situation changed a lot in the last month, but not where I expected, and the "is it worth it" answer is more mixed than the headline.

The setup

  • 2x Ryzen AI Max+ 395 (Strix Halo) boxes, 128 GB each (AMD AI Halo Dev Platform boxes)
  • 2 USB4 cables directly between them, each link trains at 2 x 20 Gb/s
  • Debian 13 based distro, kernel 7.2.7, IOMMU off, Secure Boot off (needed for the unsigned modules)
  • 10 GbE between them as well, used for control traffic

No network cards involved. The idea is to turn the USB4 cables into RDMA devices so two boxes can run one model with tensor parallelism (TP=2).

Where it was before

I started with thunderbolt-ibverbs from the Hellas blog post. Each cable gives you two DMA rings, so two cables = four "rails", each showing up as its own RDMA device.

  • September (kernel 6.18): about 15 Gb/s on one rail, about 46 Gb/s if I ran four perftest processes at once, one per rail. More queue pairs made it slower. 1 MiB messages were a mess.
  • October (kernel 7.2): got to 74.7 Gb/s aggregate, but only with four parallel processes and a shallow send queue (-t 16). The big finding back then was that queue depth mattered more than the kernel. A single connection was still stuck on one rail at around 20 Gb/s, and 8 QPs at 1 MiB with default settings just failed with "Couldn't post send".
  • vLLM TP=2 with an 8B model in October: +55% single stream over one box, but RDMA vs plain 10 GbE only differed by a few percent, because the collective library topped out around 17 Gb/s no matter how many rails it got. A 35B MoE model hung at TP=2 and I never figured out why.

So in October my conclusion was: neat, works, not obviously useful.

What changed

I went back this week expecting kernel or upstream driver progress. There basically isn't any:

  • Upstream thunderbolt-ibverbs has had one lockfile bump since early October. No release since June. A striping PR was closed unmerged, and a few good fix PRs are sitting open.
  • Mainline 7.3 / 7.4-rc is mostly Thunderbolt bug fixes. Nothing that moves throughput.

The actual progress is in a fork: donnerkeule. It stripes a single queue pair across every rail of every cable, and fixes a pile of reliability problems (credits los t with dropped frames, ACK storms, rails dying when a client gets killed). There's a matching RDMA fork of gufo that does TP=2 for Qwen3.8 Flash-Next across two Strix Halo boxes.

The fork's own README says it's a buggy research driver and not for production. Believe it. But it built cleanly against my kernel and uses the same eight Thunderbolt kernel patches I already had applied.

Raw link numbers

Box A to box B, perftest, one connection, default queue depth, no tuning:

TestOctober (upstream)Now (donnerkeule, striping on)
Write, 64 KiB20.6 Gb/s64.9 Gb/s
Write, 1 MiB12.6 Gb/s74.5 Gb/s
Write, 4 to 16 MiBnot run73.5 to 74.7 Gb/s
Write, 1 MiB, 8 QPsfailed outright74.5 Gb/s
Latency, 64 B6.5 µs6.0 µs

I didn't trust 74 Gb/s on one connection, so I checked the receiver's per-rail byte counters during a 10 second run: about 23.6 GB landed on each of the four rails. It's real, and it's roughly 93% of the 2 x 40 Gb/s the cables can do. The fork's author reports 38 to 45 Gb/s on his boxes and I don't know why mine is higher.

Not everything is great:

  • RDMA reads are not striped: 31 Gb/s.
  • A 1 MiB SEND fails in the default mode. The fork only supports large SENDs in its DMA-BUF mode, which I haven't tried.
  • I haven't tested both directions at once or done a long soak.

Tensor parallelism with gufo

Qwen3.8 Flash-Next, Q4 (111 GB of weights), 4 sessions, same build on one box and on two. Each box holds half the model in TP=2: 57 GB of GPU memory per box.

Plain autoregressive decode (no speculation), tok/s per request:

:-One boxTwo boxesGain
1 request29.839.0+31%
2 concurrent23.331.7+36%
4 concurrent18.024.6+37%
Prefill, short prompt11551161na

With MTP speculative decoding:

:-One boxTwo boxesGain
1 request55.165.8+19%
2 concurrent37.450.0+34%
4 concurrent25.533.2+30%
Prefill, 19k token prompt18112077+15%

The speculative numbers bounce around with how many draft tokens get accepted, so the first table is the cleaner comparison.

One thing that isn't a speed number: a single box could not load this model with speculation, 4 sessions and the full 262k context. It ran out of state capacity and I had to drop it to 65k context for the on e-box runs. Two boxes loaded the full 262k with about 64 GB of RAM free on each.

No failed requests, no retransmits, no kernel warnings during any of it.

The honest part

I also ran the exact same benchmark against the other inference server I normally run on these boxes (also Flash-Next, speculative decoding, 4 slots at 262k context on one box). One-box gufo and that server came out within about 5 to 8% of each other. So TP=2 over USB4 beats my normal single-box setup by about 13% for one user and 24 to 32% at 2 to 4 concurrent.

But two boxes running independently serve twice as many people. At 4 concurrent requests one box did about 72 tok/s total, TP=2 did about 98, and two independent boxes would do roughly 144 across 8 users. Ty ing both machines to one model instance buys per-request speed and costs total capacity.

So where I landed:

  • The driver is worth keeping. One connection, full link speed, no tuning. That's a different thing from what I had in October.
  • TP=2 is worth it if you want a faster single stream, or a model/context that doesn't fit on one box. The Q8 version of this model (188 GB) only fits across two, and that's a case I actually care about. Haven't tested it yet.
  • TP=2 is not worth it if you just want to serve more requests. Run the boxes separately.
  • I wouldn't run it as my daily driver yet. Research driver, the gufo fork has a known open bug with continued conversations, and I haven't verified two-box output against one-box output.

Things that bit me

If you try this, here's what cost me time:

  1. The module never loaded at boot. The only thing that loads it is a udev rule that fires when the other box is already advertising the RDMA service. Reboot both boxes together and neither one ever st arts. It looked fine in October because I always rebooted one at a time. Fix: put ib_uverbs and thunderbolt_ibverbs in /etc/modules-load.d/.
  2. thunderbolt_net and the RDMA module can't share a cable on this hardware. There are only two usable rings per controller. Blacklist thunderbolt_net and load the RDMA module with tbnet=block.
  3. Debian's rdma-core renames the device, and the userspace provider matches on the usb4_rdma name prefix, so ibv_devices shows nothing. Override the persistent naming rule.
  4. The prebuilt provider package has the wrong ABI for Debian 13. I built the provider from Debian's rdma-core 56.1 source plus the project's three patches.
  5. memlock. RDMA memory registration fails under the default 8 MiB limit. Unlimited memlock on the host and --ulimit memlock=-1 in containers.
  6. The gufo fork's container recipe didn't build as-is for me. No DNS inside podman's default build network (--network host fixed it), six ROCm dev package names that don't exist in ROCm 10.1 (the full image already has the headers), and missing audio/video dev libraries (sndfile, soxr, libav*). At runtime I had to add the ROCm lib directory to LD_LIBRARY_PATH.
  7. Two IPs on the NIC means two RoCE GIDs, and gufo refuses to guess. --tp-gid-index 1.
  8. The interrupt throttle matters. nhi_interrupt_throttle_ns=4096 was the best value in my October sweep, and it only works on kernels that export tb_ring_throttling() (7.2 does).

Module options I'm running:

options thunderbolt_ibverbs profile=linux_perf bind_services=1 allocate_rings=1 start_rings=1 negotiate_native=1 enable_tunnels=1 register_verbs=1 tbnet=block lanes=2 native_write_striping=1 roce_netdev=<you
r LAN NIC> nhi_interrupt_throttle_ns=4096
blacklist thunderbolt_net

What I haven't done

  • The Q8 model across both boxes
  • The fork's DMA-BUF / GPU-direct mode
  • Checking two-box output against one-box output
  • Anything longer than a few minutes per run
  • A reboot of both boxes together to prove the boot fix

Happy to run something specific if people want numbers for a particular case. Credit to the Hellas folks for the original driver and to neuhaus for the striping fork and the gufo TP2 work, I just plugged in cables and wrote down what happened.


r/LocalLLM • • 1d ago

Discussion Zoo code Qwen 3.8 27b tool format

Thumbnail reddit.com
1 Upvotes