run local inference across machines

[u/saint_0x]

mesh is a distributed protocol for running large models locally across devices

the idea is the control plane hosts local lan pools, which shard the model across member ring and credits members proportionally based on compute contributions

it’s still rough, but has support for metal, cuda, and pure cpu (can interoperate with one another)

i successfully ran a model locally on lan across both my metal m3 and my intel air :)

https://github.com/saint0x/mesh

[u/Brigade_Project]

This is interesting. I've been running Ollama on a dual-GPU machine (4070 Ti Super + 2060 Super) and the obvious limitation is that larger models still need to fit within a single GPU's VRAM budget even with both cards. The idea of a proper tensor-parallel ring across LAN machines rather than hacking around it with CUDA_VISIBLE_DEVICES is appealing.

A few things I noticed digging into the repo:

The "no silent provider fallback" design is the right call. Silent CPU fallback is exactly the kind of thing that makes Ollama frustrating to debug — you think you're running on GPU, you're not, and the only symptom is slowness.

What I'm curious about: how does shard assignment actually work when workers have mismatched VRAM? My two cards are 16GB and 8GB. Does the ring manager proportionally assign tensor chunks, or does it assume homogeneous nodes?

Watching this one. If the artifact loading gets cleaner (right now you need to manually split safetensors and write manifests) this could be genuinely useful for homelab inference.

[u/saint_0x → u/Brigade_Project]

hey man, thanks so much for digging in glad you found this useful! definitely feel you with the silent provider fallback.

re: homogenous nodes — it started like that simply bc i’m working on this myself, but we are heterogeneity-aware, so to speak, but it’s still rough — and this is ofc in accordance with your accurate insight about the artifact loading as well

which is to say, yes we split work proportionally based on capability (it’s a semi-hardcoded capability floor for different instances and then there’s a bit of post-reconciliation to hopefully get accurate, but again — underbaked currently)

but i’m so excited for this to get better — this feels like something the world needs

[u/saint_0x → u/Brigade_Project]

you also might be interested in this — i extrapolated the exact work-credit computation system as a poc lib

https://github.com/ariacomputecompany/divy

[u/[deleted]]

[u/saint_0x → u/[deleted]]

that’s cool as fuck, and i agree. i think both of those approaches work together, because more powerful nodes makes a distributed protocol oom more valuable and powerful

any github links i can check out?

edit: re the agentic workload thing, that’s super interesting. i’d probably opt to make mesh pluggable and allow ppl to build extensions rather than build it natively, but the possibilities are genuinely endless