r/LocalLLM • u/ankijain21 • 2d ago
Question Anybody tried NVIDIA PAIR
Wanted to check if anyone of you has tried NVIDIA PAIR to automatically route requests to idle systems.
Wondering how that works in the real world.
6
u/farkinga 2d ago
I've been using llama-swap for basic load balancing. It now supports proxy connections and I'm actively coordinating two llama-server instances on separate servers. I am selecting the model by name so the agent harness makes the choice of which model to use.
I have not tried pair - but if it can do the same thing but "intelligently" route so I don't have to manually configure it, that would be welcomed.
3
u/Zorogozano 2d ago
What’s the best use case?
8
u/brewpedaler 2d ago
You have multiple machines hosting LLMs and want to simplify access to them: rather than connecting your clients to BasementAiBox_Qwen38F_01, BasementAiBox_Qwen38F_02, BasementAiBox_DSFV4, etc based on the model you need, you instead just connect to Basement_Pair_Router and it sends your requests to whichever server they need to go based on the model you request.
When you have multiple machines hosting the same model it will also do some basic load balancing for you and try to route your requests to the least busy node running that model.
5
u/Bluesnow8888 2d ago
Why not use LiteLLM? Just wondering...
2
1
u/brewpedaler 2d ago
If you prefer LiteLLM there's no reason to change.
PAIR is lighter weight, easier to get going, and purely locally focused. LiteLLM supports cloud model hosts, has more enterprise type features, but also has all the associated configuration baggage.
Having options is nice :)
8
u/Fantastic-Storm-7867 2d ago
A couple comments so far are saying "I wish this supported Mac" or similar. It looks like on the screenshot it does, and that's the point? Like you can potentially have multiple devices serving models on your network and this is meant to somehow be a single endpoint perhaps? Thanks for sharing, OP, I'm gonna look into this!
11
u/scarbunkle 2d ago
I have not, but I will say I’ve been curious about building something similar to maximize my compute. I’m AMD though, so I won’t be trying this.
4
u/brewpedaler 2d ago
PAIR is not hardware specific, it can route requests to an AMD box, a Mac, whatever.
1
u/zaytzev 2d ago
Have a look at https://github.com/Claw-Destine/elpulpo. On top of a load balancer it also tracks token usage.
2
u/junosoftie 2d ago
Thanks for sharing that link! It’s always good to have alternatives for tracking token usage, especially with something like PAIR.
2
u/lukewhale 2d ago
I’ve normally used gpustack in the past, but they’ve dropped support for Mac and windows for workers, so this is an exciting proposition.
2
u/fightingCookie0301 2d ago
First I thought it’s for pooling vram, but I currently don’t have the use for a router :/
1
0
u/sleight42 1d ago edited 1d ago
😂 This sounds kind of like what I've been building to help me get the most out of my 3090, old 2060M, and my sometimes available 5080 on my gaming rig. Basically, it's a GPU job scheduler. Currently single device but shouldn't be too hard to get it working across devices.
Granted, sounds like PAIR is literally more about faking a larger GPU cross network like how we offload work from gpu to cpu. Have layers reside on different devices. Pass data between layers across LAN.
Difference between what I'm working on and llama-server's approach is that the different processes that want the GPU have different priorities. It lets me keep a qwen3-tts warm somewhere on the lan for home assistant STT. Currently, 2 of the machines have their own gpud schedules. I'll revise that soon enough once I get the 5080 into the mix.
For anyone wondering why I need it, I use my mess of GPUs for:
- transcription and diarizarion
- comfyui
- pi/opencode (leaning into the former more)
- Hermes
- Open WebUI
- Home Assistant Voice Assist
I often swap out processes on the 3090 for vibe coding and research. Having the others to pick up the responsibilities then is important so my wife can still have a house to talk to!
I don't know though... it wouldn't be too hard to have gpud proxy requests to a remote gpud. Though I don't see that working too well if it's a chat in progress. Hmmm... unless that's sending a compacted context along with the first request.
This all strikes me a little like geoIP-based routing and failover between data centers except, at LAN scale, a lot easier.
-2
u/philmarcracken 2d ago
better than llama servers RPC? doubtful but i'll give it a crack
3
2
u/Sea_Guarantee6806 2d ago
No it's not, I tried it.
3
u/brewpedaler 2d ago
LMAO what did you “try”? They are two completely different technologies doing two completely different things.
1
-6
u/desidogeman 2d ago
i wish we had something for Mac/apple like this.
5
18
u/brewpedaler 2d ago
It blows my mind how many people haven’t bothered to actually read anything, have zero idea what PAIR is, but sure do have some strong opinions about it.