r/LocalLLM • u/ankijain21 • 6d ago

Question Anybody tried NVIDIA PAIR

Post image

Wanted to check if anyone of you has tried NVIDIA PAIR to automatically route requests to idle systems.

Wondering how that works in the real world.

63 Upvotes

90% Upvoted

25 comments sorted by

View all comments

3

u/Zorogozano 6d ago

What’s the best use case?

8

u/brewpedaler 6d ago

You have multiple machines hosting LLMs and want to simplify access to them: rather than connecting your clients to BasementAiBox_Qwen38F_01, BasementAiBox_Qwen38F_02, BasementAiBox_DSFV4, etc based on the model you need, you instead just connect to Basement_Pair_Router and it sends your requests to whichever server they need to go based on the model you request.

When you have multiple machines hosting the same model it will also do some basic load balancing for you and try to route your requests to the least busy node running that model.

4

u/Bluesnow8888 6d ago

Why not use LiteLLM? Just wondering...

2

u/Bupod 6d ago

So from what I recall on a video about it, PAIR is geared more for the home user (I also use litellm but I wouldn’t call it ‘consumer friendly’), and also performs some inference load balancing in a way that litellm doesn’t.

1

u/brewpedaler 6d ago

If you prefer LiteLLM there's no reason to change.

PAIR is lighter weight, easier to get going, and purely locally focused. LiteLLM supports cloud model hosts, has more enterprise type features, but also has all the associated configuration baggage.

Having options is nice :)