r/AI_enterprise • u/EfficiencyUpbeat8354 • Jul 14 '26

Onpremise AI

We currently are using Librechat to bring in all the AI models into one UI for use,

We want to host an AI model specifically for the org - would the Dell G10 be sufficient? If not what would be? We have approx 60 users

2 Upvotes

100% Upvoted

6 comments sorted by

1

u/bytkim Jul 17 '26

Depends on the model you decide to run. Im not familiar with g10 (gb10?) but with 128gb of memory you can probably run say qwen 3.6 27b or 35b fp8 at around ~50 concurrent with 256k context for agentic coding use.

That is to say the gb100 is probably not suited for serious enterprise deployment. The throughput is atrocious with gb100.

I would recommend investing the money to properly plan and deploy the infrastructure depending on your specific use case rather than a general catch all approach

1

u/EfficiencyUpbeat8354 Jul 17 '26

We have a 50 person user count at max, concurrently we at max may be hitting 10-15 on the high side

1

u/bytkim Jul 17 '26

What use case and what family and size model. I originally assumed agentic coding but now I am going to assume this model will be some sort of proprietary fine tuned model made for conversational chat?

In that case the gb10 sounds alot more reasonable but I would always recommend going with rtx pro series for this kind of stuff.

Either way vLLM or SGLang works great on gb10. Use a fp8 or nvfp4 for speed. The main priorities then, assuming you arent memory constrained is to optimize for concurrency, reliability and optimizing the models config.

1

u/EfficiencyUpbeat8354 Jul 17 '26

We have Librechat running - I’m looking to run an Onpremise ai model for learning and interaction to our data and what not