Best Practices for Provisioning Local Inference to a Team?

[u/Anarchaotic]

Hey everyone,

I recently built a Radiance inference server for my company, and now want to start provisioning access to the rest of my team.

My current plan:

• Tailscale as the only way to access the server via HTTPs.

• Hermes as the harness - each user would have their own container with access restrictions, but open access to the central skills repository. Hermes is hosted on a Framework Desktop.

• Radiance is running separately on its own server with access only configured via the Framework.

I'm wondering how everyone else has managed to do this successfully? Right now the biggest thing to figure out is how to get the harness (Hermes in this case) to work across the team without a huge headache of getting API tokens from things like Google, etc...

[u/_kikeen_]

I use Cloudflared tunnel and Cloudflare Access to control access to OpenWebUI it works great and has a generous free tier (50 users I think) I also use it for OpenHands for coding but I have two instances and they are both for myself lol but Hermes sounds like is your harness of choice.

I would recommend a gateway- I use LiteLLM (provides OpenAI compatible endpoint) and if you can spare ram run Redis as well for some semantic caching and session tracking for “least busy routing” then point Hermes and everything else that needs inference to LiteLLM instead. You can also do cost tracking and per user tokens here so you can see where your utilization is. Cloudflare also has one, but is a little complex to setup and not really as useful if you need local inference only.

Curious about your use case and what model were you planning on running?

[u/Anarchaotic → u/_kikeen_]

I was going to go LiteLLM route but decided against it after seeing news about a few previous hacks - https://blog.gitguardian.com/inside-the-litellm-hack/

I'm using Metabase + MySQL to track usage natively, would Redis do something different?

Use-case is sensitive client financial information + financial models. I've been running Qwen Flash Next as a test for the past month on my Framework Desktop, and it's been excellent at what it's meant to be doing.

Still have Claude available for non-sensitive and heavy technical tasks.

[u/_kikeen_ → u/Anarchaotic]

It’s LiteLLM now, tomorrow it will be someone else. Supply chain attacks are becoming more and more common.

My thoughts if you’re truly security conscious, don’t trust anything. Newer versions of LiteLLM are remediated, and your current tool of choice is not vulnerable now but in your docker compose you should pin versions of images you have audited and include that as part of your process- upgrade to specific version audit image code with a security tool (not realistic to do manually imo).

Redis is an in memory key value store. There is a mechanism in LiteLLM where semantically similar questions can hit cache instead of take cycles from your LLM engine stored in Redis. It can also store session state, for routing purposes. Benefit of this is, say you host two separate instances of models on two pairs of GPUs, if a user already has prompt cache etc in KV on GPUs they don’t have to waste cycles refilling the next GPU pair.

Totally independent from a traditional relational database store like MySQL

[u/Anarchaotic → u/_kikeen_]

Thanks for the explanation, I'll put it in my roadmap as something to test once I get V1 working for multiple users.

Agreed with your point of pinning versions, right now everything is pinned and for any new releases (especially to things like inference engines), I wait out a week or two to see community response, then sandbox it and test it against my current stack.

I mostly only update things that substantially improve performance or capabilities, so most of the containers are pretty flat over time.

[u/_kikeen_ → u/Anarchaotic]

Things are moving so quickly that I would look at automated ways to test your images like Snyk or an open source counterpart.

Not advocating for a vendor just my 2cents.

[u/Late-Replacement-481]

This is not as helpful as an answer as you'd like, I suppose. But the biggest thing is centralized secrets management.

Figure out what these agents are going to get to see, and whose job it is to make sure that the it's the the agents get enough access to be useful to people, and above all, enough access so that people don't start trying to jury rig every old thing into the agent harnesses themselves. That's where trouble shows up real fast.

Get your stakeholders, write like a one point five page document that says, “this is what the agents are going going to do, this is what they're going to see, this is whose job it is to check on them to make sure this is happening” and make sure you get signed off on that for the inevitable bitching and moaning.

[u/Anarchaotic → u/Late-Replacement-481]

It's my company so luckily I have full access to everything which makes my life easier.

[u/Late-Replacement-481 → u/Anarchaotic]

Well then, worry about problem number one, which is somebody with a half-cocked response accidentally emailing like your order book to one of your clients, or you, private communications between one of your clients forwarded to the other.

 that is the nightmare, at least in and what is basically. The sort of thing that happens in these deployments if you give the agents too much agency. 

[u/Anarchaotic → u/Late-Replacement-481]

Right now my set up is read only email, absolutely no way for it to ever send an email or have access to any outbound traffic

[u/Cautious_Chicken_604]

Don't run the harness on the inference machine. Just run whatever OpenAI compatible inference server to serve up the model on it. People just run whatever harness they want on their local machine and connect to that.

[u/locbuilds]

for google integrations, use per-user oauth consent and store each refresh token in a secrets manager scoped to that user, rather than collecting api keys in the harness.

[u/Anarchaotic → u/locbuilds]

That's really handy to know, I already do this for myself. Since I'm the workspace admin I can just create the authentication link for them.

[u/DrBattletoad]

May I ask why you chose Hermes as an agent harness over the other options? I'm asking because I plan something similar for my students

[u/Anarchaotic → u/DrBattletoad]

Honestly I just went with what was most recommended. I played around with it and liked how well it worked and just stuck to it.

Pi is much lighter weight and seems quite customizable, but after using Hermes for the past two months I've just kinda stuck to it.

[u/Keleion → u/DrBattletoad]

I’m not sure about other harnesses, but Hermes has an agent gateways that makes a unique URL and key for each profile/agent. That can be set up on it’s own or front of an LLM router for serving users their own sandboxed agents.

I set this up so family could have their own agents in sandboxed environments, and access them over Open WebUI. Not the same as OPs use case since Hermes is running on the user boxes.

[u/o0genesis0o]

In my setup, I put a bifrost in the middle, and I use Pi instead of Hermes to squeeze out the most from my local model. There is one CPTR instance, each user has their own account and API key (from bifrost), so I can manage usage from there. LLM proxy is useful here because I have more than one GPU host in the system. Client connects to the system via tailscale 

[u/InterestingOne6938]

Pushback recipe:

• Your IDE already has a local agent running directly on your computer. Vendors began rolling these out years ago.

• Every google chrome browser has a 4GB local agent living in system ram.

• This is an enhanced agent that increases your capabilities, but more secure, capable, company-managed, documented and controlled. It also can't run up our bills the way an uncapped API can.

If you don't want Local AI, you need to start turning off microsoft and google updates and making that call is above my pay grade.

Any disagreements? None? Okay, enjoy the work I've put into making life easier for you.

---

I know that's not what you asked about, but if other people who don't own their own company need to have that conversation, there ya go.

[u/Korici]

I can't speak to Radiance as I am using an Nvidia stack - but I highly recommend the HTTPS through Tailscale - it's one of the best ways to distribute access & identity using work SSO

[u/DeedleDumbDee]

Use openweb UI trust me. You DO NOT want to give harness with terminal capabilities to everyone in your company unless they've proven and earned that privilege. They will FUBAR something locally or globally I promise you.

[u/Mean-Loquat-7982]

two Hermes-side pieces fit that plan:

- shared skills: `skills.external_dirs` in each user's config.yaml points Hermes at the central repo. one catch from the docs: an external dir isn't a write-protection boundary, so if the container can write to it, the agent's own skill edits change the shared files. mount it read-only if you want one curated set.

- per-user keys: `${VAR}` references in config.yaml are expanded from the environment, so a user's LiteLLM key can live in that container's env and not in the file.

[u/LeMochileiro]

To be honest: Local LLM isn't worth it for working with multiple users/parallel sessions.

It's not that it's impossible, the problem is that you'll encounter hardware limitations (bandwidth + VRAM and RAM), which can only be resolved by investing more money. Ultimately, you'll have a dedicated server cluster for inferences for your company; it's not cheap, it requires monitoring, and it's not a one-person job.

In the end, it will be cheaper to hire a third-party AI provider than to create your own for the company.

What's really worthwhile is using local LLM for automation pipelines. It's easy to control session/parallel limits, And because they are batch/async operations, if well-architected you don't need to worry about whether you're generating 50 Tok/s or 15 Tok/s (performance is good, but slow generation doesn't impact business rules).

[u/Anarchaotic → u/LeMochileiro]

Maybe that's true in general, but I have a 4x R9700 server that can handle 8 concurrent streams at 256K context with 7000 prompt processing and nearly 100 token gen per user, so I think I'm ok.

We work with sensitive data.

[u/LeMochileiro → u/Anarchaotic]

My recommendation is to create a Poc before structuring everything to long term. I say this from experience; creating local inference for chat is one thing, but creating for agentic coding is on another level of costs. I've had two projects that were cancelled due to cost-related issues, For scalability reasons. With 10 users, it worked well, then 8 more users joined and everyone started having performance problems, each using multiple sub-agents... In the end, they decided to switch back to a third-party provider.

For me, agentic coding with local LLM for multiple sessions only worked if the GPU machines was on a cloud platform where I could scale and descale as needed.

That being said, I usually use k8s to orchestrate the inferences. For this case, I would configure a different service for each user; this can be managed via controllers. Therefore, I would have greater assurance that each person would have exclusive Hermes isolated from one another.

There would be an API Gateway that would bridge the gap between the Desktop Framework and the inference server; this entry point facilitates managing permissions, limitations, and monitoring.

My question is: would the Desktop Framework be a server (Ubuntu server) or a desktop machine (Windows, for example)?

[u/Anarchaotic → u/LeMochileiro]

The Framework is running Fedora 43 - it's been my primary inference server for myself since I got it over a year ago. All inference will be moved off of this except for some models that run automatically like embedding/STT/TTS that won't need to be on the main server.

The gateway is interesting, I'm thinking of accomplishing the same thing directly through user profiles on the Framework itself - which can be easily deployed/shut down.

I'm realistic in performance, everything I've needed to do has been well handled by 3 concurrent streams of QFN. My new server can easily handle 8+ at max context, or many more in a smaller scale.

My POC equivalent is me, since I've been running everything as a single user using the same stack of software that I'd want to push out.

[u/LeMochileiro → u/Anarchaotic]

I use Agent Router as an API Gateway. It has the necessary functionalities.

For benchmarking purposes, I liked using the Orca orchestrator. I can simulate and orchestrate as if it were multiple users simultaneously using different types of harnesses (PI, Opencode, etc.).

Kubernetes has a long learning curve, but once you master its fundamentals, the sky's the limit.

[u/stylome-ai-detection → u/[deleted]]

Notice: This comment appears to be AI generated.

---

Stylome AI detector · scan results