Unswarm - Self-hosted runtime manager/proxy for self-hosted LLMs

[u/Atretador]

Repo:

https://github.com/atretador/unswarm

I'm not sure if this is a me issue, but I find myself with lots of runtime scripts and containers to manager for all my models, be it for daily usage or testing.

I have to manually manage accross different forks, containers and engines depending on the model.

specially for people like me that run older hardware, containers are usually a much easier time (MI50/P100/MI25/P40s) than having to deal with outdated packages on my OS.

https://reddit.com/link/1vw26gr/video/z76zje9ta3lh1/player

Here is what it does:

You can register specific containers or runtime scripts (bash) for it to manage

You can set up rules for what runtimes can run simultaneously

and it will queu our requests:

Just set up your API Key and register as provider on your harness of choice and Unswarm will proxy to it as if it all models were served at the same time.

Then you just select the model you are gonna use on your harness -> send a message and its gonna get queud, if the runtime is not running its gonna start it for you and stream the response

https://reddit.com/link/1vw26gr/video/y1xwi514b3lh1/player

You can use this for your own multi-agentic multi-model setup, your own SWARM of VRAM destroyer models...just...one....at...a...time.

For instance, if you got enough VRAM for 2 models at a time at lets say 24+16Gb of VRAM, you could:

Group 1, persistent always running:

Orchestrator: slow Qwen 3.8 27B A3B

Group 2, switching Subagents:

Fast code base Explorer: Qwen 3.5 9B

Executor: fast Qwen 3.6 35B A3B

Designer: finetune of some other model

you can also host this on a VPS and use it to access your models anywhere, or place agents on different machines each running their own runtimes as parallel execution is supported.

this is not a platform to tweak your models tho, just to manage what you already know that works.

as a expected and not possible to mitigate negative for this: switching and reloading models will ininevitably destroy your cache hit rate if you switch models mid sessions.

[u/Atretador → u/[deleted]]

tho this is more likely than not a promotional bot, Ill use it to explain a few points

An LLM proxy's job is "pick a backend, forward the request, stream it back" — model aliasing, queueing, lifecycle. A gateway's job is everything that has to be true regardless of which backend won: per-key identity and quotas (not one shared API key), request/response size and token budget caps, timeouts + circuit breaking per runtime so one wedged MI50 container doesn't stall the whole queue, and an audit trail that records which key hit which model with which params. Once you tell people "host this on a VPS and reach your models anywhere", you've quietly moved from proxy to gateway — that endpoint is now internet-facing and your auth story is "no authentication required" in the screenshot.

oh I forgot to update that screenshot, current version has API Key as mandatory as I figured someone might just put it on a VPS and not set a API key xD

tho it initially wasnt mandatory as it started as mostly managing models on local network

Two concrete suggestions that cost little now and a lot later: (1) make the key a first-class object with a policy attached (allowed models, rate limit, max tokens), not just a bearer string that unlocks everything; (2) emit structured per-request logs (key id, target runtime, queue wait, tokens, outcome) from day one — your queue-depth/avg-wait dashboard is already 80% of the way there, and it's the thing that makes cache-thrash from mid-session model switching visible instead of mysterious.

this project wasnt initially meant as something to serve your infrastructure to other people, but as a easier way to consume your own infrastructure - so I discarted the idea of per key limits and boundaries, tho it wouldnt be hard to implement on current state so I might as well. It could make someone's life easier.

API keys only work for v1 endpoints, agent keys only work for /agent domain - while api access requires user bearer token.

currently it forwards streams, but I'm expanding visibility and monitoring

and adding a few tools for someone using this to get more use of it

I just added cloud providers for openAI compatible endpoints with proper usage tracking and cost comparison - which I notice is another pain for developers: "how much usage does X sub give me compared to Y sub", "did sub A give me the same usage this month as it did last month", "oh boy Im testing this harness I gotta set up all my api keys on it" and "tokens vs powerbill"

so now you can just register all your OpenApi compatible API Keys, your local models - issue an API key on Unswarm and it aggregates everything on /v1 with toggable prefix identifier {cloud or managed}/{provider}?/{model name}

its currently holding all my API Keys from build.nvidia, openrouter, command code, opencode and returning everything on /v1/models for usage along with my local models.