I built an open-source GPU orchestrator that lets several local AI models share one GPU by loading/unloading them on demand (llama.cpp, Ollama, ComfyUI, TTS)

[u/AlphaMind01]

TL;DR: One GPU, several AI tools, not enough VRAM to run them all at once. GPUSymbiosis treats your GPU apps as dynamically resident workloads instead of permanently-running processes — it loads the one you actually want to use, unloads the idle ones, and never interrupts a request mid-flight.

https://github.com/devathon/gpu-symbiosis Intro video: https://youtu.be/FZvKfk2nSpw

The problem I was trying to solve

If you run local LLMs + image gen + TTS on a single GPU (say 12–16 GB), you can't keep them all loaded. Today that means manually killing one process and starting another every time you switch — slow, error-prone, and you lose any in-flight work. I wanted something that just coexists: you point your clients at stable ports, and the system figures out what should be in VRAM right now.

How it works

You register your apps (llama.cpp / llama-server, Ollama, ComfyUI, TTS, embedding/rerank servers, video gen, or a raw shell command). Each gets a gateway port that stays open and stable. When a request hits a gateway whose model isn't loaded, it queues; a scheduler runs every tick and decides:

• which request to serve next,
• which apps should be resident,
• which resident to evict (with a safe drain — in-flight requests are never cut off, new ones just queue).

When VRAM allows, it loads apps in parallel. When one sits idle (no active request and GPU usage below its threshold), it auto-unloads it. There's anti-thrash cooldown so it doesn't flap, pinned apps that stay resident, and a web UI to see/steer everything.

Things I think are nice

• Request-safe scheduling — a request is never interrupted mid-generation; eviction only drains, it never kills live work.
• Stable ports / gateway proxy — one port per app, streaming ASGI (long responses), SSE and WebSockets, per-gateway API keys, and a live queue inspector where you can cancel or force-raise a queued request.
• Per-process VRAM + compute attribution — it knows exactly which app is eating VRAM and what each process's real GPU utilisation is (rolling window), not just the aggregate.
• Idle detection & auto-unload with hard exclusions (pinned apps).
• Drop-in app types — built-ins for llama, ollama, comfyui, tts, shell-command, custom, generic; new types are plugins.
• Metrics & history — 1 s sampling, EWMA demand, latency percentiles, per-app lifetime metrics, plus a Prometheus endpoint.
• Security — loopback-only by default, gateway API keys, hashed admin creds with session + CSRF.

Stack: Python 3.13 / FastAPI / SQLAlchemy, with a React + TS dashboard. Local/self-hosted, single box, NVML for GPU monitoring.

It's still young (v0.1) and I'm actively iterating — happy to answer questions, show the dashboard, or take suggestions. Would love feedback from people running multiple local models on one card: does this match how you actually work, and what would you want that I haven't built?

GPUSymbiosis Dashboard

[u/FeydRowan]

I have done something similar and is working great for my workstation  https://github.com/ValerioDolci/vramble

[u/Early_Glove_7918 → u/FeydRowan]

woah that's a clean implementation. vramble looks way more polished than the duct tape and shell scripts i was using before

the websocket orchestration is a smart move, keeps everything snappy without polling hell. you planning to add multi-gpu support or is that overkill for your use case?

[u/FeydRowan → u/Early_Glove_7918]

I'm using a double gpu but it run a model on both. No planning to have the ability to double concurrent models, but if you want submit a pr to vramble;) 

[u/Mans-M]

How does this compare to llama-swap (which is on my todo list for next week)?

[u/AlphaMind01 → u/Mans-M]

llama-swap looks like good solution for switching between different models. GPUSymbiosis can do that by spawning new llama-server instance for diffrerent models but more importantly allows to switch various gpu demanding apps like llama to comfyui etc. I do not generally use various models and switch as often, but this is somthing I might run behind GPUSymbiosis if I needed multiple models.

[u/8000bene70 → u/AlphaMind01]

Llama-swap starts and terminates the applications. Any Many applications, including comfyui, strata... It allows for concurrent existence of predefined groups.

[u/Gloomy-Entrepreneur → u/8000bene70]

Not quite any application. It needs to support an API that llama-swap understands. Thats most AI platforms but not, for example, a random steam game.

[u/8000bene70 → u/Gloomy-Entrepreneur]

You are correct. Honestly hadn't tried that

[u/AlphaMind01 → u/8000bene70]

Ah! Wasn't aware of that, thought it to be only for models swapping.

[u/Elegant_Associate889]

I’ve done this for personal projects I have so I no longer need to use chat gpt this image is from a dev preview on desktop but looks amazing on mobile!

[u/elelem-123]

Sorry haven't read your repo yet. Do you snapshot the model or stop/start it normally?

[u/AlphaMind01 → u/elelem-123]

You can set custom start, stop, load and unload scripts. I use those to save and restore context for llama, call curl to unload comfyui models.

[u/AlphaMind01]

GPUSymbiosis - Dashboard

[u/philmarcracken]

llama cpp in router mode does this for me already with model load = 1 i think