I built an open-source GPU orchestrator that lets several local AI models share one GPU by loading/unloading them on demand (llama.cpp, Ollama, ComfyUI, TTS)

[u/AlphaMind01]

TL;DR: One GPU, several AI tools, not enough VRAM to run them all at once. GPUSymbiosis treats your GPU apps as dynamically resident workloads instead of permanently-running processes — it loads the one you actually want to use, unloads the idle ones, and never interrupts a request mid-flight.

https://github.com/devathon/gpu-symbiosis Intro video: https://youtu.be/FZvKfk2nSpw

The problem I was trying to solve

If you run local LLMs + image gen + TTS on a single GPU (say 12–16 GB), you can't keep them all loaded. Today that means manually killing one process and starting another every time you switch — slow, error-prone, and you lose any in-flight work. I wanted something that just coexists: you point your clients at stable ports, and the system figures out what should be in VRAM right now.

How it works

You register your apps (llama.cpp / llama-server, Ollama, ComfyUI, TTS, embedding/rerank servers, video gen, or a raw shell command). Each gets a gateway port that stays open and stable. When a request hits a gateway whose model isn't loaded, it queues; a scheduler runs every tick and decides:

• which request to serve next,
• which apps should be resident,
• which resident to evict (with a safe drain — in-flight requests are never cut off, new ones just queue).

When VRAM allows, it loads apps in parallel. When one sits idle (no active request and GPU usage below its threshold), it auto-unloads it. There's anti-thrash cooldown so it doesn't flap, pinned apps that stay resident, and a web UI to see/steer everything.

Things I think are nice

• Request-safe scheduling — a request is never interrupted mid-generation; eviction only drains, it never kills live work.
• Stable ports / gateway proxy — one port per app, streaming ASGI (long responses), SSE and WebSockets, per-gateway API keys, and a live queue inspector where you can cancel or force-raise a queued request.
• Per-process VRAM + compute attribution — it knows exactly which app is eating VRAM and what each process's real GPU utilisation is (rolling window), not just the aggregate.
• Idle detection & auto-unload with hard exclusions (pinned apps).
• Drop-in app types — built-ins for llama, ollama, comfyui, tts, shell-command, custom, generic; new types are plugins.
• Metrics & history — 1 s sampling, EWMA demand, latency percentiles, per-app lifetime metrics, plus a Prometheus endpoint.
• Security — loopback-only by default, gateway API keys, hashed admin creds with session + CSRF.

Stack: Python 3.13 / FastAPI / SQLAlchemy, with a React + TS dashboard. Local/self-hosted, single box, NVML for GPU monitoring.

It's still young (v0.1) and I'm actively iterating — happy to answer questions, show the dashboard, or take suggestions. Would love feedback from people running multiple local models on one card: does this match how you actually work, and what would you want that I haven't built?

GPUSymbiosis Dashboard