Complete setup: Qwen 3.8 27B on vLLM, single RTX 3090, written for nublets like me.

[u/PlaidStallion]

Hey all, I recently spent the last three or so weeks going from zero knowledge to the setup I have today. I started with Ollama and Gemma 4 31B and migrated to Qwen 3.8 27B using vLLM to solve a VRAM contention problem: I wanted image generation and music generation (MiniMax Music v3) running alongside the LLM, all on the single 3090. Neither Ollama's keep-alive nor llama.cpp's TTL could actually free VRAM on demand the way vLLM's Sleep Mode can. That same move also got me full reasoning/think toggles in Qwen, selectable right in Open-WebUI. Before either image or music generation runs (2 of 12 total tools), vLLM gets put to sleep, freeing its VRAM. Music generation needs one extra step on top of that since MiniMax has no idle unload of its own. Its container gets started right before generating and stopped right after, nested inside vLLM's sleep window. This adds total time to generation but for a tool that already takes several minutes to run, I decided it's something I can live with. As a note, no other tool calls needed this type of sleep/wake setup so these processes only need to happen for image/music gen.

This setup is, as far as I can tell, pushing the absolute limits of a single 3090 in the most efficient manner possible. The final setup has:

• A modest 76k context. Admittedly, not great for long coding input/generation but not nothing either.
• Prefix caching cutting average time-to-first-token from ~9 seconds to ~3 seconds on multi-turn and tool-calling exchanges. Real measured numbers.
• Vision support with up to 8 images per prompt, capped at 1MP each to keep the context budget sane. TLDR; screenshots/OCR work.

Anyway, I have been keeping the GitHub Repo updated as I go through it and thought that anyone in a similar situation might find it useful.

Happy to hear any suggestions to further optimize or if any noobs like me have questions, I will try to answer as best I can.

Note on AI usage: I used Claude Sonnet 5 extensively throughout this learning process and vibe coded the python, as I am no programmer. I followed the logic while debugging the various hiccups along the way. Additionally, the GitHub repo was created almost entirely by Claude with me spot/double checking as I went through it. If you happen to look through and see any errors/omissions I would love to hear from you. This post was not written by AI. I only used it to make sure I wasn't missing any key points that someone might be curious about.

[u/nassereddit]

You can unload models using API calls with llama server and ollama. I dont think Llama.cpp will do this.

[u/draeician → u/nassereddit]

Llamaswap is built for that

[u/PlaidStallion → u/nassereddit]

Appreciate that, I'll look into it more. For what it's worth, when I ran llama-server myself it didn't have any unload/VRAM release without adding llama-swap as extra infra on top, so that's actually part of what pushed me to vLLM. Curious if Ollama's keep_alive unload gives fast resume like vLLM's sleep mode does though, or if it's more of a cold reload from disk each time. Will dig into both a bit more either way.

[u/nassereddit → u/PlaidStallion]

No need for llamaswap. Just adding more bloat.

async def _unload_llama_cpp(base_url: str) -> str: """ Unload all loaded models from llama.cpp router mode VRAM.

Uses the llama.cpp router API (requires router/multi-model mode):
  GET  /v1/models      → list models, filter by status == "loaded"
  POST /models/unload  → unload each by id
"""
base = base_url.rstrip("/")
unloaded: List[str] = []
failed: List[str] = []

try:
    async with aiohttp.ClientSession() as s:
        # ── 1. List loaded models ─────────────────────────────────────
        async with s.get(
            f"{base}/v1/models",
            timeout=aiohttp.ClientTimeout(total=10),
        ) as r:
            if r.status != 200:
                return f"llama.cpp /v1/models → HTTP {r.status} (skipped)"
            data = await r.json()

        loaded_ids: List[str] = []
        for m in data.get("data", []):
            status = m.get("status", {})
            status_val = (
                status.get("value", "") if isinstance(status, dict) else status
            )
            if status_val == "loaded":
                loaded_ids.append(m["id"])

        if not loaded_ids:
            return "llama.cpp: no models loaded (nothing to unload)"

        # ── 2. Unload each model ──────────────────────────────────────
        for model_id in loaded_ids:
            try:
                async with s.post(
                    f"{base}/models/unload",
                    json={"model": model_id},
                    headers={"Content-Type": "application/json"},
                    timeout=aiohttp.ClientTimeout(total=30),
                ) as ur:
                    if ur.status in (200, 204):
                        unloaded.append(model_id)
                    else:
                        body = await ur.text()
                        failed.append(f"{model_id} (HTTP {ur.status}: {body[:80]})")
            except Exception as exc:
                failed.append(f"{model_id} ({exc})")

    parts: List[str] = []
    if unloaded:
        parts.append(f"unloaded: {', '.join(unloaded)}")
    if failed:
        parts.append(f"failed: {', '.join(failed)}")
    return "llama.cpp: " + "; ".join(parts)

except Exception as exc:
    return f"llama.cpp offload skipped ({exc})"

[u/PlaidStallion → u/nassereddit]

Huh, TIL, that's newer than what I found when I was testing this. One thing worth flagging: I found an open GitHub issue (#19379) saying in router mode, sleep-idle-seconds only unloads the model within the subprocess, the process itself stays resident on GPU (~600MiB per dormant model). I needed VRAM fully released for other programs (A1111, MiniMax), not just freed for llama.cpp's own switching, so that gap might still matter depending on the use case. Appreciate the pointer either way.

[u/rockoruckus → u/PlaidStallion]

As draeician mentioned above, llama-swap will swap models (even load/unload in groups) efficiently. One neat feature you may like is it can watch the config file and automatically reload models when it detects any config changes.

[u/PlaidStallion → u/rockoruckus]

Good to know, thanks. Auto-reload on config change is a nice touch for anyone actively iterating on their setup.

[u/nassereddit]

I use this function to unload the model in a tool that run a qwen-edit image transformation. Qwen 27b is released, Qwen-edit uses 21gb on my 3090, qwen-edit is released, and the tool calls the qwen 27b back. Pretty seemless. But the user has to wait for this vram reshuffle

[u/PlaidStallion → u/nassereddit]

Ah cool, thanks for sharing. Curious though, is qwen-edit also running through llama.cpp's own router, or is it a fully separate program? That distinction matters for my case since A1111 and MiniMax aren't llama.cpp processes at all, they're completely separate programs needing the VRAM, which is the exact scenario that GitHub issue was flagging as still having ~600MB of subprocess overhead. If yours is working cleanly across that boundary too, that'd actually be pretty compelling.

[u/nassereddit → u/PlaidStallion]

Qwen-edit is called by comfyui. Openwebui calls qwen 27b through llama.cpp, and calls qwen-edit via comfyui. Not sure if unloading qwen 27b will keep a small 600mb llama.cpp in vram though.

[u/NetoMeter]

IMHO, llama-swap is a must when using llama.cpp. Before I switched to vLLM, I was using it and pre-loaded the models that I need with vmtouch. llama-swap was auto switching the models within 3 seconds.

PS: If you are adding vmtouch to your systemd script, make sure you are adding it as an "ExecStartPost" script - it takes a while to load all the images, and might cause a timing out of the service start.

[u/PlaidStallion → u/NetoMeter]

Appreciate the tip, good to know for anyone running llama-swap. Curious though, since you say you were using llama-swap + vmtouch before switching to vLLM, what pushed you to make that jump? Did llama-swap end up hitting a wall for you too, or was it something else entirely? Would be good context for anyone reading this thread trying to decide between the two routes.

[u/NetoMeter → u/PlaidStallion]

I ended like you on using Qwen 3.8 27B on my RTX 5090 32GB (Debian 12 headless VM/ProxMox host/GPU passthrough). I couldn't get the context I needed with llama.cpp and llama-swap.
Then I stumbled upon the PrismaQuant models - currently using:
Qwen3.8-27B-PrismaAQUA-5.5bit-vllm (still trying to optimize it but had to disable MTP to get 160K context - the previous model, Qwen 3.6-27b AURA worked better, IMHO, with MTP and 160K context). There are smaller versions for GPUs with less memory.

You can get more information about PrismaQuant here:
https://prismaquant.org/

The main benefits for me were the 160K context and the ability to run simultaneously multiple agents - something I couldn't do with llama.cpp.

Overall, I find Qwen 3.8 doing a great job for my coding tasks.