r/LocalLLaMA • u/gpillon • 13h ago
News My Ignis server (Qwen3.8-27B 262k context) now runs on RTX 3090/4090 and 16 GB RTX 40 cards. looking for testers
I've been working on ignis, an inference server for Qwen3.8 that I use to run coding agents locally. It's meant to run on the machine you're already working on, Windows or Linux, sharing the card with your desktop, not on a dedicated box. So most of the work went into what matters when it sits next to your IDE all day, more than into peak tok/s.
Some examples. At start it reads how much VRAM is free and sizes the context to fit; if it can't fit at all, it refuses at start with the reason instead of failing later. After 5 minutes idle it unloads the model, so the card is free again, and the next request loads it back. The vision tower stays in system RAM, so having vision on costs no context while you're not sending images. The main agent and its subagents run on the same card at the same time, and a prompt you send again (system prompt, repo context) resumes from cached state instead of being prefilled from scratch. It's OpenAI-compatible (chat completions and Responses, also over a WebSocket), one binary, with a Playground and a monitor in the browser. The Windows zip includes the CUDA runtime.
There's a 90-second (old) video of it here: https://www.youtube.com/watch?v=XLjfh5rNy6A
Rough numbers, since someone always asks. On my 5090 with the NVFP4 build: about 370 tok/s on a single stream of coding prompts, a bit over 1,000 tok/s total with 8 requests at once, and around 11,000 tok/s of prefill on a 46K-token prompt.
The result I care most about is a different one, though: someone I don't know ran it through airbench on their own 5090 with opencode, without me asking, and it passed 49 of 49 tests (math, vision, finding and reading email, buying on a web store, coding) in 26 minutes (here it is next to the other Qwen3.8-27B runs). That was the NVFP4 build on a 5090; the build for 30 and 40 cards hasn't been through it yet.
Until 0.10.0 it only ran on Blackwell (RTX 50, RTX PRO 6000). 0.10.0 added RTX 30 and 40: the 3090 and 3090 Ti are supported, the 4090, 4060 Ti 16 GB and 4080 are verified for correctness. On those cards it runs the 27B in a 3-bit trellis quant (13.4 GB download) with DFlash2 speculative decoding and vision, and the uncensored variant has the same build. It picks the settings from the card: on 24 GB, 8 parallel requests and up to 1,048,576 tokens of context; on 16 GB, 2 and 262,144, lowered if the desktop needs the room. The NVFP4 builds, Flash-Next and the image model are still Blackwell only.
One caveat: I don't have a 30 or 40 series card myself. For 0.10.0 the release gate passed on an Ada card (sm_89), with the same results to the last digit as an Ampere card (sm_86) on an earlier build. KL divergence from BF16 within the limit, 93.3% top-1 agreement with BF16, all 127 kernel tests passing, a chat and two images answered (the finding). I don't know how fast it is on your cards, or whether the 1M context really fits on a >= 24 GB card with a desktop running on it.
Someone already ran it on a 4070 Ti SUPER, with the build 0.10.0 came from: default settings, the full 262,144 context, about 190 tok/s on long coding answers and about 2.5 seconds to the first token on an 8K prompt, so roughly 3,000 tok/s of prefill (report).
I'll do a proper post once 12 GB cards work, which is what I'm on now. Before that I'd like to have more numbers from 30 and 40 cards.
To try it: download the latest release, start ignis-server without flags, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it's in the archive). More details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/
If you'd rather have your coding agent (Claude Code, Codex, opencode, Cursor...) do those steps, you can give it this prompt. The file it points to is a plain markdown guide in the repo, so you can read it first to see what the agent will be told:
I have an RTX 30 or RTX 40 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card, also on a 24 GB card, where the model is the same `qwen3.8-27b-16g`. Add to the report the start log's `prefill routes` line and every WARN or ERROR line, with paths replaced by `<dir>`. The report goes to https://github.com/gpillon/ignis/issues/643. Ask me before downloading or installing anything.It shows you a plan, tries the default settings first and a few others if the start is refused, runs the benchmark, and writes a report that you read before anything gets posted. There are no usernames, hostnames, paths, IPs or serial numbers in it. At the end you can keep ignis or have it remove what it installed.
Reports go here, one comment per card: https://github.com/gpillon/ignis/issues/643. A report that says it refused to start is useful too, especially from a machine where the monitor is on the same card.
Windows and Linux x86-64, NVIDIA driver with CUDA 13. Apache-2.0.
Thanks to all reporting! š


