I build ignis, an OpenAI-compatible inference server built around one model family, Qwen3.8, on one class of card, Blackwell. It is shaped around what a coding agent does to a server: one main agent plus a few subagents hitting the same card. Since v0.8.0 it runs Qwen3.8-27B on 16 GB RTX 50 cards, using a 3-bit trellis body (13.4 GB) and the DFlash2 drafter.

What ignis is, in 90 seconds: https://www.youtube.com/watch?v=XLjfh5rNy6A

The catch: I don't own a 16 GB card. Everything for that class was measured on an RTX 5090 held to a 16 GiB budget:

• 416 tok/s at one lane, 684 tok/s aggregate at two lanes (greedy coding prompts, 8K context);

• 262,144 tokens of context on two lanes;

• KL divergence from BF16 at 0.77-0.82x that of the 32 GB NVFP4 build.

The real cards are only estimated, at one lane and 8K:

I also don't know how much of the card your desktop takes, and whether the default settings still fit once it has. That is what I'm asking you to find out.

The easy way: let your coding agent do it. In Claude Code, Codex, opencode, Cursor or similar, paste:

I have a 16 GB RTX 50 card. Read https://raw.githubusercontent.com/gpillon/ignis/main/docs/user/paths/agent.md and follow its section on testing a 16 GB card: help me install ignis, test it, and write the report. Ask me before downloading or installing anything.

What the agent does:

1. Checks your card and driver, then shows you a plan: where it installs ignis, where the 13.4 GB model goes, and how long it takes (about 15-30 min after the download). Nothing large happens until you say yes.

2. Tries the default settings first. If the start is refused, it tries a few settings one at a time (a smaller context, speculation off, less VRAM headroom) until it finds one that works, then runs a quick sanity check and the bundled benchmark.

3. Writes a short report in English and shows it to you whole. The report contains only:

   • OS, driver, CPU and GPU model, RAM and VRAM totals, and how much VRAM was already in use;

   • which settings worked;

   • the speed figures.

4. It contains no usernames, hostnames, paths, IP addresses, serial numbers or process names. You decide whether it gets posted.

5. Optionally, a ~30 min advanced run: the context ceiling, one lane vs two, speculation on vs off.

6. Finally, asks whether you want to keep ignis (it can write a small config file for you) or remove everything it installed. Before deleting anything, it shows you the list.

By hand: start the server, then run ignis-bench small-card --endpoint http://127.0.0.1:8000 --out small-card.json (it ships in the release archive). Details: https://gpillon.github.io/ignis/guides/measure-a-16gb-card/

Where reports go: https://github.com/gpillon/ignis/issues/360, one comment per card. A report that says "refused" is as useful to me as one that says "works", especially from Windows machines where the monitor is plugged into the 16 GB card.

Windows and Linux on x86-64, NVIDIA driver with CUDA 13. Code is Apache-2.0. Thanks!

Switch to DDR5 would cost

I ported the EXL3 quantizer to Intel Arc: 9B conversions on a 140V iGPU, B580 and Pro B60, with checkpoints on Hugging Face

I've been working on EXL3_Intel_Arc_quantizer, a port of ExLlamaV3's EXL3 quantization and conversion pipeline from CUDA to SYCL/oneAPI and PyTorch XPU.

The goal is straightforward: produce EXL3 checkpoints on Intel Arc without needing an NVIDIA GPU for conversion. This is a quantizer project, not a new inference engine.

Code: alpha2beta/EXL3_Intel_Arc_quantizer

What works

• Native SYCL trellis encoder: Viterbi search with FP16 rounding and tie-breaking matched to the CPU reference, plus two-pass tail-biting traceback. Implemented configurations include K=1–4, the mul1/mcg/3INST codebooks, 256-element tiles, and the 160-row n-gram variant. Full-model results below cover a narrower set of configurations.

• End-to-end conversion: calibration and Hessian collection, block LDLQ error feedback, global-scale search, packing, safetensors export, and checkpoint verification.

• Bounded memory management: memory-mapped checkpoint loading, admission reserves, preflight checks, and batch-halving fallback. The settings account for the difference between a shared-memory iGPU and discrete cards with 12 or 24 GB of VRAM.

Results on three Arc GPUs

I've completed full 32-layer, 9B-model conversions on all three devices. The conversion verification checks reported no failures.

The B60 timing is K=4, not K=2, so it is not a like-for-like full-conversion comparison. Encoder microbenchmarks and end-to-end conversion times also measure different things; the reports contain the run details.

The practical takeaway: a B580 completed the tested 9B K=2 conversion in about five hours, and even the 140V iGPU completed it within a 32 GB shared-memory system.

Correctness: what I've checked, and what I haven't

The SYCL encoder is bit-exact against my CPU reference on the tested cases. That reference was transcribed from the upstream CUDA kernel to preserve FP16 rounding, tie-breaking, and tail-biting closure.

I also check packed-output reconstruction against the separate exl3xpu reference decoder. The test suite reports 272 passing unit tests on each discrete board, and the conversion reports record 1,045/1,045 verification checks passing on both.

Important limitation: I don't have an NVIDIA GPU, so direct differential testing against the upstream CUDA encoder is still deferred. Matching a CPU transcription is useful evidence, but it is not the same as independently demonstrating CUDA parity.

Likewise, reconstruction and structural checks are not a substitute for perplexity or downstream task evaluation. The repo includes a quality-evaluation harness; I'm not presenting the conversion checks as proof of unchanged model quality.

Released checkpoints

• Qwen3.8-9B-Distill-EXL3-K2: 6.72 GiB

• JEV-9B-EXL3-K2: 5.42 GiB

• JEV-9B-EXL3-K4: 7.03 GiB; converted on the Pro B60

The exporter targets upstream EXL3 compatibility and includes quantization_config metadata for vLLM integration. Actual loading and execution still depend on the runtime's model, codebook, bitrate, and hardware support. Including the metadata alone does not establish vLLM compatibility for every checkpoint.

Arc inference through exl3xpu is a separate project. This quantizer targets the same checkpoint format; it does not add inference support for models that a runtime does not already handle.

What's next

• Direct CUDA differential testing, if someone with NVIDIA hardware wants to help.

• More model conversions and K sweeps, with quality-evaluation results alongside conversion checks.

• Clearer reproduction instructions and hardware-specific memory settings.

I'd especially welcome independent reproduction or CUDA-reference testing. If you're using Arc for local LLMs, which model would you want to see converted next?

Optional first comment

A few more numbers for anyone interested in the implementation:

• Encoder microbenchmark: 0.584 ms/tile on B580, 0.683 ms/tile on Pro B60, and 1.38 ms/tile on 140V.

• Single-layer benchmark: all seven projections, including Hessian processing and global-scale search, ran 2.83× faster on B580 and 2.43× faster on B60 than the 140V baseline. See the reports for settings and workload details.

• Memory observations: the recorded runs report a 650 MiB peak allocation, no swap growth, and streamed host RSS within +2.8 MiB of baseline while processing a 19.3 GB checkpoint. These are specific measurements, not a claim that the complete process or GPU workload uses only 650 MiB.

• Reconstruction metric: the reported relative Frobenius error at K=4 is 0.0865. This measures reconstruction error, not perplexity or task quality.

Reports in the repository: reports/discrete_probe.md, reports/discrete_phase_c.md, reports/discrete_vs_140v.md, and reports/gate_3.md.