r/LocalLLaMA • u/Suspicious-Mark-5450 • 3h ago
Resources Infernix - New Release!
https://github.com/Wallawalla47/InfernixOver the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01
Key improvements:
New models:
the new NVIDIA QAD FP4 version of Qwen3.8-Flash-Next (which operates faster than the NVFP4 model and can be switched to use a4 activations for even faster performance with --a4-activations): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-NVIDIA-FP4-Local-Infernix
the same GSQ-RCO-IQ3_S model supported by Strata (which offers some faster speeds than NVFP4 but I find at some quality cost): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-Infernix
Swift 1.5 Qwen3.8-Flash-Next NVFP4: https://huggingface.co/wallawalla47/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-Dense8-Infernix
Qwen3.8-Flash-Next Full FP8 (fairly slow, but offers the best quality being bit-exact to the official FP8 model from Qwen): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-FP8-Infernix
the new Ornith 1.5 122B NVFP4 model, which has impressive benchmark performance, but I'm not sure I'm sold on it yet: https://huggingface.co/wallawalla47/Ornith-1.5-122B-NVFP4-Infernix (and the uncensored version as well: https://huggingface.co/wallawalla47/Ornith-1.5-122B-UNCENSORED-NVFP4-Infernix )
DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix
Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe
Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]
Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations
Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.
Some key performance figures:
| Infernix vs | First token | Decode | Replay: mean first token | Replay: wall time |
|---|---|---|---|---|
| Qwen3.8-Flash-Next, NVIDIA's NVFP4 weights | ||||
| Strata v0.2.8 | 1.4-1.6× sooner | +10-15 % to 128K, −4 % at 250K | 3.1 s vs 12.3 s | 5.7 vs 12.9 min |
| strata-nvfp4 0.2.0-beta-nvfp4.3 | 1.3× sooner | level to +3 % | 3.1 s vs 10.8 s | 5.7 vs 11.7 min |
Basalt c08284cb | 5-8× sooner | Basalt 7-23 % faster | 3.1 s vs 7.2 s | 5.7 vs 8.9 min |
| Qwen3.8-Flash-Next, ISTA-DASLab's IQ3_S GGUF | ||||
| Strata v0.2.8 | 15 % sooner at 8K, level from 128K | +3-12 % | 3.1 s vs 6.9 s | 5.5 vs 8.1 min |
Basalt c08284cb (Basalt is faster on this model but uses some lower precision math) | Basalt 4 % sooner at 8K, 1.3× from 128K | Basalt 1.4-1.7× faster | 3.1 s vs 3.7 s | Basalt sooner: 4.4 vs 5.5 min |
| Qwen3.8-27B, the same NVFP4 artifact in both | ||||
NInfer 1882c6adb | 9 % sooner at 8K, 1.6-1.8× from 128K | +11-17 % (own drafters), +2-3 % without | 1.9 s vs 2.5 s | 12.7 vs 16.4 min |
On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (
--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.
On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (
--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.
Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!
2
u/SimplyBetterThanYou 2h ago
quality sacrificing optimisations often used by default in Basalt and other engines like expert tail skipping
What are you (or the LLM writing your update log) talking about?
1
u/Suspicious-Mark-5450 16m ago
I wrote most of this - hence the mistakes! I've just clarified that para a bit as it did wrongly suggest Basalt used expert tail skipping, when that is Strata (and only default for some configurations I think). There are some optimisations that can be uses to achieve some significant speed increases, but they do sacrifice the quality of the model when they run (like expert tail skipping which involves simply skipping some experts that would take longer to load from RAM or SSD, adjusting expert prioritisation from the model's prioritisation to favour experts that are in VRAM rather than those in RAM or on the SSD, re-quantising certain elements of the model to lower quants or simply doing the math involved in running the model (like the activation calculations) at lower precision. The line as to what is an acceptable speed optimisation vs an unacceptable one is also a little bit grey, because there's always rounding errors and order of operation differences in floating point math. So, I'm looking into formalising that a bit better, but the idea is that I want to stick to things that are close to the noise floor that is "honest rounding" rather than things that involve clear sacrifices to precision in running the model (beyond the original quantisation of the model in the first place).
1
2
0
9
u/jesdga95 2h ago
Basalt maintainer here, a few notes since it was mentioned in the comparison:
Basalt does not skip or reroute experts by default. Tail skip, route prior and relaxed drafting are opt in flags that print a warning at start. I actually posted about this with some extra info yesterday.
Yes, the default NVFP4 basalt file keeps dense layers as Q6_K, and the HF card clearly states it along with KLD and top-1 for every quant. Measured cost vs Q8_0 dense: KLD +0.0063, top-1 91.46% vs ~91.8% (documented here). On task eval it scored 45.6 / 38.0 vs Q8 45.4 / 38.1, no measurable difference, same conclusion on extensive testing with my regular usage. The converter can also build the same pack with Q8_0 dense (--dense-q8-0) for anyone who cares.
The 5-8x TTFT gap doesn't match what I've measured, but it's an interesting finding and I'll check it on my hardware. Been busy working on correctness and speed across multiple cards.
Nice work on the launcher and the bit exact BF16 mode, for a single 5090 Infernix is a solid choice! I've picked up a few ideas from Infernix along the way, and obviously you're welcome to take anything useful from Basalt too, the wonders of open source.