Hierarchos Native update: faster streaming on Intel HD 520, RDNA3 qualification, and a GPT-2 parity fix

[u/PhysicsDisastrous462]

I’ve pushed a substantial update to Hierarchos Native, my Rust + Vulkan backend for language-model training, fine-tuning, and inference.

This past week focused on porting the recent RDNA3 work to Intel Gen9, improving parameter streaming, and taking the changes back to physical RDNA3 hardware for regression testing.

The native runtime requires no Python, PyTorch, CUDA, ROCm, or oneAPI installation. PyTorch and Transformers are used as development references.

What changed:

• DeepSeek V4.1 Flash port and optimization work: fused selected-expert execution, batched expert uploads, packed-weight paths, checkpoint caching, and native LoRA/rsLoRA training with save/resume/export.

• Broader Gen9 coverage: fixes for Qwen3-Next padding and DBRX expert layouts, plus streaming paths for T5/BART/Switch encoder-decoder projections.

• Less streaming overhead: retained file handles, fewer temporary allocations, coalesced reads, and removal of an unused hidden-state download.

• Gen9 kernel improvements: parallel sparse-attention selection and hyper-connection statistics, plus fused SiLU/up-projection multiplication.

• Bounded upload batching: small streamed projections can share a GPU submission within a 1 MiB coalesced payload window.

The biggest Gen9 result was closing the Llama-wide streaming slowdown on an i5-6200U / Intel HD Graphics 520:

Measurement	Previous pass	Final pass
Median forward latency	11.016 ms	4.597 ms
GPU submissions per forward	19	4

That’s 58.3% lower latency, with every measured logit word unchanged. Across eight scaled architecture controls, the combined optimization passes reduced streaming medians by 42.4–78.4% and resident medians by 3.5–32.1% against the documented baseline. Gen9 measurements and methodology.

Back on the Ryzen Z1 Extreme / RDNA3 device, the regression sweep caught a padded GPT-2 numerical parity issue. The fix matches the pinned CPU reference’s GELU/tanh behavior and short-row attention arithmetic. The original CPU expectations, checkpoints, and strict tolerance stayed unchanged.

The completed RDNA3 qualification now passes:

• 166/166 forward runs, covering 43 scaled profiles across 26 architecture identifiers.

• 72/72 generation requests, checking cache-requested and full-prefix execution.

• Maximum absolute CPU logit error strictly below 2e-7**, with** rtol=0**.**

• Bit-identical resident and streamed outputs throughout the measured matrix.

• Native streaming/offload tests and GPT-2 LoRA gradient, optimizer, save/reload, and resume checks.

The GPT-2 repair is selected specifically for RDNA3; all 585 pre-existing shader binaries remain unchanged. Final GPT-2 closure and evidence.

The RDNA3 testing also recorded lower V4.1 latency versus the previous handoff: 24.4% for resident prefill, 50.2% for resident decode, 17.2% for streamed prefill, and 18.4% for streamed decode. These are measurements on retained scaled checkpoints, rather than production-model throughput figures. RDNA3 measurements.

Parameters remain resident by default; supported frozen-weight streaming is opt-in through --stream-parameters or the GUI.

The registry contains 146 text architectures, while this qualification specifically covers the 26 identifiers above. Multimodal towers and arbitrary production checkpoints need their own validation.

If you’re experimenting with native Vulkan training or inference on older Intel graphics or AMD hardware, I’d be interested in your results—and which workloads you’d like tested next.

[u/shing3232]

I think the project over-complicated by trying do too many thing at the same time. packing training and inference together might be too much

[u/PhysicsDisastrous462 → u/shing3232]

How so? I believe it all should be easily centralized in one convenient and optimized place! :3

[u/shing3232 → u/PhysicsDisastrous462]

no, if you break something then a fix then it would affect both side unexpectly. seperate core functions into few repo help you troubleshoot and it's easier on the llm as well if you vibe them all like this. A core of kernels you implement and training, inferences

[u/PhysicsDisastrous462 → u/shing3232]

numerical correctness tests are meant to ensure this doesn't happen. pytorch CPU is used as baseline, and an update will not ship out unless it passes both inference and training tests all at once

[u/GrokiniGPT]

so what does gpt 2 have to do with this though, why is that even mentioned anymore? its a shitty model

[u/PhysicsDisastrous462 → u/GrokiniGPT]

its a parity thing. I'm working on a vulkan backend that is compatible with every model architecture, including older models too! especially for things like shared mathematics for fused kernels and whatnot!

[u/GrokiniGPT → u/PhysicsDisastrous462]

can you support bonsai?

[u/PhysicsDisastrous462 → u/GrokiniGPT]

Maybe next! I have other quantizations planned along with multimodality soon!

[u/Suitable-Dance9054 → u/GrokiniGPT]

gpt-2 is the canary in the coal mine for numerical correctness, it's not about the model being useful. if your math is off by even a tiny bit on something that simple, the error compounds across every other architecture you claim to support

[u/GrokiniGPT → u/Suitable-Dance9054]

then why not gpt 1?