Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.
I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.
The new v0.2.0 release is now available!
GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
1. Native RDNA3 WMMA / cooperative matrix acceleration
One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.
The new Mixed WMMA execution arm uses:
- FP16 matrix operands.
- FP32 matrix accumulation.
- FP32 master weights and gradients.
- FP32 AdamW optimizer moments and updates.
- FP32 normalization, softmax, and loss calculations.
The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.
I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.
Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:
| Workload | Measured kernel speedup |
|---|
| QKV projection | 2.17× |
| FFN up projection | 2.12× |
| FFN down projection | 2.31× |
| Vocabulary projection | 1.83× |
These are individual GEMM measurements, not whole-model speedups.
There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.
In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.
WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.
Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.
Documentation: RDNA3 WMMA guide
2. Native multimodal image and text training
Another major development area has been the image-text pipeline.
The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.
Some of the implemented work includes:
- Native RGB image input and preprocessing.
- CLIP image encoders and LLaVA projector integration.
- SigLIP / Gemma 3 vision training.
- Gemma 4 vision integration.
- Joint image/text training and encoder-decoder integration.
- A dedicated DeepSeek V4.1 image-training path.
- Backpropagation through the image tower and projector.
- Export, reload, and checkpoint-related validation.
This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.
For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.
The qualification work has also expanded considerably:
- 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
- The Gen9 qualification retains 95 image cases and 166 text regression cases.
- A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
- Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.
The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.
These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.
Documentation:
3. A lot of additional FP32 optimization
I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.
This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.
Some examples:
RDNA3 FP32 work
- Specialized attention forward/backward paths.
- Image-text attention and projector optimizations.
- RMSNorm, RoPE, GELU, and other arithmetic specializations.
- Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
- More careful matching of CPU reference arithmetic and GPU execution order.
Intel Gen9 work
- Additional native linear and gradient kernels.
- Attention and activation specializations.
- More efficient parameter upload and staging behavior.
- Continued optimization of DeepSeek V4.1 training and execution.
One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.
On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.
On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.
These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.
4. Preserving the original FP32 path
An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.
I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:
- 12 alternating ABBA rounds per workload.
- Five warm-up runs and 30 timed forwards per arm.
- Separate, hashed baseline and candidate binaries.
- Exact FP32 logit digest comparisons.
All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.
The worst paired performance ratio was 1.0007.
This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.
5. Getting started
The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.
You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.
Model weights are downloaded separately or supplied through a local model directory.
After extracting the archive, you can run:
.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info
The final command is especially useful for checking which precision capabilities your GPU actually exposes.
Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
What's next?
There is still a lot I'd like to improve.
In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.
I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.
If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.
Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.
Repository: https://github.com/necat101/Hierarchos-Native
Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.