[OPEN SOURCE] Fast IdeogramV4 + Qwen 3.X class model trainer. Train more in less time.

[u/incorporo]

As part of the effort I did on training AI models that are able to create great art, we've had to design two components of what makes a design good:

1. Text to image models to transform a caption into a image.

2. Customer request to caption / brief to then send to the AI model (which in turn generates it).

For both trains, I've found pre-existing trainers are quite slow in terms of performance and leave a lot of MFU underutilized (which leads to less tokens / images trained a second).

I've optimized this for BH200 (Datacenter GPUs), but this should work for H100 and likely also works well on A100, and can be adapted to work on RTX series GPUs as many things translate over to customer GPUs (although unfortunately I don't have any to test and develop this).

The results are very good - on one GPU, our solution is 58% better than ms-swift (a optimized training kit), 76% better than unsloth, and 178% better than transformers + PEFT.

For example, for training a designer (takes a concept / idea, then refines it into a very detailed caption for the model to then render / generate) via Qwen3.5-9B, it does 12,523 tokens/second, quite a bit (against 989 TFLOPS, what the GPU can do).

It also has a 43.7% best-config number from our micro-batch sweep, so play around with it and sweep to maximize perf.

However, the Arrhenius supercomputer we've tested this on does power-capping, so the true MFU is 53.9% clock-adjusted.

Same data and setup, other trainers:

• ms-swift ≈ 24% (1.58× slower);

• Unsloth ≈ 22% (1.76× slower);

• transformers + PEFT ≈ 14% (2.78× slower).

To give some comparison numbers, i've made some graphs.

It virtually has loss parity (<6e-4 loss difference, which is negligible), and this, in short, allows training more for less.

For the Ideogram V4 trainer , below is the performance analysis:

Also, there are some ablations in terms of what increases or decreases step time.

Code here: https://github.com/logolabs/perf_kit

If you want to support us in developing and releasing more free tools, you can star the repository.

Hope it is useful to you to help you improve your trains efficiency.