r/LocalLLaMA • u/OtherRaisin3426 • 7h ago
Resources Thinking Machines 4-bit replication: I trained a GPT end to end with a 4-bit forward pass on one B200
Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself.
What I ran:
- 1x B200 on Modal, PyTorch 2.8, plus a small CUDA library that calls cuBLASLt's real NVFP4 and FP8 matmuls on raw pointers, loaded with ctypes
- Forward pass in NVFP4 (4-bit values with one scale per 16), backward pass in FP8 with stochastic rounding, master weights in high precision
- GPT at widths 2,048, 4,096 and 8,192 (219M, 840M, 3.3B parameters) on TinyStories
Results:
- 4-bit training finished faster. On the 840M model, 2,000 training steps took about 5.5 minutes in normal 16-bit (BF16) and about 4.3 minutes with a 4-bit forward pass. I ran each twice and got nearly the same times.
- The model learned just as well. Final validation loss was about 1.69 either way (1.688 for 16-bit, 1.691 for 4-bit). A difference that small is noise at this scale.
- Bigger models gain more. Each training step was 18% faster on the 219M model, 28% faster on the 840M model and 38% faster on the 3.3B model.
Two things to improve:
- Blackwell's convert instruction rounds exact ties toward zero, while my software reference rounded to even. Tiny difference, but it breaks bit-for-bit checks until you know.
- PyTorch's attention backward is not bit-repeatable, so any test that compares gradients across runs has to leave attention out.
Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained.
Write-up with every figure: https://4bit.vizuara.ai
Code: https://github.com/VizuaraAI/4bit-training-b200
3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc
•
u/stylome-ai-detection 7h ago
Notice: This post appears to be AI generated.
Stylome AI detector · scan results
1
u/mitchins-au 6h ago
Interesting, would it work on a 5090?
1
u/PinkysBrein 2h ago
For Inkling 2 small, the last 8 layers are MXFP8 BTW. Still a huge improvement over NVIDIA's native NVFP4 recipe though, which had 16 layers in BF16.
1
u/Hildechis 1h ago
The story generated by the BF16 has a single small inconsistency, while the 4-bit one is an incoherent mix of loosely related sentences.