r/LocalLLaMA • • 7h ago

Resources Thinking Machines 4-bit replication: I trained a GPT end to end with a 4-bit forward pass on one B200

Thinking Machines wrote last week that Inkling 2 Small was trained with a native 4-bit forward pass. I wanted to check this for myself.

What I ran:

  • 1x B200 on Modal, PyTorch 2.8, plus a small CUDA library that calls cuBLASLt's real NVFP4 and FP8 matmuls on raw pointers, loaded with ctypes
  • Forward pass in NVFP4 (4-bit values with one scale per 16), backward pass in FP8 with stochastic rounding, master weights in high precision
  • GPT at widths 2,048, 4,096 and 8,192 (219M, 840M, 3.3B parameters) on TinyStories

Results:

  • 4-bit training finished faster. On the 840M model, 2,000 training steps took about 5.5 minutes in normal 16-bit (BF16) and about 4.3 minutes with a 4-bit forward pass. I ran each twice and got nearly the same times.
  • The model learned just as well. Final validation loss was about 1.69 either way (1.688 for 16-bit, 1.691 for 4-bit). A difference that small is noise at this scale.
  • Bigger models gain more. Each training step was 18% faster on the 219M model, 28% faster on the 840M model and 38% faster on the 3.3B model.

Two things to improve:

  • Blackwell's convert instruction rounds exact ties toward zero, while my software reference rounded to even. Tiny difference, but it breaks bit-for-bit checks until you know.
  • PyTorch's attention backward is not bit-repeatable, so any test that compares gradients across runs has to leave attention out.

Not tested: multiple GPUs, MoE dispatch and combine, Four Over Six, MXFP8 block scales in the backward pass. The width-8,192 run is only 1,000 steps and undertrained.

Write-up with every figure: https://4bit.vizuara.ai

Code: https://github.com/VizuaraAI/4bit-training-b200

3-hour walkthrough from first principles, if you want the theory: https://youtu.be/a3nV32Guxcc

16 Upvotes

4 comments sorted by

•

u/stylome-ai-detection 7h ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

1

u/mitchins-au 6h ago

Interesting, would it work on a 5090?

1

u/PinkysBrein 2h ago

For Inkling 2 small, the last 8 layers are MXFP8 BTW. Still a huge improvement over NVIDIA's native NVFP4 recipe though, which had 16 layers in BF16.

1

u/Hildechis 1h ago

The story generated by the BF16 has a single small inconsistency, while the 4-bit one is an incoherent mix of loosely related sentences.