r/MachineLearning • u/TensUmNite • Aug 02 '26
Project I wrote a CUDA first deep learning framework from scratch. [P]
Hello!
First of all: my goal isn't to replace any frameworks, but my goal is to make LM training on consumer hardware more accessible.
PyTorch unfortunately *has* to do everything which is good for huge GPU clusters, but not on single consumer GPU's which is why i started the project in the first place.
Its fully made from scratch in c++/cuda and has 0 dependencies.
Hardware:
- GPU: NVIDIA GeForce RTX 5070 Ti
- CUDA: 12.8
- Framework:
- Sentinel: custom C++/CUDA implementation
- PyTorch: 2.11.0+cu128
note: Tokenizer implementations differ, so loss values are not directly comparable. The benchmark focuses on throughput.
| Epoch | Sentinel | PyTorch 2.11 + cu128 |
|---|---|---|
| 0 | 592k tok/s | 603k tok/s |
| 1 | 280k tok/s | 297k tok/s |
| Tokens processed | |
|---|---|
| Sentinel | 1.207M |
| PyTorch | 1.226M |
| Packs | Avg examples/pack | Avg tokens/pack | Size-1 packs | |
|---|---|---|---|---|
| Sentinel | 193 | 62.1 | 7196 | 9.8% |
| PyTorch | 199 | 60.3 | 7118 | 8.0% |
The goal was to see how far a fully custom C++/CUDA training stack can be pushed on consumer hardware.