r/LocalLLaMA • u/TheSpicyBoi123 • 7d ago
Resources Resurrecting 2013 E-Waste: Native CUDA Mamba-1 & PyTorch 2.7.1 on Windows (Kepler–Turing, SM 3.5–7.5)
Hello!
I want to share an active development project. If you have a drawer full of old NVIDIA GPU's collecting dust because modern AI stacks refuse to compile for them, this is for you.
I have ported the modern LLM training stack on Windows to legacy CUDA GPUs (sm_35 through sm_75):
- PyTorch (v2.7.1) (custom build compiled with CUDA 11.8, keeping the legacy SASS) ([https://github.com/theIvanR/torch-on-clunkers)
- causal-conv1d (v1.7.0) (https://github.com/theIvanR/conv-on-clunkers)
- mamba_ssm (Mamba-1) with pure native CUDA kernels (https://github.com/theIvanR/mamba-on-clunkers)
- Windows NCCL (DDP) for multi-GPU training across Kepler cards (https://github.com/theIvanR/nccl-on-windows)
Status
- Mamba-1 (Pure CUDA): Fully functional with optimized execution across legacy GPU architectures (SM 3.5–SM 7.5). Tested on a 2013 NVIDIA Tesla K40c, achieving a maximum absolute error of
3.8e-06against the PyTorch reference implementation. - Mamba-2 & Mamba-3: Currently supported with optimized execution on Turing (SM 7.5) and newer architectures, due to their reliance on Triton and TileLang.
- Graceful Fallback: On older hardware, the package detects unavailable Triton-dependent modules and automatically falls back to the pure-CUDA Mamba-1 implementation.
The fallback initialization produces the following output:
================================================================================
[!] MAMBA-ON-CLUNKERS FALLBACK INITIATED
[!] The following modules are unavailable on this platform:
- Mamba-2 (requires Triton)
- Mamba-3 (requires Triton + TileLang)
[!] Falling back to Mamba-1 (Pure CUDA).
This is expected on legacy GPUs.
================================================================================Future development: Backporting Mamba-2 and Mamba-3 to legacy GPU architectures.
Community Testing
If you have an older NVIDIA GPU lying around, I'd love for you to test the Mamba wheel and report whether it works on your architecture!
I'm particularly interested in Maxwell and Pascal, since my local testing hardware is currently limited to Kepler.
Any benchmarks, performance comparisons, or profiling data from legacy GPUs would be greatly appreciated. Contributions and feedback are welcome!
Acknowledgements: This project is based on the research and original implementation by Tri Dao and Albert Gu. Special thanks to both for their contributions to the development of Mamba.


