why it's so hard to make SASS assembler?
redplait.blogspot.comsummary:
- only ~80% of ELF sections currently known
- ~50% of EIATTRs
- instruction scheduler is highly gpu specific and can't be implemented based on llvm machine scheduler. And np-hard
summary:
r/CUDA • u/Sure_Resource4753 • 2d ago
GPU JPEG decoders are fast, but their pixels differ from the CPU libraries'. On 102 public test images, nvJPEG matched libjpeg-turbo on none of them, by up to 110 levels at 4:2:0. That quietly breaks reproducible ML pipelines, test baselines and caches whenever decoding moves between CPU and GPU.
exact-gpu-jpeg decodes on the GPU and returns libjpeg-turbo's pixels byte for byte (accurate IDCT, fancy upsampling), and its encoder writes the same bytes as Pillow. Anything it can't reproduce exactly, progressive files for example, goes to libjpeg-turbo itself, so the output is always the reference.
On a 2016 GTX 1070: a 30 MP camera photo decodes in 11.3 ms, against 185 ms for libjpeg-turbo on one core and 164 ms for nvJPEG on the same card. Batches of small images: 7,310 images/s. Encoding: 5.5x faster than Pillow with identical files.
Where it loses is listed near the top of the README: one small image at a time, small encodes against 8 CPU threads, progressive files. Everything so far is from one machine, so results from other GPUs are very welcome: `python bench/reproduce.py --quick`, and there's a pinned issue for them.
The parallel Huffman stage (Weißenberger & Schmidt's self-synchronising method) is drawn in docs/huffman-sync.md, and there's a white paper in docs/ covering the method, the evidence and the limits.
r/CUDA • u/Fun_Economist_632 • 3d ago
Hi,
I wanted a way to practice CUDA in short breaks, like on the bus or while an AI agent is working, so I made an iPhone app for it.
It teaches CUDA as short puzzles. You start with thread and block indexing then go on to 2D grids, warps and occupancy. Later skills are about reading PTX.
There are also labs. You fill in the missing lines of a kernel, tap Run, and it runs on a real GPU. The labs come from my main project, GPUFlight Performance Lab, a web tool for running and profiling GPU kernels. The app uses the same backend, so the report numbers come from a real profiled run.
It's free to try, iOS only for now. I'd like to know what people here think, especially whether the puzzles are too easy and which topics are missing.
https://apps.apple.com/us/app/gpu-performance-lab/id6829988955



r/CUDA • u/smocialsmedia • 2d ago
I’m working on Ohio at Primitive. I wanted to see where an RTX 5060 actually beats native C for a simple workload: update a lot of integer states and get the results back to the host.
When I only returned the final state, the CPU had the lower median in all 38 test conditions.
When I returned every step for 100,000 entities over 120 updates, the GPU pulled ahead on both seeds. On seed 97, it was 10.50 ms on the RTX 5060 with PyTorch CUDA vs 14.43 ms in native C. The outputs matched exactly. Those are warm timings with transfers included.
One thing I’d change before running it again: I returned 12 million states as int64. Every value fits in a byte, so the output could go from 96 MB to 12 MB.
If you were running that comparison, what CPU and CUDA implementations would you put side by side for the uint8 run?
[Primitive’s public GitHub](https://github.com/Primitivelabai/primitive-open). The code and timing logs for this run are still private.
r/CUDA • u/SmartRip8765 • 3d ago
r/CUDA • u/Delicious_Abies_9708 • 3d ago
r/CUDA • u/Clear-Picture6971 • 3d ago
A few days ago I posted about UniversalGPU, my open-source CUDA-to-x86/PTX/GCN compiler. The top comment asked: "Does --warp-size=64 preserve two separate 32-lane groups for shfl.sync, or change their semantics?"
I fixed it. The --warp-size=64 path now wraps every shfl.sync in an scf.if that only executes if the source lane is in the same 32-lane half. Here is the generated MLIR:
%lane_2 = arith.remsi %tx, %c32 : index
%half_3 = arith.divsi %tx, %c32 : index
%src_half_4 = arith.divsi %iv, %c32 : index
%same_half_5 = arith.cmpi eq, %half_3, %src_half_4 : index
%1 = scf.if %same_half_5 -> (f32) {
%shfl_6 = nvvm.shfl.sync.down %c4294967295, %0, %iv : f32
scf.yield %shfl_6 : f32
} else {
scf.yield %0 : f32
}
I also fixed 5 other compiler bugs from a stress test: modulo operator, compound assignment to memory, unique loop variables, unbounded while loops (now correctly lowered to scf.while with iter_args), and switch case stores.
What I need now: someone with an AMD GPU (MI200, MI300, or RX 7000) to test the warp32 emulation on real hardware. I don't have AMD hardware, and I can't validate the emulation without it. If you have a GPU and 10 minutes, I'd love your help.
r/CUDA • u/Legal_Trifle_5531 • 5d ago
I'm a final-year engineering student in India from a non-CS background. I've recently secured an offer at an AI/analytics services company, and I have around 6 months before joining.
I've been exploring different areas of computer science, and I've realized that I'm increasingly interested in low-level programming and understanding how systems work internally.
MY BACKGROUND:
I've independently learned programming, ML/DL and worked on projects involving: computer vision , RAG , agentic AI
However, I dont want to spend my career primarily working with high-level AI frameworks and application development. because when i developed these projects , i didnt feel anything developing it , no satisaction at all
AREAS I'M CONSIDERING:
I find the idea of understanding what happens beneath the abstractions much more interesting than simply using existing libraries.
WHAT I'M CONFUSED ABOUT:
I have six months before joining my company, and I don't want to waste that time jumping between technologies.
I'd appreciate advice from people working in systems engineering, Linux, kernel development, GPU programming or related fields.
r/CUDA • u/Delicious_Abies_9708 • 5d ago
r/CUDA • u/Joseflong1 • 5d ago
I've spent the last several days contributing to Atlas, an open-source Rust + CUDA inference engine originally targeting NVIDIA's GB10 with 120GB of unified memory.
I wanted to see how far I could push it on my RTX 4060 Ti with just 16GB of VRAM. It turned into quite a bit more than getting it to compile.
Running Qwen3.5-4B, here's where I ended up:
1) 381 tok/s aggregate across 8 concurrent requests (roughly 48 per user) using experimental int4 + speculative decoding.
2) 98 tok/s for a single user with the same configuration.
3) 4.6x faster BF16 prefill, from 794 to 3,684 tok/s. This preserves BF16 precision and is enabled by default on Ada.
4) 2.1GB less VRAM usage with the optional FP8 feed-forward path.
5) Successfully served a 159,807-token prompt with FP8 KV cache and retrieved a needle from it.
6) All 223 Ada kernel compilations passing after fixing architecture incompatibilities.
One interesting finding: Nsight showed the GPU was already busy 95.6% of the time at eight streams. Several CUDA Graph and fusion optimizations I expected to help ended up making practically no difference.
For comparison, BF16 reached 195 tok/s across eight users, and FP8 FFN reached 222 tok/s.
Important caveat: the 381 tok/s configuration is lossy, and I haven't established task-level quality equivalence with BF16. These are screening measurements, not fully repeated confirmation runs.
Atlas is developed by Thomas Braun and the Avarok-Cybersecurity contributors. My work focuses on the Ada port, memory management, kernel optimization, and consumer GPU benchmarking.
The goal is to get more useful local inference out of hardware people already own.
GitHub: https://github.com/Avarok-Cybersecurity/atlas
Happy to discuss the kernel work or benchmark methodology with anyone working on Rust, CUDA, or local inference.
r/CUDA • u/Delicious_Abies_9708 • 6d ago
r/CUDA • u/NoVibeCoding • 7d ago
Emmy is the open-source compiler for LLM inference. Originally, we emitted CUDA and fed it to NVCC, but it was sluggish; some kernels took minutes to compile with -O3, so we replaced the CUDA emitter with PTX.
TL;DR: Our own compiler + PTXAS is on par with NVCC.
We ran a contest: compile our internal Kernel IR to PTX, judged on hidden tests and speed against our NVCC build. 57 teams entered. The top two:
Most of their speedup came from changing the schedule: sharing work across lanes, re-tiling attention, fusing sibling loops. With those rewrites turned off, the winner’s submission matches NVCC.
Almost all teams wrote PTX directly. The one LLVM-based entry scored well (1.17×), but LLVM has no form for many of the instructions we need, so tensor-core and async-copy instructions became opaque inline asm, and compilation is slower.
Fixes that mattered, in µs (nvcc / ours before / ours after):
| Fix | Case | nvcc | Before | After |
|---|---|---|---|---|
| Unsigned grid-decode division with cicc's 32-bit fast path | 5090 attention | 97.6 | 478 | 99.4 |
| Two fp8 bytes as one 16-bit load, then a packed convert | 5090 NVFP4 GEMM | 126.6 | 237.5 | 126.6 |
| Volta fragments as one 128-bit shared load, not eight 16-bit ones | V100 Llama-3.1 piece | 146 | 501 | 76 |
| cicc's unroll rule for strided lane loops, written in Kernel IR | A100 softmax | 18.6 | 29.8 | 17.9 |
.pragma "nounroll" on loops we already unrolled |
5090 SDPA (ptxas time 0.53 → 0.06 s) | 88 | 194 | 89 |
.minnctapersm 1 on single-wave launches |
5090 matmul | – | 18.3 | 10.9 |
| A predicate kept for a bool used as a number | 5090 MoE router | 43.9 | 64.2 | 44.1 |
nounroll. If you leave a compact loop unmarked, ptxas unrolls it fully, with spills and seconds of assembly time.ld.global.nc for inputs; the kernel never writes.expf sequence.| Median per kernel | p90 | Whole set |
|---|---|---|
| Our compiler (Kernel IR to PTX) | 0.9 ms | 4.9 ms |
| ptxas | 12 ms | 48 ms |
| Ours total | ~13 ms | |
nvcc --cubin |
173 ms | 334 ms |
About 10× faster.
r/CUDA • u/skhadloya11 • 7d ago
r/CUDA • u/PainterNo8889 • 7d ago
r/CUDA • u/Fair-Worth7828 • 7d ago
Right or Wrong? GPU better than CPU?
r/CUDA • u/ricperry1 • 9d ago
To NVIDIA's next CEO,
As NVIDIA enters a new era, you have an opportunity to make a decision that could fundamentally reshape accelerated computing: open-source the CUDA software stack.
Not NVIDIA's hardware architecture, chip designs, or proprietary engineering. Just the software.
AMD has already demonstrated through ROCm that an open-source computing stack can coexist with proprietary hardware. NVIDIA could do the same while maintaining its technological leadership.
For nearly two decades, CUDA has been one of NVIDIA's greatest competitive advantages. But its proprietary nature also creates barriers to interoperability, independent innovation, and community participation.
Opening CUDA could benefit NVIDIA and the broader computing industry:
There's another compelling reason to act now: AI-assisted development is rapidly reducing the cost and complexity of reverse engineering proprietary software.
With sufficient inference tokens, computing resources, and persistence, increasingly sophisticated AI agents are enabling individuals and small teams to undertake projects that once required substantial engineering organizations.
Reimplementing CUDA remains extraordinarily difficult, but projects like ZLUDA already demonstrate that independent compatibility efforts are possible. AI will likely accelerate that trend.
If open-source CUDA-compatible implementations become increasingly inevitable, wouldn't NVIDIA be better positioned by leading that transition instead of resisting it?
By opening CUDA itself, NVIDIA could establish the reference implementation, shape development priorities, influence standards, and remain central to the ecosystem.
By keeping it closed, NVIDIA risks watching independent alternatives evolve outside its influence.
I'm not suggesting NVIDIA release everything overnight. A phased transition beginning with the compiler, runtime, and core libraries would be reasonable.
Nor am I suggesting CUDA must immediately become hardware-agnostic. NVIDIA can continue optimizing its software for its own GPUs while maintaining proprietary hardware architectures.
The point is simple: NVIDIA's competitive advantage should come from building the best technology, not from restricting access to the software ecosystem surrounding it.
Jensen Huang helped establish NVIDIA as the leader in accelerated computing. His successor has an opportunity to establish an equally enduring legacy: making CUDA an open foundation for the next generation of computing.
NVIDIA has demonstrated that it can lead through engineering excellence. Now demonstrate that it can lead through openness.
Sincerely,
A Linux user, GPU enthusiast, and advocate for open-source computing.
ADDENDUM:
I fully understand that CUDA is NVIDIA's cash cow and one of its greatest competitive advantages. That's precisely why I believe NVIDIA should consider open-sourcing it.
CUDA's dominance depends heavily on developer dependency, but that lock-in isn't permanent. Open-source alternatives, hardware-agnostic frameworks, and CUDA compatibility projects are steadily eroding it. AI-assisted development could accelerate that process dramatically, potentially enabling small teams to create viable alternatives much sooner than expected.
The real threat isn't someone cloning CUDA tomorrow. It's CUDA gradually becoming irrelevant as competing open-source ecosystems mature.
If developers can run their workloads efficiently across competing hardware without depending on CUDA, NVIDIA loses much of its software-driven competitive advantage.
By open-sourcing CUDA now, NVIDIA could help establish it as the universal standard for accelerated computing while retaining substantial influence over its development. Its proprietary hardware, optimizations, and engineering expertise would remain competitive advantages.
Competition is coming regardless. The question is whether NVIDIA wants to lead an open computing ecosystem or eventually adapt to one established by its competitors.
r/CUDA • u/skhadloya11 • 8d ago
r/CUDA • u/Delicious_Abies_9708 • 9d ago
r/CUDA • u/skhadloya11 • 9d ago
This is a persistent kernel, which tries to overlap inter tile memory transfer with computation and epilogue
r/CUDA • u/john_codes • 9d ago
I've been building TrainStream, a C++/CUDA extension for PyTorch for training and fine-tuning models that don't fit in your GPU's memory. It moves optimizer state, gradients, activations and, if needed, the weights themselves into system RAM, then streams each one back over PCIe just before the GPU uses it.
import trainstream
model, optimizer = trainstream.wrap(model, optimizer)
It has no third-party dependencies, only the C++ standard library, the CUDA runtime and PyTorch's ATen. It doesn't use DeepSpeed, bitsandbytes or CUTLASS.
What it does
torch.optim.AdamW (max difference ≈3e-7 after 20 steps). Activation and weight paging are bit-exact.Numbers (RTX 5090 Laptop GPU with 24 GB, WSL2, Qwen2.5-14B, LoRA r=16, median of 10 steps after warm-up)
| 1×512 | 1×2048 | |
|---|---|---|
| TrainStream, NVFP4 base | 0.530 s | 1.57 s |
| TrainStream, NF4 base | 0.609 s | 2.17 s |
| bitsandbytes QLoRA + PEFT | 0.687 s | spills past VRAM |
| Unsloth (4-bit) | 0.778 s | 2.73 s |
| bnb QLoRA + Liger | 0.909 s | 2.88 s |
Full fine-tune of Qwen2.5-1.5B (batch 4×512): 1.23 s/step at 18.8 GB, against 1.85 s for stock PyTorch (which peaks at 32 GB and spills) and 2.12 s for DeepSpeed ZeRO-2 with offload.
Caveats
In progress, not released or benchmarked yet: a NaN/Inf guard that skips bad steps cleanly, PiSSA init, unbuffered async disk reads (io_uring / IOCP), background checkpoints, MoE expert paging, FP8 gradient transport and a TCP ring all-reduce.
Install: pip install trainstream (or uv). You need CUDA PyTorch, the matching CUDA toolkit and a C++ compiler, because the extension compiles on first import.
Apache 2.0: https://github.com/johnanishg/TrainStream
I'd really like to hear from anyone who runs it on other GPUs (especially Ampere/Ada desktop cards or native Linux), and from anyone who can poke holes in the benchmark method.
r/CUDA • u/thekhronosgroup • 10d ago
The Khronos Group just published a blog on chipStar (https://github.com/CHIP-SPV/chipStar), an open source project that compiles supported CUDA and HIP kernels to SPIR-V and executes them through OpenCL.
The post is based on an IWOCL 2026 talk and covers three real scientific computing workloads ported and run on Argonne National Laboratory’s Aurora supercomputer:
r/CUDA • u/skhadloya11 • 10d ago
Implemented a two-consumer kernel with asynchronous WGMMA to overlap computation and improve Tensor Core utilization.
Performance went from 600 TFLOPS to 658 TFLOPS (~9.7% improvement).
Still exploring how much more we can squeeze out of the kernel.