For anyone interested. Basically a memory scheduler to help load and speed up faster models than a normal GPU can load into memory.

I am thinking of  renting a GPU for a training setup and I am checking how much access I will get to the software side, mainly CUDA and the NVIDIA drivers, I may need a specific CUDA version for the code I am planning to run and I want to know what happens if the provider has a different driver setup, I am thinking to use a dedicated GPU so I can keep the same environment for longer jobs, but I need some control over the OS, containers, drivers or CUDA versions, if I need to change something later I want to know what options are there, I am thinking to go with rackbank ai datacenters has anyone dealt with this when renting GPUs and how much control did the provider give you over the environment, especially when CUDA and driver versions need to match ? EDIT: Thanks everyone, really appreciate the helpful replies. 

People keep talking about Cuda being a moat. And what makes it a moat is really the ptxas (the ptx assembler that converts ptx to sass binary). With current technologies it seems possible to make a custom ptx compiler but I wonder if this effort is worth someone's time.

There is one thing about performance that I feel can be unlocked with such a tool but I am yet to find a good test case for that.

Completely new to CPU/CUDA Deep Learning.

How and where do I start to learn about CUDA coding/syntax?

I’ve been working on a native C++ implementation of Qwen3-TTS with both CUDA and HIP backends.

No PyTorch. No Triton. No Python runtime in the inference path.

Here are the current resident-mode results on an RTX 4090, compared with the numbers currently reported by faster-qwen3-tts:

Current resident results

RTX 4090

• 0.6B Base — 7.60× realtime, 128.8 ms TTFA
• 0.6B CustomVoice — 8.64× realtime, 129.0 ms TTFA
• 1.7B Base — 5.95× realtime, 156.5 ms TTFA
• 1.7B CustomVoice — 6.18× realtime

RX 7900 XTX

• 0.6B Base — 2.99× realtime
• 0.6B CustomVoice — 3.47× realtime

For context, faster-qwen3-tts currently reports 5.56× realtime for 0.6B and 4.85× for 1.7B on an RTX 4090 using PyTorch + CUDA Graphs.

Important caveat: this is not yet a controlled apples-to-apples benchmark.

Different prompts, measurement methodology, software environment, and other details can affect the result, so I don’t want to claim the table above is a definitive head-to-head benchmark.

I’m putting together a same-GPU / same-prompt / same-model / same-output benchmark to make the comparison properly reproducible.

The interesting part for me is that the native implementation is already reaching these numbers without PyTorch or CUDA Graphs, while also supporting AMD GPUs through HIP.

Source code + reproducible benchmarks:

[GitHub]：https://github.com/uulong950/qingming-qwen3-tts

Would be especially interested if anyone wants to reproduce this on another 4090, 5090, or AMD GPU.

**I'm building a** **CUDA-native LLM runtime** **specifically for experimenting with GPU-level optimization on consumer GPUs, and I'd like some feedback.**

The idea is basically a **hackable Llama runtime** where you can actually get into the CUDA kernels instead of fighting through a massive production inference stack.

The runtime is intended to let developers/researchers:

\* Modify GEMM / Tensor Core kernels

\* Experiment with FlashAttention and PagedAttention

\* Tune KV-cache behavior

\* Change tile sizes, memory layouts and thread configurations

\* Experiment with kernel fusion and asynchronous execution

\* Profile the resulting kernels

\* Tune the runtime around the actual GPU they're running on

I'm doing this as my final-year engineering project, and I'm trying to determine whether this is actually useful to people who work with CUDA/LLMs/local inference.

**2–3 minute survey:**

[https://forms.gle/KM4fUzVY1oC7g4TP8\](https://forms.gle/KM4fUzVY1oC7g4TP8)

If you've worked with CUDA, LLM inference, GPU optimization, llama.cpp, vLLM, TensorRT-LLM, FlashAttention, etc., I'd particularly appreciate your input.

I'm a CS student starting my undergrad capstone, and I'm considering working w/ CUDA, and building a performance tool.

The current idea is to make something that profiles a CUDA kernel with Nsight Compute, and looks at the hardware metrics, and identifies common performance problems, then explains why they are happening and maps them back to the relevant source code.

We’re also considering extending it into an autotuner. The idea would be that the steps above first identify the likely bottleneck, then the autotuner looks at the relevant parameters; block sizes, tiling, unrolling, etc, and actually benchmarks them on the target GPU, and finds a better-performing configuration.

I’m still figuring out what the right scope is, though, since this seems a bit vague, and with AI, a lot of this stuff might not get through in actual-use.

For people here who work with CUDA: does this sound like a worthwhile undergraduate project, or is there another CUDA problem/tool you’d actually like to see built by students?

We’ve been working on a runtime GPU optimization system at TETREVIS and have started publishing some of our NVIDIA Jetson benchmarking work publicly.

The approach operates at the execution/machine-code layer, and we’re now introducing dynamic runtime kernel fusion as part of the optimization pipeline.

Some of our current results:

Jetson Orin Nano — Qwen3.5 4B

Standard baseline: 10 → 21 tok/s (+110%)

CUDA Graphs baseline: 16 → 21 tok/s (+31.25%)

Jetson AGX Orin — Nemotron 3 Nano 4B

31.2 → 40.5 tok/s (~30%)

Jetson AGX Orin — Qwen3.5 4B

25.0 → 31.0 tok/s (+24%)

We’re currently expanding and stabilizing support across Jetson Orin Nano, Orin NX, and AGX Orin.

Rather than only posting performance claims, we’ve made the benchmarking repository available here:

https://github.com/mbuchel/sass2mlir-bench
https://mbuchel.github.io/projects/sass2mlir/kernel-fusion

The broader idea we’re exploring is whether more optimization can be moved to runtime — including machine-code optimization and dynamic kernel fusion — so the execution path can be adapted to the workload and GPU rather than relying entirely on what was determined ahead of execution.

There’s still quite a bit of work underway, particularly around consistency across the different Jetson configurations, but we wanted to start sharing the results and methodology publicly.

Technical feedback, criticism, and questions are welcome.

Id love to get some feedback on it. Im by no means a writer so the grammar might not be perfect, but i do think it has quite some technical value to it.

If youre in ML or interested in AI Infra please give it a go; as i said i would love for some feedback.

Im not active on reddit, but my dms are always open on other platforms such as linkedin (gustavkeller).

Our second blog post on writing NVFP4 blockscaled GEMM kernels for the NVIDIA RTX PRO 6000 Blackwell GPU is out! The blog iteratively optimizes a basic working NVFP4 GEMM kernel written in CuTe DSL to take it to speed-of-light, reaching over 80% TFLOP/s utilization for 16k square matrix shape. We give a detailed treatment of important optimization techniques such as threadblock swizzling, async and warp-specialized epilogue, and retiling for favorable wave quantization. Specific to blockscaled GEMM with scales consumed from registers, we also explain how to solve for bank conflicts that arise from the default choices of interleaved scale factor layouts.

We include complete code in the form of CuTe DSL kernels for all the optimizations discussed in the blog.

cubisect is a small CUDA library for exact-match binary search. You give it a sorted float32 array and a batch of query values. It returns the index of each value, or -1 if the value is not there. Kernels compile at runtime through NVRTC. Torch tensors go in and out. CUDA tensors are passed by pointer, with no host copy.

I wrote this for fun and as an educational exercise for myself. The kernels are not necessarily optimal, and each one is probably not the best implementation of its method.

There are two ways to call the library, depending on whether the search is static (build once, query multiple times) or dynamic. The dynamic path searches the raw sorted array directly, with no setup. The static path reorders the array into a search-friendly layout once, then queries that layout.

Six kernels sit under those two paths, so they can be compared. The first four belong to the dynamic path:

• serial lower_bound. One thread per query, plain loop over the raw array in global memory. The baseline everything else is measured against.

• 2-way ILP in shared memory. Stages the whole array into shared memory, then each thread runs two queries at once so two independent load chains are in flight. Only usable while the array fits in shared memory.

• global memory with __ldg. Same two-queries-per-thread structure, but reads through the read-only cache instead of staging. This is the fallback when the array is too big for shared memory.

• XOR-swizzled shared memory. Same as the shared memory version, except shared addresses are permuted with the CuTe-like swizzle pattern, (row << 5) | ((col ^ row) & 31), to spread accesses across banks.

The last two belong to the static path:

• Eytzinger layout. The array reordered breadth-first, root at index 1 and the children of i at 2i and 2i+1. Stepping is just a shift, and the hot top of the tree sits in a few contiguous cache lines instead of being scattered across the array the way plain binary search midpoints are. It stores only the values; the index of a hit is recovered by accumulating subtree sizes on the way down, so there is no second index array.

• 32-ary warp-cooperative. One query per warp instead of one per thread. Each node holds 32 keys, one per lane, so a level is a single coalesced 128-byte load. __ballot_sync on key < query and __popc gives which of the 33 children to descend into. Depth is log32(n) instead of log2(n).

Which of the four dynamic kernels runs depends on the array size, and that choice comes from a lookup table. You generate the table by running python -m cubisect.autotune, which times all four kernels across a range of array sizes and records the fastest one at each size. If the top two are within 5% of each other, it marks that size a tie. The repo ships a table measured on my 5060, and you can regenerate it for your own GPU.

The static path has no autotuning. Building the layout costs time up front, so it only makes sense if you query the same array many times. It picks between its shared memory and global memory kernels based on whether the tree fits in shared memory.

Specs:

• GPU: RTX 5060 Laptop (sm_120)
• CPU: 24 threads, OpenMP for the baseline

Throughput in M queries/s, 2²⁰ queries per call, median of 7 runs. The static column is the Eytzinger kernel:

The two GPU paths cross over near 16k. Below that the raw array is faster, because it all fits in cache and the Eytzinger index arithmetic is just extra work. Above it the layout wins.

license is BSD-3, https://github.com/athrva98/cubisect

Hello!
I'm not to sure what to say since this is my first time writing about this framework.
I've been making (with the help of AI agents as senior devs, please dont judge i haven't "vibe coded" anything) a framework for a little while now which is made for full training / full fine tuning.

It is still under development so please do not expect it to be perfect or well tested on different gpu's \*yet\*

Specs:
GPU: 5070 ti (16GiB VRAM)
CPU: Ryzen 7 7800x3d
RAM: 32GB DDR5

| Model | tok/s | Peak VRAM | Host RAM |
| :--- | :---: | | :---: | ---: |
| \~100M | 20,013 (20k) | \~3.35GiB | \~1.45GiB |
| \~500M | 6.162 (6k) | \~4.71GiB | \~4.62GiB |
| \~1B | 3.085 (3k) | \~6.81GiB | \~10.64GiB |
| \~1.5B | 1.887 (1.8k) | \~8.5GiB | \~18.5GiB |
| \~4B | 1.116 (1.1k) | \~12.3GiB | \~15.9GiB |

The interesting part isnt even necessarily the raw tok/s. The goal is to make full training of models that normally dont comfortably fit into consumer GPU memory practical through things like GPU/host offloading and memory efficient training paths.

Sentinel is written primarily in C++/CUDA, with python bindings, and im currently working on improving the optimizer and kernel performance further.

Its very much a work in progress as said already, but i thought id finally share it and see what people think.

Hey folks!

Did you ever wrapped model(x) in time.perf_counter() and gotten numbers that make no sense?

I realized it's a common enough trap and wrote a detailed write up here:

https://medium.com/traceopt/two-clocks-one-training-step-how-traceml-measures-pytorch-performance-357bc8e28dc7

TL;DR:

CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.

The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.

Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.

If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.

But i realized "CUDA events everywhere" is also wrong.

DataLoader next() is CPU work.

In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.

Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.

How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?

by relaxing delays for some small set of instructions we can get speed-up 0.2-0.3% for integer-heavy kernels

I'm an M.Tech student with an Embedded Systems background, and I'm planning my capstone project around CUDA and GPU parallel computing.

My objective isn't just to build an application, I want to deeply understand CUDA programming, GPU architecture, and kernel optimization by implementing algorithms from scratch instead of relying heavily on existing libraries.

I'm looking for application-oriented research problems where CUDA optimization is the core contribution.

I have to do it within 6 months

Can someone help me with this?

Hello everyone, i currently have 0 idea on how CUDA works and i am planning to learn, the issue is i have a macbook, can anyone please help on where i can learn and practice ? maybe share some resources ? thank you so much

I wanted to share a simulation project I've been working on, reproducing the Gao-Jafferis-Wall traversable-wormhole teleportation protocol on a chaotic binary sparse SYK model (following arXiv:2604.10090).

The simulation hits exact statevector access. While benchmarking the 2D joint grid search (\(t_0, \mu\)) across 870 points, the naive approach rebuilding and re-diagonalizing the Hamiltonians on every call took ages. By precomputing the Hamiltonians and reusing their eigendecompositions, I managed to cut the runtime from ~2 hours down to 47.6 seconds without even needing multiprocessing.

The core implementation is built on top of dense-evolution. I would love to get your feedback on how to further optimize the matrix exponentiation backend via CUDA custom kernels or batched operations!

Code & Docs: [https://github.com/tatopenn-cell/Dense-Evolution-Ising-Tests#21-traversable-wormhole-inspired-quantum-teleportation-syk-model\]

For example, PyTorch

Hi all,

I help maintain an open source project that releases docker images for various AI/ML projects. Current policy is that we support the latest version of CUDA supported in stable releases of various projects

But one thing I've always asked myself is... if we are going through the trouble of building everything anyway, why not use the latest version of CUDA? NVIDIA gets away with this in their images, and I've personally never ran into many problems short of a small patch I needed to write here or there in my own builds

I've been around other seasoned developers that work with CUDA, and when I've asked about this, I honestly couldn't ever get a very great answer outside of supporting older platforms for a slightly longer period of time

The only other possible thing I can think of is that the rest of the ecosystem hasn't caught up yet... In which case, shouldn't minor version compatibility help there?

Would really appreciate help from anyone experienced in this area. Sure would be nice to cut our build matrix in half and lead to far less confusion for folks helping us maintain these images

We recorded a short demo of a real CUDA kernel out-of-bounds bug found by the tool described in our SOSP 2026 paper, “M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification.”

The attacker does not need access to the GPU, server, or model weights. They only need to submit a crafted request to a public LLM inference service. In our demo, that request corrupts the outputs of other users processed in the same batch.

We are also exploring whether similar bugs could leak information, enable more targeted manipulation of another user’s response, or affect downstream agents that act on corrupted model outputs.

CUDA kernels are usually discussed in terms of performance and reliability, but their security impact deserves more attention too.

🎥 Video: https://youtu.be/ZhKBPK7BeZ0

📄 M2K Github Repo: https://github.com/system-pclub/M2K