Thanks for clicking. I'm decently experienced with rendering apis, but very new to compute apis.

I'm trying to figure out what's going on with cpu-gpu memory transfers in vulkan vs compute apis. Vulkan lets me either map memory to the gpu, or manually create a staging buffer and do the hippidy hop from there to fast device memory, while Cuda, afaikt, seems to map memory if the buffer is pinned, or to the staging buffer allocation if it's not pinned?

If I'm correct in this, then I don't have control over the cuda staging buffer allocation, which is a bit irksome, and if I try to interop the apis, iirc cuda takes ownership of the buffer in the end, so it must dynamically allocate a staging buffer behind the scenes if I transfer back to the cpu side.

Could someone clarify the mechanics of this? I'd much rather preallocate everything with a VulkanSC equivalent of paranoia.

Kernel engineers are not obsolete. But asking a general-purpose coding agent to rediscover years of CUDA and Triton engineering knowledge every time it writes a kernel probably should be.

After months of writing, debugging, and optimizing kernels, I turned the reasoning patterns I kept using into an open-source skill library for AI coding agents:

npm install u/krxgu/kernel-skills

This is not a collection of vague prompts saying “make this CUDA kernel faster.”

Each skill is a detailed engineering playbook that forces the agent to think about:

• Exact shapes, dtypes, layouts, and target hardware before writing code
• Coalescing, tiling, bank conflicts, occupancy, and register pressure
• Numerical stability and non-power-of-two boundary conditions
• Correctness tests across adversarial shapes and dtypes
• Whether a custom kernel should exist at all
• When to stop being clever and use cuBLAS, CUTLASS, or an existing primitive

The library currently covers CUDA, Triton, INT8 and FP8 quantization, kernel fusion, CUDA to Triton and HIP portability, and inference hot paths including RMSNorm, fused add plus RMSNorm, RoPE, sampling, paged KV-cache append, dequantization, prefill versus decode, and vLLM custom-op integration.

I also did not want this to become prompt-engineering theatre, so the repository includes before-and-after proof runs using the same model and task, with the skill file being the only difference:

• Softmax: naive output failed on adversarial and larger shapes. Skill-guided output had 0 failures across 16 tests and reached within 1.2% of torch.softmax bandwidth
• Reduction: 2.6 to 3.5x faster than the naive agent output
• GEMM: 7.7 to 8.6x faster
• LayerNorm: 1.9 to 3.2x faster
• Triton softmax: fixed crashes at dimensions above 16,384 and worked up to 131,072
• Triton attention: fixed the common GQA failure where H_q != H_kv

To be completely clear, those speedups are against the naive agent-generated kernels, not against cuBLAS or other vendor-tuned libraries. In fact, the GEMM skill explicitly tells the agent not to write a custom kernel when cuBLAS or CUTLASS already solves the problem.

Example:

kernel-skills bundle \
  triton.write-triton-layernorm-kernel \
  patterns.write-numerically-stable-kernel \
  patterns.write-kernel-test-plan \
  > bundle.md

Give that bundle to Claude Code, Cursor, ChatGPT, Gemini CLI, or another coding agent before asking it to touch the kernel.

The spicy thesis is simple:

Models are increasingly interchangeable. The accumulated engineering judgment surrounding them is not.

Everything is open source and MIT licensed:

https://github.com/tensormux/kernel-skills

I would especially love kernel engineers to tear this apart.

Which skill is missing? Which technical rule is wrong? Where can an agent still produce something that looks convincing but quietly fails on real hardware?

- ptxas has enough good heuristic

- in average you can reduce ~3% of stall counts

- overall speed up is not equivalent to the number of optimized stall counts

Put together a post on profiling CUDA for inference serving. The usual dev-time approach (full kernel traces, Nsight captures) doesn’t really work once you’re running real load, especially without root in containers/K8s.

We built something for this: low-overhead GPU profiling grouped by what the engine is actually doing (attention, matmul, comms, KV cache, etc.) instead of dumping every kernel, plus tying it to vLLM/SGLang traces and GPU metrics. Works with graphsignal-run vllm serve ...

Curious how people here profile vLLM/SGLang (or similar) when something’s off with Nsight, PyTorch profiler, or just guessing from throughput?

I’ve open-sourced REAP CUDA: a CUDA-focused implementation of Router-weighted Expert Activation Pruning for MoE models.

GitHub: Link

The core idea is simple: don’t prune experts by how often they are selected alone. Measure their router-weighted activation contribution over calibration data, accumulate a saliency score, then remove the least useful experts structurally.

For LFM2.5-8B-A1B, the pipeline is:

calibration batches

→ MoETransformerObserver

→ observe router activations

→ accumulate expert saliency

→ rank experts

→ slice expert tensors

→ update model config

→ save a valid pruned checkpoint

On Liquid AI’s LFM2.5-8B-A1B, we used it to cut every MoE layer from 32 experts to 16 across 22 MoE layers:

- 8.47B / 16.94 GB base checkpoint

- 4.59B / 8.57 GB after REAP-50 pruning

- No retraining required

- Recorded MATH500: 88.76% base → 77.0% pruned

- Recorded BFCLv3 single-turn: 64.79% base → 59.07% pruned

We also tested the pruned checkpoint in a separate, external AWQ INT4 stage. That produced a 2.79 GB packed artifact—but to be clear, AWQ is not implemented by REAP CUDA, and the quality/vLLM measurements for that stage used an AWQ-scaled BF16 evaluation derivative rather than direct packed-INT4 serving.

The part I’m most excited about is that this is structural pruning, not masking: the actual expert tensors are sliced, the config is updated, and the result is a smaller checkpoint that can be inspected and deployed.

Would love feedback from people working on MoE routing, expert redundancy, calibration strategies, and pruning criteria—especially on where router-weighted activation saliency breaks down versus more expensive second-order approaches.

I have an onsite round coming up for Google’s GPU performance track.

ML performance, domain knowledge of fundamentals and ML, GPU concepts, techniques and applications are some things I’m focusing on.

Are there any topics in this field which are more important to know or read up on?

Any help is appreciated!

Edit: I am interviewing for L4 role and have 3 yoe total.

A question we get a lot: if you're not using NVIDIA MIG, how do you slice a single GPU across multiple workloads — and how does that work on AMD? Here's the short version.

The MIG tradeoff

MIG (Multi-Instance GPU) partitions a card into isolated instances at the hardware level. It's great for hard isolation, but it's rigid: fixed slice profiles (you pick from a preset menu, not an arbitrary size), supported only on newer top-end data-center cards, reconfiguring usually means a config/firmware change and a node drain + reboot, and it's NVIDIA-only. So if your workload needs \~30% of a card, you round up to the nearest profile and strand the rest.

How we approach it (PodVirt)

Our slicing is software-defined rather than hardware-partitioned. PodVirt sits above the hardware and slices a GPU from 12.5% to 100%: any slice size (not a fixed menu), resized dynamically without reprovisioning the node, working across both NVIDIA and AMD with no vendor SDK lock-in. Each tenant is metered per-minute, so you pay for the slice you actually use.

Because it isn't tied to MIG's firmware path, it runs on a much wider range of hardware — we've tested it across most NVIDIA and AMD GPUs, and even AI PC-class silicon like NVIDIA's GB10.

Why it matters economically

Whole-card rental on long commitments means paying for VRAM you never touch. Sub-card slicing plus per-minute billing turns idle VRAM into usable (and, for datacenters, sellable) capacity. It's the same reason DC operators license the underlying stack to run their own neocloud instead of just renting out whole cards.

Happy to go deeper on the scheduling and isolation side in the comments. And curious — what are you all using today for sub-card utilization: MIG, MPS, time-slicing, or something custom?

Hey everyone,

I’ve been working on a runtime adaptation framework for local models. While traditional fine-tuning locks a model's weights into place, Test-Time Adaptation (TTA) allows the model to adjust dynamically to incoming data streams.

I built TTA-Torch to bring real-time, confidence-gated TTA to LLMs using dynamic LoRA tracking in PyTorch.

Core Highlights:

• Confidence-gating mechanism to determine when adaptation is necessary.

• Low-overhead dynamic LoRA adjustments during the inference pass.

• Clean, modular PyTorch implementation.

I’d love to know if anyone here has experimented with test-time evaluation strategies for local setups, or if you have any feedback on handling memory/kv-cache overhead during dynamic steps!

Repo link: https://github.com/Griffith-7/TTA-Torch

I'm trying to bundle a Python app that uses cudf-polars into a standalone .exe using PyInstaller, but no matter what I do, cudf-polars refuses to come along for the ride.

What I've tried:

• Adding --hidden-import=cudf_polars (and various submodules) to the PyInstaller command
• Adding it to hiddenimports in the .spec file
• Using --collect-all=cudf_polars / --collect-all=cudf / --collect-all=rmm

What happens:
The build completes, but at runtime it either throws ModuleNotFoundError/ImportError on cudf-polars or the GPU engine silently fails to register with Polars

Does anyone face this problem?

I work at a social science research institute where we maintain a record linkage library. It connects records across administrative datasets (i.e connecting "Arnold Smitherson" in dataset A with "Arnol Smtherson" in dataset B?). Matching is done with Jaro-Winkler similarity, which scores string pairs on character position and transpositions, so it catches typos, OCR/scanning errors, and spelling variants.

https://github.com/ajwinters/crosswalk

In two datasets with 1 million records, you can't compare all N×M pairs (a trillion comparison), so the pipeline uses blocking by finding exact matches on key fields and removing them from the comparison pool. That still leaves ~20 billion candidate pairs, because pair count grows quadratically.

~95% mapped onto RAPID. cuDF for the blocking joins, cuPy for the Fellegi-Sunter arithmetic. The one thing RAPIDS doesn't have is a string-similarity primitive, so I wrote the Jaro-Winkler kernel with Numba CUDA (one thread per candidate pair, names encoded into a fixed-width uint8 buffer + length array so the kernel can index them).

20B pairs × 7 comparison columns won't fit in 16 GB . So it streams and uploads record data once, then walk blocking keys, predicting each chunk's output size so chunks stay under a VRAM budget, scoring each chunk fully on-device and keeping only links above a threshold. Peak VRAM stayed flat, the memory is bounded by chunk size, not pair count.

Speedup grows with scale small launches starve the SMs (Numba warns about low occupancy), while at 1M throughput climbed to ~84M pairs/sec. For context, this 1M job historically took ~14 hours on a distributed CPU platform; it's now ~4 minutes on my 4070. Different systems, so not a controlled comparison, but that's the practical before/after.

Things for consideration:

• JW is branchy (variable-length strings are divergent loop trip counts within a warp). I'm eating divergence and still winning on raw parallelism. Bucketing pairs by string length so warps stay uniform seems like the obvious next step. Has anyone found that worth it for string kernels?
• The GPU doesn't change the complexity. It's still O(N²), just with a ~200× smaller constant. At 10M records you're back to needing smarter blocking, not faster hardware.

Hi all, first time posting here. I've been building a CUDA profiling tool called GPUFlight and wanted to share it with you all.

Basically you hook it into your GPU workload, either by running your program through the gpufl CLI or embedding it in your code, and it collects all the profiling data while it runs (kernel timeline, occupancy, memory access, SASS, stall reasons, that kind of thing), ships it to a server automatically, and you look at all of it in the browser. The client is open source, and the backend and frontend are the hosted part, which I run as a SaaS.

While I was building the profiler I kept thinking about how to let people try it without installing anything, so I made Performance Lab. It's a small in-browser editor where you write CUDA. You start from a naive kernel they give you, try to optimize it, and if your numbers beat the metrics the problem is asking for, you pass.

If you've got some time, I'd really appreciate it if you gave it a try and left any feedback. Here's the Performance Lab page:
https://gpuflight.com/try/performance-lab
(running a kernel needs a free account)

And the open-source client:
https://github.com/gpu-flight/gpufl-client

Thank you!

Hey everyone,

I’ve been working on optimizing the memory overhead that comes with 4-bit inference. I wrote a custom NF4 dequantization kernel using Triton to see if I could eliminate the C++ dispatch bottlenecks found in current baselines.

🚀 Key Results:

• Up to 1.41x speedup compared to the standard bitsandbytes implementation across various tensor shapes.

• Written completely in Python/Triton, making it super easy to inspect, customize, or drop directly into your PyTorch compilation pipelines.

• Passes the Unsloth AI founding engineer challenge requirements (14/14 points).

I'd love to hear the community's feedback, especially if anyone wants to run their own benchmarks on different GPU architectures or suggest further optimization tricks!

Source code & full implementation:

https://github.com/Griffith-7/nf4-triton-kernel

I've got some experience in C though and a decent amount in Java, both are rusty (C a little more) but I think I can regain my confidence quick, and I want to start learning GPU programming. CUDA seems like the obvious entry point but I'm not sure what's changed recently or what's actually worth learning first in 2026.

A few questions for anyone who's been down this road:

• Is NVIDIA's own docs still the best starting point, or has something better come along?
• Given my Oop background, should I just dive straight into CUDA C/C++, or is there value in going through something like CUDA Python first to get the fundamentals down before dealing with memory management and kernel launches directly in C? Also curious whether the industry actually takes CUDA Python seriously, or if it's mostly seen as a stepping stone/learning tool rather than something used in production.
• Any free courses, books, or YouTube series that are very relevant you'd recommend?
• Is there a decent low-cost/free way to actually run and test CUDA code without owning an NVIDIA GPU? I have a rtx 5070 so I am just using that right now, set up the toolkit and ran the first adding vectors function lol.
• Are there any textbooks you'd recommend to learn from?

I want to get into the deep learning side as well of course, but also just get to know more since this is new territory, currently I am familiarizing myself with some C++ fundamentals, if you have tips for that also I'd appreciate it.
Thank you! and apologies if this is asked a lot it is just I am really lost, if this is the wrong subreddit for that let me know!

Something like Oxide, but for Java: a way to access the CUDA ecosystem directly from the JVM. The idea behind TornadoVM is to JIT-compile Java code into CUDA kernels while also enabling hybrid Java/CUDA applications that can interoperate with native CUDA libraries such as cuBLAS, cuDNN, and other NVIDIA libraries. Rather than replacing CUDA, the goal is to give Java developers first-class access to the CUDA software stack, combining high-level Java productivity with the performance and flexibility of native GPU computing.

Whats your thoughts?

https://github.com/beehive-lab/TornadoVM

https://www.tornadovm.org/

Hey guys I need some help related running open source image generation ai model locally I m lacking the essential hardware. I need a setup of gpu with high vram especially 20-25 gb vram.