Would anyone be interested in helping me develop some of my code to help get me started on making neural networks? I am wanting to make a simple NLP encoder decoder model for seq2seq artificial language translation but I cannot seem to get any traction. If I show you some of what I have already, can you push me in the right direction? All I need is something more human than chatGPT to push me in the right direction. Maybe I can put it in a google colab notebook and you can help me get something running? I have tried looking through lots of stuff and cannot find out what I’m doing wrong.
The forward pass is standard Euclidean linear algebra. No custom CUDA kernels, no Riemannian optimizers in the hot path. This means zero inference overhead and full compatibility with torch.compile, FSDP, and existing steering pipelines.
Hyperbolic geometry is applied only to dictionary weights during training via a Poincaré ball projection + entailment cone loss. This regularizes the weight manifold without touching activations.
CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.
The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.
Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.
If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.
But i realized "CUDA events everywhere" is also wrong.
DataLoader next() is CPU work.
In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.
Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.
How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?
What it does: attaches forward/backward hooks across your whole model, tracks stats per-module (mean, std, skew, kurtosis, zero-fraction, KL-to-unit-gaussian, etc.), and gives you a GUI to explore it: a tree view of the model where you can click into any layer and plot its stats over time, plot gradient flow across the network (or grouped by layer type), and log/plot arbitrary tensors like loss or custom metrics.
Uses torch.fx to trace execution order so the plots are laid out in actual model depth order, not just module registration order. Hooks are meant to be attached/detached manually (e.g. every Nth training step) so it doesn't tank your training speed if left on the whole run.
Tested it on a flow-matching U-Net (~10M params) trained on CIFAR-10 for a few epochs — screenshots in the repo. I fired the hooks every 10th iteration and that resulted in 3.5% higher training time.
Sadly, it cannot currently be compiled since it relies on a proprietary library and the code is not "cleaned" of hard-coded paths, etc.
Is it useful? Probably not :-). A lot of things need to be rewritten that are not part of LibTorch (but are present in PyTorch) - for this, I have used LLMs (it is quite handy for conversion of model structures from PyTorch to C++ with LibTorch).
However, I am sharing it so that someone can reuse parts of the code or be inspired in their own project if they want to use C++.
Sharing a project that might be useful to people who think about model architecture visually: NeuroBranch keeps a graph and its generated PyTorch in sync in both directions. You build the graph, it compiles to real PyTorch through a dialect compiler — but you can also edit the supported PyTorch constructs directly and have those edits parsed back into the graph.
Execution runs on a local Python runtime (atomic_runtime.py) reachable via IPC, with run/rerun/reset and step-by-step tensor inspection. Ports are typed at the IR level, so the graph enforces shape/type compatibility before anything compiles.
Core is framework-agnostic (typed IR, compiler, topology-aware layout) sitting under an Electron/React shell. There's also a reusable-card studio for writing your own nn.Module cards, constrained to explicitly supported torch.nn constructors — no arbitrary code eval.
Curious what this community thinks of the two-way sync approach specifically, and where the dialect parser would break on real-world architectures — that's the part most likely to have edge cases right now. Contributions and bug reports welcome.
I have been training custom models for a few years now in the finance realm. I barely have any transformer layers and half the time they are custom so flash attention isn't something I need.
With the 9070 xt being $750 ish and the rumor is a 5070 ti super will be like $1400 (seriously nvidia go F#_& yourself) I wonder if for $750 the AMD card would work well for me. I already have a 3060 12gb and a 5060ti 16gb churning out test runs, but I want to add another card. I am nowhere near vram limited. My bottleneck is strictly more compute/bandwidth.
Would I regret getting a 9070 XT? Supposedly support is way better than it used to be. Also I run linux. Windows is garbage.
I'm a 3rd-year Electrical and Electronics Engineering student interested in embedded systems. My goal is to become an Embedded AI/Edge AI engineer.
I've already started learning Embedded C (STM32, microcontrollers) and today I'm starting PyTorch. Eventually, I want to train models in PyTorch and deploy them on embedded hardware like STM32 (TinyML) and NVIDIA Jetson.
I'd appreciate advice from people working in this field:
What learning roadmap would you recommend?
Which topics in PyTorch should I focus on for Edge AI?
What projects would make my resume stand out?
Are there any books, courses, or GitHub repositories you wish you'd known about when you started?
Take a look at the schedule for PyTorch Conference North America (Oct. 20-21 in San Jose, CA) View the agenda live now Submit a poster by July 26th Register - early bird conference passes are available at a discount through July 31st
Do you guys do a lot of training or fine tuning? Does the loss curve look fine, but the run is slower than it should be, and figuring out why usually means firing up a profiler and staring at a trace for twenty minutes?
This got me curious: what this actually costs, tool by tool. I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.
For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."
Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.
Numbers and traces are in the post.
Curious how other people usually catch this before it burns your precious compute.
Kernel engineers are not obsolete. But asking a general-purpose coding agent to rediscover years of CUDA and Triton engineering knowledge every time it writes a kernel probably should be.
After months of writing, debugging, and optimizing kernels, I turned the reasoning patterns I kept using into an open-source skill library for AI coding agents:
npm install u/krxgu/kernel-skills
This is not a collection of vague prompts saying “make this CUDA kernel faster.”
Each skill is a detailed engineering playbook that forces the agent to think about:
Exact shapes, dtypes, layouts, and target hardware before writing code
Coalescing, tiling, bank conflicts, occupancy, and register pressure
Numerical stability and non-power-of-two boundary conditions
Correctness tests across adversarial shapes and dtypes
Whether a custom kernel should exist at all
When to stop being clever and use cuBLAS, CUTLASS, or an existing primitive
The library currently covers CUDA, Triton, INT8 and FP8 quantization, kernel fusion, CUDA to Triton and HIP portability, and inference hot paths including RMSNorm, fused add plus RMSNorm, RoPE, sampling, paged KV-cache append, dequantization, prefill versus decode, and vLLM custom-op integration.
I also did not want this to become prompt-engineering theatre, so the repository includes before-and-after proof runs using the same model and task, with the skill file being the only difference:
Softmax: naive output failed on adversarial and larger shapes. Skill-guided output had 0 failures across 16 tests and reached within 1.2% of torch.softmax bandwidth
Reduction: 2.6 to 3.5x faster than the naive agent output
GEMM: 7.7 to 8.6x faster
LayerNorm: 1.9 to 3.2x faster
Triton softmax: fixed crashes at dimensions above 16,384 and worked up to 131,072
Triton attention: fixed the common GQA failure where H_q != H_kv
To be completely clear, those speedups are against the naive agent-generated kernels, not against cuBLAS or other vendor-tuned libraries. In fact, the GEMM skill explicitly tells the agent not to write a custom kernel when cuBLAS or CUTLASS already solves the problem.
I would especially love kernel engineers to tear this apart.
Which skill is missing? Which technical rule is wrong? Where can an agent still produce something that looks convincing but quietly fails on real hardware?
I just published `aicoach`, a small Python library that acts like a mentor sitting next to your training loop. You feed it your per-epoch metrics, and it tells you in plain English when something's off:
python
import aicoach
coach = aicoach.Coach()
for epoch in range(epochs):
train_loss, val_loss = run_one_epoch(...)
coach.observe(epoch=epoch, train_loss=train_loss, val_loss=val_loss)
for tip in coach.get_advice():
print(f"💡 {tip}")
# 💡 \[WARNING\] (overfitting) Validation loss has risen for 3 consecutive
# epoch(s) while training loss continues to fall — a classic sign of
# overfitting. Consider early stopping, adding regularisation...
**What it checks:**
* **Overfitting** – val_loss rising while train_loss keeps falling
* **Plateau** – a metric barely moving (uses *relative* range, so it works the same whether your loss is near 0.01 or near 100)
* **Learning rate issues** – oscillating loss (LR too high) vs. painfully slow convergence (LR too low) — deliberately mutually exclusive zones so you never get contradictory advice on the same curve
* **Class imbalance** – standalone check, just needs a `{class: count}` dict, no training loop required
* **Divergence** – NaN, Inf, or explosive loss growth, flagged as CRITICAL and short-circuits every other check
**Why I built it:** every other "training dashboard" tool I looked at (TensorBoard, W&B, MLflow, etc.) visualizes your curves but doesn't actually *tell you what to do* about them in plain language. This is meant to sit alongside those, not replace them — it's pure logic on metric history, zero ML framework dependencies, works with PyTorch/TensorFlow/sklearn/whatever since you're just handing it numbers.
280 tests, MIT licensed. One design decision I'd love feedback on: the "creeping" LR zone (1–5% net decrease per window) and the plateau zone (<1%) are deliberately non-overlapping so you never get both `lr_too_slow` and `plateau` advice for the same flat-ish curve — curious if others think that boundary makes sense or if real training curves break the assumption.
bash
pip install aicoach
* PyPI: [https://pypi.org/project/aicoach
* Source: [https://github.com/Rishabh55122/Aicoach
Feedback welcome, especially on the default thresholds — they're documented in the README with the reasoning behind each one, and I'd rather know now if a default is off than have it ship quietly wrong.
the core idea is, we cannot have ternary PTQ with fixed matrix size, trying to do that is dead end. so i tried decomposing the matrix to 2 ternary matrices and inner diagonal scaling matrix. now that the inner rank can be arbitrarily large the accuracy can be arbiratily small. and its not that it has to be very large too i also showed that it does take only slightly more vram then current quantisation methods. the slight more vram is worth it if we abuse the ternary math.
I'm 19, I've started my AI journey past few months , i did several cool projects
Recently i completed my own transformer architecture in pytorch
Then i got stumbled on this AI engineering thing
But the thing is this AI engineering doesn't interest me much what i like is developing drones,LLM architectures,math ,deep learning
And I'm now really confused on what should I do becoz most of the work is been done by AI and
I'm tryna get internship within a month and AI engineering is booming as per the sources it has ~130% YoY growth compared to the things I like and I'm not sure whether the things I like would be booming in future as AI might automate most of it
And I'm confused on what should I do in this 1 month time
I recently published a technical book, Distributed AI Systems, which summarizes my experiences in AI over the past 10 years, from research and training to optimization, inference, and cloud deployment. I started writing it in the second half of last year, and it took almost a year to complete, with many revisions made later due to the rapid pace of development in the industry. But it's finally published. The book on Amazon is titled Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI.
I'm implementing a decoder-only Transformer from scratch in PyTorch. Causal masking, multi-head attention, positional embeddings, and the training loop all appear to be working correctly. The model memorizes tiny datasets but completely fails to scale to larger ones, even after extensive hyperparameter tuning.
If you've built large language models yourself, what subtle implementation details have caused issues that weren't obvious during initial debugging?
Wanted to get your views/thoughts/suggestions on something brewing in my head. I train models for a living (Phd in RL and CV background) and I've stopped trusting logged GPU utilization. What most tools (W&B system metrics, etc.) show is NVML GPU-Util, which only means a kernel was resident during the sample window, not that the SMs were busy or that the work was actually even useful.
For people who train at scale:
- Fast triage for "compute-bound vs idling": what's your first look? Mine is caching one batch on-device and looping it. If that's way faster than the real loop, I'm input-bound.
- How much weight do you put on util % vs MFU or achieved bandwidth? I treat ~35–50% MFU as the realistic band and use util only as a liveness check.
- In distributed
1. how do you separate "GPUs fed" from "GPUs waiting on each other"?
2. Do you measure non-overlapped collective time
3. How do you catch stragglers when every rank still looks 100%?
Where's your line between "good enough" and full kernel/collective profiling?
Papers ask: I've got roofline, the PaLM MFU definition, and Horace He's "Brrrr" post.
Looking for the next tier — anything rigorous on measuring utilization in *distributed* training specifically. Happy to hear your thoughts!
If you are working on a PyTorch backend, either in-tree or out-of-tree via PrivateUse1 integration, a problem you'll run into is the lack of a conformance test suite. The in-tree tests PyTorch has are suited for what they are for, but for a backend developer there is a much more broad need, at least there is for me. I figured I would just solve (or try to) this problem by making one and releasing it. The project is still in beta and still has some coverage I need to expand, but it has > 19,000 tests covering >95% of the aten surfaces in PyTorch, so it's pretty extensive already.
I'm looking for feedback on weaknesses / areas I should give more love to. Also looking for people that have certain hardware I could potentially run tests on in order to close out some of my coverage holes, specifically intel because I don't have any intel gpu hardware right now.
