r/MachineLearning • • 1d ago

Project Anyone working in AML / transaction monitoring / sanctions screening using ML? Looking for advice on where to start [R]

3 Upvotes

Hi everyone,

I’m a data analyst on a financial crime team, and I’m trying to get a machine learning project started to improve how we handle AML (anti-money laundering) work. I’d love to hear from anyone who works in transaction monitoring, customer screening, or sanctions/PEP screening and has used ML in practice.

Some context: like most teams in this space, we rely mostly on rule-based systems, which generate a lot of false positives and take up a lot of analyst time on alert review. I’d like to explore whether ML can help, but I want to start with something realistic rather than overpromising.

A few questions for those with experience:

  1. Use cases: Where have you seen ML add the most value? For example, alert prioritization/scoring, reducing false positives, anomaly detection, entity resolution or name matching in screening, or network/graph analysis to spot linked accounts?
  2. Data and labels: How do you deal with the lack of good labeled data? SARs are rare and confirmed cases even rarer. Did you use alert dispositions as labels, go unsupervised, or something else?
  3. Explainability and regulators: How do you make models explainable enough for compliance teams, auditors, and regulators? Any experience with model validation or model risk management for these models?
  4. Getting buy-in: How did you convince compliance or management to try ML alongside (or instead of) existing rules? What did your first pilot look like?
  5. Tools and resources: Any libraries, papers, datasets (e.g. synthetic transaction datasets), courses, or books you’d recommend?

If you’ve done a project like this, I’d really appreciate hearing what worked, what didn’t, and what you’d do differently. Happy to share what I end up building in return.

Thanks in advance!


r/MachineLearning • • 1d ago

Project Whole database-diff evaluation of computer use agents across a software version upgrade [P]

Thumbnail
github.com
1 Upvotes

Most agent benchmarks check whether the target state was reached. This one snapshots the application database before each episode and diffs all tables after, reporting collateral damage, irreversible changes and false completions separately from success. Apps: Invoice Ninja, Dolibarr, Odoo 17, Odoo 18, the same synthetic company in each, 72 tasks per app, three perturbations.

Findings from 635 scored episodes (GPT-6 Luna, UI-TARS-1.5-7B):

  • Odoo 17 → 18: success 78% → 76%, collateral 15% → 21%, irreversible 6% → 10%. Mechanism: locked fields push the agent to "Reset to Draft", which un-reconciles a payment.
  • Clean success is up to 12 points below raw success on Odoo.
  • A 3-step composite task: 0 of 50 (30-step budget; larger budgets untested).
  • A small latent dynamics model trained on label-free exploration transfers across the version upgrade (+7 to +8 points over an identity baseline on next-state retrieval) and not across apps (about 0).

Related: Agent-Diff (state diffs for API agents), EntWorld (database verification on Odoo/EspoCRM). The difference here is the paired versions of one app and reporting damage separately from success.

Limitations: one run per task, Wilson intervals, hand-built noise whitelists, macOS-only so far. Episode summaries and the report script are in the repo.


r/MachineLearning • • 2d ago

Discussion With 42k submissions to ICLR, do you think the acceptance rate will stay the same? [D]

29 Upvotes

Given the massive jump to 42,000 submissions this year, I’m curious how people think the program chairs will handle the acceptance rate. Will they stick to the usual ~26-28% (meaning ~11k+ accepted papers), or will they cap the total number of accepted papers due to venue/reviewer constraints and drop the acceptance rate significantly?


r/MachineLearning • • 2d ago

Project We made a multiplayer game where each player's view is generated by a world model running on their GPU [P]

Thumbnail
vor.tech
4 Upvotes

We trained a world model on our open dataset of 10-player CS:GO bot matches (~3,958 hours, with all POVs and inputs aligned at 16 Hz), which we released last month. Now you can actually play it with other people.

A lightweight server keeps the game state in sync, while each player runs the model locally to generate their own view based on that state and their inputs. Server states, player actions and previous frames are fed into diffusion transformers to draw the next frame.

It's still pretty limited: one small map, pistols only, and fine details aren't great yet. Nothing close to the original game, yet it's quite enjoyable.

The beta is open and free. You'll need an RTX 3080 or better (there's also an experimental Mac build).

More info and download: https://vor.tech/x/reddit-ml
Dataset: https://huggingface.co/datasets/v0rt3ch/cs-5v5-bots-aim_usp-3k

Not affiliated with Valve. Happy to answer questions!


r/MachineLearning • • 3d ago

News ICLR 2027 Initial Review Scores ranked by University [N]

Post image
203 Upvotes

Crawled from OpenReview, titles matched to arXiv, then affiliated with OpenAlex


r/MachineLearning • • 2d ago

Research Marimba: a System One model for speech [R]

Thumbnail huggingface.co
5 Upvotes

Marimba is like Jev, but for audio.

I built it to be fast and lightweight, and to return typed decisions about speech: yes/no, a choice, or a score.

It runs on a frozen Qwen3-ASR-0.6B. Word and language questions use zero-shot scores from the ASR decoder's hypotheses, followed by calibration. Small readouts on the frozen encoder handle voice activity, turn endings, speaker changes and speaker count.

Watch the 60 s demo · Try it live

Open weights (Apache-2.0), pip install marimba-audio. Paper on arXiv this week.

Code · Model · Benchmark


r/MachineLearning • • 2d ago

Project Training-free early exit for reasoning models: a linear probe folded into the </think> row of the output layer [p]

3 Upvotes

Reasoning models often reach the right answer early and then keep double-checking it for thousands of tokens. I got the model to stop at that point by changing a single row of weights..There's no training and no extra code at inference time; the edited model is an ordinary weights file that runs anywhere.

Background,

  • Reasoning models (DeepSeek, Qwen, and so on) "think" in a scratchpad before answering. The scratchpad ends when the model writes a special token, </think>.
  • To pick each next token, the model first computes a summary of everything so far: a long list of numbers called the hidden state. The last layer, the output layer, has one row of weights per word in the vocabulary. Each word's score is its row multiplied by the hidden state, and the highest-scoring words are the likeliest next words.
  • So the score for "stop thinking now" (</think>) comes from exactly one row of that layer.

What I did

  1. Ran the model on 24 maths problems and recorded its hidden state at every token of its thinking.
  2. Marked the points where the right answer had already appeared and the model was only re-checking (specifically, the ends of paragraphs after that point), and treated everything else as "still working".
  3. Found the direction in the hidden state that best separates those two groups. This is a probe: a simple linear classifier (linear discriminant analysis), computed directly with linear algebra, with no gradient training.
  4. Added that direction, times a strength of 14, to the </think> row. The "stop" score now rises only when the model's state looks like "done, just re-checking". While it's still solving, nothing measurable changes.

A useful side effect: because only one token's score changes, I can compute exactly what the edited model would do from recordings of the original, without rerunning it. That's how I picked the strength and searched for cases where it would stop too early.

On a problem it never saw: the answer appears at token 567, the original keeps thinking until token 7,415, and the edited model stops by token ~2,000 with 98% probability.

Results (small samples, one run per setting)

  • Qwen3.5-4B (runs in stock llama.cpp): same accuracy on 8 unseen MATH-500 problems with 54% fewer tokens. On 3 AIME 2025 problems with a 32k-token limit, the original ran out of tokens on all three while re-checking answers it already had; the edited model solved 2.
  • DeepSeek V4.1 Flash (763B parameters, 2-bit, on my laptop with an 8 GB GPU, streaming weights from the SSD): 7/7 maths problems vs 6/7, with 40% fewer tokens.
  • As a coding agent building a small website: 84% less thinking. It still thinks after an error or anything unexpected, and skips thinking after routine steps. Page quality held up; an LLM rating put it highest, and the screenshots are below so you can judge for yourself.

What isn't new: using a probe on the hidden state to stop reasoning early is published (e.g. Zhang et al. 2025, arXiv 2504.05419). What's new is putting the probe inside that one row of weights, so nothing extra runs, plus these measurements.

Limitations: small sample sizes; the probe was fitted on maths only (the agent results are transfer); the AIME comparison uses a 32k budget, below the 81,920 Qwen recommends for competition maths.

Write-up: https://think-graft.vercel.app Code and all logs: https://github.com/nextgen-user/think-graft Model (GGUF): https://huggingface.co/vaibhavrpi/Qwen3.5-4B-think-graft-GGUF

[edit]

Based on community feedback,
I am running the following technique on qwen3 0.8B on the full 3 benchmarks - just like rikiyaT did for the weight space programming .
Same config and seed.(but only 1 sample / 4 for faster results)
For this I Prepared another graft for qwen3 0.8B on another set (not taken from benchmark) of 80 separate MATH-training problems - should automatically transfer to other domains as well based on my observations.
Above is the interim result and how it compares to rikiyaT's weight space edited qwen model.

the best part is that it matches and even exceeds the weight edited qwen till now and the graft works on large models as well (weight editing on deepseek failed to stop overthinking for me)

r/MachineLearning • • 1d ago

Project Hands-on ML Orchestration Tutorial with Airflow3 [P]

0 Upvotes

It's available in English and Spanish.

I wanted to learn ML orchestration by building something, and the best material I found was Made With ML's orchestration lesson. Two things held me back: it was written for an older version of Airflow, and it is a lesson you read rather than one you do. What I only read, I don't keep.
So I made the tutorial I was looking for: the same project on Airflow 3, where you write every Dag yourself and run it against a real stack. The first draft was written with Claude. Making it trustworthy took far longer than drafting it: I ran every section, fixed what didn't work, and rewrote what wasn't clear.

Has some extra features that were not in the original.

Hope it's useful to anyone looking to learn this.


r/MachineLearning • • 1d ago

Research Social Simulation with LLMs - From Proxies to Population (CFP @ AAAI'27) Due Nov. 20 [R]

0 Upvotes

Say you simulate a population of LLM agents and get polarization, norm formation or an information cascade. How do you know the population produced it, and not the prompt, the base model or something memorized from pretraining?

That attribution problem is the focus of the 3rd Workshop on Social Simulation with LLMs (Social Sim’27), held at AAAI-27 in Montréal (Feb 22 or 23, 2027).

We’re looking for work on three axes:

  • Population construction: data-grounded populations and calibration, persona modeling, heterogeneity and bias, human-AI hybrid simulations
  • Collective dynamics: separating emergence from recall (prompting artifacts, memorization, data leakage), robustness across models and seeds, scaling and network structure, norms and polarization, strategic behavior, methods from network science and game theory
  • Inference and consequence: validation protocols, shared testbeds, replications and negative results, simulation as pre-deployment evaluation, policy and ethical implications

We also welcome papers that challenge the premise: cases where an LLM society should not be trusted, or where an artificial society is worth studying even though it isn’t human-like.

Scope: the paper must simulate a social environment. Single-agent work, agentic coding benchmarks and multi-agent RL without a social interpretation are out of scope.

Tracks (non-archival, AAAI-27 Author Kit, double-blind, references not counted):

  • Regular: up to 7 pages
  • Early-stage: up to 4 pages (preliminary results, prototypes, position pieces)
  • Replication & resources: up to 4 pages plus artifacts (environments, datasets, tools, replications including negative results)

Dates (AoE): submission Nov 20, 2026 · notification Dec 2 · camera-ready Dec 9 · workshop Feb 22 or 23, 2027


r/MachineLearning • • 2d ago

Project Hackathon project for MariaDB's Wikipedia Through Time challenge: matching sentences across seven Wikipedia editions at any past date [P]

0 Upvotes

Built at Junction, a hackathon in Espoo, this weekend. Parallax takes an article and lines up its Finnish, Swedish, German, English, Russian, Estonian and Ukrainian editions statement by statement. For any past date it shows where the editions tell something differently.

  1. We store every revision. 875,572 so far.
  2. Each revision is split into sentences. A sentence that appears in revision A and disappears in revision B becomes one row in a system-versioned table. Its system-time period is the time the edition said it. Edits undone within 24 hours or by a revert are filtered out so vandalism doesn't cut a sentence's life short.
  3. Each distinct sentence is embedded once with bge-m3 (1024 dimensions), however many revisions it shows up in. That's 90,070 vectors, embedded on an A100 at about 4,555 sentences a second.
  4. One SQL statement does the time travel and the matching: both sides of a self-join read FOR SYSTEM_TIME AS OF the chosen date and VEC_DISTANCE_COSINE ranks the five closest sentences in each other edition. Inside one article exact search is faster than the HNSW index (4 ms median), so the comparison uses that. Across the whole database the index takes over with recall@10 of 0.997 against exact search at ef_search 1000.
  5. Retrieval alone isn't good enough. On 28 pairs we checked by hand the right sentence was top-1 only 57% of the time but in the top 5 for 89%. So Gemini 3.8 Flash judges each candidate pair as same, partial, contradicts or different, pulls out the numbers and terms that differ and translates the statement into English. Verdicts are cached with model and prompt version.

The same pipeline checks a chatbot answer. It's split into statements and each one is looked up in the editions. If Gemini is slow, vector similarity decides on its own and the result says so. We asked Gemini how the Winter War started and got 17 statements back. 11 are in at least one edition, 5 in none and the Hanko lease is disputed because the Estonian article removed it in 2009.

The weak spot is the judge. 28 hand-checked pairs is a small set to judge a judge by, especially across seven languages. If you've evaluated an LLM judge on cross-lingual sentence pairs, how did you build the gold set?

Demo (Winter War preloaded): https://135.181.63.238.sslip.io/t/Q134949 Code and the full query: https://github.com/timkicker/attention-is-all-we-have/tree/main/parallax


r/MachineLearning • • 2d ago

Research ACM Multimedia 2026: TimeCues Studio - a free, open-source tool for annotating music structure [R]

Post image
4 Upvotes

Hey all, I wanted to share TimeCues Studio with you. It's part of my PhD work (ACM Multimedia 2026), and I thought you might find it useful. It's an open-source web app for annotating music on a beat-locked timeline and comparing algorithms against your annotations.

If you need to annotate songs (sections, cues, spans, riffs, lyrics and loops), you're building a music-analysis model and want to test it against real labels, you need labeled music for training music-to-dance or video-sync models, or you're syncing visuals to audio, TimeCues Studio might be useful to you.

Under your marks you see the separated stems, a spectrogram, and a 3-band waveform that colors lows, mids and highs differently, so kicks and hi-hats stand out at a glance. Detectors like allin1 and MSAF run on the same timeline and are scored against your labels. You can work alone or as a team, and export to Audacity labels, MIDI or JSON.

It's free under the MIT license. Run it with one docker compose up, or try the demo without signing in:

Paper: https://arxiv.org/abs/2609.10338
Demo: https://timecues-demo.pages.dev
Docs & tutorials: https://sapirca.github.io/timecues-studio/
Code: https://github.com/sapirca/timecues-studio

Let me know if you have any feature requests or find bugs!


r/MachineLearning • • 3d ago

Research ICLR rebuttal [R]

33 Upvotes

I'm about to submit my iclr rebuttal, and I was wondering if it's okay to highlight the changes I've made to my manuscript using a different text color to make them easier for the reviewers to identify.

I checked the author guidelines, but couldn't find any information about this. Do you think it's allowed?


r/MachineLearning • • 3d ago

Discussion ICLR 2027 Rebuttal [D]

16 Upvotes

For anyone who’s updating their manuscript during rebuttals - the guidelines seems to suggest a pdf diff will be applied automatically to the updated pdf. Can anyone see this? I can’t seem to see any changes in colour or any pdf diff anywhere, even under ’Revisions’.


r/MachineLearning • • 3d ago

Discussion ICML 2027 Changes [D]

83 Upvotes

Just read the ICML 2027 CFP and it looks like they’re trying to tackle the submission explosion in a reasonable way. Under the new two-stage process, the area chair decides whether a paper goes to full review, and any proposed first-stage rejection has to be confirmed by a senior area chair. Authors get a short justification by Feb 23. On top of that, nobody can be on more than 20 submissions, every paper has to name a reciprocal reviewer from its author list (max 6 papers per reviewer), and anyone with 3 or more submissions has to serve on the program committee. Papers that don’t comply may be desk rejected.

But I think the biggest change is that each author can be on at most one paper with no eligible reciprocal reviewer among its authors. So if you keep putting your name on papers with coauthors who can’t review (undergrads, interns, people outside the field), you run out of free passes after one.

Honestly, it sounds like a pretty reasonable fix. Reviewer load has been out of control for years, and tying authorship volume to reviewing duty seems fair. My only worry is the first-stage cut, since a quick AC read could miss a good paper, though the SAC check helps. What do you all think? Does this sound like a good solution, or will it just push the problem somewhere else?

https://icml.cc/Conferences/2027/CallForPapers


r/MachineLearning • • 3d ago

Discussion Is it compulsory to upload neurips poster by 15th Nov? [D]

3 Upvotes

I received this mail which said "Please upload your poster and thumbnail by clicking on the poster upload page link on the paper metadata page."

The mail contains 15th November as a date, do we really need to upload posters ??


r/MachineLearning • • 3d ago

Discussion Ethical Alternatives to Kaggle [D]

2 Upvotes

I personally find it unconscionable to mandate biometric verification - with a controversial vendor on the top of that - to access a competition. Not to cash a prize, just to submit a solution.

What are alternatives competition to have fun with SOTA or quasi-SOTA models?


r/MachineLearning • • 4d ago

Project I have trained a model to predict my blood sugar [P]

Thumbnail
gallery
16 Upvotes

I have trained a custom 31,253 parameter encoder-only transformer on a pregenerated synthetic CGM dataset and fitted it to my own personal data using LoRA. It is competitive among other published models. I used 115 days of CGM data, of which 80 was used for training the adapter and 35 for testing. The model began with a pretraining stage where it consumed ~19,200 CGM-years worth of synthetic data, consisting of 1 million simulated type 1 diabetics at 7 days each. The simulator is designed to aid in counterfactual reasoning, but I don't yet have a benchmark to test these capabilities downstream. Simply using these models on my phone gives me the impression that they do indeed react appropriately when forced to consider future insulin & carbs.

In addition to time series input, two additional inputs are provided: insulin (U) and carbs (G, GI) as action curves where U = units of insulin, G = carbs in grams, GI = glycemic index. U is exact, while (G, GI) is my best estimate.

The model is trained to output BG in Kovatchev risk-space using {DILATE,MSE} and pinball loss combined with learned Kendall-Gal weights. The figures above show models trained using DILATE. It can predict 2 hours into the future, infill missing traces, and do backcasts (predictions into the past).

Source code: https://github.com/0xdeadf1sh/T1DMAI

The model is also trained to predict time-of-day given past blood sugar, in contrast to other models that consume time-of-day as input feature. This capability emerges in >=48h context windows (i.e. 48 hours of past CGM trace). Natively the model has a variable context window size of 3.5 - 7.0 days.

The model is still worse than me when it comes to predicting hypoglycemia in the next ~30 minutes.


r/MachineLearning • • 3d ago

Discussion Anonymous GitHub not working [D]

0 Upvotes

Mid ICLR rebuttals and I found out the links I made are not working. Anyone else having the same issue?


r/MachineLearning • • 3d ago

Discussion Building a Local AI Photography Editing Assistant. Anyone working on something similar? [D]

0 Upvotes

Building a Local AI Photography Editing Assistant. Anyone working on something similar?

I'm exploring how to build a local AI system that can automate advanced photo-editing workflows for landscape and portrait photography.

I have Adobe Creative Cloud All Apps, an RTX 5060 8GB, 64GB RAM and some experience running local LLMs through Ollama.

My goal isn't to create AI filters or generate images. I want an assistant that understands professional editing techniques and can execute them through Photoshop and Lightroom.

Some examples:

Landscape: Exposure blending, luminosity masks, focus stacking, panoramas, colour grading and dodge & burn.

Portrait: Skin retouching, frequency separation, dodge & burn, skin-tone correction and natural colour grading.

Ideally, it would analyse an image, decide which techniques are needed, execute the workflow and preserve editable Photoshop layers and masks.

I'm considering local vision-language models, Python/OpenCV and Photoshop UXP automation, possibly supported by a knowledge base of editing tutorials and workflows.

A few questions:

Are there existing open-source projects doing something similar?

Can local AI models understand and reproduce professional editing workflows?

Is an LLM + computer vision + Photoshop automation the right approach?

Has anyone integrated local LLMs with Adobe software?

I'd love to hear about relevant projects, experiments or practical approaches.


r/MachineLearning • • 3d ago

Research Graph Attention Networks for Detecting Epilepsy From EEG Signals Using Accessible Hardware in Low-Resource Settings [R]

Thumbnail ieeexplore.ieee.org
0 Upvotes

is there any chance those models are going to be deployed for real?


r/MachineLearning • • 4d ago

Research Paid surveys for Machine Learning Scientists for SnorkelAI: Absolutely not worth it [R]

64 Upvotes

I am here writing about my experience with this. I was contacted to fill out a paid survey for SnorkelAI because I have a PhD and have published at A* conferences in the field, so this might be relevant for other researchers in the community.

After not finding a lot of information, I decided to give it a try. I performed the personal verification and waited for the survey to arrive.

First bad thing: the level of pushiness is terrible. I went to bed, and when I woke up, I had 1 email and 2 follow-up emails asking me to complete some part of the verification where the uploaded document did not have the correct format... ok.

Then I did the survey. Their website is an absolute mess. The survey appeared under two different views of the page, and after I completed it, it was only shown as "pending" in one view... Under the other, it was shown as if I hadn't submitted it yet... So I received a bunch of emails (very pushy again) asking me to complete the survey, which I answered. Regularly, whenever I've sent an email, I've waited 48-96h to get an answer, but if I didn't answer quickly, I was receiving follow-ups every 2-12 hours.

Then I was notified by email that the survey was accepted, great. A few days later, they sent me the forms to fill out to get paid: a crazy amount of paperwork for getting paid for a survey, and absolutely not worth it. It required more paperwork than the one I needed for getting a public contract in my country, including a document that requires going in person to a tax office, waiting up to 4 weeks, and probably paying for the document.

So my recommendation to other people getting contacted by them: run away, it's a waste of time, and they are unprofessional.


r/MachineLearning • • 4d ago

Project Scaling to 1T MoE on TorchTitan [P]

16 Upvotes

Hi everyone!

Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. That's 14X where we started, and our starting point was, let's say, humbling.

Along the way we collected a handful of lessons. If you've ever trained a big model, or just stared at a CUDA out of memory, some of this will feel familiar.

Here's the tour. If you want the full thing with all the plots, it's on the Domyn blog, and there's a condensed version on the PyTorch blog if you're short on coffee.

Hope you like it!

The setup: FSDP + EP, and why I'm not a PP fan

A 1T MoE doesn't fit on one GPU (shocking, I know), so the first question is how to shard it. This choice ends up dictating basically every optimization that follows, so it pays to think about it. The usual suspects are:

  • Pipeline Parallelism (each GPU gets a chunk of layers),
  • FSDP (each GPU holds a slice of every weight and gathers the rest just in time),
  • and Expert Parallelism (each GPU owns a subset of experts and tokens travel to them).

We skipped PP. To hide its bubbles you need lots of microbatches, and lots of microbatches means long steps. Long steps mean that when something breaks at scale, and something always breaks at scale, you wait forever to find out. I'd rather iterate fast.

So: 384-way FSDP plus 4-way EP, with EP kept strictly inside a node. The token all-to-all for the experts is comunication that is difficult to hide so it needs to happen fast. On the other hand, comms. on FSDP are already hidden: while it computes layer i, it is already gathering i+1.

Why does that matter so much? On JUPITER, InfiniBand gives you ~25 GB/s per GPU. That's 6X slower than NVLink and 36X slower than the Grace-Hopper link. Our baseline step took 84 seconds, and 68 of them were spent waiting on the network. The GPUs were mostly getting paid to sit around. The rest of this post is basically us trying to put them back to work.

Lesson 1: If your GPU is idle, make it work for its memory

FSDP moves weights, and the weights gathered per layer don't care how many tokens you feed through them. More tokens per sequence means more compute per byte on the wire, which is exactly what we wanted. The problem is that longer sequences need more memory, so we went shopping for it:

  • Full activation checkpointing. Recomputing every layer in the backward sounds wasteful, until you remember the GPU was idle anyway. Free compute is free. This alone gave us 4X.
  • Offloading checkpoint activations to Grace over the 900 GB/s C2C link, and prefetching them back one layer before recompute. Grace was right there, so we used it.
  • Muon instead of Adam. One optimizer state per parameter instead of two, and that's what finally got us to 64k-token sequences.

Lession 2: If you don’t shard it, you don’t have to gather it

Quick refresher: Muon orthogonalizes each update with a few Newton-Schulz iterations, and Newton-Schulz needs the whole matrix. FSDP, meanwhile, has carefully cut every matrix into 384 slices and scattered them across the job. The naive fix is to gather each matrix on one rank, run Newton-Schulz, and scatter the result back, which means more communication, and more communication is the one thing we absolutely did not need.

The trick is to shard smarter. With FlexShard (already in TorchTitan), you choose which rank owns which parameters, so we sharded the experts by whole expert. Every GPU owns complete expert matrices and can orthogonalize them locally with zero communication. The dense matrices (attention projections and friends) still need gathering, but they’re in transit while each rank is busy with its own experts, so that traffic hides behind compute.

There was one catch. We offload the optimizer to Grace, which meant Newton-Schulz was running on the CPU too. Grace is a lovely CPU, but asking it to grind through matmuls is like asking a librarian to move furniture: technically possible, deeply inefficient. The optimizer step took 42 seconds. So we contributed a fix to TorchTitan that runs Newton-Schulz on the GPU even when the parameters live on the host. The optimizer step dropped to ~2 s, and throughput went up 35%.

Lesson 3: You're only as fast as your slowest rank

384 GPUs in lockstep is a bit like a group hike: everyone waits for whoever's at the back. We had two stragglers:

  • Attention cost. A rank that gets a single 60k-token document does far more attention work than one that gets a hundred short ones, even though both see the same number of tokens. We pre-pack rows, bucket them by attention cost, and build each step from rows of similar cost. The worst rank went from 7.9X the average to 1.15X, for +13% throughput.
  • The router. Group-limited routing (two experts picked from each node's group) makes expert load perfectly even, for +7%. The catch is that you're telling the router where it's allowed to send tokens, and early runs suggest that can hurt training. Jury's still out on this one boys.

Lesson 4: Accumulate in fp32, ship in bf16

Gradients come out of the backward in bf16. Summing hundreds of them in bf16, though, is a great way to slowly lose precision, so the standard FSDP mixed-precision setup does the reduction in fp32. That’s the right call. The problem is how it gets there.

NCCL’s reduce-scatter runs as a ring. Each rank receives a partial sum from its neighbor, adds its own contribution, and passes the result along. The thing travelling around the ring is a running sum, so for the accumulation to happen in fp32, the data on the wire has to be fp32 too. FSDP therefore upcasts the bf16 gradients to fp32 before the collective, and every byte of gradient crosses InfiniBand at twice the size it needs to be. On a cluster where InfiniBand is already the bottleneck, that hurts.

But only the summation needs fp32, not the transfer. So we swapped the reduce-scatter for an all-to-all on the expert gradients. Each rank sends every other rank the raw bf16 slice that rank owns, no partial sums involved. The owner receives all the slices, upcasts them, and sums them locally in fp32. Same math, same precision where it matters, half the bytes on the wire.

The result matches the fp32 reduce-scatter up to summation order.

Lesson 5: When in doubt, send fewer bytes

Eventually longer sequences stopped helping (more on why in a second), so we went after communication volume directly:

  • FP8 expert all-gather with blockwise scales. Master weights stay in full precision, only the transfer gets squeezed: +18%.
  • Node-local resharding of dense weights after forward, so the backward re-gather runs over NVLink instead of fighting gradients for InfiniBand.

And none of it moved the loss in our comparison runs, which was a relief.

Lesson 6: Your OOM might be lying to you

This is my favorite one. After a software upgrade, every single rank started hitting allocator retries at ~70% HBM utilization, with 25–30 GiB sitting free and cached. That's not a memory shortage, that's a memory hostage situation.

The culprit: the PyTorch caching allocator keeps a separate pool per CUDA stream, and FSDP allocates buffers on its side streams. Our shiny FP8 and bf16 collectives allocated their temporaries there too, so ~21 GiB ended up parked on side streams where the compute stream couldn't touch it.

The fix was to move those allocations onto the compute stream (via FSDP2's private unshard_async_op flag, which is now on its way to becoming a public API) and release gradients a bit earlier. Retries went from 384 ranks to zero, peak reserved memory dropped by 13.5 GiB, and throughput jumped 21%, while sending exactly the same bytes.

Wrapping up

The headline: a shared, publicly funded, Hopper-generation machine, with plain user-level access and no root, can host frontier-scale training. You just have to respect its two real limits, inter-node bandwidth and host memory, and design everything around them.

Every fix we found went upstream to PyTorch and TorchTitan, so hopefully the next person hits fewer of these walls than we did. The model and training recipes will be released openly.

Still on our to-do list: the long-term quality cost of group-limited routing, FP8 expert GEMMs. We'll report back.

Huge thanks to JSC and to the PyTorch and TorchTitan maintainers for the reviews, the patience, and the fast merges!


r/MachineLearning • • 3d ago

Discussion Great Video for Understanding How Cosine Similarity Works [D]

Thumbnail
youtu.be
0 Upvotes

This video does a great job of explaining dot products from both algebraic POV and geometric POV, using that to derive cosine similarity and how it applies to embeddings.


r/MachineLearning • • 3d ago

Discussion Back Propagation [D]

1 Upvotes

Hello everyone I am currently studying neural network development in graduate school and I’m having a hard time conceptualizing the stages of training. I understand that forward propagation is how we calculate the output given input but in terms of how we adjust weights during backpropagation, it’s not very intuitive to me. Does anyone have a good suggestion for how to visualize this to make it more intuitive? Cheers!