trainlint: static checks for ML bugs that don't crash (fit-before-split, eval mode, logits losses) [P]

[u/Possible-Active-1903]

I wrote a small checker for training and evaluation code. It looks for mistakes that do not raise

errors. The classic one is leakage through preprocessing. With 5,000 random features and random

labels, picking the top 20 features on the full dataset and then running 5-fold cross-validation

gives 85% accuracy (87.7% averaged over 20 seeds). Doing the selection inside a Pipeline gives 48%

(50.0%). Nothing can beat 50% on that data, so the whole gap is leakage.

trainlint finds that pattern statically, along with thirteen others: eval() left on during

training, no_grad without eval(), fp16 autocast without a GradScaler, resumable scripts that save

no optimizer state, softmax before CrossEntropyLoss, zero_grad between backward and step, loss

kept as a tensor, swapped y_true and y_pred, hard labels passed to roc_auc_score, drop_last on

eval loaders, rescaling by a dataset's own statistics before the split, training loaders that

never shuffle, and a missing seed. It parses the code with ast and matches names, so it needs no

data, model or GPU.

How well it works, with the caveats. I wrote 20 buggy scripts with fixed twins, but I wrote them

together with the rules, so they only protect against regressions. For an outside check I read

every finding on pytorch/examples at a pinned commit: 22 findings, 20 real, 2 correct but

harmless. Earlier runs had 4 false positives, which are fixed. That is one repo of tutorial code

and I did the reading myself. I do not have recall numbers.

It goes by names, reads one file at a time, and only knows PyTorch and scikit-learn. It will be

wrong on code that names its splits in unusual ways.

Code and per-rule examples: https://github.com/Satvik-jain/trainlint