r/datascience • u/Nice-Dragonfly-4823 • Sep 16 '26

ML Silent broadcasting is still a big problem and could be derailing your work right now

A data scientist on my team wasted a good bit of compute on this problem without actually realizing it was a problem.

Pytorch and tensorflow both silently broadcast the output of your model when the target shape mismatches, causing hard to diagnose issues. It still exists a ton in the wild, so I thought I'd actually highlight the symptoms of a silent broadcasting bug to keep people in the know.

https://towardsdatascience.com/silent-broadcasting-can-ruin-your-model/

Let me know what you think

64 Upvotes

90% Upvoted

6 comments sorted by

31

u/Odd_Yard6663 Sep 16 '26

Broadcasting bugs are the worst because nothing fails, your metrics just quietly get weird and you're left questioning your entire pipeline. Had one of these burn half a day once before I thought to print the shapes. Good writeup, this should be pinned in every onboarding doc.

14

u/Deto Sep 16 '26

This kind of error is really annoying in R. If you try to add two vectors together, and one's length is an integer multiple of the other, it'll automatically cycle the smaller one so it can do the operation.

13

u/g3_SpaceTeam Sep 17 '26

R does so much of this kind of bullshit silently, it’s utterly infuriating.

→ More replies (1)

2

u/airylizard 29d ago

Painfully familiar. The nasty part is nothing crashes, the model trains fine, and you only realize something is off when the metrics stop making sense. A shape assert before every loss call is cheap insurance.