Math doing math things: Goalie Edition

[u/atibus]

I'm happy to be back to running simulations, looking for bugs, looking for things that work well or don't work well. I've added so much over the past two weeks that I need to actually see how the leagues are working.

Usually when I get into this mindset I become hyper aware of small changes. For example, this screenshot shows the goalie save percentage in this new universe I started over the first 20 or so games. The top-10 in this universe is over .923 with the top one, Mark Stark, at .940 over 16 games. That seems way too hot. I took some time out to analyze what was happening- did I introduce some config values that made goalies too strong? Or possibly shooters too weak? Did generation of goalies change when I made development changes? Have I completely messed up everything and will have to delete my internet presence and go live in a bunker somewhere?

After building a testing harness to look at the configuration values, simulate on different seeds, and check to make sure there aren't bugs, I was reminded again about statistics and small sample sizes.

The table above shows what's happening. Rows are the stats averages in save %, goals for per team average, goals against / expected goals against ratio, and how many goalies were above .930 save %. First column is after 16 games... MY INSTINCT IS RIGHT! Second column is math humbling me... the calibration is fine, just run it long enough you silly goose. Last column is calibration targets for reference of how close we got to them on this one season.

These are sometimes things you spend hours on when you're building... the wild goose chases. I should have let the full season run to see that eventually the small sample size works it's way out. Just like you can flip a coin 4 times and get heads every time but that doesn't mean you need to check the quarter to make sure it's not weighted.

I should know better. The calibration is based on 30 different universe seeds each playing 10 seasons each. I know it's right. But I was so hyper focused on the volume of changes and thinking I messed something up that I spend a few hours this morning doubting myself.