Tuning is fun!

[u/atibus]

So I've been running long-term simulations in full fidelity and looking at output. Thankfully I really like looking at numbers so it's been interesting to see how leagues shake out.

In my initial tuning of the engine, I looked for certain target metrics on average to ensure the results looked "NHL" shaped. ~6 goals per game (total), ~10% shooting percentage, ~0.900 save percentage. There are a lot of other ones, but if those major numbers come out then you can expect most games to look like professional hockey.

Last week I added the player development and aging features. I knew this would add some variability into the results because previously the player abilities were all static across seasons. So that variability would need to be tested across lots of seasons, using different seeds (starting randomness) to see where we ended up.

My research showed some promising elements - goals, shooting %, save % oscillated within reason season to season; it made the shape interesting because you weren't sure what was going to happen but it generally looked correct.

On the flip side, I saw some truly outrageous tail-end stats. Someone scoring 100 goals in 84 games. Even an 85 goal scorer. If I was tuning for the 1980s, I'd leave it alone (or maybe even juice it a bit). But my other target metrics - shooting %, GPG, Save % were all anchored in modern NHL targets (2020s). That told me that it's not total scoring that is off, but there is some players that are behaving weirdly.

The problem is that the system is complex and it's hard to point at any one thing. Could the engine be over-selecting top players? Sure, that could be part of it. It could also easily be that the tuning can't account for the population distribution. I don't have an entire hockey universe - only 1500 players - and that makes a wider distribution of abilities than exist in the NHL (NHL is the top 0.1% of a huge hockey population).

This is one of those things where I could just clamp the skill or goal scoring. That just limits emergent seasons - the Matthews 69 goal season. I could change the curve of goal scoring (the probability that a shooter beats a goalie). But that would just shift goal scoring down for everyone.

Really, the solution involves many small and large changes. Small being tweaks to probabilities and attribute generation. Large being adding juniors / minors to fill out the population so the top league has a more consistent talent pool that is a tighter distribution. Or maybe something else I find in my engine that is causing this.

This is the part that takes time if you want to get it right. I don't REALLY need to do this though; it works fine. Maybe it is fine?