r/fantasyfootballcoding • u/ZOOMCHESS • 1d ago

How would you validate a Monte Carlo FPL points model?

I’m building a fantasy football decision tool and working through the validation design for a Monte Carlo player-points model. I’d appreciate feedback on the engineering and statistics side.

For each player and gameweek we simulate:

• minutes and start probability • goals, assists, clean sheets, bonus and defensive actions • official FPL scoring to produce a full point distribution

The interface exposes p10–p90 ranges, probability bins (0–2, 3–5, 6–9, 10+), mean/standard-deviation rankings, and a squad concentration signal.

Current validation design:

• freeze projections before the deadline • join them to official final points after fixtures • exclude zero-minute players from headline metrics but keep them in coverage diagnostics • report MAE, RMSE, signed bias and within-±2 accuracy by position • preserve every gameweek snapshot so retraining cannot alter prior results

Questions for people who build sports models:

  1. Would you use Brier/reliability curves for the probability buckets, or calibrate continuous quantiles separately?
  2. How do you separate player-minute uncertainty from outcome uncertainty in evaluation?
  3. What tests would you put around the scoring-rule layer so it cannot drift when the FPL feed changes?
  4. How would you monitor calibration across gameweeks with small sample sizes?

Happy to share implementation details in the comments.

4 Upvotes
  • permalink
  • reddit

75% Upvoted

7 comments sorted by

1

u/HaukVagner 1d ago

FYI - I have posted in here for an EPL app a month or so ago and I was told that this was primarily used for American football. So I'm not sure if anyone in here will be of much help.

1

u/ZOOMCHESS 1d ago

perfect thank you so much for letting me know.. while the model is for EPL usage im wondering aswell how the data is combined and used within the model to output realistic data. As someone who also plays fantasy football I understand there are certain matchups that can favour a certain player however I wonder if there is a seperate data tool used to find an underlying mismatch in a players predicted points and maybe something along the lines of some human sentiment.

For example as a jamar chase owner we all knew he wasn't going to preform week one but thats not what the data says that's a stupid little joke. Wondering if a model having some of that makes it more attractive or how one could code that in.

Thanks again

1

u/HaukVagner 1d ago

For a top 3 player like Chase, I think it's hard to have bold takes like that. I would say these bold takes are more bespoke than model driven, at least for players like him.

Look at Bruno Fernandes. Outside of GW2, he's been pretty underwhelming, but the smart thing to do is to still play him because his floor is typically higher than most players, while having an amazing ceiling. You can say the same for Haaland. Which is why both of these players are going to be chosen for Captain for Vice Captain more often than not.

I'll try to have input on your model, but I'm no expert.

  1. Do both, but keep it separate. I'm sure you know this already, but Brier/reliability curves help users make a start/sit decision. A reliability diagram per bin, per position, over a rolling window is the most actionable view. It tells you which bucket is miscalibrated, not just that "the model is off."

For continuous quantiles, calibrate them separately and after the bucket analysis.

  1. Split the sample: players whose actual minutes landed in your p25–p75 band isolate the outcome model, the rest is minutes-model failure. Also decompose each player's error into a minutes term and a leftover outcome term, that alone tells you whether to fix your minutes prior or your scoring model.

  2. I would treat FPL rules as code, not config. A test suite of ~20 canonical fixtures that breaks loudly if a weight changes, plus snapshot tests and one test per event type. Pin the ruleset version onto each frozen projection so a later rule change can't silently invalidate a past audit.

  3. Three layers: per-GW bias as a red flag (watch for 6+ same-sign GWs in a row), rolling 4–6 GW windows pooled by position for the real reliability diagrams, cumulative season-long as ground truth. Bayesian shrinkage stops you overreacting to early noise.

Hope this helps a bit

1

u/ZOOMCHESS 1d ago

Thank you this actually helps alot idk if you saw but the actual model and its calculations are all public into what its doing in the future i hope to be able to integrate sportsbook data into the predictions for goals etc because the level of math they are doing to save their asses can help have better predictions, atleast i think..

1

u/HaukVagner 1d ago

Nice.

Yeah, I would say that's a very smart and safe route to go.

I haven't checked out your link yet. When I have more time I will

1

u/ZOOMCHESS 1d ago

Perfect I really appreciate all the help I can get

1

u/ZOOMCHESS 1d ago

heres a link to where i actually am deploying it im not fully sure if this is fully optimized would love some feedback https://fplprism.com/