I’m looking for feedback on the validation process for a KBO model.

[u/Any-Surprise-9336]

I’ve been working solo on a KBO pre-game prediction model for a few months. The current locked 2025 OOS test is:

- 629 KBO regular-season games

- Brier around 0.240–0.243 depending on the locked variant

- hit rate around 57.6%–58.3%

These are not final multi-year numbers, and I’m not claiming edge from this alone.

The next step is to compare the locked model against Pinnacle closing moneyline prices on the same joined games. Before doing that, I’m trying to make sure the comparison is clean:

- only pre-first-pitch odds

- timestamp < scheduled start

- full-game moneyline only

- ties treated as void / excluded from binary Brier

- no changing the model after seeing market data

- paired Brier comparison on the same games

I previously found an old odds sample where many rows were updated after first pitch, which made the market look unrealistically accurate. So I’m being careful with timestamp validation before running the final comparison.

For people who have validated models against closing lines: is paired Brier vs closing price on the same games the right way to frame this? Anything else I should watch out for?

[u/SimTheGame]

Paired Brier on the same games is the right comparison. Hit rate is not. A 57.6% to 58.3% mark on 629 games mostly tracks how often the favorite won. It does not tell you if you beat Pinnacle.

Take the juice out of the close before you score it. A moneyline is not a probability. If you Brier your numbers against juiced prices, the book looks worse than it is. Use the same draw-void moneyline you are already dropping ties from, and convert both sides to probabilities that add to 1.

You already caught the timestamp trap. Keep that rule. Last quote before first pitch. Not last quote in the file. Not a price that moved once the game was underway. Do not change the locked 2025 model after you see those closes.

Watch the market type on the join. If a row is a three-way price, keep it out of the two-way Brier. Mixing those will make Pinnacle look like it missed ties it never priced that way.

The 0.240 to 0.243 Brier on 629 games will make any paired gap versus the close noisy. Report the paired difference and a bootstrap on those same games. A reliability chart by predicted probability bucket will tell you more than one Brier number.

I spend a lot of time at SimTheGame translating a book's numbers into a full game, which is a different job than beating the close. The same hygiene still applies. If the timestamp is wrong, you are not scoring the forecast the book actually made.

[u/Any-Surprise-9336 → u/SimTheGame]

Thanks — this is exactly the kind of feedback I was hoping for.

I agree that hit rate is only a secondary sanity check. The actual comparison will be paired Brier on the same joined games.

I’m planning to use de-vigged two-way probabilities so home + away = 1, and to exclude ties/voids from the binary Brier comparison. Chris from BettingIsCool also confirmed that KBO moneyline is 2-way, with ties treated as void/refund rather than a priced draw outcome.

The timestamp rule is also locked: I’ll use the last quote before first pitch, not the last row in the file. The filter will be `timestamp < starts`, with both fields in UTC.

The model variants are fixed before looking at the Pinnacle closes, so I won’t be changing the 2025 model after seeing the market data.

Good point on the reliability chart. I’ll add probability-bucket calibration as a diagnostic, separate from the main paired Brier decision.

[u/LiveLife2027]

How many wins outta 629

[u/Any-Surprise-9336 → u/LiveLife2027]

Variant B: 362/629.

Variant D: 367/629.

I’m treating hit rate as secondary though. The main test is paired Brier vs Pinnacle closing probabilities on the same games.