Overfitting
Give a flexible model enough features and it will memorise last season, including its noise. It scores beautifully on data it has seen and worse on data it has not. That gap is overfitting, and it is the default outcome of model building, not a rare accident.
The three sets
| Set | Used for | Rule |
|---|---|---|
| Train | Fitting parameters | Everything before a cut-off date |
| Validation | Choosing features and settings | Later data, looked at repeatedly |
| Holdout / test | The final performance estimate | Looked at once; never tuned on |
In sports, splits must respect time. A random shuffle lets the model learn from the future. Walk-forward testing refits on everything up to each period and scores only the next one — the closest thing to how a model is actually used.
Leakage
Leakage is any path by which information from the outcome sneaks into the inputs — a stat that includes the game being predicted, a feature built from closing prices, an identity join that pulls in a later record. Leaked models post spectacular backtests and fail in production.
The PBE Fight Model card lists its leakage checks and states that no model feature comes from a sportsbook price. The NHL methodology froze its release criteria before the first fit — beat the baselines on log loss and Brier, stay calibrated, hold up season by season, show zero leakage findings — and keeps even a passing model in shadow until validated.
A clean holdout estimates past performance under past conditions. Rule changes, new data sources and roster shifts can all move a sport away from what the model learned.