A forecasting model can look sharp on historical data and fall flat the moment the month changes. Start by fixing the question: what would you actually have known before the match, and how would you have judged the decision after it? An honest test captures both moments.
Recreate the decision in chronological order
Train on January through March. Record simulated April predictions before looking at April outcomes, complete with timestamps. Move forward one window, train through April and then test May. Set the process in advance and keep a final period untouched, even if April makes you want to tweak the model. A 2026 study on pre-match LaLiga markets shows how temporal controls can be built into a forecast evaluation.
Audit when each field became available
Record when the team sheet was announced, when an injury was reported and when each price was observed. A closing price fed into a morning prediction gives the model information it would not have had. The same happens with a rolling average that includes future matches or with duplicate events shared between training and evaluation. Log the time, market, rules, source and exclusion criteria for each simulated decision.
Keep different questions in different columns
Suppose you assign a 70% probability to each of 20 simulated cases, and 12 of them occur. The observed rate is 60%. That is a reason to investigate and a small sample from which to draw a conclusion. The scikit-learn calibration guide explains how to group probabilities and compare them to observed outcomes. Report sample size, costs, misses and results by period alongside any performance figure. Closing line value compares like-for-like prices and does not establish profitability. Start with a constant simulated stake to isolate forecast performance. Kelly sizing introduces another variable and depends heavily on the quality of the estimated probabilities.
Finish with one proposed improvement, the observation that would disprove it and the date you intend to run the next evaluation. Label a change in market or source as a fresh test.


