2026-10-12
What a lucky holdout taught me about model validation
A clean holdout can still be optimistic
For the MSCapital forecasting project, I reduced 804.5 million raw market rows to 292 engineered features per sample. The evaluation used expanding walk-forward training with an embargo. Across five folds, the ensemble averaged +0.14088 cosine similarity, with 0.0041 fold-to-fold standard deviation.
Six untouched months then produced +0.15171 cosine. That result was measured once, but one untouched period can still be an unusually favorable sample of market conditions. Independence from model fitting does not guarantee representativeness of the next period.
Record a forecast before seeing the answer
The project recorded a forecast around 0.143 before its graded submission; the single LightGBM submission scored 0.128. A later ensemble trained on four additional months scored 0.129. Both were below the optimistic holdout value, and the shipped ensemble remained below the leaderboard field median.
Rather than editing the narrative to make the holdout look predictive, I kept the discrepancy visible and treated it as the next research question.
Test explanations instead of selecting a story
Six hypotheses were investigated against the gap. One confirmed finding was period difficulty: the holdout sat at the 83rd percentile of period difficulty, and adjusting for it produced +0.14084, close to the independently computed walk-forward mean. A spread-regime difference also contributed, but less than anticipated. Their estimated shares of the gap overlap and must not be added as independent explanations.
Four other proposed explanations were falsified at the widths actually tested. The later release also evaluated a learned sequence model, regime features, extra ensemble members and feature pruning. None demonstrated a reliable improvement over the shipped tabular ensemble. A positive point estimate alone was not enough to overturn paired uncertainty.
Practical lessons for an ML delivery team
Keep a written prediction of external performance before the result appears. Prefer repeated time-based comparisons over one favorable period, inspect baseline noise, and measure candidate changes as paired differences against the current model. Report null or negative experiments with the same care as wins.
This was competition research, not an investment strategy or evidence of trading profitability. The reproducible notebooks, source repository and model card show the training, evaluation and limitations, including tests that did not support the original hypotheses.