2026-10-12
Why random cross-validation overstated fraud performance
The split is part of the question
A fraud model is not just a classifier trained on rows. It is a decision system evaluated on transactions that arrive after training. If training and validation share time-dependent identity patterns, a random split can answer an easier question than the one a review team will face.
An earlier IEEE-CIS Kaggle notebook described a random-versus-chronological split comparison, but its public notebook URL is currently unavailable and those historical scores have not been reproduced from accessible source. They are therefore not used as current validation evidence; consult the published time-holdout validation report below instead.
Keep the production evaluation separate
The portfolio's deployed fraud pipeline has its own time-based holdout: 118,108 transactions. Its measured ranking results are ROC-AUC 0.9134 and average precision 0.5354. Those figures refer to this held-out pipeline and should not be silently substituted for the earlier split-comparison scores.
The model is used to prioritize human review, not to automatically decline transactions. On its documented holdout, reviewing the highest-scoring 5% of transactions captures 58.32% of fraud labels at 40.13% precision. The operational question is whether that workload and precision are acceptable, not whether one score looks impressive in isolation.
A good ranking is not necessarily a probability
Weighted training changes the scale of the classifier output. In the documented holdout, its mean score was 17.96% while observed fraud prevalence was 3.44%. Its Brier score was 0.0636, worse than the 0.0332 reported for a constant base-rate forecast. Treating these raw scores as probabilities would therefore be misleading, even though they can still rank transactions for triage.
Another warning comes from risk bands: the High band's observed fraud rate was 44.59% on the fitted portion but 30.34% on the holdout. Showing only the first value would exaggerate expected field performance.
The review rule
I would first choose validation splits that reflect when predictions will be made, then measure both ranking and the human review budget. I would report risk-band rates only on unseen time periods, separate score ranking from probability calibration, and keep assumptions in any cost-benefit simulation explicit.
The evidence does not establish that a particular review threshold will save a bank a specified amount. It supports a transparent comparison of thresholds under stated assumptions. The linked project repository, time-holdout validation report and operational documentation contain the currently inspectable measurements and their limitations.