Validation Protocol and Reported Forecast Accuracy: Holdout, Time-Series Cross-Validation and Walk-Forward Compared

Abstract: Reported forecast accuracy depends on the validation protocol used to compute it, and papers rarely state which they used. This paper holds the model, features, hyperparameter grid and dataset fixed and varies only the protocol. A support vector regression forecasting Nigerian FDI one quarter ahead, evaluated on 13 held-out quarters from 2021Q3 to 2024Q3, returns RMSE 902.72 under a single holdout with ordinary three-fold cross-validation for tuning, 901.95 under the same holdout with time-series cross-validation, and 653.34 under expanding-window walk-forward evaluation. Out-of-sample R-squared moves from -1.90 to -0.52. The direction here runs opposite to the standard warning: the naive protocol understated accuracy here, it did not overstate it. Switching the tuning procedure from ordinary to time-aware cross-validation changed almost nothing (902.72 against 901.95), which locates the effect in the evaluation step and not the tuning step. We explain the mechanism, which is evaluation variance rather than leakage bias, and argue that this makes protocol disclosure more important, not less, because the direction of the effect cannot be inferred from the reported number.

Keywords: Validation Protocol; Walk-Forward Evaluation; Forecast Accuracy; Evaluation Variance.