Reporting Uncertainty in Permutation Feature Importance: Why the Usual Dispersion Measure Is the Wrong One

Abstract: Permutation feature importance is usually reported as one mean value per feature and ranked, even though the standard implementation produces a dispersion measure alongside every mean. This paper argues that the dispersion normally reported is the wrong quantity, and demonstrates the consequences on a real model. For a support vector regression forecasting Nigerian FDI one quarter ahead, evaluated on 13 held-out quarters, we report permutation importance three ways: the mean, the standard deviation across 30 permutation repetitions, and a bootstrap 95% confidence interval from 1,000 resamples of the test observations. The three disagree in a way that matters. A screening rule based on the ratio of standard deviation to mean, which some authors have proposed, would clear four of six features as reliable. The bootstrap intervals show that not one of the six has an importance whose sign is established, including current FDI, which is the strongest variable in the companion econometric model. The reason is that permutation dispersion measures Monte Carlo noise in the shuffling procedure, which the analyst controls through the repetition count, while the bootstrap measures sampling uncertainty over the test observations, which the analyst does not. We recommend reporting the repetition count, and reporting bootstrap intervals wherever the holdout permits.

Keywords: Permutation Importance; Uncertainty Quantification; Bootstrap; Model Interpretation.