Analysis · Finance
When a Forecast Test Finds a Difference, Has It Found a Winner?
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Two forecast models can have similar average losses yet distribute their mistakes differently. That creates a subtle problem for claims of superior accuracy: detecting a difference in error patterns is not the same as establishing that one model is better across all error thresholds.
Hossein Hassani and Emmanuel Sirimal Silva’s 2015 methodological paper proposed the Kolmogorov-Smirnov Predictive Accuracy test, or KSPA. It compares distributions of absolute or squared forecast errors. The researchers assessed the method through simulations and historical forecasting examples, rather than an experiment identifying causes of market movements.
The procedure uses observations withheld from model training to evaluate predictions. For each error threshold, it calculates the fraction of errors at or below that value. The two-sided test measures the largest gap between the resulting cumulative curves. Its question concerns identical error distributions; the benchmark Diebold-Mariano test instead concerns equal average losses. These are different hypotheses.
That distinction matters when interpreting the paper’s 10,000 simulation replications. A test that detects distributional differences more frequently is not automatically a universally better test of average accuracy. The authors reported favorable KSPA results, but some findings about correlated errors were described without tables, limiting scrutiny of their broader robustness claims.
There is another boundary in the paper’s directional test. Its stated alternative requires one cumulative curve to exceed the other somewhere. That alone does not establish that it stays above the other everywhere. The imports example even describes exceptions to the apparent ordering. Evidence at some thresholds therefore needs a narrower interpretation than an across-the-board winner.
For a market forecast comparison, ask what the test actually establishes, alongside the evaluation period, horizon and error measure. Failure to reject equal mean losses or identical error distributions does not establish that the models are equivalent. A detected difference, meanwhile, needs interpretation before it becomes a claim of superiority.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts Hossein Hassani, Emmanuel Sirimal Silva
