Analysis · Finance
Fifteen Forecast Scores, Almost the Same Leaderboard
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Change the scoring rule, keep the winner: a large forecasting comparison found that 15 accuracy measures produced remarkably similar overall rankings. Yet agreement about which method ranks first does not tell us how much better its forecasts are, or whether a particular market estimate deserves confidence.
In On the selection of forecasting accuracy measures, first published online in 2021, Diamantis Koutsandreas and colleagues compared 14 forecasting methods using historical data from the M4 forecasting competition. Their analysis covered 89,894 yearly, quarterly and monthly series across six domains, including finance and industry.
The researchers trained methods on the available training records, then scored predictions against held-out observations. Forecast horizons were six years, eight quarters or 18 months, depending on frequency. They ranked methods within each series, averaged those ranks and used statistical comparisons to assess differences. This was an empirical evaluation of forecasting methods, not an experiment on market behaviour.
Across the full dataset, correlations between the rankings from different measures exceeded 0.97. ARIMA ranked first and a combination of three smoothing methods ranked second throughout. Some agreement was mathematical: dividing every method’s absolute error by the same positive baseline within a series cannot change their order. Multiple scores therefore need not provide independent confirmation.
The boundaries matter. The researchers excluded certain constant test sequences and series with nonpositive forecasts to keep measures defined. They studied non-intermittent data and point predictions, not uncertainty ranges. Rankings also conceal the size of performance gaps. Within industry data, the leading method changed with the score, showing that overall agreement can hide narrower disagreements.
For checking a market forecast, the useful lesson is to examine the evaluation behind its accuracy claim: which series, which horizon, which baseline and which exclusions? Agreement across scores can support a comparative ranking under those conditions. It does not validate a growth forecast’s assumptions or establish its eventual accuracy.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- On the selection of forecasting accuracy measures Diamantis Koutsandreas, Evangelos Spiliotis, Fotios Petropoulos …
