Analysis · Finance
A Forecast’s Error Score Depends on How You Test the Past
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
A forecasting model can look reassuringly accurate when its test lets it learn from observations later than those it predicts. The order of training and testing matters: an error score can describe a task quite different from forecasting an unseen future.
Vítor Cerqueira, Luís Torgo and Igor Mozetič examined this problem in their 2020 study, supplied here as a 2019 manuscript. They compared 11 methods for estimating forecast error using 62 real-world time series and three stationary synthetic settings. This was a comparison of evaluation methods, not a test of any market growth forecast.
Their design tested the error estimates themselves. The first 70% of each series supported training and internal testing; the final 30% provided a later validation period. After retraining on the first portion, they measured the model’s error on that later period and checked how closely each evaluation method had anticipated it.
Across the real-world series, repeated holdout testing had the best average rank for estimating error accurately. It used several training windows followed by later testing windows. Cross-validation methods generally underestimated eventual error, while holdout and sequential methods tended to overestimate it. The finding concerns the reliability of an error estimate, not an improvement in the underlying forecast.
The pattern depended on the data. Cross-validation remained competitive for stationary series, whose statistical properties stay stable over time. Holdout approaches performed better among non-stationary series. That association supports the authors’ concern about changing conditions, but does not isolate why each method succeeded. The study focused on predicting the next value from one variable’s past, with results mainly reported for one forecasting algorithm.
For a market forecast claim, a useful question follows: did every test use only information available beforehand, and were several periods examined? Those details help assess an accuracy claim. They do not establish that a long-term sector estimate or company projection will hold.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Evaluating time series forecasting models: an empirical study on performance estimation methods Vítor Cerqueira, Luı́s Torgo, Igor Mozetič
