Analysis · Finance
The Same Forecasts Can Put a Baseline First or Last
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
A simple forecasting baseline ranked first under one accuracy measure and last under another, using the same predictions. That reversal illustrates why a claim of superior forecasting needs more than an impressive error score: the scoring rule can change the verdict.
In their 2017 methodological study, Chao Chen, Jamie Twycross and Jonathan M. Garibaldi proposed UMBRAE, the Unscaled Mean Bounded Relative Absolute Error. They evaluated it using constructed examples and 3,003 time series from the M3 forecasting competition, comparing 22 forecasting methods plus a naïve benchmark across the first six forecast horizons.
The reversal involved mean relative absolute error, which divides each forecast error by the benchmark error before averaging. When the benchmark was almost exactly right, that division produced enormous ratios for competing methods. These extremes helped put the naïve benchmark first. UMBRAE instead placed it last, with all 22 alternatives scoring better on that measure.
UMBRAE divides each absolute forecast error by the sum of that error and the benchmark’s absolute error. This bounds each contribution between zero and one; if both errors are zero, the contribution is set to one-half. The average is then transformed so that one represents benchmark performance. In the study’s 3% trimming check, UMBRAE’s rankings stayed unchanged.
That stability has limits. Bounding contributions reduces the influence of very large misses, so the score does not directly represent their full magnitude. Frequent perfect benchmark forecasts can also bias the comparison. The synthetic examples were designed to expose scoring problems, not represent typical data, and the authors assumed rather than tested reliability and sensitivity.
For assessing market forecast claims, the practical lesson is to request the benchmark, scoring rule and treatment of extreme errors together. This study evaluates ways to measure past forecasting performance; it does not establish which model will predict a particular market best.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- A new accuracy measure based on bounded relative error for time series forecasting Chao Chen, Jamie Twycross, Jonathan M. Garibaldi
