Analysis · Finance
A Better Annual Forecast Score Can Hide Worse Winter Coverage
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Which forecast range is better: one that contains 93% of outcomes, or one that contains 91%? If both promise 95%, the first looks closer to its target. But a 2024 methods study by Ricarda Duerst and Jonas Schöley shows why that annual comparison can miss an important weakness.
Using weekly death records from 23 European countries, the researchers repeatedly fitted a forecasting model to five years of data and predicted the following two years. They used errors from earlier forecasts to build uncertainty ranges, allowing the spread and asymmetry of errors to change with the season. Separate validation periods tested those ranges against unseen observations.
The conventional negative-binomial intervals achieved 93% coverage across the year, compared with 91% for the fitted empirical intervals. Neither reached the stated 95% target. Yet the conventional intervals contained only 85% of winter observations and 99% of autumn observations. Their annual result combined ranges that were too narrow in winter with ranges that were too wide in autumn.
The empirical method brought winter coverage to 89% and autumn coverage to 94%. The researchers also used a score that penalizes both unnecessarily wide ranges and observations falling outside them. On this measure, the empirical method performed better overall and in every season except summer. Closer annual coverage and a better balance of width and misses were therefore different achievements.
This was a comparison of forecasting methods, not an experiment establishing causes of mortality. The empirical approach needs long records and assumes future errors resemble past errors. Its remaining undercoverage matters. Findings for individual weeks also do not establish uncertainty for monthly or annual totals.
For market forecast readers, the transferable question is specific: what does the reported performance measure reward, and which periods does it average together? Requesting both coverage and interval width by relevant period makes the claim more assessable. The paper tests mortality forecasts; any business application would need its own validation.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Empirical prediction intervals applied to short term mortality forecasts and excess deaths Ricarda Duerst, Jonas Schöley
