Analysis · Finance
The Forecast That Rarely Won but Performed Best Overall
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
The strongest overall forecast rarely won an individual prediction. In a 2022 study of US COVID-19 mortality forecasts, Estee Y. Cramer and colleagues found that a model combining multiple teams’ predictions achieved the best aggregate accuracy, even though it finished first less often than any other evaluated model.
The researchers evaluated forecasts submitted across 79 weeks from April 2020 through October 2021. Their main comparison covered predictions one to four weeks ahead. A simple baseline predicted that weekly reported deaths would stay at the most recent week’s level, with uncertainty based on past weekly changes.
Accuracy included more than the distance between a predicted number and the outcome. The weighted interval score penalized both wide uncertainty ranges and observations falling outside them. Comparisons also adjusted for differences in the difficulty of the forecasts each model submitted. On this measure, the ensemble recorded 39% less probabilistic error than the baseline overall.
Consistency was its distinctive strength: it was the only model to rank in the top half for more than 75% of its predictions. Yet strong overall performance concealed weaknesses. During selected waves, forecasts tended to underestimate rising deaths and overestimate falling deaths. Some uncertainty ranges missed outcomes more frequently than their stated coverage implied.
This was an evaluation of operational forecasts, not a controlled experiment isolating which modeling choices caused better performance. Teams changed models, submitted different sets of predictions, and faced reporting anomalies and revised death counts. Eligibility rules also excluded some approaches with limited coverage. The findings concern reported deaths, not financial or business outcomes.
For readers assessing market estimates, the transferable lesson is a method of scrutiny: request a simple baseline, a defined forecast horizon, and performance across the full record. A few impressive successes cannot establish dependable accuracy, and even the strongest average result deserves inspection at moments when trends change.
Sources used for this article (1)
Publisher reports used to prepare this article. Sources with unavailable links are marked below.
- Source 1
- Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States — source link unavailable. Link checked . ['Estee Y. Cramer', 'Evan L Ray', 'Velma K. Lopez', 'Johannes Bracher', 'Andrea Brennen', 'Alvaro J. Castro Rivadeneira', 'Aaron Gerding', 'Tilmann Gneiting', 'Katie House', 'Yuxin Huang', 'Dasuni Jay
