Analysis · Finance
A Forecast Can Win or Lose Depending on How You Score Its Uncertainty
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Two forecasts can miss the same outcome, yet disagree about which performed better depending on the scoring rule. A 2021 methods paper by Johannes Bracher, Evan L. Ray, Tilmann Gneiting and Nicholas G. Reich shows why evaluating uncertainty requires more than checking the headline prediction.
The authors explain established scoring methods for forecasts reported as a median plus prediction intervals. Their weighted interval score combines interval widths with penalties when observations fall outside them. Wider ranges incur a cost; narrower ranges risk larger penalties for misses. Lower scores are better, and the mathematical design encourages forecasters to report their actual uncertainty.
A constructed example makes the tradeoff concrete. Two forecasts have expected counts of 60 and 80, but the observed count is 190. The logarithmic score favors the first forecast because it assigns more probability to that extreme outcome. The weighted interval score favors the second because its reported quantiles sit closer to the observation. These are illustrative calculations, not experimental findings.
The paper also compares historical forecasts from the 2016–2017 FluSight Challenge. Across 26 models, weighted interval scores agreed almost perfectly with a related measure using the full predictive distribution. Agreement with logarithmic scores was weaker: some models had fairly accurate central predictions but assigned too little probability to extreme outcomes.
The limits matter. That historical agreement does not establish that the metrics are interchangeable elsewhere. The method addresses central prediction intervals, and averages can be dominated by larger targets or longer forecast horizons. The authors also acknowledge that the costs of different mistakes may justify different scoring choices.
For readers assessing market forecasts, the transferable lesson concerns evaluation, not market performance. Ask what probability a range represents, how misses are penalized, and whether comparisons use matching horizons. This paper offers a way to assess forecast uncertainty; it does not demonstrate better business or investment outcomes.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Evaluating epidemic forecasts in an interval format Johannes Bracher, Evan L Ray, Tilmann Gneiting …
