Analysis · Finance
A Good Average Forecast Can Hide Unreliable Extremes
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
Train the same forecasting network 100 times, and its overall scores can look reassuringly similar while its predictions for extreme events vary sharply. That happened in a wind forecasting case study: ordinary performance checks barely registered differences in the reliability of predictions for unusually strong winds.
Jakob Benjamin Wessel, Maybritt Schillinger, Frank Kwasniok and Sam Allen reported this in their 2026 methodological study, Enforcing tail calibration when training probabilistic forecast models. They examined ways to change what models are rewarded for during training, using simulations and a weather forecasting application.
The application used forecasts issued 72 hours ahead for 124 UK weather stations. Models learned from April 2019 through December 2020 and were evaluated on January 2021 through March 2022. The main extreme-event threshold was 12.5 metres per second, approximately the 97.5th percentile of training observations pooled across stations.
The key distinction was between reliability across all outcomes and reliability above that threshold. Extreme-event forecasts need two checks: whether predicted exceedance probabilities match observed frequency, and whether predictions of severity beyond the threshold are calibrated. Training against severity alone improved that particular diagnostic but led models to strongly overpredict how often the threshold would be crossed.
Adding penalties that considered both aspects, or giving extreme outcomes more weight in the training score, improved tail reliability in the test data. These adjustments also reduced variation across repeated neural-network fits. But gains depended on the model and its starting performance; better tail calibration could come with worse overall calibration or accuracy.
This was a weather methods study, not evidence that the approach improves market forecasts. Data scarcity prevented testing much higher thresholds, and exceeding the chosen threshold did not necessarily mean damaging weather. For assessing market forecast claims, the useful question is therefore specific: what evidence tests the stated extreme scenario, beyond the model’s average score?
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Enforcing tail calibration when training probabilistic forecast models Jakob Benjamin Wessel, Maybritt Schillinger, Frank Kwasniok …
