Analysis · Finance
When a 95% Forecast Interval Covers Only 55% of Outcomes
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
A forecast range labelled 95% can miss almost half the outcomes it is supposed to cover. In a 2025 electricity demand study by Hussein Ahmad Ahmad and colleagues, one model’s day-ahead intervals contained actual demand just 55.3% of the time. The label described the intended coverage; the test measured what it delivered.
The researchers compared recurrent neural networks using hourly electricity load, generation and market data from 2015–2018. They separated training, validation and testing chronologically and evaluated forecasts one, six, twelve and twenty-four hours ahead. Inputs included past demand, calendar information and an existing load forecast.
They constructed uncertainty ranges in two ways. Deep Ensembles combined five independently trained networks. Monte Carlo Dropout repeatedly ran a network while randomly omitting neurons. Both methods used the spread of their predictions to form nominal 95% intervals, then checked how often observed demand fell inside those bounds.
For the GRU network, dropout intervals covered 92.7% of outcomes one hour ahead, falling to 55.3% twenty-four hours ahead. The corresponding ensemble covered 62.9% and 36.2%. Dropout produced wider ranges and better coverage in this comparison, but its day-ahead result remained far below the stated target. Better than a competitor did not mean adequately calibrated.
These are retrospective forecasting results, not evidence that deployment improves grid operations or causes financial savings. The methods assessed model-related uncertainty without explicitly separating inherent data noise. Stronger seasonal statistical benchmarks were left for future work, and the study does not establish that dropout will outperform ensembles elsewhere.
For anyone assessing a forecast claim, the useful question is how often its range actually contained subsequent observations at the relevant horizon. Interval width matters alongside coverage: a narrow range can look precise while missing frequently. Here, the day-ahead coverage figures place a clear limit on claims of operational reliability.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Interpretable Deep Learning for Short-Term Electricity Load Forecasting with Calibrated Prediction Intervals Hussein Ahmad Ahmad, Taha Benarbia, Seyyed Kasra Mortazavi …
