A calibrated 97.5% VaR model gives a one-year backtest about six exceedances to work with. In Coelho's 250-day windows, the median was nine days when either of two models was breached. With that little evidence, a Diebold-Mariano test tends to reward the less conservative model. At τ = 0.025, it wins 73.1% of significant window results. Coelho gives this desk intuition a formal footing through theorems and an application to seven equity indices. The findings should change how risk teams choose between models.

The evidence arrives on miss days

Consider two interval forecasts evaluated with the interval score. When the outcome falls inside both intervals, their score difference is simply their width difference, already known before the close. Only a miss moves the difference away from that baseline.

Quantile scores behave the same way. So does the FZ0 loss of Patton, Ziegel and Chen, which evaluates (VaR, ES) pairs. Outside exceedance days, the forecasts alone determine the loss gap. The count of exceedances therefore sets the informative sample size. Coelho illustrates this with 99% intervals: ten thousand rounds provide little more than a hundred informative observations.

To use that information, Coelho constructs an empirical-Bernstein confidence sequence, a confidence interval that remains valid as observations accumulate. It compares the score difference with the predictable width gap, estimating variance from deviations on miss days alone. Validity holds at every sample size without a distributional assumption. When expected scores were equal, it incorrectly excluded zero in at most 0.1% of replications.

A lower bound in the paper applies to every test. For some forecast pairs, reducing the tail level α does not reduce the required number of misses. The required number of days then grows like 1/α. Simulations follow that pattern. For a narrowed Gaussian interval, the empirical-Bernstein sequence separates the forecasts after a median of between 191 and 237 misses as α moves from 0.2 to 0.005.

Bounds calibrated to the range of the score difference, the Hoeffding-type procedures, exact a steep price. At α = 0.2, they require 1.2 to 1.7 times as many rounds as the empirical-Bernstein sequence. At α = 0.005, the multiple rises to 50 to 73. For that smaller tail level, the requirement is about 1.4 to 2.7 million rounds.

Why does the narrower interval win?

Imagine a 250-day sample without a miss. Each day rewards the narrower forecast by its width advantage, and the variance estimate captures only variation in widths. The resulting t-statistic is large. The paper explicitly says its bound on this event identifies the mechanism while leaving its size open. In the Gaussian design at α = 0.01 and T = 250, the probability of no misses is below 0.1%. Yet the test favours the narrower interval in 12% of samples.

When neither forecaster is better, a two-sided 5% test should reject in each direction 2.5% of the time. At α = 0.01 and T = 250, rejection in favour of the narrower interval occurs in 12.0% of Gaussian samples. With Student-t errors and random scale, the rate reaches 18.6%. Rejection in favour of the wider interval never exceeds 2.8%. Every design with fewer than 25 expected misses has an error rate above 5% toward the narrower interval.

Conventional small-sample adjustments change the rate by at most 1.5 points. For the Student-t case above, the Harvey-Leybourne-Newbold correction produces 18.4%, and the Coroneo-Iacone fixed-smoothing test produces 17.4%. These adjustments correct variance. The distortion comes from skew: the narrow forecast wins a little on most days, while the wide forecast wins heavily on a few. A bootstrap-t version captures that skew and stays at or below 3.6% except with four expected misses, where it reaches 6.4%.

Seven indices, with qualifications

The equity application covers daily returns from January 2000 to September 2026 on the S&P 500, FTSE 100, DAX, Nikkei 225, Euro Stoxx 50, Hang Seng and Bovespa. Six standard models supply (VaR, ES) forecasts at τ = 0.025 and 0.01: two historical simulations, EWMA, two GARCH variants and filtered historical simulation. Coelho applies Diebold-Mariano tests to non-overlapping 250-day windows. Each level has 2,130 window-pair tests, and each significant sign is checked against the mean loss difference across all remaining days.

At τ = 0.025, significant results account for 17.8% of tests; 73.1% of them favour the less conservative model. Among those results, 18.4% have the opposite sign to the remaining-days mean. The corresponding share for results favouring the more conservative model is 6.9%. At τ = 0.01, the less conservative model wins 80.2% of significant results, with wrong-sign rates of 21.1% against 10.5%.

Those figures need their qualifications alongside them. The remaining-days mean is an estimate, so a sign disagreement identifies a possible error without proving one. The windows are also dependent. Coelho resamples calendar years to obtain 95% intervals for the gap in wrong-sign rates. They are [1.7, 19.7] points at τ = 0.025 and [0.0, 18.9] at τ = 0.01; he describes the latter as borderline. He also acknowledges that the tilt at τ = 0.025 is modest, given that the less conservative model already wins 66.7% of significant full-period results.

He judges the tilt large at τ = 0.01. There, the less conservative model wins 80.2% of window results, compared with 50.7% over the full period.

The application measures forecast losses, not P&L.

We did not find a position-sizing rule in the paper, and we could not backtest it either. SPY with a VaR-scaled position would be the natural substitute. That would require a sizing rule the paper does not supply, and percentages from its seven-index application would not carry over to a single ETF. Such a backtest would be a different study.

Bootstrap before moving the risk budget

With circular blocks of 10 days, the bootstrap-t version removes most of the tilt. The less conservative model wins only 51% (τ = 0.025) and 60% (τ = 0.01) of significant window results. Compare its wins with wins for the more conservative model: at τ = 0.025, the wrong-sign rates are 12.9% against 9.5%. At τ = 0.01, they are 14.2% against 6.3%. The remaining asymmetry is concentrated where exceedances are scarcest. The bootstrap has no finite-sample guarantee.

The confidence sequence provides that guarantee at a cost in power. In the Gaussian simulation with plentiful misses, consider the horizon at which a known-variance test reaches 80% power. Diebold-Mariano achieves 74% to 84%; the tuned sequence achieves 12% to 38%. Reaching the same power takes the tuned sequence 1.9 to 3.3 times as many rounds as Diebold-Mariano.

For ES, the asymmetry is structural. Without assumptions, a less conservative ES can be certified worse using the days when it was less conservative. Certifying it better requires a bound on the loss tail, whatever the test. Certification never succeeds within a 250-day window. Over the full period, it succeeds for 47.6% (τ = 0.025) and 43.8% (τ = 0.01) of pairs. Diebold-Mariano agrees in 48 of 50 and 36 of 46 cases and never takes the opposite side.

Flooring returns at twice the more conservative ES is a tuning decision. It changes power while preserving validity. With floors of 1.5, 2 and 3 times the more conservative ES, certification rates span 48% to 50% (τ = 0.025) and 42% to 44% (τ = 0.01).

Before a 250-day Diebold-Mariano win sends capital toward a lighter VaR model, count the exceedances and rerun the test with a bootstrap. I would reconsider the bootstrap half of that rule if a longer panel kept showing separated wrong-sign rates at τ = 0.01, as this one does at 14.2% against 6.3%.