Almost any monthly US import-disruption index beats the realized-volatility benchmark that Salisu, Ogbonna, Gupta and Gil-Alana use to forecast WTI volatility. The breadth of that success weakens the case. Umbrellas, cocoa and photographic goods perform about as well as the aggregate index. A soft benchmark explains much of what we see here. For a desk, the paper offers a reason to test supply data against its own risk models.
What goes into the forecast?
Liu et al. (2023) build the Supply Disruption Index (SDI) from about 200 million shipment-level import transactions in US seaborne trade since 2007. The monthly index measures the fraction of established importer-exporter pairs, those that have traded often, that temporarily stop trading. Its average is scaled to zero before January 2020. Read it as a relative probability: how much less likely a recently active pair is to trade again than before the pandemic. The aggregate has a sample mean of 0.56 and a standard deviation of 0.85. Sub-indexes cover 4 census regions, 25 countries of origin, 8 product categories and 88 HS 2-digit industries.
The index captures broad import friction. Counting pairs gives a stalled furniture supplier the same weight as a stalled fuel supplier. Petroleum is the subject of only the Energy category and its HS counterpart, mineral fuels and oils.
GARCH-MIDAS combines a GARCH(1,1) short-run variance term with a slow long-run term by multiplying them. Monthly SDI enters the long-run term through a one-parameter beta lag weighting. Estimation uses a rolling window and normal innovations. The target consists of daily EIA WTI spot log returns from 2 January 2013 to 30 August 2024: 2925 observations, with kurtosis of 42.29. In the benchmark, realized volatility (RV) occupies the same long-run slot.
The authors propose that disruptions create rare-disaster jump risk and an output shock. The latter lowers prices and feeds the leverage effect. The aggregate SDI's in-sample slope is 0.1056, with a standard error of 0.0405 and a t near 2.6. We did not find a test separating those channels.
Forecast MSE falls 20% to 24% against RV at 20 and 60 days
Starting out-of-sample on 2 January 2020, the aggregate SDI model records MSE of 23.9575 versus 29.9877 for the benchmark at h=20. The reduction is 20.1%. At h=60, MSE is 18.8288 versus 24.6980, a 23.8% reduction. The advantage narrows at h=120 to 11.7703 versus 12.5085, or 5.9%. Modified Diebold-Mariano statistics (DM*, where negative values favor SDI) are -5.4258, -5.1473 and -3.1090, all at 1%.
These horizons cover 20, 60 and 120 days of a daily variance process. The reductions concern forecast errors; no P&L accompanies them. Economic-significance results appear in Supplementary Table S1, outside the paper body.
Cocoa, umbrellas and the oil-input explanation
Slopes across the indexes range from 0.092 to 0.108, with standard errors around 0.040. At h=20, DM* ranges from -5.2910 for Plants to -5.5610 for Animal products. Umbrellas delivers -5.4076, close to the aggregate's -5.4258.
The authors recognize this uniformity and draw the opposite conclusion. They describe "a stable pattern of predictability across the overall and various sub-indexes of SDIs." For them, it highlights "the paramount role played by oil as an input" across the aggregate economy, its regions and its industries.
MSE provides a cleaner test of that explanation. Every SDI model retained in the 90% MCS at h=20 lies between 23.8769 for Photographic goods and 24.0365 for Clocks, watches. The range is 0.16, compared with a 6-point gap to the benchmark. The paper gives no h=20 MSE for the eight excluded models. Energy records 23.9371, behind photographic goods and just 0.036 below Umbrellas at 23.9733. An explanation based on oil-input exposure should give the oil-relevant index some separation from the rest. It has none.
Other monthly series achieve much the same result against RV. At h=20, the New York Fed's Global Supply Chain Pressure Index (GSCPI) produces DM* of -5.4013. Baumeister et al.'s global economic conditions index (GECON) produces -5.4170. Neither is an SDI series.
Replacing RV with almost any slow-moving monthly series in the long-run term appears to account for most of the improvement.
The authors also appeal to persistence, interpreting it as evidence for supply content. They write that GSCPI and GECON "exhibit comparatively weaker persistence," and emphasize "the importance of persistent supply-side information." The aggregate SDI's long-memory estimate is 0.4267, with a 95% interval of (0.2018, 0.6515), after a dummy removes the 2020:04 break. Yet GSCPI, which the paper finds less persistent, achieves nearly the same improvement over RV. Its DM* statistics are -5.4013, -5.1225 and -2.9943; SDI's are -5.4258, -5.1473 and -3.1090. Removing the RV driver tracks the improvement more closely than any property of its replacement.
There are also reporting issues. Utilities and Residues and food industry waste have identical statistics in Tables 1 and 2: slope 1.0585E-01 and DM* -5.5103 at h=20. Their Table 3 MSEs are identical too, at 23.9115, 18.7810, 11.6938, although their Table 3 v_M statistics differ. Energy and Mineral fuels, oils follow the same pattern, both reporting MSE 23.9371. Table 1 also lists 130 monthly observations for a window covering 140 months.
RV survives the 90% model confidence set
The authors say "over 92% of the SDIs-based GARCH-MIDAS models qualify as candidate models in the 90% MCS" and contrast that result with the RV model's "low p-values". Those values are 0.1092, 0.1257 and 0.6663. Each exceeds 0.10. The benchmark therefore remains in the 90% model confidence set at every horizon. The table note instead excludes models with p below 0.01, which corresponds to a 99% set.
MCS membership says equal accuracy could not be rejected. It cannot, on its own, establish that a retained model beats another retained model. Pairwise DM* near -5.4 and an MCS unable to exclude the benchmark thus give different accounts of the results. At h=120, the benchmark's p of 0.6663 puts it closer to the top models than DM* suggests. Mexico and seven HS industries leave the set at h=20 and h=60, then return at h=120.
What would earn a place in a CL risk system?
The model comparisons are harder to interpret than they first appear. GSCPI scores DM* of -5.4013, -5.1225 and -2.9943 against RV, almost matching SDI. In direct comparisons, SDI beats GSCPI at 3.6014, 3.9374 and 10.5281. It beats GECON at 14.8217, 15.0211 and 19.8703. Diebold-Mariano divides by the variance of the loss differential. Highly correlated forecasts can therefore turn a small difference into a large statistic. We did not find MSE figures for the GSCPI and GECON models, leaving the size of that difference unknown.
GSCPI and GECON are the basis for the abstract's claim that SDIs outperform other widely used monthly measures. A footnote reports a geopolitical-risk model tying SDI at all three horizons. A climate-policy-uncertainty model ties it at h=20 and h=60. Details of both are available on request.
The forecast target deserves attention too. The paper uses EIA WTI spot, with an out-of-sample window beginning 2 January 2020 that includes the April 2020 dislocation. We did not find an explanation of how negative spot prints entered the log returns.
We could not test this ourselves. Our data contain neither the SDIs constructed from those 200 million shipment records nor the EIA WTI spot series. Using CL futures would change the forecast target while leaving the predictor unavailable. Such a run would not replicate the paper.
We would put the Energy sub-index on our test list after one result: SDI beating a plain GARCH(1,1) or a HAR model fitted on the same 2925 days, by more than the 0.16 MSE points separating every retained SDI model at h=20.