AQAI QuantAI research lab for systematic strategies

Automated analysis

This analysis was drafted by our research engine and has not been checked by a human editor. It may contain errors. It separates the paper’s own results from our tests, and any figures called ours come from our own backtest.

Our automated analysisOur backtest

GBM wins with a linear hedger and loses with an MLP

A replication note on decision-compatible scenario generation, and the four choices the paper leaves to the implementer

2026-09-08 · 7 min read · US equities and listed equity options (end-of-day).

Reviewing: Rethinking Synthetic Scenario Realism: Compatibility, Not Fidelity, Drives Hedging Performance · Ryuji Hashimoto, Masanori Hirano, Ryota Ozaki et al. · Read it on arxiv

Our backtest of this idea

Our automated quick test, not the paper's

Rolling Decision-Compatible Deep Hedging for US ETF Options

Backtest period 2020-01-01 to 2024-07-01 · hypothetical, net of modelled costs

Why these figures are not the paper's (1)

Our own audit found this run does not follow the paper faithfully (15)

  • deviation left undescribed by the audit (invalidates: All paper N225-specific realism values and Figure 3 statistics; direct numerical comparison with Tables 1-4; the paper's reported best generator-hedger pairs)
  • deviation left undescribed by the audit (invalidates: All fixed-period N225 numerical results; Figure 3 statistics; Tables 1-4 rankings and winners)
  • deviation left undescribed by the audit (invalidates: The paper's task-specific generator rankings and reported European and Lookback winners)
  • deviation left undescribed by the audit (invalidates: The paper's hedger-class-specific learnability values; Table 3 winners; Table 4 rankings)

11 further finding(s) are described in the note.

These are our findings about our own implementation, not criticisms of the paper. Read the figures below as a description of what we ran.

Jan 2020Total 5.9%Jul 2024
Sharpe
1.78
Total Return
5.9%
Max Drawdown
-0.6%
CAGR
1.3%
Volatility
0.7%
Beta vs SPY
0.01
Trades
635

If you train a deep hedger on simulated paths, you cannot choose the simulator before you have fixed the network. The evidence is a rank table that turns over when the hedger architecture changes.

Hashimoto, Hirano, Ozaki and Imajo start from a distinction that is easy to state and awkward to measure. Realism asks whether generated paths reproduce the statistical properties of the market. Compatibility, their term, asks whether the generator preserves the loss landscape induced by the strategies the hedger can actually express. Formally the compatibility gap is the supremum over admissible strategies of the absolute difference between the risk of that strategy under the true law and under the generator. Theorem 1 bounds the excess risk of the deployed hedger by the learning error, measured under the synthetic law, plus twice that gap. Theorem 2 is a construction. Score realism with a finite-dimensional linear class of test functions that does not span the loss direction of some admissible strategy, and you can build a pair of measures with realism discrepancy exactly zero and a strictly positive compatibility gap.

The experiment is a full crossing. Four generators: geometric Brownian motion, Merton jumps, Heston with Andersen's quadratic-exponential scheme, and a VAE whose encoder and decoder are three-layer MLPs with 16 hidden units and latent dimension 16. Three hedgers: a linear policy, the no-transaction band network, and a four-layer 32-unit MLP. The band network predicts a band from Black-Scholes delta and the previous position, and trades only when the position falls outside it. Four tasks: European or lookback call, transaction cost of 1e-4 or 1e-3, spot and strike both 1.000, maturity 20 days and therefore 20 decision steps. Three risk functionals, trained and evaluated matched: entropic risk at lambda 1 and 50, and CVaR at 0.90. Data is daily Nikkei 225, calibrated on 2018-01-01 to 2022-12-31 (1,218 trading days) and tested on 2023-01-01 to 2026-02-28 (771 trading days). The test window has kurtosis 15.943, skewness of -0.733, a Hill index on absolute returns of 2.557 and a Hurst exponent on absolute returns of 0.642. Heavy tails, persistent volatility.

The money question for a desk is which engine to train the book's hedging policy on. The paper's answer is that the question is not well posed until you name the architecture and the payoff.

Two terms in the budget, one of them estimated

The bound is the part I would keep. It licenses something practitioners already do and rarely defend: deliberately training on a simplified law. The authors say it outright, that a generator introducing a compatibility gap can still win if the induced simplification cuts the learning error enough. Their own numbers make the case. GBM is worst on almost every realism metric (Kolmogorov-Smirnov distance 8.619e-2, kurtosis gap 16.032, Hill gap 4.047). It also has the lowest generalization gap in every hedger class. For the linear hedger that figure is 2.847e-2, against 7.296e-2 for Merton, 3.599e-2 for Heston and 3.552e-2 for the VAE.

The second term never gets estimated. The authors concede as much in their limitations: compatibility cannot be measured directly and is only assessed through indirect proxies. Their proxies are the generalization gap and the epoch-to-epoch variation of validation loss over epochs at or beyond 100. Both are diagnostics of the learning error term in Theorem 1, not of the gap. So the compatibility half of the budget is inferred residually from which generator happened to rank first. The authors name the measurement problem as future work. Their proposal is compatibility-aware deep hedging, where the generator and the hedger are jointly designed to optimize decision performance. A review has to argue against that proposal rather than ignore it. The theory as printed is a decomposition you cannot yet fill in.

Which simulator wins?

No generator wins realism outright. Heston takes the KS distance (4.315e-2) and the absolute-return autocorrelation (0.809e-1). The VAE takes mean absolute return (1.120e-4), raw autocorrelation (1.773e-2) and Hurst (0.678e-1). Merton takes kurtosis (12.752) and Hill (0.237).

The interaction claim is where the paper earns its title. The best pair differs in all four task configurations: Merton with the MLP hedger for the European call at cost 1e-4, GBM with the linear hedger for the European call at 1e-3, Heston with the MLP for the lookback at 1e-4, GBM with the band network for the lookback at 1e-3. GBM ranks 1.00 on the lookback at 1e-4 for the linear hedger and 4.00 for the MLP hedger on that same task. Heston beats Merton in all four tasks for the linear hedger (1.67, 3.00, 2.00, 3.00 against 3.50, 4.00, 4.00, 4.00). Switch to the band network and that ordering breaks down: the paper notes the Heston advantage diminishes there, and Heston ranks 4.00 against Merton's 3.00 on the European call at 1e-4. Learnability is jointly determined too. The band network's generalization gap runs from 7.782e-2 on GBM to 19.627e-2 on Merton, while the MLP hedger's is nearly generator-invariant at 4.295e-2 to 4.579e-2.

Hedging performance is reported only as average ranks over ERM(1), ERM(50) and CVaR(0.90). No P&L, no entropic-risk or CVaR level, no Sharpe. So the economic size of any of these reversals is unknown. A cell can move a full place if one of three runs flips. Cells like 1.33 and 3.50 tell you how thin the underlying counts are.

What we built on SPY and QQQ

The figures below are from our own run, one automated pass at the deployment version of this. Over 2020-01-01 to 2024-07-01 on daily bars it returned 5.86% in total, with a Sharpe of 1.78, a Sortino of 2.44 and a Calmar of 2.17. Maximum drawdown was -0.59% and volatility 0.74%. 5.86% total over four and a half years is a thin result. The drawdown never exceeds 0.59% and volatility is 0.74%, because a single episode at a time against a 10% position cap leaves the account mostly in cash. The same cap compresses the return and flatters the risk numbers at once.

Our window covers part of the authors' 2018 to 2022 calibration period and about the first eighteen months of their 2023-01-01 to 2026-02-28 test period, on SPY and QQQ rather than the N225. The paper publishes no absolute performance for itself, only ranks, so there is no like-for-like number of theirs to set against ours.

The book shorts one observed near-ATM SPY or QQQ call per episode: strike over spot between 0.95 and 1.05, delta within 0.20 of 0.50, and 20 to 35 calendar days to expiry. Each episode runs exactly 20 daily decisions. We hedge at each close with a policy trained on rolling synthetic paths from all four generators and all three hedger classes, with the position projected into [0,1]. Calibration uses the 1,218 closes ending no later than entry. We charged four tenths of a cent a share with a one dollar order minimum and configured no slippage.

The run is short gamma by construction. Two of our choices matter more than anything in the paper. We hedge at the daily close with zero modelled slippage, and we cap the hedge at one share per share of notional. Neither is the authors' setup. A weak result here is evidence about our pass first.

Four settings the paper never pins down

These four decided our run, and we did not find any of them fixed in the text:

Two further features of the design would move my reading if changed. The parametric generators are calibrated by minimizing a stylized-facts distance and then scored on eight overlapping stylized-facts metrics, so Table 1 is partly a report on the calibration objective. And the VAE is trained on 24 tickers including VIX, TNX, BTC-USD and ETH-USD, while GBM, Merton and Heston see the N225 alone. The comparison confounds generator class with training information, and it does so in exactly the comparisons that carry the interaction claim. We have made a version of this point before about policies whose measured edge depends on the simulator they were tuned in (the LLM TWAP note).

What would change my mind is one number the paper does not print: the realized entropic risk or CVaR level per generator, in units of notional, on the same 20-step episodes. If the spread between the best and worst generator is a few basis points of notional, the reversals are a curiosity. If it is large, the selection protocol implied here becomes part of the hedging desk's model risk process, and the compatibility gap becomes something worth trying to estimate directly.

Our backtest stops at 2024-07-01, and everything after that date is deliberately left untouched so the same strategy can be checked out of sample later.

How our backtest worked

The steps the code we ran actually executed, from its strategy card. Ours, not the paper's — it is one automated implementation of the idea, not the authors' own.

For each eligible SPY or QQQ episode origin:
  1. Require 1,218 valid closes ending no later than entry, 10 prior returns,
     and a complete 20-decision-step realized path.
  2. Query that date's observed option chain and retain calls with:
       - 20–35 calendar days to expiration,
       - strike/spot in [0.95, 1.05],
       - |delta - 0.50| <= 0.20,
       - exactly 20 available daily decisions through expiration.
  3. Break ties by delta distance, log-moneyness, expiration, then option name.
  4. Calibrate GBM, Merton, Heston-QE, and VAE generators on point-in-time data.
     Generate a 10-return feature burn-in and 20-step training paths.
  5. For each generator, task, cost level, risk configuration, and hedger class,
     train the policy under the same ERM or CVaR functional used for evaluation.
  6. For the listed-call task, short one contract at its observed option close.
     For the lookback task, create only a shadow liability and no option order.
  7. At each daily close, compute point-in-time features, obtain the proposed
     hedge, project it into [0,1], and trade the observed underlying close.
     Hedge shares = contracts × 100 × projected position.
  8. Reject a new episode if its allocation exceeds the 10% position cap or
     the 4× leverage constraint. Skip any fill or episode lacking an observed
     execution price.
  9. At expiration, settle the contractual terminal payoff and unwind the
     underlying at its observed close; voluntary early exercise is ignored.
 10. Rank generators separately by realized matched-risk loss for every
     underlying, hedger, task, cost level, and risk configuration.