An agent can optimise its reward to the last decimal while trading a preference nobody approved. Tavakoli and colleagues put a number on that mismatch for risk reviewers. It is worth measuring. Getting a trustworthy measurement is expensive.
The missing continuation hedge
The investor's target is terminal mean-variance equilibrium. At each date and state, it maximises conditional terminal mean minus φ times terminal variance, taking its future selves' policies as given. Terminal variance includes the covariance between today's return and the value the strategy adds later. That covariance gives the correct action a continuation hedge.
Keep the variance penalty and drop the continuation-covariance term, and a myopic reward loses the hedge. More data cannot restore it. In the authors' tabular study, policy distance from equilibrium remains 0.5871 at 8,192 samples per time/state, down from 1.497 at 128. With the same samples, the continuation-aware learner reaches 0.1649, down from 1.436. Their maximum one-step deviation gains are 0.003319 and 0.000273, respectively.
The familiar ΔX minus κ(ΔX)² reward has a different defect. Its second-moment penalty penalises squared expected return too. The paper shows that no constant κ eliminates the discrepancy when wealth grows at s≠1.
The paper contributes a certificate: a bound on how much a change of action at one date and state could improve the stated objective. It takes a frozen policy and critic, and keeps the agent's future policy fixed. Global strategy regret falls outside its scope. The certificate uses the critic-based action gradient and the critic's Bellman residuals. It propagates the residuals' spread across states backward through a mixing coefficient. Only the spread matters. An error constant across every state causes no harm; these offsets changed the bounds by under 1.7×10^-15 across twenty cases.
A constructed failure makes the case for this check. Zero critic-based action gradients made a policy look converged, yet its true deviation gain was 0.10263. The certificate put the bound at 0.89674. A loose bound, but it caught the problem.
The learner uses two independent draws from the same state. Its reward takes their average return and penalises the squared difference of returns plus continuation-critic values. In expectation, the gradient equals the critic-based equilibrium gradient. With an exact continuation critic, that is also the true gradient.
The tests use a simulated two-regime market. Regime-persistence probability p in {.5,.8,.95} and horizon T in {4, 12, 24} produce nine markets, tested across twenty seeds. A separate proximal policy optimisation (PPO) arm uses T=12. The historical study instead fits a one-period long-only allocation to Kenneth French's 10, 30 and 49 value-weighted industries. Its 432 months run from January 1990 to December 2025. It uses a 120-month window, φ=5 and a 10bp L1 cost.
Continuation changes the policy; centring helps the audit
After 3,000 updates in the shared neural actor-critic, mean policy distance is 0.6102 with the myopic reward. Adding continuation brings it to 0.2507 with an ordinary critic and 0.2493 with a centred critic. The paper's 59.2% reduction refers to the centred variant. Maximum deviation gain falls 81.4%, from 0.00320 to 0.00059. The paired distance difference is −0.361, with seed-bootstrap interval [−0.390, −0.329].
Centring barely changes the deployed book. It improves distance by 0.57%, a difference of −0.00143 with interval [−0.00253, −0.00030]. Zero falls outside that interval. Even so, the shift is 0.00143 against a distance near 0.25. The audit benefits more: centring tightens the certificate by 27.1%, from 0.00223 to 0.00163.
The PPO results move in the same direction. Distance falls from 1.7280 to 1.1525, a 33.3% reduction. Maximum gain falls from 0.02349 to 0.01180; the paired distance difference is −0.575, with interval [−0.739, −0.404]. The authors describe this as a structured PPO hybrid. Compute is matched only within each panel, and this arm covers two markets and ten seeds. Its higher distances cannot support a verdict on PPO within the authors' claims.
Can 786,432 audit draws per policy secure sign-off?
For one policy at one tolerance, they cannot. The authors choose τ = 0.001 in normalised utility units. For seed 0, an empirical-Bernstein bound comes to 0.004992 after 786,432 fresh observations. The simulator's true maximum gain is 0.000506, comfortably within τ. The audit nevertheless returns inconclusive. Hoeffding alone produces 0.041214. At the looser tolerance τ = 0.01, the Bernstein bound would pass.
Across twenty fixed policies in the T=12, p=.8 market, true mean maximum gain is 0.000902. Mean bounds come to 0.001257 with the exact model, 0.004911 with Bernstein and estimated mixing, and 0.036141 with Hoeffding. The authors disclose that they added the Bernstein and mixing refinements after seeing the Hoeffding results. They also disclose that they applied the tolerances retrospectively.
Even with 786,432 draws, seed 0's Bernstein bound of 0.004992 is about 10 times its true gain of 0.000506. The authors acknowledge the expense. They call large audit requirements, and abstention when variance cannot be certified, "substantive properties of the method."
The guarantee requires independent conditional draws in every time/state cell. The policy and critic must be frozen before the audit, return support must be known, and returns must be scalar. A historical return series supplies one autocorrelated path. The authors acknowledge that dependent observations require new statistical guarantees. Their generative-data setting also makes stronger demands than a single historical trajectory.
We therefore could not run the certificate ourselves. Our data are ordinary market histories, and a historical allocation backtest cannot establish the audit's assumptions. We did not rerun the one-period industry study either. Today, the certificate can be used with a simulator you trust. That leaves the model risk with the simulator.
The check runs only in a two-regime simulated market.
Calibration reaches the fitted target's 0.570 Sharpe
The historical study stands apart from the sequential tests. It audits a fitted one-period target without running the sequential learner. Joint calibration rescales the reward's risk and cost terms together. Across all 21 covariance/universe cases, it reaches a mean distance within 1.60×10^-7 of its own fitted optimum. Distances for the raw reward range from 0.00827 to 0.02249. A fifty-state borrowing and shorting test gives joint distance of 1.51×10^-7. Leaving financing charges unscaled gives 0.00997.
For ten industries, joint calibration reports allocation distance 0.00000 and reference gap 0.0000bp. Excess Sharpe is 0.570 and realised MV is −8.497bp, identical to the fitted MV reference row. The misaligned raw second-moment reward records 0.573 and −7.182bp. Minimum variance records 0.645 and 5.274bp. It also delivers the highest realised MV among quadratic comparators in all 21 cases. Across all eighteen comparisons of a covariance-model target with the fixed-diagonal target, paired 95% intervals contain zero. Those comparisons span six estimators and three universes.
The authors state the limit directly: "Improved alignment therefore does not establish improved historical investment performance." The abstract makes the same point: joint calibration preserves each fitted target but "does not improve realised investment performance." They argue that alignment deserves measurement separately from backtest performance. I accept that argument.
The 49-industry market-factor case shows the distinction. Changing the covariance model shifts the target by 0.09392, compared with a raw-reward discrepancy of 0.02174. That covariance change is about four times the raw-reward discrepancy. Joint calibration leaves a discrepancy of 1.36×10^-7.
This is a model-risk control for sequential mean-variance agents. Before treating the certificate as routine, I would want it to pass a preregistered tolerance on a multiasset sequential problem with fewer than 786,432 audit draws per policy. The authors leave a realistic sequential multiasset test open. They also say operational users must fix the tolerance before inspecting audit outcomes.