Residual PPO trailed TWAP by 0.019 bps at the default ±10% bound across 1,000 paired out-of-sample IBM episodes. The 95% interval was [−0.110, 0.071]. Since it crosses zero, the result is a statistical tie. Its narrow width also puts a useful bound on the possible gain. And that outcome is close to what the cost model guarantees before training begins.
Where would the timing edge come from?
The RL execution studies cited in this paper report gains over TWAP or analytical schedules. Their pitch is straightforward: a learned policy observes the market during liquidation and moves volume toward cheaper moments. Timing earns the money. Tosi, Barbierato and Gatti test this proposition using 1-minute IBM midquotes from 1998 to 2025, covering 7,030 trading days with one sell episode per day. The agent must liquidate 10,000 shares across 8 intervals each day.
For each interval, the fill calculation begins with a representative midprice, taken as the average of its two boundary midprices. The simulator subtracts a half-spread of 0.01. No unit is given; at IBM prices, this reads as a cent. Temporary impact subtracts another 0.05 times the fraction of the order sold in that interval, and uniform noise of ±0.002 is then added. Historical prices supply the paths. The authors fit no price model.
PPO (Proximal Policy Optimization) controls the schedule. The reported results use what the authors call the residual formulation: PPO chooses a multiplier around TWAP. A slice equals the 1,250-share TWAP amount times (1 + δ), with δ restricted to ±0.10. The eighth interval clears the remaining inventory. Five inputs reach the agent: elapsed time, remaining inventory, the midprice and two lagged returns.
Every strategy receives the same test days and noise draws, allowing pathwise comparisons. Across 5 seeds, all bootstrap intervals overlap zero; the Wilcoxon test on pooled differences gives p = 0.57. With the bound widened to 0.20, the result is 0.008 bps and the interval is [−0.081, 0.094]. The authors report that preliminary unconstrained runs generated extreme front- and back-loaded schedules. They also say some PPO and DDQN runs appeared profitable on average until paired evaluation reduced the gains. We found no table or figure quantifying those runs.
Taken alone, a p of 0.57 tells us the test could not detect a gain. The interval ceilings are more informative. At the 0.10 bound, the 95% interval limits the edge to 0.071 bps; at 0.20, that ceiling is 0.094 bps. Calling this equivalence would require a margin that the paper does not specify. We also did not find the frictions themselves expressed in bps in the text.
Equal slices start with the friction advantage
Equal slices minimize friction in this model. A learned schedule would need forecastable drift to beat them on total IS. Per-share friction is s + k·q/Q0, giving total friction of s·Q0 + (k/Q0)·Σq². Once Σq is fixed at 10,000, equal slices minimize Σq². Impact on a 1,250-share slice comes to $0.00625 a share.
Small departures from TWAP cost very little, too. Alternating ±10% across every slice adds (0.05/10,000)·8·125², or about $0.63 for the entire order. The friction penalty for a ±10% wiggle is therefore about $0.63 on the whole order. At that size, drift decides the outcome. Any gain must come from Σq(P0 − P̄t), the drift term, and earning it requires a forecast.
The authors regress next-step returns on the two lagged returns and describe the R² as close to zero, without printing its value. PPO is thus being asked to recover an optimum that the simulator already gives to TWAP. It succeeds to within about a tenth of a basis point.
The authors supply the information side of this argument themselves. They do not identify equal slices as the friction minimum under linear costs. They write that "the negative result should therefore be read mainly as an information result", and explicitly caution against treating it as a verdict on RL for execution generally. Their defence is that different impact curves or state-dependent spreads would not necessarily change the conclusion. In their account, persistent outperformance requires information about future prices, liquidity, or impact, and "the state variables used here do not appear to contain that information."
That reading holds for price. Liquidity is a different matter: the state already includes the clock t/T. A spread that changes by time of day would be forecastable from the state vector alone, allowing a non-flat schedule to outperform TWAP without a directional bet. The constant s = 0.01 shuts off precisely that channel.
What the clamp buys
The residual bound delivers the promised stability. Schedules in the paper's figure remain near TWAP. The authors put the distinction carefully: "The residual formulation addresses the stability issue; the performance issue remains open." For a desk accountable for its fills, smoother schedules are a useful control property. The paper describes the residual policy as a bounded perturbation of TWAP.
The last slice needs attention.
Because the eighth interval takes all remaining inventory, only seven slices obey the ±10% bound. Put the first seven at the same edge and the final slice can range from 375 to 2,125 shares, as much as 70% away from TWAP.
Implementation decisions we had to make
Our own build uses only a simulator and minute-bar proxies rather than the authors' quote data. Several details remain unresolved:
- We did not find the train/test split, the method for choosing the 1,000 test episodes, the PPO hyperparameters, or the training sample size.
- Table 2 gives a discount factor of 0.10, although the text describes an undiscounted objective. Across 8 steps, γ = 0.10 gives the last reward a weight of 10⁻⁷, leaving the agent nearly myopic. We follow the text with γ = 1.
- Time and inventory are normalized in the state. The paper calls the third input the current midprice but gives no normalization. We divide it by the arrival price.
- We did not find the session hours covered by the 8 intervals.
- Table 1 reports mean daily volatility of 7.36 bps and an intraday return standard deviation of 8.99 bps. We could not reconcile those units.
- The body describes the raw data as proprietary and unavailable for redistribution. The data availability statement says they are openly available on GitHub.
The limit of our minute-bar build
We are building the residual mechanism on US equity minute bars, which contain neither quoted spreads nor realized impact. Our version uses bar-price proxies and takes the paper's 0.01 and 0.05 as assumptions. We cannot confirm a match to their IBM quote data or their interval definition. The build can check whether residual PPO behaves as described within a simulator. It cannot assess real execution cost, and the paper's figures do not carry over to our universe.
The evaluation protocol is worth copying. At the 0.10 bound, shared paths and noise draws yield an interval about 0.18 bps wide over 1,000 days. TWAP's own IS varies widely across those days and crosses zero in the paper's Figure 1. Pairing is what makes the comparison informative. A tail metric would be the next addition; the authors themselves name risk-sensitive objectives as future work.
One result would change my reading. Run the exact protocol again with a time-of-day spread, making the cost-minimizing schedule non-flat. If residual PPO still ties TWAP in that setting, the null says something about RL itself.