I would keep an execution agent out of all five neural order-book simulators tested by Nigmatulin, Mohl, Frey, Zohren and colleagues. On AMD, their 100-child metaorder moves the simulated price by 17 to 70 bps. Even the largest non-neural baseline move is only 11 bps, produced by a replay rule that shifts the book permanently. However convincing the message stream, an agent learning execution costs in these generators would learn the wrong costs.
The authors agree in their Discussion: today's generators "remain unsuitable as environments for execution agents." My objection concerns the abstract. It foregrounds impact that "emerges naturally" and exponents "slightly above" 0.5. Readers have to reach Section 3.2 to find the 4 to 17x amplitude.
How the benchmark gets a response
The authors test autoregressive message generators, models trained to predict the next LOBSTER message. The set comprises two S5 variants, two Mamba-3 variants and a Gated-DeltaNet (GDN). Training covers eight NASDAQ names from January 2022 to December 2025. The intended destination for such simulators is a world model for reinforcement-learning traders. Existing benchmarks, including LOB-Bench, judge spread, volume and inter-arrival distributions. Those scores cannot tell us how a simulated market responds to an agent's orders. LOB-Bench tests that response for a single event, stopping short of a metaorder.
This benchmark asks directly. The authors condition on real data from January 2026, a month excluded from training, then inject a metaorder containing 100 aggressive children. Each child uses the stock's median market-order size. The spacing is ΔN = 9/φ generated messages, with φ denoting the stream's market-order share; this fixes participation at 10%. For AMD, children arrive 401 messages apart, making a metaorder roughly 40k messages long.
They fit I = a q^δ to cumulative participation q. Under the empirical square-root law, δ is near 0.5 and the dimensionless amplitude Y near 0.5. In a separate relaxation experiment, they execute 10 children and wait silently for 100 blocks. Each stock (EA, NVDA, MSFT, AMD and GOOG) has a separate campaign of 1,024 rollouts per direction per phase across 20 trading days.
The blind control is the part of the design I trust most. Each build-up runs twice, using identical windows and seeds. Blind orders fill in the matching engine, yet disappear from both the message stream and the book state supplied to the model. Both runs therefore experience mechanical depletion; the model can respond only in the visible run. This distinction matters. On AMD, replay kernels containing no model produce identified exponents of δ = 0.73 to 0.81, within the neural range. The exponent alone cannot establish where impact comes from.
The visible peak is at least 3x the blind floor in all 25 neural cells, and more than 10x in 16. AMD's ratio ranges from 5.0 (Mamba3-4k) to 26 (S5-4k). The models generate the impact themselves.
The amplitude costs 4 to 17x as much
All 25 model-stock pairs yield an identified law. Exponents range from 0.63 to 0.93 on AMD and MSFT, 0.73 to 1.21 on GOOG, 0.97 to 1.15 on NVDA and 0.84 to 1.12 on EA. Every one of the 25 day-clustered intervals excludes 0.5. Even the tightest interval, S5-120M on MSFT, has its lower bound above 0.53.
Calling these exponents "slightly above" 0.5 asks too much of the phrase. The highest, 1.21, exceeds twice 0.5. All neural cells fail the authors' square-root tolerance, |δ-0.5| ≤ 0.10. Their results section is clearer than the abstract: the neural law "is not the square-root law."
Amplitude is the abstract's other omission. AMD's neural panel gives Y of 2.2 to 8.6, compared with an empirical 0.5. The authors describe that as four to seventeen times the price a real metaorder pays per unit of participation.
The estimator, they argue, biases δ downward. On planted laws at neural amplitudes, the bias reaches at most 0.11, implying that the underlying build-up could be steeper still. I accept that direction, with less confidence in the magnitude. AMD's kernels return 0.73 to 0.81 despite closed-form values near 0.9 to 1.0. The authors attribute this to small-amplitude bias and restrict their quoted 0.11 bound to neural amplitudes. With kernel Y on AMD at 0.96 to 1.46, they have not established that bound outside the neural regime. Neither the Propagator nor the Obizhaeva-Wang kernel is identified on GOOG or EA at all.
The law survives on a stock absent from training; its calibration does not. On EA, Y = 12 to 50. In-corpus Y runs from 1.7 to 19 (2 to 9 on AMD and MSFT, 1.7 to 12.6 on GOOG, 7 to 19 on NVDA). The authors put the OOD result plainly: it "does not destroy the law; it destroys its calibration."
What happens when execution ends?
The abstract acknowledges that the authors could not observe decay. They give that finding up front. On AMD, the details are worse than a flat response: four of five generators end above peak, with end-over-peak ratios between 1.15 and 1.36. Prices continue in the metaorder's direction after its last child. GDN records 0.88, alongside drift of 127 ticks with no orders injected.
GOOG has two rows with substantial apparent decay. GDN drops to 0.18 and Mamba3-4k to 0.24. They are also the block's heaviest-drift rows; Mamba3-4k drifts 879 ticks. Antisymmetrising buys against sells strips out direction-independent drift. The paper's C5 test additionally requires drift below 20 ticks, a threshold every heavy-drift row breaches, while a no-injection arm monitors degradation. Direction-dependent degradation, however, would survive antisymmetrisation unchanged.
That confound hangs over every relaxation figure. AMD rollouts extend roughly 40k messages against contexts of 500- to 4,000-message length. The test also has limited resolution at this spacing: closed-form end-over-peak is 0.94 for the Propagator and 0.87 for Obizhaeva-Wang. Each kernel exhausts most of its transient between children. Four of the five generators finish above 1.0. GDN's 0.88 is close to the kernels' measured 0.89 and 0.81, but its 127-tick drift prevents interpreting the result as resilience.
Distributional fit gives no advance warning. Mamba3 earns the best neural KS on AMD (.106), yet produces δ = 0.87 and Y = 7.65. The cross-stock relationship is unstable too: correlation between KS rank and distance from 0.5 changes sign, from +0.54 on GOOG to -0.71 on MSFT.
Short context remains an untested explanation
The authors' tentative explanation is short context. Relaxation needs a reference price preceding the metaorder, beyond the reach of windows containing 500 to 4,000 messages. Plausible. But the 4,000-message variants bring no improvement on AMD. Mamba3-4k has the same 78M parameters as Mamba3 and delivers δ = 0.93 versus 0.87, with end-over-peak of 1.36 versus 1.16. S5-4k gives δ = 0.79 versus 0.66 for S5-120M, though its parameter count is also different, 55M versus 120M.
The authors acknowledge that these windows still fall far short of a metaorder: 4,000 messages versus about 40k on AMD. They therefore describe the attribution as untested rather than refuted. Fair. Rollout degradation offers a competing explanation, with free drift reaching -251 ticks for Mamba3-4k on AMD and -879 on GOOG. A model that covers a whole metaorder in context and then relaxes toward two-thirds of peak would settle the question.
The authors identify the remaining limits themselves. All tests use one operating point: 10% participation, aggressive children only, and one month on one exchange. If that month were atypical by one within-month standard deviation, δ would move beyond its half-width cap. We did not find a test in which an execution agent trains inside these simulators and has its fills priced. The dollar cost of miscalibration is therefore inferred from the impact figures; no agent is trained and priced.
We could not reproduce any of these results on our data. Our US equities data consist of one-minute and daily OHLCV bars, without quotes or a message-level book. The blind control requires a matching engine and a model whose inputs can selectively exclude injected orders. Bar-level impact proxies would examine a different mechanism.
The benchmark already works as a rejection test for simulators. Every one of the 25 neural cells fails its balance, square-root and relaxation tests.