A design Sharpe of 1.69 meant little once the clock moved forward. An LLM agent built a contrarian RSI and volume z-score book, re-ranked monthly, that beat equal-weight buy-and-hold's 1.15 during 2017-2021 on the same 453-stock universe. Across the held-out 2022-2025 period, it made +4.7% at Sharpe 0.18, with confidence interval [-0.80, +1.14]. The buy-and-hold benchmark gained +40.8% in that window. SPY made +51.5%. Such collapses are familiar.
Gençay's sharper finding comes from the design window itself. The agent recorded 102 backtests. Once the search ended, the deflation threshold had reached 1.21 and the Deflated Sharpe Ratio was 0.86, below the 0.95 level. The Deflated Sharpe Ratio compares an observed Sharpe with a threshold that increases as the number and dispersion of trials rise. Here, 102 drove the result.
The author states both headline claims in his abstract. The Sharpe-35 oracle "survives Deflated Sharpe and probability-of-backtest-overfitting testing completely", while honest evaluation "rejects every LLM-discovered strategy". The question is whether four-year windows and a hundred-candidate budget can support that rejection. His closing line, "That is not a failure of the system. It is the system working", earns half my agreement. The rest needs qualification.
A registry beside a trial ledger
Each strategy is a declarative plan, a typed specification that the system executes as written. It identifies features, may connect them through a signal graph of named nodes, and assigns weights to the resulting signals. The portfolio then follows one of five archetypes: a linear tilt, a cross-sectional rank long/short, a volatility-targeting overlay, a random-weights null, and a stateful rule engine.
The agent writes no code. It assembles plans with six registry-backed tools. Every call is checked in-loop for valid parameters and resolved references. Structured errors return to the model for correction. The registry contains a deliberately leaky feature family, though the agent cannot select it. Look-ahead, as the paper puts it, is inexpressible.
Alongside the registry sits the trial ledger. Every strategy evaluation passes through one entry point, leaving a complete account of the search. The trial count N and Sharpe dispersion V therefore come directly from the recorded evaluations and determine the deflation threshold. An evaporation curve places the best design Sharpe after the first k trials against the deflation bar implied by k. In E3, the equity discovery run, the best result rises from -0.07 to 1.69 as the threshold follows it to 1.21.
The data comprise daily bars from Tiingo for 453 liquid US large caps plus SPY over 2015-2026. Every 21 trading days, the system re-selects the tradeable universe point-in-time, taking the top 200 names by trailing 63-day dollar volume. The design period is 2017-2021 and evaluation covers 2022-2025. A separate universe of 39 multi-asset ETFs spans 2005-2026, with design over 2007-2016 and evaluation over 2017-2025. Trading costs include 1bp commission, 2bp spread, square-root impact against trailing 21-day dollar volume, and 50bp annualized borrow on short notional. The models are gpt-4.1 and, in the replication, claude-sonnet-5. Certification requires the held-out Sharpe interval to exclude zero.
The advance over the two cited audits is specific. Li, Kim, Cucuringu and Ma test published LLM timing strategies across a broader cross-section and longer horizon, where the advantages disappear. Yao and Zheng inspect the execution assumptions in thirty studies and provide a checklist without re-running them. Gençay instead uses logged search intensity to deflate the result.
Deflation misses contaminated information
E1 supplies the paper's cleanest finding. Tomorrow's return is planted as an available feature in a literal configuration arm, beyond anything the agent path can express. The oracle records design Sharpe 34.7, evaluation Sharpe 51.5, and DSR 1.00. It passes every statistical screen.
Deflation adjusts for selection across honestly computed backtests. A corrupted information set lies beyond its reach, and this one survives out of sample by construction. The registry is what removes it.
The two guardrails therefore perform separate jobs. In the paper's wording, asking an LLM through a prompt to avoid look-ahead bias supplies weak protection. If the execution environment exposes future information, a text instruction cannot prevent a generated strategy from using it. The author describes the oracle as an intentionally extreme counterexample. Practical leakage often comes from subtler alignment errors, which this example does not model. Its purpose is to establish the boundary of any statistics-only defence.
The validation counts belong beside that boundary. A matched 60-iteration critic emitted free-form plan text without in-loop validation and produced 0 of 60 usable proposals. With the tool loop, 99 of 100 gpt-4.1 proposals and 80 of 100 claude-sonnet-5 proposals became valid plans in the hundred-candidate searches. The twenty claude-sonnet-5 rejections consisted of sixteen duplicate feature roles and four empty plans. Validation caught them before evaluation.
Search context changes the verdict
The verdict applies to the search that generated each strategy. The paper's tables show how much this matters. Safe momentum long/short receives DSR 0.35 in E1 and 0.22 in E2. Its construction and window are unchanged, while its ledger company differs. In E2, the factor grid joins the ledger at N=9 with PBO 0.83. PBO measures the probability that the search's in-sample winner falls below the median candidate out of sample.
E3 reports benchmark DSR values of 0.45 for equal-weight buy-and-hold and 0.31 for SPY. The paper immediately says those arms were never searched. Because they are pre-registered references, each enters at N=1. Their verdicts reduce to undeflated Sharpe probabilities of 0.99 and 0.98. Strip away that note and the table appears to say buy-and-hold failed.
Gençay is explicit about the statistical weak point. DSR treats trials as exchangeable draws. Hypothesis hints rotate and wrap after twenty, allowing duplicate candidates into the E3 ledger, where they are counted at face value. 102 trials amount to far fewer than 102 independent bets. He calls those decisions conservative for the agent. For duplicates, the argument holds because padding N raises the threshold.
A separate dependence appears in the five multi-seed runs at N=18 each. Three of five gpt-4.1 repetitions generated near-identical best strategies, so the multi-seed dispersion makes the search look broader than it was. PBO is also coarse with roughly twenty candidates.
The ledger reaches beyond the agent loop. E2 includes its hyperparameter grid at N=9, while E4 records cost and universe variants as arms. Choices fixed before the system runs remain outside it when they are never backtested internally. Those include the 2017-2021 and 2022-2025 split, the 0.95 certification cutoff, and the cost inputs. We have previously flagged unlogged human decisions of this kind (review).
E5 earns its place
E5 prevents the suite from becoming an expensive demonstration of the random walk. On 39 ETFs and a nine-year held-out window, SPY produces evaluation Sharpe 0.85 and +249%, with CI [+0.24, +1.53]. The 39-ETF equal-weight portfolio reaches 0.71 and +108%, with CI [+0.07, +1.46]. Both certify. Time-series momentum makes +55% over those same nine years, with evaluation Sharpe 0.49 and t around 1.5, yet fails certification. The random-weights null returns +23% at evaluation Sharpe 0.55, beating several active arms after a design Sharpe of -0.57 and DSR 0.0008.
A nine-year out-of-sample gain came from random weights. One of the literature's most documented premia was still under-powered over those same nine years. The paper gives the power arithmetic directly: a true Sharpe of 0.6 requires roughly eleven years of out-of-sample data to reach t near 2. Rejection of every LLM-discovered strategy in this suite therefore says something about four-year windows as well as the strategies.
Five independent gpt-4.1 runs at N=18 each converged on the volatility-breakout family. Every one made money out of sample, posting evaluation Sharpe 0.59-0.70 and +26-33%. All five failed, with DSR 0.15-0.29.
Our run cannot test the claim
We could not use the paper's generator. The exact model versions and prompts used for autonomous candidate generation were unavailable to us. Commission, spread, borrow and impact calibrations are not native historical fields, so the cost model is our own, including a 10bp square-root impact coefficient chosen by us. We did not calculate the DSR, the PBO, the bootstrap intervals or the evaporation curve. The search-aware evaluation layer is one of the paper's two structural corrections, and we did not build it. Our figures below do not test the paper's claim.
We ran one candidate instead: a 252-day cross-sectional momentum rank long/short. It held 20 longs and 20 shorts selected from a point-in-time top-200 liquid universe, re-ranked every 21 days at 100% gross and zero net, from 2020-01-02 through 2024-07-01. The strategy lost 10.81% at Sharpe -0.10. Volatility was 23.55%, maximum drawdown reached -47.78%, and the book generated 2,828 trades.
The drawdown accounts for the loss. A 40-name portfolio capped at 10% per name offers little diversification. Despite running at zero net, it retained beta -0.26 to SPY. Our sample includes the March 2020 crash and the November 2020 winner-loser reversal, the two worst episodes of the decade for twelve-month momentum. The market-neutral book's -0.26 beta put it on the wrong side of the second one. Although 56.84% of trades won, the profit factor was 0.92. Losing trades were larger. Neither episode falls inside the equity experiments' 2022-2025 evaluation window.
The closest paper arm is its 126-day momentum long/short reference, which reports design Sharpe -0.07 and evaluation Sharpe 0.01 at -1.6%. Our version records -10.81% at Sharpe -0.10. Different samples and constructions make those figures unlike-for-like. The headline +4.7% at Sharpe 0.18 belongs to the agent's RSI-and-volume contrarian book, another strategy object altogether.
The signs agree; the magnitudes do not. Verified differences include the window and strategy object. The likely remainder comes from the 252-day lookback versus 126-day and the 20/20 concentration. Our result describes our implementation alone and says nothing about whether the framework certifies correctly.
Questions the suite leaves open
The author acknowledges that the equity constituents come from a fixed current list containing only eight mid-sample delistings. In his account, the resulting survivorship favors active strategies and therefore works against his null. He also reports that the system includes news-sentiment and point-in-time fundamental features, though the headline discovery runs exclude them. The agent searched only price features.
The evidence covers two models and budgets capped at 100 candidates, with most experiments at N=9-21. The author's own arithmetic says a 10^4-candidate search would face a threshold roughly 1.5x higher. The suite reaches a verdict for this search intensity and these windows, as the paper itself states.
I found no sensitivity analysis for the 0.95 cutoff. E3 lands at 0.86 and E2's SPY at 0.94. The prose identifies both as near-misses, while every arm retains its continuous DSR. Reducing the universe from 200 to 100 names shifts momentum's DSR from 0.22 to 0.62 at evaluation Sharpe 0.71. That movement exceeds the effect of doubling or halving all costs. The fragility lies there.
The practical response is cheap. Record every candidate scored by the search, publish N and dispersion next to the Sharpe, and remove leaky features from the callable surface instead of relying on a prompt. E6b extends the same instruments to a human system: an 84-node port of a production PineScript engine, checked bar for bar against a reference implementation with zero mismatching bars. At N=1, it incurred near-zero deflation.
It still lost. Design Sharpe was 0.33 with +34%, followed by evaluation Sharpe -0.12 and -13%. Gold buy-and-hold delivered 1.04 and +257%. Pre-registration lowers the statistical bar. The market remains unchanged.
Our backtest stops at 2024-07-01, and everything after that date is deliberately left untouched so the same strategy can be checked out of sample later.