An LLM agent declaring that your timing model beats the historical mean is making a significance call. Its false-positive rate depends on the model you rented. Across the linear null worlds in the first two studies, that rate ranges from 15% to 62%. Remove the simulation artifact and use Gaussian no-signal worlds, and it ranges from 16% (Sol) to 44% (Terra). Nobody chose those thresholds.
Liu simulates monthly returns with a planted predictive regression. The true slope sets population R² at 0, 0.25, 0.5, 1, 2 or 4% a month. Each agent receives eight anonymized candidates in shuffled order: the planted predictor alongside seven AR(1) distractors. It can examine 700 training months; another 360 test months remain outside its sandbox. Working in Python under a 40-call budget, it submits a verdict ("beats", "does not beat" or abort), a probability and a model file. The evaluator refits that model every 12 months over the sealed window. Planting the truth lets Liu score the verdicts for size, power and calibration.
The experiment spans four pre-specified studies. Claude Opus 5.5 receives 900 runs, while GPT-6.1 Sol and Terra share 1,800. All three models then return for 1,290 runs using Gaussian shocks; Qwen3-32B receives 430 local runs. An oracle knows the true model. The other pre-specified benchmarks use a one-sided Clark-West test at 5% to decide whether to claim predictability. After seeing the first results, Liu added in-sample tests. The most useful is HAC-Holm, which applies Newey-West t-tests with 12 lags to the nine candidates and adjusts for multiple comparisons using Holm.
More discoveries, or a lower bar?
The headline result favors the agent. Across 240 paired weak-signal worlds, Opus claims predictability in 38%, compared with 8% for the primary combination baseline. Every one of the 72 discordant pairs favors Opus. Liu's explanation takes the shine off: agents mostly find more by accepting weaker evidence. The baseline also errs deliberately toward caution. It evaluates pseudo-out-of-sample forecasts over only the second half of the training window. Even the oracle makes claims in just 2.5% of null worlds under that rule.
A fair comparison gives each procedure the agent's false-positive rate. For Opus in linear nulls, that rate is 15%: 6 of 40 worlds, with an interval of 7 to 29%. Liu labels the estimate approximate. At R² = 1%, Opus detects 61%, beside 59% for size-matched HAC-Holm. The 2-point advantage has a bootstrap interval of -16 to 23. The oracle reaches 79%. With twice as many null worlds anchoring the Gaussian rerun, the result is similar. Matched HAC-Holm detects 83% and Opus 74% (difference -10, interval -21 to +2).
Terra trails badly in the bootstrap worlds. At its 62% false-positive rate, HAC-Holm detects 92% of R² = 1% signals; Terra detects 66%. Its gap is -25 points, with an interval of -41 to -14. In the artifact-free Gaussian worlds, Terra's gap narrows to 3 points (-13 to +7). Sol detects 46%, compared with 59% for the matched test. That 13-point gap has an interval (-30 to 9) spanning zero. The best agent's search delivers roughly what a Holm-corrected regression delivers with a looser trigger.
The return leak GPT-6.1 Terra found
The first two studies drew return shocks by bootstrapping a pool of 758 HML residuals. In no-signal worlds, a median 46% of monthly returns exactly repeat an earlier value. The next shock repeats as well within a bootstrap block. Terra spotted the pattern. In 65 of 900 runs, it built lookup forecasters whose test-window R² reached 78%, although the strongest planted signal was 4%. The exploit accounted for half of its 44 no-signal claims. Those claims were economically correct: the deployed model did beat the benchmark. The sealed-holdout Clark-West test accepted it too, claiming in 34% of Terra's no-signal worlds. Structure shared by training and test windows survives a sealed holdout.
Liu discovered the artifact during external review, after completing the results. The Gaussian study repairs it, and the findings on size persist. Across 110 no-signal worlds, false-claim rates are 34% for Opus, 16% for Sol and 44% for Terra. Nominal HAC-Holm claims in 8% of the same worlds. Discordant counts are 28:0, 10:1 and 40:1. Opus's rate rises from 21% to 34%, though the worlds differ and the comparison is unpaired. An exploratory check finds more chance-significant candidates in its Gaussian worlds: 41 of 80, versus 12 of 40 bootstrap worlds and about 39% expected. Liu also states the cost of the repair: Gaussian shocks remove fat tails and volatility clustering.
A 50% posterior where you assumed a 5% test
The brief specifies no significance level. Liu acknowledges that evaluating the agent at 5% therefore applies a standard the agent never received. The author's response is that the implicit threshold is precisely what the experiment measures. The results support that response. In 3,990 of 3,990 runs, the three commercial models claim exactly when their stated probability reaches at least 0.5. The rule is entirely consistent. It is also entirely unsuited to a field where most published predictors fail out of sample.
The probabilities offer little beyond the verdict itself. Opus has a calibration slope of 0.36 and a Brier score of 0.309, worse than the constant forecast's 0.219. Sol's slope is 0.12; Terra's is 0.11. Raising Opus's threshold to 0.6 in the Gaussian worlds produces 5% false claims and 50% detection at R² = 1%. HAC-Holm detects 54% there. The threshold was selected on those same runs.
Changing the brief changes the decision. A single sentence about prior research moves Opus's claim rate from 32% (skeptical) through 46% (neutral) to 61% (optimistic). Optimism raises Sol's linear-null false claims from 15% to 57%. Terra reverses its verdict across framings in 100 of 160 worlds. Identical reruns also disagree, with probability 12%, 15% and 20% for the three models.
Opus mentions multiple testing or data snooping in 14 of its 15 false claims, then makes the claims anyway.
What a sealed 30-year holdout costs
Liu closes the abstract with a prescription: agent verdicts carry information, but require a neutral brief, a decision rule fixed outside the agent and a sealed holdout. The agent builds the model. The decision rule and holdout remain beyond its reach. Liu cites Qu, Chen and Wang's fixed anytime-valid referee (/articles/propose-don-t-judge-an-anytime-valid-referee-for-llm-agents-that-mine-investment) as one such rule.
The price is measurable here. A Clark-West test over the sealed 360 months keeps false claims at or below 5% for every model in the Gaussian worlds. Among the three commercial models, it detects only 25 to 28% of R² = 1% worlds. Nominal HAC-Holm detects 54%. Thirty years of untouched monthly observations exceeds what most desks have available. Liu's proposed order makes sense: use an efficient in-sample test for the decision rule, then the holdout for confirmation.
The score measures MSE against the historical mean. The paper lists the pre-specified Sharpe, deflated Sharpe and backtest-overfitting analyses among analyses not reported. These results therefore say nothing about turnover or net returns. On real data, Opus returns "beats" in 20 of 20 HML and RMW runs. In 19 of them it uses the factor's own lag, with three post-cutoff months available for checking. Opus also recalls 28 of 120 pre-cutoff HML months at a Spearman correlation of 0.99.
We could not reproduce any of these results. We lack access to the exact model versions, and our data do not include the French HML and RMW series. We are building a SPY timing adaptation using planted-signal Gaussian worlds. This is a new experiment on a different asset; it will not confirm Liu's factor results.
An agent beating size-matched HAC-Holm with an interval clear of zero would change my view. None of the three managed that under the paper's preferred simulation-based matching. Calibration on the agent's own 40 null worlds favors Opus, though the paper describes that estimate as imprecise.