User approval tells you nothing about whether an agent's backtest adds up. A simulated investor approved 35 of 150 sessions in FinCosmos, the benchmark built by Cao, Peng, Qian and colleagues. Every one of those 35 failed at least one programmatic check on the work delivered.
What does the agent have to deliver?
A hidden strategy starts each task. It comes from a grammar covering six components: stock selection, market timing, portfolio construction, risk control, execution and transaction cost. Before parameters are assigned, the grammar allows about 3.8 million combinations. Fresh tasks can therefore replace ones that leak. An LLM acts as the investor, taking the role of a retail investor, a generalist analyst or a biotech specialist. Its opening request gives only a vague idea; details such as moving to cash at a 20% drawdown emerge only if the agent asks or proposes them.
No data come with the task. The agent must fetch prices and filings from the Web or a Financial Modeling Prep server with 247 APIs, then write a backtest subject to a fixed data cutoff. The required output is code, a trade log and a report. The ten frozen tasks all trade US stocks, with universes ranging from small baskets to the whole market. Each session allows six turns of 300 seconds.
The grading has no reference answer. Programs verify that dated rows fall on or before the cutoff and fills follow their signals. They reconstruct returns from the submitted trades, then check the report against them. The tolerances are 0.02 for returns and drawdown and 0.5 for Sharpe. An LLM judge handles semantic grading, while an item-by-item comparison measures the delivered strategy against the hidden one. Vetoes account for 15 of the 45 dimensions. Acceptance requires both user satisfaction and a clean pass on every veto.
The trades never reproduce the reported returns
Earlier benchmarks kept part of the problem closed. In the authors' comparison, AlphaForgeBench grades executable strategies without a reference answer, while supplying the task and data. τ-banking conceals user intent within a closed database. FinCosmos opens intent, data and the timeline together. The agent has to enforce the cutoff, and a program checks whether it did. For a desk, the useful difference is narrower: the evaluator reconstructs returns from the trades instead of trusting the reported equity curve, and checks every reported metric against that reconstruction.
The paper tested four vendor coding agents able to search, call APIs and execute code: Claude Code on Claude Opus 5.5, Codex on GPT-6.1 Sol, Gemini CLI on Gemini 3.8 Flash, and Kimi Code on Kimi K3. They completed 120 sessions, covering ten tasks with three runs each. In 25 sessions, the delivered strategy matched the user's hidden strategy. Of those, 15 also respected the data cutoff, and 14 also avoided look-ahead. Zero supplied trades that reproduced the reported returns.
The first drop starts with elicitation. Agents brought up only 54 to 64% of the details withheld from them. For details never stated, their strategies mismatched 66 to 83%. Even a tool-less Claude Opus 5.5 delivered the user's strategy in 9 sessions, matching Codex, though its data score was 0.0.
Missing evidence, then bad arithmetic
The counts distinguish reports that cannot be checked from reports whose numbers fail the check. Of the 35 approved sessions, 29 omitted either a trade log or prices. Reconstruction was impossible. Those 29 supplied none of the required evidence.
The other kind is worse. Returns could be partly reconstructed in 5 sessions, all from Codex. Every reconstruction disagreed with the report, and each session scored 1 of 4. Comparing reported metrics with the agent's own equity curve gave another view: they agreed in 10 of 35 sessions, contradicted the curve in 11, and came without any curve in 12.
A user-approved Kimi Code session supplies the worked example. Its report gives maximum drawdown as -18.96% and says the metrics were recomputed from the equity curve. The curve instead shows -21.91%, beyond the user's 20% stop. The agent had calculated drawdown from a peak that reset after the stop fired. It also ranked stocks by 21-day momentum despite the user's request for short-term reversal.
The polish conceals the failures. Approved sessions earned 50.8 on pre-veto quality, the 0 to 100 rubric average before a veto zeroes the score. The other 85 agent sessions scored 21.0. User experience showed a similar gap, 69.6 against 35.0. Using only the four vetoes decided by the LLM judge would accept 17 sessions, or 11% of all 150. Gemini CLI shared the highest sign-off count at 10. Every one of those sessions failed even the LLM-judged vetoes.
How much does the clock explain?
The time limit may explain some of the zero. Agent turns reached the 300-second cap 70% of the time, and the authors acknowledge that longer turns may allow more progress. Thirty minutes is tight for sourcing whole-market data and backtesting it.
The ten-task sample also produces wide intervals. Codex earned 37.2 [30.6, 43.3] on pre-veto quality, beside Claude Code's 37.1 [27.8, 45.0]. They are indistinguishable at p=0.99. Neither Gemini CLI (22.2) nor Kimi Code (22.3) separates from the tool-less model's 20.2 (p≥0.44). Claude Code's lead over that tool-less model also falls short of significance (adjusted p=0.10).
Two strict completeness checks bear heavily on the headline: they fail in 120 and 119 of the 120 agent sessions. The authors say these checks do not distinguish the agents. Removing them would allow 2 of 150 sessions to pass, both from Codex. Those two clear every other veto, yet their partly reconstructed returns disagree with the reports. That leaves me reading the zero as near zero for substantive reasons.
Expert review supported the benchmark. Two experts, paid team members who had not built FinCosmos, assigned responsibility to the agent in 53 of 60 failed sessions (88%). They attributed 3 to the simulator and 4 to the evaluator. Across 30 calibration sessions, the LLM judge agreed with the experts at κ=0.87 on the four vetoes it decides. Agreement between the experts was 0.93. The authors flag that the simulator, judge and one evaluated agent belong to the same model family.
A desk can use the acceptance gate
The acceptance rule transfers directly to a research desk using agents. Three checks account for 33 of the 35 catches. Adding a fourth, source traceability, costs nothing:
- Require a trade log with fill prices and the underlying price series, or give no credit. Recompute every reported metric from those trades using code the agent did not write.
- Check that every dated row falls no later than the cutoff and every fill follows its signal.
- Require a specification table covering each strategy component stated by the user.
- Attach a URL or an API call with its parameters to every series.
I would tighten the Sharpe tolerance of 0.5: it permits an overstatement of half a unit and still awards a pass.
The memory experiment calls for a careful reading. With persistent memory and leak-free feedback, a Claude Code agent eventually passed all ten tasks after 506 sessions. The median was 34.5 sessions per task; one required as many as 132. Median failed vetoes dropped from 6.3 per session during the first five attempts to 2.6 during the last five. The feedback clearly teaches something. But each task was practised until acceptance.
Strategy ambiguity was the final veto cleared in 6 of 10 tasks. It failed in 77% of the 496 pre-acceptance sessions. The authors acknowledge that feedback scores can reveal information about the hidden strategy over repeated attempts, and therefore recommend frozen agents for leaderboard results. Their claim is that feedback makes the ten tasks learnable. The data support it. Transfer remains untested: we did not find a run on freshly sampled seeds, which the grammar makes cheap.
Why we could not rerun FinCosmos
We could not reproduce the benchmark. Our platform confines agents to its own data, removing the open-Web sourcing the benchmark tests. Any acceptance rate from that setup would therefore be incomparable. Our data also lack the insider buying and FDA catalyst proximity needed for the paper's worked example. A faithful reproduction requires the released tasks, evaluator and simulator paired with the same agent models.
Thirty-five reports won their reader's approval. None supplied trades that reproduced its reported returns.