Point-in-time search and leakage filtering shrink backtest-to-live gap: 0.1203 to 0.0285

Point-in-time search and an LLM leakage filter reduced the mean absolute backtest-to-live gap from 0.1203 to 0.0285 across nine U.S. model-task pairs. That result should matter more to buyers and builders of AI research tools than any leaderboard position here. A closer match means less cheating in the backtest. The authors claim as much in their abstract. Our objection: the gap remains positive in every pair, and the test extends only to those nine U.S. pairs. Whether any of these forecasts deserve trading capital remains open on the paper's own metrics.

Testing around each model's cutoff

Liu, Li and their coauthors tackle a familiar backtesting problem: a language model can earn credit for remembering an outcome already in its training data. They construct test instances around each model's declared knowledge cutoff and divide them into three regimes. T1 contains forecast dates before the cutoff, when recall can pass for forecasting skill. T2 contains dates after the cutoff with outcomes already resolved, allowing immediate scoring. In T3, forecasts are recorded live and scored later.

The suite contains 1,119 records covering Chinese, U.S. and Hong Kong equities. Its Overall score combines four forecasting tasks: price targets (VAL, 244 records), peer return ranking (RANK, 383), direction following earnings or macro events (EVT, 138) and undisclosed fundamentals (FUND, 294). Counterfactual reasoning (CF) and multi-step calculation (CALC) are separate diagnostics, with 30 items each. T2 scores cover twenty-two model-harness pairings.

Within FinanceHarness, the paper tests three retrieval policies on three models. Unrestricted search comes first. Point-in-time (PIT) control then rejects pages whose source timestamps fall after the forecast date. The final policy adds a DeepSeek-V4-Flash classifier to reject pages containing later information or ambiguous dates.

The advantage from the pretraining period appears consistently: T1 outperforms T2 in 25 of 27 comparisons across RANK, EVT and FUND. On Chinese FUND, Claude Sonnet 4.5 earns 0.9997 before its cutoff and 0.5655 afterward. Gemini 3.1 Flash-Lite achieves the highest T2 Overall score, 0.7099, through the general-purpose GPT-Researcher harness. Every FinanceHarness pairing trails it.

What supports the 0.0285 figure?

Nine comparisons.

That average covers three models and three tasks, all within FinanceHarness, during matched one-week U.S. windows. The wider benchmark covers three markets and 22 configurations; the reported T3 results cover only the U.S. We did not find counts of instances in the live windows or confidence intervals for any gap.

The direction of the gap matters more than its size. In all nine pairs, filtered T2 scores exceed T3 scores, with differences ranging from 0.0037 to 0.0864. The 0.0285 average therefore measures an optimistic bias, with no pair crossing zero. Filtering also makes some comparisons worse. For RANK, the gap widens from 0.0748 under PIT alone to 0.0864 for Gemini, and from 0.0537 to 0.0689 for Claude. The authors report these results and describe RANK agreement as "more task-dependent".

They acknowledge that T2 is "a practical retrospective proxy, not a substitute for T3 live evaluation." Their response is that stricter controls let backtests "closely approximate live forecasting scores in the evaluated tasks". The last four words set the limit of that claim.

Weak rankings, striking event scores

RANK averages normalized Spearman and a top-third hit rate. Random ordering scores about 0.5 on the first component and 1/3 on the second, or roughly 0.42 overall. The scores for all 22 configurations fall between 0.4428 and 0.5023. Although the paper lists long-short spread as an auxiliary measure, we did not find spread figures in its results tables. These results establish no stock-selection alpha.

EVT uses a Brier skill score under which a constant 0.5 forecast earns zero. The live scores, 0.5805 and 0.5926, imply Brier near 0.10. In the paper's own EVT example, the prediction covers a single open-to-close session. Such a low Brier on equity direction would be extraordinary. We found no market-based baseline against which to assess it.

FUND raises the reverse concern. Copying Vanke's disclosed Q2 revenue into a Q4 forecast earns about 0.9345 utility in the worked example. Without a persistence baseline, which we did not find, the live FUND scores of 0.7194 to 0.8563 are difficult to interpret.

FinForesight leaves several judgments unresolved

T1 runs before 2024-06-01, whereas T2 runs after 2025-07-31, confounding memory with market regime. VAL also runs against the broader pattern. Of its 9 T1-minus-T2 deltas, 7 are negative, including -0.1948 for Claude in Hong Kong. Citing noise in short-horizon prices, the authors exclude VAL from the focused analysis and report it in an appendix.

The filter itself needs an audit. DeepSeek-V4-Flash serves as both classifier and contestant. The appendix defines precision, recall and F1 for page-level leak detection, but we did not find the resulting values. It also rejects pages with ambiguous dates. Claude's EVT score falls from 0.8232 to 0.6006, against a live score of 0.5926. An implementer cannot determine how much of that fall reflects leaks removed and how much reflects evidence removed.

To support the claim that models "understand the forecasting tasks", the paper relies on live results, small filtered T2-T3 gaps and aggregate score stability. The gaps have already been addressed. Stability provides the weakest support.

On T2 EVT and FUND, prompt rewrites materially alter 16.83% to 23.67% of predictions for GPT-4.1-mini and Claude Sonnet 4.5 across two harnesses. Numerical tweaks alter 7.00% to 9.17%. The mean signed score change remains below 0.0026 in magnitude. But offsetting changes at the instance level can disappear in a signed mean, and the paper gives no absolute score change. Under paraphrase, GPT-4.1-mini in GPT-Researcher materially changes 35% of its Hong Kong FUND answers, compared with 10% under numerical tweaks.

Reproduction was out of reach

We could not reproduce this. Exact comparisons between models and harnesses require the evaluated model versions, their declared knowledge cutoffs and the retrieval configurations. We could not establish that these were available to us. Choosing an LLM ourselves and limiting retrieval to our records would produce a different system. Such a restriction would also leave untouched any outcomes memorized during pretraining. A fair substitute would require documented cutoffs and forecasts recorded prospectively.

FinForesight should be treated as an audit protocol for AI backtests. A live run in China and Hong Kong, with filtered T2 gaps whose intervals straddle zero, would change that verdict.