A trader should read FinanceHarness as a retrieval improvement. A 27B open-weight model equipped with one search tool scores 25.3% on FinanceGym, rising to 32.4% inside the authors' harness. Most of that 7.1-point gain comes before the cutoff. The pre-cutoff result moves 9.6 points, from 36.1 to 45.7, while the post-cutoff result gains 3.1, from 8.7 to 11.8. Better evidence gathering and citation discipline explain the improvement far more than forecasting does.

We did not reproduce these results, and we could not. Replication would require the authors' agent harness, their point-in-time retrieval sandbox, a corpus of 100+ million dated articles, the entity-relation extraction pipeline, the benchmark rubrics, and LLM infrastructure at their scale. The corpus has not been released. Private rubrics on the authors' leaderboard handle the grading. An in-house RAG or LLM sentiment workflow over news and transcripts could borrow the idea, but the reported FinanceHarness figures would remain unavailable. Every assessment here comes from the paper's own tables.

The paper uses financial deep research to mean an agent that searches and reads before producing a cited, analyst-style long-form report. It must connect companies and sectors to events and dates, then decide what may happen next. The authors illustrate the task with an equity analyst covering a semiconductor name: review past earnings reports, assess current competitor dynamics, forecast demand and macro conditions, and finish with a rating. The second half, the view, is what a paying user wants.

FinanceHarness runs this process. Its serving layer contains the orchestrator backbone and a lightweight reader that extracts evidence from long documents. Beneath them, a runtime control plane contributes no model intelligence. It manages the bounded agent loop, schema validation, tool-result chaining, recovery and run-level budget limits.

Tools arrive in tiers. Point-in-time search, source reading, citation composition and report finalization remain loaded throughout the core loop. Deferred tools appear through a compact catalog, with complete schemas loaded after the model chooses a tool family. Prompt modes sit above that layer (research, analytical, automatic), alongside workflow skills that declare their expected tools and specialist context. The harness builds numbered citations from sources the agent visited. A grounding-review pass asks the backbone to qualify or attribute claims that lack support from a read source or tool output.

The score has two parts. Pre-cutoff rubric items reward facts that could have been found before the question's cutoff date. Post-cutoff items assess whether the report anticipated developments confirmed only afterward. A dated environment exists largely to make that second tier possible.

Every system struggles there.

Xiao and co-authors put FinanceGym on a 400-question dated sandbox

The paper offers three contributions: the point-in-time sandbox, the FinanceGym benchmark and FinanceHarness.

The sandbox comes first. The authors assembled a 2025 English-language web corpus containing more than 100 million articles from thousands of public domains. htmldate extracts publication dates, while trafilatura produces clean text. Qwen3-Embedding-4B supplies the embeddings, which are served through a FAISS IVF-SQ8 index with cutoff-date access control. For a question dated April 2025, the agent can retrieve only documents published by that date. Detected leakage voids the run.

FinanceGym rests on this corpus. Using Gemini-3.5-Flash, the authors extracted an entity-relation graph with 5.74M raw edges, later filtered to 4.37M. It covers 1.11M entities drawn from 1.20M source articles. They mined the graph for "situations" in three modes. These included cross-category linkages and temporal narrative arcs, plus polar divergences such as upgrades versus downgrades. A z-scored objective over event volume, entity diversity and relation entropy chose cutoff dates. Eventful days receive higher scores; single-entity batch artifacts incur a penalty. From each (situation, cutoff-date) pair, an LLM writes an analyst-style question, a reference investment thesis and the two-tier rubric.

Selection removes most candidates. The process began with 29,669 unconstrained generations and retained a 2,078-question pool. An integer linear program chose 500, balancing topic and sector as well as reasoning type and monthly cutoff. Those 500 went to an external professional annotation vendor. The curation bar admitted 411, an 82% pass rate, after roughly 1.2 hours of expert work per sample. The released benchmark contains 400 questions and 2,464 rubric items, with a mean of 6.16 per question across 12 monthly cutoffs from December 2024 to November 2025.

Gemini-3.5-Flash serves as the LLM judge, assigning each criterion a score from 0 to 4. Questions receive equal weight regardless of rubric length. Seventeen baselines were each run once, along with the harness. Claude-Opus-4.7 ranks first at 34.1% (50.1 pre, 9.9 post). Gemini-3.1-Pro follows at 33.2%, ahead of FinanceHarness on Qwen3.6-27B at 32.4% and TTD-DR at 31.5%. Nobody exceeds 40%, leaving what the abstract describes as substantial headroom.

Why the sandbox deserves copying

Filtering retrieval separately for each question date is sound design. In the paper's comparison table, FinanceGym alone is marked both reproducible (fixed corpus) and PIT (per-question publication-date cutoff). DeepScholar-Bench carries the PIT label without being marked reproducible. BrowseComp-Plus and DeepResearchGym are reproducible, though they lack per-question cutoffs.

The setup yields another useful result: fine-tuned deep-research models underperform. Tongyi-DR scores 28.2%, OpenResearcher 27.2% and MiroThinker 21.2%, each trailing plain GLM-5 with search at 30.4%. The Limitations section gives the necessary caveat. These specialist models face an out-of-distribution test, using a corpus-backed retriever instead of the live-web stacks on which they trained. Appendix C.1 explains the likely mechanism: the scores measure transfer, and "their learned reporting interface is mismatched with a long-form, citation-sensitive financial rubric." Buyers should remember that result before training an agent against Serper and Jina, then connecting it to an internal document store.

The harness ablation also holds up. With the same backbone and retriever across four configurations, the naive harness captures 4.3 of the 7.1 points. The complete harness contributes another 2.8. GRPO training on 172 machine-curated instances adds 0.4, below the 0.8 bootstrap standard error. The authors appropriately call it "a refinement rather than a headline result."

Does a rubric measure foresight?

The post-cutoff tier supports the paper's main claim: forward-looking judgment is the hardest part of this task. The authors acknowledge the gap directly. Their conclusion summarizes 17 baselines and FinanceHarness: "every system stays below 40% overall and shows a persistent pre-/post-cutoff gap, suggesting that financial deep research is not addressed by stronger retrieval alone." Appendix C.2 states the disparity more sharply. Training leaves post-cutoff performance at about 12%, compared with 46% pre-cutoff.

The paper treats this gap as a selling point for the benchmark. Because all agents use the same dated corpus, pre-cutoff scores several times higher than post-cutoff scores are presented as evidence of core difficulty rather than a retrieval artifact. The numbers leave that interpretation open to challenge. In Table 2, Gemini-3.1-Pro has the highest post-cutoff score at 12.8%, while gpt-oss-120b records the lowest at 5.2%. All 18 rows fit within a 7.6-point range. The table's note places bootstrap standard errors on the Overall score, where they span 0.6 to 1.1. Each agent received one run over 400 annotated questions.

Difficulty alone offers little value when a tier compresses 18 systems into 7.6 points.

Resolution is only part of the concern. Judge anchors are defined as 0 (not addressed), 1 (mentioned), 2 (partial), 3 (substantive) and 4 (fully grounded). Probability and confidence appear in none of them. Comparing a thesis with subsequent events evaluates one realized draw from a distribution. A hedged forecast and an unqualified assertion are therefore judged through grounding and specificity, leaving calibration unseen. An LLM generates the criteria together with the question and reference thesis, then the judge scores them while holding both pre-cutoff and post-cutoff edge evidence. Experts filtered the records and assigned feasibility a mean of 3.89 and clarity a mean of 3.97. The experts did not write the records.

The temporal contract has another boundary. As described, point-in-time enforcement applies to the search server, where each result needs a publication date no later than the cutoff. The protocol does not state when backbone pretraining ended relative to cutoffs running from December 2024 to November 2025. Post-cutoff results between 5% to 13% suggest that parametric memory offers little help. The leakage channel and the metric's ability to distinguish systems both appear small.

Judge, backbone and authorship

Gemini-3.5-Flash is the judge. The paper distinguishes it from the Gemini-3-Flash backbone used in five agentic scaffolds, although both belong to the same family. Gemini-3.1-Pro also places second overall at 33.2%.

The GRPO reward blends two signals: 0.6 comes from rubric-style scoring against generated criteria, and 0.4 comes from an LLM judge assessing report coherence, grounding and trajectory quality. Evaluation and optimization consequently depend on similar signals, despite keeping the 172 training instances separate from FinanceGym.

TTD-DR, the strongest agentic scaffold at 31.5%, is attributed to Han et al. Han is also a co-author of this paper. FinanceHarness leads it by 0.9 points. Their standard errors are 0.8 and 0.7, based on one run each.

FinanceHarness itself combines tool tiers, prompt modes, workflow skills, citation passes and grounding-review passes. The paper emphasizes its shared contract with the evaluation environment, allowing a trained model to encounter the same tool distribution at test time. We found no held-out set used to select those design choices.

Nothing here prices a trade

The reported results say nothing about forecast calibration, recommendation accuracy versus prices, or any connection between a higher rubric score and later returns. The paper also supplies no per-report cost or latency figure. Its nearest evidence is the guardrail ablation. Turning off URL pre-fetch raises the visit error rate from 2.1% to 39.4%. The final score changes little because the backbone repairs its mistakes at substantially greater trajectory cost. Search-budget results tell a similar story: harder questions attract heavier searching, so caps constrain spending.

The abstract does not claim any of the missing outcomes. Appendix case studies also carry the authors' warning that "the generated financial claims have not been independently verified."

Return to the fixed-backbone Qwen3.6-27B ablation in Table 4. FinanceHarness improves the pre-cutoff tier by 9.6 points, from 36.1 to 45.7. Its post-cutoff gain is 3.1, from 8.7 to 11.8. Overall, it adds 7.1 points. The best post-cutoff result anywhere in the table is 12.8%.

A small, inexpensive test could change my view. Use the 400 questions to extract each report's directional call and confidence, then compare both with realized prices across the horizon already implied by the rubric. Outcomes are available because the cutoffs extend through November 2025. Until that test exists, the post-cutoff tier remains a judge's hindsight assessment, while 32.4% measures report craft.