The specified gate discards all but 9 to 14% of post-break signals and leaves violations near 20%. That is a weak trading control. Talukdar, Shankar and Prabhu acknowledge the discrepancy, yet their abstract still says adaptive gates accept 25 to 35% after the break and keep errors among accepted signals near 10%. Gating also reduces risk-adjusted return in their design. Their defence depends on that 10% error rate, which the stated specification fails to reproduce.

The measurement beneath the gate holds up. Before the break, per-field 90% intervals cover the entire extracted eight-factor vector just 47.9% of the time; after it, coverage is 42.9%. Those are the simulation's rolling-calibration results.

Upstream of the trade

The authors place an extraction layer ahead of any strategy: a model reads text and produces a factor vector. A reviewer audits a 10% sample against source documents, with labels arriving two quarters later. Conformal calibration divides each audited error by a fitted uncertainty estimate and takes an empirical quantile of the resulting scores. Multiplying that quantile by the fitted scale gives the interval's half-width.

The gate operates on the weighted composite, the number the strategy trades. It accepts a signal if the composite's half-width is below a tolerance τ, fixed at go-live at the 75th percentile of half-widths then observed.

No LLM runs in the main study. The authors simulate an extractor over 300 firms, 60 quarters and 8 dimensions, using 200 seeded replications. Errors follow Student-t with 5 d.f.; hallucinations account for 2% of entries. Stress regimes multiply the error scale by 1.6. At quarter 36, a model update multiplies error by 1.8 and introduces a 0.4σ bias, both on dimensions 1 to 4. The comparison covers four calibration schemes:

The output feeds a top/bottom 20% rank long-short and an operator that sizes positions inversely to interval width. Simulated returns give the true composite an IC of 0.10. The authors explicitly disclaim "real-world alpha, coverage or economic benefit."

Coverage survives. Slow breaks escape the alarm.

The max-score rectangle takes a single quantile of the vector's worst normalized error. Joint coverage reaches 89.8% before the break and 88.2% after it. Its pre-break half-width is 2.22, versus Bonferroni's 2.99, making it 26% narrower. After the break the comparison is 2.80 vs 3.49, or 20% narrower. Even so, the half-width exceeds two factor SDs. The authors describe the rectangle as "better read as an audit statement than as an informative forecast."

I agree.

Frozen calibration deteriorates to 79.9% coverage after the break. Coverage averages 69.2% in its worst four-quarter window and falls to 63.4% in post-break stress quarters. Delayed ACI averages 89.8%. Every scheme falls within 79 to 83% in the break quarter and the next: post-break labels have yet to arrive. Extending the lag to four quarters takes ACI's worst window down to 75.2%.

The label-free monitor applies a KS test to quarter-on-quarter changes in extracted factors. It raises an alarm in the break quarter in 83.5% of runs; audit-based monitoring does so in 2.5%. A gradual shift is much harder to catch. Spread the same change across twelve quarters, and the alarm fires at quarter 36 in only 3.5% of runs. The abrupt-break figure in that same supplementary code is 78.5%.

Can the specified gate deliver a 10% post-break error rate?

The authors say it cannot. Their table reports 25.4% post-break acceptance for delayed ACI and a 10.1% violation rate among accepted signals. The ungated violation rate is 23.1%; the frozen gate's is 16.4%. Following the written gate specification instead produces 9 to 14% acceptance with roughly 20% violations.

On that implementation, about nine signals in ten are discarded while the error rate remains close to the ungated level. The risk-control argument rests on 10%, and the specified gate produces about 20%. The authors pre-specified a τ sweep, then withheld it for the same reason.

They disclose the problem honestly. The abstract nevertheless repeats the claim that accepted-signal errors stay near 10% after the break, without the caveat. Duplicated draft sentences remain there too, giving accuracy figures of 94.9% and 95.5%, alongside detection figures of 83.5% and 78.5%.

The price of keeping 20%: a ratio of 0.66 versus 1.42

The gating-cost experiment differs from the τ gate. Each quarter it retains a fixed share of signals, choosing those with the narrowest intervals. Operator A's ratio falls from 1.42 at full acceptance to 1.13 at 60%, then to 0.66 at 20%. The ratio is measured per quarter in idiosyncratic-SD units. Costs, turnover and capacity limits are absent, as the authors acknowledge in their limitations. With IC fixed at 0.10 and interval-width ranking supplying little return information in this design, lost breadth is the obvious suspect. The authors identify breadth as the likely mechanism but say they did not test it.

The error-linked design sets a harder test: an overstated composite loses money. Without an error penalty, the supplementary code shows the ratio falling from 1.44 to 1.13 when acceptance is gated to 60%. Even at the heaviest penalty (κ = 2k), that gate reduces the ratio from 0.79 to 0.74. Making errors more expensive shrinks the gating cost from 0.31 to 0.05. The cost remains.

At full acceptance, raw scale sizing earns a ratio of 1.49, ahead of calibrated-width sizing at 1.34 for rolling width and 1.25 for ACI width. Width sizing also silently reduces gross exposure by 32 to 41% after the break. The authors cite Singh and Rezaeimanesh's gain from matched sizing on real FactSet news. They regard the results as compatible because the designs differ. Our Conformal Kelly review examines a related application of conformal widths to sizing.

A pilot with 492 companies and one reporting period each

For the pilot, gpt-oss-120b reads 8-K press releases from 492 companies between January 2024 and March 2025. SEC XBRL values provide the scoring reference. The model extracts 1,956 of 2,113 reference figures; 95.5% of the extracted figures are within 1%, with a company bootstrap of 94.4% to 96.5%. Fixing a revenue tag reduces large errors from 81 to 60. None of the 22 errors removed by that fix received a hand check.

The remaining 60 also carry uncertainty. Forty-five use fourth-quarter references calculated as annual minus nine-month values, and none were adjudicated. The models agree with each other in 24 of the 45 fourth-quarter cases, a pattern consistent with a bad reference rather than a misread. Disagreement across Llama 3.3 70B and gpt-oss-20b flags 27 of the 60 errors, along with 8.6% of correct figures. Agreement on a wrong label would leave those cases invisible to the detector. If those agreed cases are label errors, its hit rate on true misreads may therefore exceed the reported 45%.

Temperature-zero decoding also proves unstable. A rerun moves 53 figures by more than 5%, a count comparable to the entire large-error tally. Across all 1,956 figures, the model's GAAP flag never changes. A self-consistency proxy showed no signal and was dropped during development; verbalized confidence was never collected. The authors acknowledge that richer self-reported signals remain untested. Their 10,000-run Monte Carlo produces 3.1% to 1.9% large errors at 90.3% acceptance. It is fitted to the pilot, and the authors say it supplies no real evidence beyond that pilot.

We could not test any of this ourselves. Reviewer-audited extraction labels are missing. As-reported statements serve as proxies, requiring reconciliation of GAAP, period and units. Our usable text history begins only in 2020, far short of 60 quarters. Running a modern extractor over that history also risks recall of figures seen in pretraining, so the exercise would not provide prospective evidence.

The gate has never been run through time on real filings, and no version has incurred a cost. The authors' own replication protocol already calls for that run and requires operator results to be reported only after transaction costs. I would change my view on a multi-quarter real run, net of costs, where the specified gate keeps violations among accepted signals near 10% while accepting more than 9 to 14% of signals.