Our review of the paperwe backtested it

Market returns replace human labels in this sentiment model

A replication note on a reward trained on same-day moves the backtest never touches

2026-08-05 · 4 min read · US equities

Read the paper on arxiv

Our backtest of this idea

Our automated quick test, not the paper's

FinSMART-Style Market-Aligned News Sentiment Long-Short US Stocks

Backtest period 2020-01-01 to 2025-10-08 · hypothetical, net of modelled costs

Why these figures are not the paper's (2)

This is not a replication of the paper (2)

A full reproduction of the Llama-3-8B GRPO fine-tuning pipeline may not be native to the strategy backtesting environment; the practical backtest would likely use available text, embeddings, and a lighter supervised or reinforcement-style classifier unless external LLM training infrastructure is provided.
The platform has news and transcripts primarily from around 2020 onward, so rolling retraining and out-of-sample evaluation are limited to the available post-2020 text history.

The figures below measure what we could run, not the paper's own method, so they are not evidence for or against its claim.

Our own audit found this run does not follow the paper faithfully (3)

Volatility filter excludes high-volatility names (per-day realized vol > 0.1): An invented volatility gate systematically removes the highest-volatility stocks; the paper applied no such filter and instead intentionally evaluated across volatile regimes. (invalidates: The predicted property that the evaluation window 'including the COVID-19 market crash' assesses FinSMART robustness under highly volatile conditions no longer holds, because the high-volatility names that constitute that stress test are filtered out of the traded universe.)
Transaction cost stress test (0-5 bps, rt = Rt − k*Turnovert): Cost model and bps grid are imported from the related FinDPO paper, not FinSMART; FinSMART's own main portfolio deducts no transaction cost. (invalidates: Any FinSMART headline return figure (Table 1/Table 2 cumulative, annualized, Sharpe, Sortino, Calmar) interpreted as net-of-cost; the paper's figures are gross of transaction costs.)
The continuous sentiment score is the paper's defining construct for ranking (FinDPO's logit-to-score converter, cited in the paper); the spec permits a fallback substitute probability_margin_score = P(Positive)−P(Negative) as an 'implementation_note' operational deviation, but this is not recorded in the fidelity items with an `invalidates` — if the fallback is used the RankIC and return-level comparisons to Table 1/2 no longer describe the same signal.

These are our findings about our own implementation, not criticisms of the paper. Read the figures below as a description of what we ran.

Jan 2020Total -48.2%Oct 2025

Sharpe: -0.46
Total Return: -48.2%
Max Drawdown: -75.2%
CAGR: -10.8%
Volatility: 18.1%
Trades: 90,506

What the paper reports for its own strategy

Cumulative return 264.9%, annualized 91.5%, Sharpe 1.97, Sortino 2.40, Calmar 4.23, RankIC +0.061 (static FinSMART, out-of-sample Jan 2019-June 2021, next-day open-to-open returns, costs not stated)
Retrained FinSMART: cumulative 406.2%, annualized 125.7%, Sharpe 2.41, Sortino 2.96, Calmar 5.65, RankIC +0.065 (out-of-sample Jan 2019-June 2021, next-day returns)

The interesting move in FinSMART is who does the labeling. Instead of paying humans to tag financial news as positive or negative, the authors let the realized stock reaction on publication day supply the label, and post-train Llama-3-8B-Instruct against that label with GRPO. Everything downstream, the 264.9% cumulative return, the 1.97 Sharpe, follows from that one substitution. So the question an implementer has to answer is whether returns are a better annotator than people, or just a cheaper one that happens to overfit the day the news prints.

Before any of our own figures, the disclosure you are owed. We could not run the paper's actual method. A full GRPO fine-tune of an 8B model (13.6M LoRA parameters, group size 8, KL beta 0.1, eight hours on a single A6000) is not native to our backtesting environment, so we did not reproduce the RL-trained model at all. We ran a lighter sentiment classifier over available text, and the traded signal in our run may be the fallback P(Positive) minus P(Negative) rather than the market-aligned logit-to-score converter the paper credits for its edge. Our text also comes from a different corpus (news, press releases and earnings transcripts) rather than The Motley Fool and MarketWatch, and our history starts around 2020, so our window barely overlaps the paper's.

What our run did, and it did badly

Our backtest, 2020-01 to 2025-10, top-500 US names, 35% long / 35% short, one-day open-to-open holds, lost 48.21% cumulatively at a Sharpe of -0.46 across 90,506 trades, net of four tenths of a cent a share in commissions. A sign flip against the paper's reported +264.9% and Sharpe 1.97 over Jan 2019 to June 2021 (their figures, gross of costs, from the GRPO model we did not build). Read our number as evidence about our setup. We did not test their claim. Their contribution is that the RL-aligned signal beats the preference-trained FinDPO baseline (264.9% vs 109.8% cumulative, Sharpe 1.97 vs 1.12). We ran neither of those models, so we measured a third thing.

The gap is only partially explained by what we can see. Four differences push the right way. Our traded signal is likely not the market-aligned one, which strips out the exact property (alignment with next-day alpha) the paper says drives returns. Our window is almost entirely out-of-sample for a 2018-trained model with no retraining, so signal decay and regime drift work against us. Our corpus and our universe filters differ, and one of ours, a gate excluding names with 20-day realized vol above 10% a day, removes precisely the high-volatility COVID names the paper leaned on. Costs on a 90,506-trade book drag too. None of that turns a large positive gross return into a 48% loss on its own, so I will not pretend we have a tidy story.

Is same-day alignment a feature or a leak?

This is the part worth arguing about, and the paper is admirably explicit. The reward is computed on publication-day returns, because that is where the signal lives: on The Motley Fool the average publication-day alpha differs by about 5.0% between positive and negative articles, and the sentiment-alpha correlation is 0.41. Shift to next-day returns and the spread collapses to 0.3% and the correlation to 0.03. So the training reward is built on information not available before the article prints. The authors argue this is fine because they only want a clean supervisory signal, and all trading results use next-day returns. Remark 2 states it plainly.

That defense holds only if a model taught to recognize event-day reactions produces a signal that still ranks next-day returns. The RankIC is where you check, and it is thin: +0.061 for FinSMART against +0.053 for FinDPO, a 15% edge. A daily-rebalanced long-short book earning a 1.97 Sharpe off a 0.061 rank IC is doing a lot of work with a weak cross-sectional signal, and I found no transaction-cost accounting anywhere in the paper to offset the turnover that implies. We also did not find standard errors or t-stats on any of the Sharpe or return figures, and the out-of-sample window is a single 2.5-year stretch dominated by the COVID crash and recovery.

The retraining result is the commercial claim

The static model is the academic point. The retraining is the pitch. Because market reactions relabel the data for free, the model can be refreshed every six months without new human annotation, and doing so lifts cumulative return from 264.9% to 406.2% and Sharpe from 1.97 to 2.41. The mechanism is genuinely attractive: your labels regenerate themselves as prices print. But the evidence for it is four retraining iterations, and the headline correlation between fresh articles and performance gain, r=0.72, rests on those four points. Suggestive, not settled.

What would change my mind is a version of the RankIC test run out-of-sample past 2021 on the retrained model, with costs charged and a standard error attached. The relabeling idea is worth taking seriously precisely because it is cheap and repeatable. Our own weak run says nothing against it; it says we traded the wrong signal on the wrong window. Someone with the GRPO pipeline and a longer post-2021 tape should be the one to settle whether same-day supervision buys next-day money.