Reading the probabilities behind a language model's sentiment label is a cheap fix worth adopting. The trading evidence here gives less support to the abstract's word "significantly".
Inside Logit Extraction
News-trading prompts commonly ask a chat model to classify a headline as positive, neutral or negative, then assign the answer 1, 0 or -1. Fabozzi and Florescu use information already available in the model's last layer: the probability of every token it could produce next. Their Logit Extraction (LE) method takes the probabilities of the three sentiment tokens and renormalises them to sum to one. The headline score is P(positive) minus P(negative). They use the Lopez-Lira and Tang prompt, which ends in "Answer:" to steer the next token toward a label.
A Financial Phrasebank sentence makes the case neatly. All four chat models classify it as "Positive". Yet GPT-4 assigns positive a probability of 50.03% and neutral 49.97% under LE, almost a coin flip. Llama assigns positive 94.09%. Discrete labels give both readings +1; sorting stocks by LE would put Llama's reading near the top and GPT-4's near the middle.
The sample contains 216,837 single-company news alerts from Refinitiv, covering 263 companies from January 2010 to October 2020. Only the headline and its first release are used. Scores are averaged by stock-day. Headlines arriving from 30 minutes before day t's open through 30 minutes before day t+1's open generate trades at the t+1 open, closed at the t+2 open. Each day's tradable cross-section consists of the stocks in the news, about 46 names. The models are FinBERT, Mistral-7B, Llama-2-7B, and the November 2023 gpt-3.5-turbo-1106 and gpt-4-1106-preview snapshots.
Even the discrete-label long-short portfolios have strong gross results. GPT-4 has a Sharpe of 2.83, followed by Mistral at 2.64, FinBERT at 2.57, GPT-3.5 at 2.14 and Llama at 2.12. FinBERT also leads the Phrasebank classification test, scoring 84.23% against GPT-4's 82.06%. The authors acknowledge that FinBERT was trained on that dataset. Their conclusion comes with a condition: chat models justify their compute only with LE.
How much does the ranking improve?
A little. Switching from discrete labels to LE raises next-day rank correlation from 2.80% to 2.96% for GPT-4 and from 2.63% to 2.88% for GPT-3.5. Mistral moves from 2.55% to 2.83%, Llama from 2.37% to 2.78%. FinBERT has the largest increase, from 1.80% to 2.22%. Improvement across every model is the paper's clearest evidence. The gains amount to 0.16 to 0.42 points of IC across roughly 124,000 stock-days. We found no standard error for any of them.
The rank correlations also complicate the paper's argument about compute costs. FinBERT with LE reaches 2.22%, below every chat model using discrete labels, including Llama at 2.37%. Even without LE, Mistral's 2.55% and Llama's 2.37% next-day rank correlations exceed FinBERT's 1.80%.
The three-day correlations look much bigger: GPT-4 reaches 21.19% rank correlation with LE. But the paper says this window begins the day before publication. Most of the 7x gap over the next-day result therefore reflects how much of the headline was priced before anyone could trade it.
The selection rule changes too
Figure 5 carries the portfolio claim. Its LE portfolios trace a frontier above each model's discrete-label point. They use the same kind of threshold grid specified for FinBERT in Figure 3, from the 10th to 50th percentile in 2.5% steps. The baseline follows a different rule. It buys every stock with net-positive sentiment and shorts every stock with net-negative sentiment, leaving leg sizes to follow the day's sentiment mix. The authors attribute the gains to LE allowing them to "take more concentrated bets on the highest and lowest sentiment stocks." That description is honest. Both the signal and the selection rule have changed.
The authors say discrete scores require the sign rule because quantile sorting on 1/0/-1 is infeasible. Their example is July 26, 2019, the busiest day in the sample: 67 companies with about four articles each, and 25% of stocks scoring exactly 0. Running the comparison in reverse is perfectly feasible, though. Trade LE scores using the sign rule. That middle case is missing.
Figure 3 keeps the model constant, with FinBERT's probability-sorted frontier above its own discrete point. Signal and selection still change together. I would want to see LE scores traded by the discrete rule: long above zero, short below. This would separate ranking from concentration. We did not find that comparison. With a 10% threshold and a 46-stock day, each leg contains four or five names.
We found no numerical Sharpe for any LE portfolio, either. Performance is presented entirely as curves. Figures 3 and 5 show risk and return relative to each discrete point; Figure 7 shows Sharpe across thresholds for Mistral and Llama with different tokenizations. The curves each cover seventeen thresholds, all evaluated over the full 2010 to 2020 sample without a hold-out period. The claim that LE Llama outperforms discrete GPT-4 rests on a visual comparison of the best points along an in-sample frontier.
The comparison across models is more convincing. Stock-level score rank correlations among the four chat models rise from 66% to 80% with discrete labels to 83% to 92% with LE. For Llama versus GPT-4, the correlation rises from 67% to 83%. These figures fit the explanation that label boundaries drive most of the disagreement, more than model size does.
Costs are absent. Training comes later.
Every position opens at one open and closes at the next. Turnover is therefore total by construction. The GPT-4 discrete portfolio returns an annualised 53% gross. Dividing that by about 252 trading days gives about 21 basis points a day, which must cover every opening-auction round trip in stocks that have just made news. We found neither transaction costs nor a portfolio weighting scheme. The 263 companies are those left after filtering for single-company alerts and merging with CRSP. The paper does not explain the small count or say whether it includes delisted names.
Look-ahead is the bigger concern. The headline sample covers 2010 to 2020; the models are 2023 releases, queried as November 2023 snapshots. Their training data could have included the subsequent price moves. The paper does not address training-data overlap, despite citing Glasserman and Lin's study of look-ahead bias in GPT sentiment. FinBERT has the least exposure to this risk. Its increase in rank IC, from 1.80% to 2.22%, is also the largest among the five models.
The book that would persuade me trades LE scores with the sign rule.
We cannot run the chat models at this scale ourselves. Our strategy test substitutes the paper's classical FinBERT-style baseline, so it will not test Fabozzi and Florescu's claim about generative models. Extracting the probabilities costs one output token and improved IC for every model in the paper. I would adopt the trick without the Sharpe charts.