An EPS beat barely moves most of these models' earnings forecasts. In Khaybullina's controlled test, four of six model files have an average signed beat-minus-miss contrast under 0.2 pp. Their mean absolute matched moves range from 0.60 to 1.82 pp. Meanwhile, fact-free opinion sentences such as "I feel optimistic about this announcement" move Phi-3.5-mini by 45.01 points on average across six pairs (28.9 for that one).

Only Qwen2.5-7B responds strongly to surprise, at 86.80 pp. Phi-3.5-mini reaches 6.57 pp; the other four are near zero. The answer depends on the checkpoint file and the format used to ask the question. The paper measures no returns, as it acknowledges.

Moving consensus, fixed earnings

Khaybullina treats an LLM earnings strategy as a function that reads a release and returns a sign, then tests which inputs change that sign. The experiment uses 24 synthetic earnings blocks spanning profit or loss, rising or falling annual EPS, and surprise gaps of 2% or 15% of absolute actual EPS. Actual and prior EPS remain fixed within each block. Stated consensus moves above or below actual EPS, allowing even a loss to beat an expectation of a worse loss.

Two other inputs vary independently: the sign of a 20-day pre-release return of 3-12%, and a commentary sentence from six authored positive/negative pairs. The prompt labels each sentence as opinion containing no additional facts. Eight sign combinations, two section orders and 24 blocks produce 384 scenarios per model. Each gets three scores: A means positive, B means positive, and U/D.

The score comes from a softmax over the log-likelihoods of positive and negative answer prefixes; there is no sampling. The headline contrast is the positive-minus-negative change in the A/B-averaged score, averaged across blocks and bootstrapped 10,000 times. All six models are Q8_0 GGUF conversions run on CPU, with base parameter counts from 0.49B to 7.61B. Khaybullina states the limit precisely: "We estimate matched changes in conditional answer-prefix scores, not calibrated market probabilities."

Tone wins in five checkpoints

Tone has a larger effect than EPS in 5 of 6 checkpoints. Momentum does too. Qwen2.5-1.5B moves 30.39 pp on tone and -0.19 pp on surprise. Phi-3.5-mini moves 45.01 pp on tone, with an interval of [33.089, 57.099], while EPS moves it 6.57 pp.

Qwen2.5-7B reverses that ordering. Its EPS contrast is 86.80 pp [81.027, 92.038], beside 10.65 for momentum and 9.84 for tone. Every matched beat/miss pair moves in the positive direction. The contrast reaches 94.08 pp in the 15% gap blocks and 79.52 pp in the 2% blocks, an exploratory between-block comparison. With commentary removed in a separate facts-first control, it reaches 96.86 pp. Adding surprise to momentum and tone in a leave-one-block-out linear fit raises R² for Qwen2.5-7B from 0.019 to 0.829. For the other five models, that change ranges from -0.000515 to +0.007589.

A signed mean near zero could conceal large moves in opposite directions. Khaybullina checks mean absolute matched changes, and the counts remain 5 and 5. Qwen2.5-1.5B's mean absolute EPS move is 1.35 pp, beside 30.39 for tone.

The 0.5B, 1.5B and 7B Qwen checkpoints give EPS contrasts of -0.15, -0.19 and +86.80 pp. Training choices are confounded with parameter count, the author warns; three points cannot establish a scaling curve.

Do they recognize the beat?

Khaybullina also asks the models directly whether EPS beat or missed. Natural-word accuracy spans 51.0% for SmolLM2 to 95.8% for Qwen2.5-7B. With A/B averaging, Qwen2.5-0.5B, SmolLM2 and TinyLlama each score exactly 50.0%. Their flat EPS contrasts could reflect failure to make the comparison. As the author puts it, weak diagnostic performance "prevents a small EPS contrast from being attributed solely to deliberate down-weighting of the surprise."

Phi-3.5-mini presents a sharper distinction. It identifies the beat 84.4% of the time in natural words, yet surprise moves its forecast only 6.57 pp. It recognizes the comparison far more often than its forecast uses it.

Format can overwhelm the signal. Swapping which letter denotes positive changes TinyLlama's score by 63.32 pp on average and flips 83.3% of its decisions. Under U/D, just 11.9% of its next-token mass lands on the two requested answers. Qwen2.5-7B moves 3.02 pp under the same letter swap. Only 2 of 6 models retain a positive EPS contrast under A, B and U/D.

The conversion checks make the exact file consequential for a backtest. Khaybullina re-exported Qwen2.5-0.5B from the pinned source. Across six selected stress cases, the author's F16 export comes within 0.668 pp of full precision. On those same six selected stress cases, the author's Q8 rebuild differs from the downloaded first-party Q8 by up to 46.136 pp; only 73 of 291 tensor entries match byte-for-byte. Compared with the pilot's saved FP32 scores, the Q8 runtime has a 15.64 pp mean absolute gap and 22.9% decision flips across 48 selected cases. The paper concludes that "these Q8 conversions are not interchangeable for the tested scores." A trader should put the file hash beside the prompt in the signal definition.

What would make this a trading signal?

The author is explicit: "The experiment measures observable input reliance and numerical recognition; it establishes neither an internal reasoning mechanism nor realized return predictability." No next-day abnormal reaction is observed, so these input responses cannot be judged as forecast errors. The paper makes this point for momentum in particular: price history can convey information in real markets. Real commentary can too, unlike the deliberately fact-free sentences here.

The proposed next step is a frozen prospective sample of real announcements, timestamped inputs, larger model families and comparisons with reasoning modes. For a live test, each release would be scored using information available at the print. Returns would then be measured from the first tradable price afterward.

For now, this is a useful specification audit with no tested signal. One of six model files reads surprise reliably, but its 86.80 pp response has yet to be checked against a single realized return. I would change my view if a prospective run showed Qwen2.5-7B's beat/miss call sorting next-day abnormal returns net of costs. Phi-3.5-mini's 45-point sensitivity to tone on real releases is the other test worth running.