Automated analysis

This analysis was drafted by our research engine and has not been checked by a human editor. It may contain errors. It separates the paper’s own results from our tests, and any figures called ours come from our own backtest.

Our automated analysisOur backtest

A one-bit rule ties S&P 500 XGBoost's 63.84%

Yuan, Leng and Zhang find 31.0% fixed labels and 49.19% accuracy on genuine sessions.

2026-10-19 · 6 min read · Machine-learning directional trading and leakage validation · US equity ETF (SPY), with the S&P 500 index available as a signal input

Reviewing: When SHAP explains a leaked target: a cautionary study of an XGBoost and multi-level SHAP pipeline for S&P 500 direction prediction · Liansheng Yuan, Yansong Leng and Yuqi Zhang · Read it on openalex

Our backtest of this idea

Our automated quick test, not the paper's

Audited SPY Next-Session Open-to-Close XGBoost

Backtest period 2020-01-01 to 2024-07-01 · hypothetical, net of modelled costs

Why these figures are not the paper's (2)

Run on a different market than the paper

The paper forecasts and backtests S&P 500 index direction; we would trade SPY because an index cannot be held. The daily price-and-volume prediction and leakage-diagnostic method can be tested on SPY, but its returns and reported performance do not transfer to the ETF.

The paper's own figures describe its universe and do not carry over to ours.

Our own audit found this run does not follow the paper faithfully (5)

Paper Eq. (1): y_t = I[P^c_{t+1} > P^c_t], with equality classified Down.: Retained only as a separately identified paper-label diagnostic; training and trades use y_t^OC. (invalidates: Paper close-to-close classification accuracy and AUC; paper close-to-close trading returns)
Paper chronological 80%/20% split and six non-overlapping outer blocks over the final 50%, with expanding and fixed-length rolling training; numerical rolling length not supplied.: Retain 80%/20% as a separate diagnostic and six-block final-50% expanding evaluation as primary. Set sensitivity rolling length from the first eligible fold's available labeled training count. (invalidates: Paper single-split and pooled six-fold numerical results)
Paper share-volume VR and Delta V inputs.: Use verified share volume if physically available; otherwise use volume_in_dollars consistently and disclose the substitution. (invalidates: Paper volume-feature values; paper volume-related individual and grouped SHAP attributions)
Paper compares close-to-close XGBoost long/flat and long/short, momentum long/flat and buy-and-hold.: Trade only the requested SPY open-to-close long/flat model; compare matched intraday directional rules and report overnight-exposed buy-and-hold separately. (invalidates: Paper Table 8 strategy returns, Sharpe ratios and drawdowns)

1 further finding(s) are described in the note.

These are our findings about our own implementation, not criticisms of the paper. Read the figures below as a description of what we ran.

Jan 2020Total 2.4%Jul 2024

Sharpe: 0.09
Total Return: 2.4%
Max Drawdown: -14.9%
CAGR: 1.1%
Volatility: 11.5%
Beta vs SPY: 0.45
Trades: 686

What the paper reports for its own strategy

XGBoost long/flat, hold-out period, as-is configuration: annualized return −11.59%, Sharpe −0.241 at 10 bps one-way cost, Sharpe +0.175 at 0 bps, max drawdown −34.7%
XGBoost long/short, hold-out period, as-is configuration: annualized return −34.29%, Sharpe −1.014 at 10 bps one-way cost, Sharpe −0.282 at 0 bps, max drawdown −40.6%
XGBoost directional accuracy (hit rate) on the 354-day hold-out: 63.84% (original run), 63.28% (re-execution); 49.19% on the 246 genuine next-day-move rows
Walk-forward pooled out-of-sample accuracy (per-fold re-optimized): 0.5980 expanding / 0.5957 rolling (as-is, N=883); 0.5133 expanding / 0.4833 rolling (trading-day, N=600)

A rule that ignores prices ties this XGBoost model's reported 63.84% hold-out accuracy. Predict "Up" when yesterday's return is exactly zero. Yuan, Leng and Zhang report the rule's result themselves; their full 32-feature model was first reported at the same figure.

Their classifier uses 32 technical features across six families: returns, trend indicators, realized volatility at 5, 10 and 20 days, volume, momentum oscillators, and lagged returns at 1, 2, 3 and 5 rows. The label is 1 when tomorrow's close strictly exceeds today's. They used XGBoost, a sequential 80/20 split and SHAP's TreeExplainer to explain the predictions. The data comes from the widely circulated Kaggle file "Time Series Forecasting with Yahoo Stock Price". It is the S&P 500 index from 23 November 2015 to 20 November 2020. The original account attributed next-day direction to short-term mean reversion and volatility regimes, with SHAP offered as a way to see those effects. On 354 test rows, the model scored 63.84% accuracy and a macro-F1 of 0.6110.

The authors then inspected the file. Its 1,825 rows cover every calendar day, and 567 (31.1%) copy the preceding session exactly: 520 weekend days and 47 holidays. Under the strict inequality, Friday receives a Down label because Saturday copies its close. Saturday receives Down for the same reason. The fill fixes 547 of 1,766 labels (31.0%) before any model sees them. P(Up) is exactly zero on all 253 Fridays and all 252 Saturdays; on the other weekdays it ranges from 0.5159 to 0.5754. The revised paper calls its contribution methodological, presents the pipeline as a cautionary case and withdraws the authors' economic readings one by one.

What wins on the filled file?

A calendar-only rule predicts Down whenever the next close is a filled copy. It scores 71.19%. In the authors' re-execution, XGBoost scores 63.28%, nearly eight points less. They attribute the 0.56-point difference from the first reported 63.84% to library-version differences in tie-breaking.

The split by label type is decisive. The model gets 95.37% of the 108 test rows with fill-fixed labels right. On the 246 rows with a real next-day move, accuracy falls to 49.19%, below that subset's 58.54% majority rate. Across the full test set, always predicting Down scores 59.32%. The bootstrap 95% interval for model accuracy, [0.585, 0.684], contains that rate.

Once the filled rows are removed, the single-split AUC is 0.527. Six walk-forward folds, with hyper-parameters picked again inside each fold, produce pooled accuracy of 0.5133 expanding and 0.4833 rolling. The majority baseline is 0.5617; AUC is 0.476 and 0.460. Across twelve pooled comparisons on the as-is and trading-day files, the model never significantly beats the majority rule. The authors report that collapse themselves.

SHAP reads the fill

SHAP puts Return(t-1) first, at mean |SHAP| of 0.4095. Exactly zero is the clue: 108 of 354 test rows have that value, marking sessions after a weekend or holiday. In the full sample, P(Up | Return(t-1) = 0) is 0.519, versus 0.325 otherwise. A zero lag tells the model that the next close will be a real observation. SHAP describes the model's use of that clue faithfully. The first submission interpreted it as mean reversion.

The lagged returns also encode weekdays. Each is exactly zero on 547 to 550 rows, concentrated on two specific weekdays: 504 of 547 for the first lag, with the remainder following holidays. Volume Change is zero on 546 rows, including every Saturday and Sunday.

The volatility features create another attribution problem. Vol 5d, 10d and 20d, ATR(14) and Bollinger Width (the spread of the 20-day bands relative to their moving average) correlate up to r = 0.947. Individual credits total 0.749, while joint attribution for the group is 0.284. Counting the credits separately overstates the cluster by 2.64x. Hierarchical clustering at |r| ≥ 0.80 takes the 32 features down to 18 groups. With one representative per group, retraining raises single-split AUC from 0.6907 to 0.7144. As the authors write of the waterfall plots, "Apparent convergence across a correlated feature set is the expected appearance of a single signal counted several times."

The backtest file remains uncertain

The diagnostic work is specified closely. The paper gives the feature formulas, a depth grid of 4 to 8, learning rates of 0.03, 0.05 and 0.1, and estimator counts of 100 to 500. Its walk-forward test uses six blocks over the final 50% of the sample, with an inner five-fold TimeSeriesSplit. It charges 0, 5 and 10 bps one-way on turnover. The trading results leave a harder question: which series produced them?

The abstract assigns the −0.24 Sharpe at 10 bps to a run after removing forward-filled rows. The backtest table says as-is. The text finds no economic value "in either configuration", though we found a table only for the as-is file. Such a backtest includes weekend and holiday rows that cannot be traded, and annualization is inferred from the calendar span. For the trade-level numbers, the as-is reading is the safer assumption.

Those numbers are poor. Long/flat returns −11.59% annualized, with a Sharpe of −0.241 at 10 bps and a −34.7% drawdown. Buy-and-hold returns +15.50%, with a Sharpe of +0.588. Long/short returns −34.29% annualized, with a Sharpe of −1.014. At zero cost, long/flat reaches only a +0.175 Sharpe.

The target needs repair too. The authors say it requires a trading-day series and make no predictability claim on one. Among their 1,199 trading-day rows, deterministic labels fall to 1.

Our SPY run

An index cannot be held, so we traded SPY instead. Our adaptation was built automatically from the paper's description and changes more than the instrument. We kept genuine sessions and used next-session open-to-close as the target, going long when forecast probability exceeds 0.5 and holding cash otherwise. The model retrains on an expanding window using the paper's grid and inner split. Data runs from 2020-01-01 to 2024-07-01; reported metrics begin on 2022-03-31. We charged $0.0040 a share with a $1 minimum and zero slippage, assuming fills at the auction price.

The run returned 2.44% across 686 trades. Sharpe was 0.09, CAGR 1.08%, volatility 11.51% and maximum drawdown −14.86%. Win rate was 55.39%; profit factor was 1.02. One flaw remains unresolved. The rule calls for a buy at the next open, yet the fill records show market-on-close execution. We cannot confirm that the intended trade was booked.

The paper's long/flat Sharpe is −0.241 at 10 bps; ours is +0.09. The sign changes across different instruments, targets and windows, so that gap tells us nothing about the paper's work. Cost likely explains much of it: the paper's own zero-cost long/flat Sharpe is +0.175, also positive. Our SPY commissions amount to a fraction of a basis point.

The data series is the likely second factor. Our genuine-sessions series resembles the paper's trading-day control, where AUC was 0.527, leaving little reason to expect an edge from either setup. Two other explanations remain hypotheses. Staying flat overnight could change what the strategy times. The 2022 to 2024 window could also differ from a hold-out that, by our count of the last 354 calendar-day rows, runs from about December 2019 to November 2020. That span includes the 2020 crash, when buy-and-hold drew down 33.9%. A beta of 0.45 at 11.51% volatility looks like half-time market exposure. With a 1.02 profit factor, slippage or market impact has little room to spare. Our one automated pass finds no edge; it is no verdict on the authors' work, and its result agrees with the paper.

Run the one-bit rule before SHAP.

Our backtest stops at 2024-07-01, and everything after that date is deliberately left untouched so the same strategy can be checked out of sample later.

How our backtest worked

The steps the code we ran actually executed, from its strategy card. Ours, not the paper's — it is one automated implementation of the idea, not the authors' own.

Audit SPY daily bars; retain genuine sessions and compute 32 features using data through close t.
Fit an expanding-history XGBoost classifier on labels already observable at decision t.
Select hyperparameters by five-fold expanding TimeSeriesSplit within training history only.
If features or sufficient valid training history are unavailable, stay in cash.
After close t, forecast session t+1.
If P(next-session open-to-close return &gt; 0) &gt; 0.5, buy at t+1 adjusted open;
otherwise stay in cash. Sell any purchase at t+1 adjusted close.
Evaluate six consecutive outer blocks covering the final 50% of eligible opportunities.

This is the specified rule. The supplied platform cost record describes daily-bar MOC execution, so the recorded fills need verification against the specified next-open entry.