Behal's simulated NSE classifiers all trail buy-and-hold on accuracy

Getting direction right 51.8% of the time offers a trader little when the index rises on 52.5% of days. An always-long position already does better. Behal reports that comparison himself. It is the paper's most useful number, yet the headline metric leaves it out.

The simulator plants the opportunity

Behal uses no real-market data. All the figures describe a simulated index whose predictability is specified in advance. The question is how much of that predictability standard classifiers can recover.

The market follows a two-state Markov regime-switching GARCH(1,1), with Student-t shocks at 5 degrees of freedom. Each day, the calm regime persists with probability 0.985; its daily drift is +0.0007 and its target daily volatility is 0.85%. The turbulent regime has persistence 0.95, drift -0.0008 and target volatility 1.9%. The planted forecasting signal supplements those drifts. It takes the sign of the trailing five-day mean return and scales it by current volatility, with coefficient β in calm markets and -1.5β in turbulent ones. At baseline, β = 0.05.

Calm markets trend; turbulent markets reverse.

The simulated index earns 10.80% annually at 18.70% volatility across 120 paths of 2,750 days. Median excess kurtosis is 7.1, and turbulent conditions account for 23% of test days. Four scikit-learn classifiers predict next-day direction using nine price-only features: logistic regression, a depth-5 random forest, histogram gradient boosting and a 32-16 MLP. Annual refits use an expanding window, producing 1,499 out-of-sample calls per path. A probability above 0.5 means a +1 position; otherwise the position is -1. Turnover costs 5bp per unit. The benchmark oracle knows the sign of the true conditional mean.

That oracle records 54.2% accuracy and gross Sharpe 1.19. Logistic regression manages 51.8%; the other three models range from 50.9% to 51.6%. Behal calls the resulting comparison a capture ratio: model accuracy minus 0.50, divided by oracle accuracy minus 0.50. It ranges from 20 to 41 percent. On turbulent days every classifier falls below 50%, with logistic regression last at 47.7%. The abstract reads these results as evidence that regime instability constrains AI-driven market prediction more than model sophistication does.

A reproducible script, with judgments still embedded

The appendix makes this unusually easy to build on. It supplies the complete script and fixes the hyperparameters in advance. Baseline seeds run from 0 to 119; each signal level uses 1000 to 1099. Two choices nevertheless deserve scrutiny.

Start with what the oracle knows. Its forecast uses the full conditional mean, including drift. Setting β to zero removes the momentum and reversal signal altogether. Even then, the oracle achieves 53.9% accuracy and gross Sharpe 1.02 on the sweep's separate 100-path set. The corresponding baseline results are 54.2% and 1.19 on 120 paths. Behal reports this result and traces it to the difference between the regimes' average drifts. He also says the models learned some of that drift, achieving about 51% accuracy without a trend pattern to exploit.

Knowledge of the current regime therefore supplies most of the capture ratio's denominator. The reported 20 to 41 percent measures regime identification at least as much as recovery of the trend rule under study. Behal acknowledges the drift effect, then still presents the capture ratio as recovery of available predictability.

The other choice is the ratio's 0.50 baseline. Buy-and-hold outperforms every classifier on accuracy, at 52.5%, as the author reports. He also acknowledges that exceeding 50% requires no active forecast. Use buy-and-hold as the baseline and logistic regression's capture is (51.8 - 52.5)/(54.2 - 52.5), about -41%. His discussion explicitly tells investors to compare a NIFTY model with buy-and-hold rather than 50 percent. Why, then, does the abstract retain 0.50 beneath its headline capture ratio of 20 to 41 percent?

What regime information did the models receive?

A proxy was all they saw.

The regime-aware version adds a flag for 20-day volatility exceeding its expanding median. It also adds the interaction between that flag and the five-day sign. Logistic regression gains about half a point of turbulent-day accuracy (t(119) = 2.65, p =.009). Its overall accuracy falls (t(119) = -2.20), as does net Sharpe (t(119) = -2.24). Changes for the other three classifiers are small and insignificant (all p >.27). The paper describes the flag as "too noisy a guide to the hidden regime," and proposes a real-time hidden Markov model in the conclusion's future work.

Behal thus acknowledges that the proxy was weak. Yet the abstract states broadly that adding explicit regime information did not help. The conclusion confines that claim to a simple regime indicator. Most readers will remember the abstract's broader wording, although the experiment supports the narrower wording in the conclusion.

The simulator already records the true state each day. Feeding yesterday's true regime to a classifier would require one extra column and settle H3, the hypothesis that regime information narrows the turbulent-day gap. If accuracy recovered, that would support the conclusion's reading that recognising regime change is the critical challenge. It would also conflict with the abstract's flat "did not help." The experiment as conducted establishes that a lagging volatility flag does not help. The abstract claims more.

There is also a statistical concern. The pooled calm-versus-turbulent test gives t(959), consistent with four models in two variants sharing the same 120 paths. These observations are dependent, so 30.9 likely overstates the evidence. I suspect logistic regression's 5.2-point gap, from 52.9% in calm markets to 47.7% in turbulent markets, would survive a per-path test. The published statistic does not establish that.

Buy-and-hold's 0.62 Sharpe sets the hurdle after five basis points

Logistic regression leads the classifier contest with gross Sharpe 0.40. The random forest delivers 0.32, the MLP 0.27 and boosting 0.16. Charging 5bp per unit of turnover reduces logistic regression's 0.40 to 0.12, with positive results on 62% of paths. Boosting switches positions on roughly a third of days; its 0.16 becomes -0.35. Buy-and-hold pays no cost and earns 0.62. Logistic regression surpasses it on 19% of paths, while the others do so on 8% to 12%.

Increasing the signal does little to change that comparison. In the separate sweep, using 100 new paths per level, logistic regression's net Sharpe moves from -0.07 at β=0 to just 0.22 at β=0.10. It becomes significantly positive only from β=0.08 (p =.002). Even at β=0.10, it beats buy-and-hold on 19% of paths, matching the baseline share. At β=0.10 the oracle earns gross Sharpe 1.75.

The trading rule creates part of this hurdle. Every probability at or below 0.5 triggers a full short, which pays away the calm-regime daily drift of 0.0007. Across paths, every model's net Sharpe has dispersion of 0.45 to 0.51. The paper itself thereby shows how easily luck can make an individual backtest look excellent.

Our SPY adaptation

The paper's market is a simulated NSE-like index. We constructed a strategy from Behal's idea on SPY because all its features use prices alone. The simulated dynamics do not transfer to this market.

We used training data from 2010 and annual expanding-window refits, requiring five years and 1,250 examples. The backtest covers 2020-01-01 to 2024-07-01. We executed regime-aware logistic regression, including the flag and its interaction. Our positions were long or cash rather than long or short, and the target was an open-to-open return. Commissions were $0.004 a share, subject to a $1 minimum.

Over that window, our net Sharpe was 0.52 on 40 trades. Volatility was about 19.4%, with a -32% maximum drawdown encompassing the 2020 crash. The number depends on choices that differ from the paper's setup. A $0.004 per-share commission is tiny for a fund as liquid as SPY. Long-or-cash also sends bearish forecasts into cash, sparing them the drift cost of a short position.

The risk profile is more revealing. Beta to SPY was 0.80, alongside 19.4% volatility and the full 2020 drawdown on 40 trades. Those figures suggest a book that stayed long most of the time. Its 0.52 Sharpe therefore likely owes more to SPY's drift than to forecasting skill. We produced no matched SPY buy-and-hold Sharpe for this window, leaving our result untested against the hurdle this review argues for. Fills were also reported at the closing auction despite the open-to-open label. That mismatch could shift the result in either direction.

Give the classifier yesterday's true regime

I would want to see the paper's own script run with the true lagged regime included as a feature. Clearing buy-and-hold's 0.62 on most paths would support the conclusion's detection reading and contradict the abstract's flat claim that regime information did not help. Continued underperformance would put the constraint on the long-only drift. Behal's abstract leaves that drift out of its headline capture ratio, built on 0.50, even as his discussion tells readers to judge against buy-and-hold.

Our backtest stops at 2024-07-01, and everything after that date is deliberately left untouched so the same strategy can be checked out of sample later.

1. Load SPY daily open/close observations from 2010; align to exchange sessions.
   Reject conflicting duplicates and nonpositive prices; do not impute missing bars.
2. After each session t closes, compute the nine baseline features.
   Set hivol = 1[vol20 exceeds its expanding median], only after 60 valid vol20 values.
   Form blind, flag_only, and flag_and_interaction feature sets.
3. Define R[t] = open[t+2] / open[t+1] - 1 and y[t] = 1[R[t] > 0].
   Use identical feature-readiness masks and observed target intervals for comparisons.
4. At each prior-year final-close cutoff, fit on expanding historical rows whose
   terminal target opens have already occurred. Require five years and 1,250 examples.
   Fit scalers only within historical training samples; freeze models for the year.
5. For MLP epoch selection, reserve the trailing 10% chronologically, purge overlapping
   training labels, and select the earliest epoch attaining maximum validation accuracy.
   Refit a fresh scaler and MLP on all eligible training rows for that epoch count.
6. Produce daily probabilities for all 12 model-feature configurations.
   The visible code defaults execution to logistic_regression/flag_and_interaction;
   it does not aggregate configurations or select a historical winner.
7. Target q[t] = 1 when p[t] > 0.5, otherwise 0; unavailable forecasts target cash.
   Retain consecutive long positions without artificial daily re-entry.
   Use available capital only, with no shorting or borrowing.
8. Specification: queue orders for the next session's open and measure the subsequent
   open-to-open payoff. Supplied platform execution metadata instead identifies MOC
   fills at the official close; treat this alignment discrepancy as unresolved.
9. Charge actual position changes and final liquidation. Report platform-recalculated
   net results separately from the specification's proportional-cost experiment.