A replay can pass its live-day check for the wrong reason. Vidovich and Zerjal found one where two bugs cancelled.
Their system trades a fixed universe of 97 to 98 US stocks intraday using Alpaca 1-minute bars. It makes its decision shortly after the open and has run on a paper account and with real money. The authors withhold the entry, exit, filter and sizing rules. They make no claim that the system is profitable. Their subject is the backtest: a replayed real-money trade passes when its result differs from live by less than 5% of the replay result or a fixed dollar floor, whichever is larger. The floor is withheld, so readers cannot assess how demanding that pass criterion is. The authors acknowledge this.
They call the backtest the parity harness. It runs the live runner's strategy code, fetches daily inputs and decision-time quotes from the broker, and is scored against live fills. Comparing live days with replays uncovered six defects: three in data, two in code, one in configuration. Each entry gives the error's size and cause, then names a test or control meant to guard against it. In all six, paths that should have agreed had drifted apart.
The question for anyone putting capital behind the repaired harness is whether it has survived enough real-money days, including winter, and whether its tests have caught a recurrence. The paper's evidence does not reach that point yet. The authors largely say so themselves.
How did a passing day conceal two bugs?
Defect 5 began with the input request. Live built it from the required-input list; the replay used the traded-stock list. After that list changed, one replay input read as zero, or was only partly computed on days when some data was present. The fault persisted in every harness run for about four months. Restoring the input changed one decision across 20 replayed real-money mornings. The authors describe that as a before-and-after comparison, unscored against live.
The repair exposed another gap: live and replay placed a size cap at different stages of sizing. On one real-money day, the two errors cancelled and the replay passed. Repair Defect 5 while leaving the cap gap, and that day fails on a sizing artifact. The cap fix is queued but unbuilt.
The authors point out that the Defect 5 repair left the other two real-money passes unchanged. They put more trust in harness decisions and aggregates than in individual trades. Those two days are a thin defence, especially given what an older harness already showed. That retired version computed inputs offline and matched live decisions on 7 of 8 real-money dates. At the same size limit, its return was less than half the parity harness's figure. The authors write, "Matching decisions is necessary, not sufficient," and the cancelled day extends the point to dollars. Two offsetting errors can place a day inside the slippage bar.
Plausible summaries, wrong inputs
The data defects left the summary looking ordinary. Defect 1 came from fetch requests with no feed parameter, which returned single-exchange IEX data for historical pulls. Cached opening-minute volume measured 0.1% to 2% of the consolidated tape. One stock had about 7,900 shares in its cached 9:30 bar against roughly 3.4 million on the consolidated tape. Correct full-day volume totals helped the problem survive for about 11 weeks.
Replay two months with each feed and the trade choices stay the same. The thin cache nonetheless understated one month's result and overstated the other's; it also made a losing trade a winner. A permutation test on that cache (p = 0.217) was retired. With one error in each direction over two months, the authors cannot separate noise from bias. Either monthly figure could be wrong without the summary revealing it.
Defect 2 pushed prices in one direction. A commit changed the adjustment from split-only to all, adding dividends, and remained in place for 3.5 weeks. A same-day scan found 37 of 98 symbols shifted downward. Every one paid a dividend; examples in the log ranged from 0.68% to 1.65%. The discrepancy surfaced when one live trade, filled in the paper account, failed to match its replay. Once unadjusted prices were pulled again, entry and exit matched to the cent.
Sizing errors and a summer clock
In the reference backtest, 20 of 61 simulated fills carried notionals the live account could never take. A later size-increasing step multiplied positions that buying power had already capped. Logged dollar amounts were about 18% too large, though the fill count stayed at 61 before and after the fix. A trade list cannot expose this capacity error when buying power binds well below the per-trade ceiling. Nine days earlier, the harness had omitted buying power altogether and oversized the real account several-fold.
The clock error changed research-scan figures without changing a verdict. A fixed "13:30" UTC key corresponds to 9:30 ET only in summer. The study included 129 winter days, 30% of its sample. In a 13-day winter sample, the scan scored 16% of stock-mornings on a pre-market bar and dropped 84%. After the fix, Study A rose from 2,207 to 3,024 stock-mornings.
No verdict flipped. Almost every number moved.
What have seven real-money days established?
The current harness has seven real-money parity days, all in US summer time. One predates the slippage bar. A live-only execution check changed the trade on two. Three pass, including one that passed only after the Defect 5 fix. One passed because two errors cancelled. Defect 4's season remains unchecked, and Defect 4 still lacks a unit test. Figures produced during the four Defect 5 months have yet to be re-run. That includes rejected ideas in the authors' research graveyard, their one-entry-per-idea record of what they killed and why.
Even matched days can differ in per-trade dollars: the harness uses a cached spread, while live uses the real quote. Defect 6 concerned configuration. An audit found 53 keys used by production code but never declared, about a quarter of the surface. The execution switch existed only in a startup file. Deploying the generated default config would silently have put live into dry-run.
We could not replay these results ourselves. That requires the authors' broker fills and decision-time quotes, and the strategy rules remain undisclosed. Minute bars alone cannot establish parity.
The checklist deserves use now, particularly the network-free test that records outgoing requests and asserts their feed and adjustment parameters. It remains a set of controls. Three real-money passes, all in summer, certify only those days. Defect 3's class has already recurred in a step beyond its test. Most log entries identify a commit, but the log is not public. A passing winter real-money replay and re-runs of the Defect 5-era figures would change my view.