A market-making desk running an automated strategy had a simple observation: positions held longer than a few hours were, more often than not, the ones that lost money. The proposed fix was equally simple — add a rule to force-close any position once it crosses that age. A quick backtest seemed to confirm it. The obvious next step was to ship it.
It didn't ship that day. Here is what an independent review actually looks like between "the backtest says yes" and "we changed the production code" — illustrative numbers, real method.
Why this matters: almost every strategy vendor, every in-house quant, every backtest has an incentive — conscious or not — to find the result that confirms the change they already want to make. An independent reviewer has no stake in the outcome. That's the entire value.
1Interrogate the claim before touching the data
The starting statistic — "positions held past a threshold lose more often than not" — sounds like a discovery. It's closer to a tautology. Positions that resolve quickly are, by construction, the ones where price cooperated fast; whatever is left in the "held longer" bucket is self-selected to be the harder cases. Before running a single line of code, the review flagged two separate risks hiding inside one number:
- Selection bias — "still open at hour X" is not a random sample; it's already filtered for difficulty.
- Win-rate ≠ expected value — a rule can look right by trade count and be wrong in dollars, if the trades it avoids losing on are small and the trades it keeps are large.
2Measure it properly — in dollars, not win percentage
The fix isn't to trust or distrust the original claim — it's to actually run the numbers, keyed to realised profit and loss rather than a count of wins and losses, across a full sweep of possible cutoff times rather than the single threshold someone happened to propose.
Across several years of tick-level market data, cutting positions materially earlier than the existing rule did produce a real, positive improvement in risk-adjusted return — and, tellingly, the win rate actually went down under the new rule. That's the useful tell: the strategy was giving up some small, easily-avoidable losses in exchange for avoiding a much smaller number of much larger ones. Win-rate alone would have hidden that entirely.
3Audit the data before trusting the number
A result this clean deserved suspicion, not celebration. Checking the underlying dataset surfaced a sixteen-month gap in the historical price archive — a stretch that had silently gone missing from the cache used to build the first-pass result. The gap didn't change the direction of the finding once filled, but it changed its magnitude materially, and it's exactly the kind of thing that turns a defensible number into an indefensible one in front of a client or an investment committee.
4Find out where the money actually came from
The full-period number still didn't smell right — the implied annualised return was far above what the strategy was actually reporting in live operation. Breaking the result down by trading regime (quiet, low-volatility conditions versus rare, large price-dislocation events) explained the gap immediately: the overwhelming majority of the strategy's total profit came from a small handful of extreme months, not from everyday operation. The backtest's headline number was true — and almost entirely irrelevant to the calm-market conditions the strategy actually spends most of its time in.
That forced the real question to be re-asked: does the proposed rule still help specifically during calm, ordinary conditions — the state the strategy is in on any given day — or does it only look good because it happens to also help during the rare, dramatic events? Re-running the comparison split by regime confirmed the edge held independently in both — a materially stronger result than the aggregate number alone, because it survived the decomposition that was designed to break it.
5Hold out data the rule was never allowed to see
A rule discovered and validated on the same dataset is not validated — it's fitted. The final step split the history into a training period and a held-out test period the analysis was never allowed to look at while choosing the rule. The rule selected on the training data was then applied, unchanged, to the untouched period. In this case, the improvement held up on data it had never seen — which is the actual bar for "this is a real edge," not "this backtest looks good."
6Formalise it: is the winning choice real, or did it just survive one lucky split?
A single train/test split is one data point. It answers "did this survive once" — it doesn't answer "how often would this survive." To go further, the review applied Probability of Backtest Overfitting (PBO) via Combinatorially Symmetric Cross-Validation (CSCV) — the method formalised by Bailey, Borwein, López de Prado & Zhu specifically to quantify this risk, rather than eyeball it.
The history was partitioned into a number of equal-sized blocks, and every possible way of splitting those blocks into a training half and a held-out half was enumerated — hundreds of distinct resamplings, not one. In each resampling, whichever variant looked best on the training half was checked against how it actually ranked on the half it never trained on. PBO is simply the share of those resamplings in which the training-picked "winner" turned out to be a below-average performer once tested blind — the rate at which picking the best backtest is equivalent to picking at random.
Across every one of the resampled splits, the training-selected choice also ranked among the strongest performers on the data it had never seen — and, notably, the training process converged on essentially the same answer regardless of which slice of history it was shown. A selection process that changes its answer depending on which years happen to be in the training set is a warning sign; one that keeps landing on the same answer, resampled hundreds of different ways, is the opposite.
One honest caveat belongs next to that result, not buried in a footnote: CSCV deliberately ignores chronological order — a resampling can train on later history and test on earlier history, which no live deployment could ever do. It answers "is this a stable, sample-independent regularity," not "could this have been known in advance." That's precisely why it complements the strict forward-only holdout in Step 5 rather than replacing it — one checks robustness, the other checks causality, and a recommendation should clear both, not either.
7Catch the near-miss: two studies that weren't measuring the same thing
Midway through, an earlier internal study surfaced with the opposite conclusion — the same kind of rule, tested months prior, had been rejected as statistical noise. Rather than treat this as a contradiction to explain away, the review compared methodologies directly: different underlying instrument, different strategy parameters, different date range, different assumed funding cost. The two studies weren't disagreeing — they weren't measuring the same thing. That distinction matters more than either individual result: two professional-looking backtests can both be correct and still be entirely non-comparable, and treating them as competing answers to the same question is its own quiet source of bad decisions.
8Stop assuming the fill — go measure it
Every backtest in this review shared one convenient assumption: if the market price touched a resting order's level, it was treated as filled, at that price. That assumption is almost always the most optimistic part of any simulation — real order books thin out precisely when it matters most, and "the price touched my level" is a very different statement from "my full size actually traded there." Rather than argue about how large that gap might be, the review went looking for a way to measure it directly.
The account's own execution history included a real, historical volatility dislocation — genuine fills, not simulated ones. For every one of those fills, the review measured, tick by tick, how far price kept moving against the position in the minutes immediately afterward — the part a naive backtest never sees, because it stops caring the instant the fill is logged.
In ordinary, low-volatility conditions, there was essentially no continued adverse move after the fill — the backtest's clean-fill assumption costs nothing there, which is consistent with calm-period backtest returns tracking live results closely. During the dislocation window specifically, real fills kept drifting materially further against the position afterward, with a heavy tail in the worst cases — confirming the optimistic assumption is real, and concentrated exactly where Step 4 showed almost all of the profit lives. Tellingly, the worst-case figure from this fill-by-fill measurement landed close to an entirely separate risk estimate the desk had derived months earlier through independent leverage stress-testing — two unrelated methods, arrived at independently, converging on the same number.
This is the piece PBO cannot give you. Step 6 tests whether a result is stable under resampling — it says nothing about whether the simulation's core assumptions are honest to begin with. A backtest can pass every overfitting check and still be quietly optimistic in exactly the conditions that generate most of its profit. Only going back to real fills answers that question, and it's the one check in this review with no formula standing in for it.
What didn't happen: no code changed in production. The review's output was a validated recommendation, a documented audit trail, and — for the one assumption that couldn't be verified from history already on hand — a live-monitoring protocol to keep measuring it going forward. Not a green light rubber-stamped on day one.
The reusable checklist
This is the actual deliverable — not the specific numbers above, which will differ for every model. Applied to any backtest-driven decision before capital moves:
- Does the headline statistic contain a selection-bias artefact — is the "interesting" subset already filtered for the outcome you're about to measure?
- Is the result being judged by win-rate (count) or expected value (dollars)? The two can point in opposite directions.
- Has the underlying dataset been audited for gaps, survivorship bias, or silent re-sampling?
- Is the aggregate result being driven by a small number of extreme periods, or does it hold across market regimes independently?
- Was the rule validated on data it was never allowed to see while being selected — genuine out-of-sample testing, not a re-run on the same window?
- Does the winning choice hold up across many different resamplings of history (formal overfitting checks such as PBO/CSCV), or does it depend on the one split that happened to be tested?
- Where real execution history exists, has it been used to measure actual post-fill slippage — not just assumed away — especially during the volatile periods the strategy's profit depends on most? Overfitting checks cannot substitute for this; they operate on the same potentially-optimistic simulation, not against reality.
- Are two backtests being compared actually measuring the same instrument, parameters, and time period — or just superficially similar questions?
- What does the backtest assume about execution and fills, and how optimistic is that assumption relative to live order-book reality?
Where this fits
This is the same discipline MOA applies to enterprise vendor proposals and architecture decisions under Independent Verification & Validation — extended to quantitative and algorithmic trading models. We review methodology and evidence; we don't sell or manage the strategy under review, and we don't provide personalised investment advice. The value is the same as everywhere else we work: no reseller commission, no stake in the answer, no incentive to tell you what you want to hear.