Why backtests beat live results: the four channels of decay

People quote a single percentage for how much performance is lost between simulation and live trading. There is no such number, and looking for one is itself the mistake. The gap is the compounded output of four separate channels, each with its own arithmetic and its own remedy. This piece decomposes them and gives the calculation for each, so you can estimate the gap for your own strategy rather than borrowing someone else's figure. The decomposition and the numbers in it are ours.

Published 28 August 2026 · AI Trading Book Editorial · Reading time about 11 minutes

Why there is no single percentage

A figure like "expect to lose a third of your backtest performance" is appealing because it is actionable. It is also unusable, because the loss is not a property of trading — it is a property of your particular strategy and, more uncomfortably, of your particular research process.

Two strategies with identical backtest curves can decay completely differently. One was found after testing four ideas, holds positions for weeks, and trades a liquid instrument at modest size. The other was found after a thousand parameter combinations, turns over daily, and is being scaled to the limit of the account. The first may lose almost nothing. The second may have nothing to lose, because it never had an edge.

So the useful question is not how much decays, but through which channel. Below, each channel is defined by what it removes, how to estimate it, and what fixes it.

Channel one: selection

The best of many results looks good whether or not anything is there. This is the channel most people underestimate, because the search does not feel like a statistical procedure while you are doing it — it feels like diligence.

Here is the arithmetic that makes it concrete. Take a set of strategies with no edge whatsoever, each producing a Sharpe ratio drawn from pure noise. Now record the best one. The expected value of that maximum rises with the number of trials, and it rises fast.

Read the fourth row carefully. If you tested a hundred variants and deployed the best, a Sharpe of 2.5 is what you should have expected from a hundred coin flips. It is not evidence of anything. It is the number you have to beat before you can start making claims.

Two things make this worse than it looks. Every abandoned attempt counts — the parameter you nudged and reverted, the indicator you tried for thirty seconds, the date range you shifted. And machine learning multiplies the count, because a hyperparameter sweep is a trial per combination, not a trial per model.

The remedy is to log every trial as you go and deflate the observed Sharpe for that count. The Deflated Sharpe Ratio developed by Bailey and López de Prado in 2014 does exactly this, adjusting for trial count, track-record length and the non-normality of returns. Full treatment on our backtest overfitting page.

Channel two: leakage

Leakage is performance that depended on information unavailable at the moment of the simulated decision. Unlike selection, it does not degrade the result — it invents it. A strategy with material leakage typically has no live performance at all, rather than reduced performance.

The forms worth checking, in rough order of how often they appear:

• Shuffled splits. Randomising a time series before separating train and test. Standard practice for independent observations, destructive here, and the usual explanation for reported accuracies above 90%.
• Global scaling. Normalising features using means or ranges computed across the whole dataset, which leaks the future distribution into past rows.
• Bar boundary errors. Using a bar's close to decide a trade executed within that same bar. Produces a working backtest with an impossible result.
• Point-in-time failures. Using reported figures at their period-end date rather than their publication date.
• Universe construction. Defining a historical universe by today's index membership, which imports a decision made later.

The remedy is chronological splits, point-in-time data, a delisting-inclusive universe, and a holdout period opened exactly once. Note that leakage is the only channel where the correct response to detection is to discard the result entirely rather than adjust it.

Channel three: cost, and why move size dominates

This is the channel everyone models and most people model wrongly, because the intuition attaches to the hit rate when it should attach to the size of the move being captured.

Work through it. A strategy with a 55% hit rate on symmetric moves produces a gross edge of ten per cent of the move size — win 55, lose 45, net 10. What that is worth depends entirely on how large the move is. Against a round-trip cost of 10 basis points:

The same hit rate, the same cost, and the outcome runs from total loss of edge to keeping four fifths of it. The variable is the move size. This is the mechanism behind the familiar observation that short-horizon strategies are harder: not that prediction is worse at short horizons, but that the cost is a larger share of a smaller move.

Now hold the move at 200bp and vary the hit rate instead:

Compare the two tables. Moving the hit rate from 55% to 65% — an enormous improvement in predictive terms, well beyond what published research achieves — takes the surviving share from 50% to 83%. Moving the captured move from 200bp to 500bp at an unchanged 55% hit rate achieves nearly the same thing. Horizon selection is doing as much work as model quality, and it is far easier to control.

Our page on prediction accuracy covers where the 55% figure comes from and why careful published work clusters in the 55–65% band.

The remedy is to model spread, commission, slippage, financing and currency conversion explicitly, then re-run with trading costs doubled. If doubling kills it, what you have is a cost-model artefact rather than a strategy. The full cost list is on our strategy building page.

Channel four: capacity

A backtest fills your orders at observed prices no matter how large they are. That silently assumes infinite liquidity, which is why a strategy can work at small size and fail when scaled without anything else changing.

The mechanism is that your own order moves the price against you, and the effect grows with size relative to available liquidity. This channel is invisible in testing and only appears when you increase position size — which is precisely when people stop being cautious, because the strategy has just been working.

It also interacts with the cost channel rather than adding to it independently. Market impact is a cost, so a strategy already retaining only half its edge after spread and commission can be pushed under water by scaling, even though nothing about the signal changed.

The remedy is staged deployment with a gate at each stage, and an impact assumption in the backtest proportional to your intended size rather than none at all.

Why the order matters

Test the channels in the order given. It saves work and prevents a specific self-deception.

If you refine your cost model on a strategy that only exists because you tested four hundred variants, you are doing precise arithmetic on a number that means nothing. Selection first, because it can reveal there was never anything to decay. Leakage second, for the same reason. Only then do cost and capacity, which reduce a real edge rather than manufacture a fake one.

Estimating the gap for your own strategy

1. Count your trials honestly, including abandoned ones, and look up the expected best Sharpe for that count in Table 1. If your result is below it, stop here.
2. Audit for leakage across the five forms listed above. Any hit means discard, not adjust.
3. Express your gross edge in basis points per trade and compare it directly to your round-trip cost in basis points. Table 2 gives the shape of the answer.
4. Re-run with doubled trading costs.
5. Add an impact assumption at your intended size, not your test size.
6. Whatever survives all five is your live expectation. It will be lower than the backtest and it will be a number you can defend.

Where machine learning changes the picture

Not by adding a fifth channel, but by amplifying the first two.

Selection rises because hyperparameter search is a trial per combination. A modest sweep over five parameters with four values each is 1,024 trials, which puts the expected best Sharpe from noise above 3.2 before any signal exists. Very few practitioners count it that way.

Leakage rises because feature engineering creates more surfaces for future information to enter. Every derived feature is a place where a scaling step or a lookback can quietly reach forward. Our page on using language models for strategy work covers an additional variant specific to LLMs, where the model has read commentary about the period you are testing on.

Cost and capacity are unchanged. They do not care what produced the signal.

Key takeaways

• Stop looking for one percentage. Decompose instead — the channels have different sizes for different strategies.
• Selection is the channel people miss and the one that can account for the entire result.
• Leakage is not decay, it is fabrication. Detect it and discard.
• Move size beats hit rate across most of the realistic range, and it is easier to change.
• Capacity appears exactly when you scale up, which is when confidence is highest.
• Test in order. Costing a strategy that never existed is wasted precision.

Frequently asked questions

Why does a strategy perform worse live than in backtest?

Four separate channels remove performance, and they compound. Selection inflates the result you chose to deploy, leakage inflates the result the model could achieve, cost removes a fixed amount per trade, and capacity removes more as size grows. Only the third is usually modelled.

How much performance is typically lost between backtest and live?

There is no reliable published figure, and any single percentage should be treated with suspicion because the loss depends on how many variants were tested, how the data was split, the turnover of the strategy and the size deployed. The decomposition here lets you estimate it for your own strategy instead.

What is the selection channel?

Reporting the best of many tested variants. It inflates results even when no edge exists: on our calculation, the expected best Sharpe ratio from 100 purely random strategies is about 2.53, and from 1,000 about 3.26. If you tested 100 variants and deployed the best, that is the number you must beat before claiming anything.

Why does the cost channel hit some strategies far harder?

Because cost is fixed per round trip while the edge scales with the size of the move being captured. A 55% hit rate on a 50 basis point move yields 5 basis points gross and dies against a 10 basis point cost. The same hit rate on a 500 basis point move yields 50 basis points and keeps 80% of its edge.

What move size does a strategy need to survive costs?

On our arithmetic, a strategy with a 55% hit rate facing a 10 basis point round-trip cost needs to capture moves larger than 200 basis points to retain half its gross edge. Below roughly 100 basis points it has no edge left at all.

Does a higher hit rate solve the cost problem?

It helps but less than people expect. On a 200 basis point move against a 10 basis point cost, a 52% hit rate is net negative, 55% keeps half its edge, and 65% keeps 83%. The move size matters more than the hit rate across most of the realistic range.

What is the capacity channel?

Your own orders move the price against you, and the effect grows with size relative to available liquidity. A backtest assumes fills at observed prices regardless of quantity, so it silently assumes infinite capacity. This is why a strategy can work at small size and stop working when scaled.

Can walk-forward testing eliminate the gap?

It addresses one channel. Walk-forward validation constrains leakage by ensuring every evaluation happens on later data, but it does nothing about selection if you ran many walk-forward tests and kept the best, and nothing about cost or capacity.

How do I estimate the gap before deploying?

Deflate the Sharpe ratio using your honest trial count, re-run with doubled trading costs, model market impact at your intended size, and check that your split was chronological. If the result survives all four, what remains is closer to a live expectation.

Is a small live sample enough to tell whether the gap is real?

Usually not. Distinguishing a genuine edge from noise takes far more observations than most people run before concluding either way, which is why the decision to stop should be set in advance as a drawdown limit rather than judged from early results.

Which channel is the largest?

It depends on the strategy, which is the point of decomposing rather than quoting one number. High-turnover strategies lose most to cost. Heavily searched strategies lose most to selection. Strategies scaled up quickly lose most to capacity.

Does this apply to AI strategies specifically?

It applies to any systematic strategy, but machine learning raises the selection channel because hyperparameter search multiplies the trial count, and raises the leakage channel because feature engineering creates more places for future information to enter.

Informational research only. Nothing on this page is financial, legal, tax or investment advice, or a recommendation to build or deploy any system. Worked examples use illustrative assumptions and are not forecasts.