Backtesting · Common Traps
A Beautiful Backtest That Loses Money Live: The Five Most Common Reasons
The equity curve climbs all the way up, you connect it to a live account, and two weeks later it is bleeding — probably the most deflating moment on the quant path. The first reaction is usually bad luck, or a market that refuses to cooperate, and back you go to tuning parameters. Far more often the problem is not luck but the backtest itself: that pretty curve never held up to begin with.
This piece is not about how to run a backtest (for the process itself, see How to Backtest a Strategy: Pulling Historical OHLCV With ccxt). It is about one thing: how a backtest lies to you. The five reasons are ordered by how well they hide — the later ones are the hardest to catch on your own — and each comes with one check you can run straight away.
Why a beautiful backtest deserves suspicion first
Start with a counter-intuitive rule: the prettier the backtest, the more it should be doubted rather than celebrated.
The reason is that a backtest is run in a world whose ending is already known. The history has finished playing out, and testing parameters on it over and over, rewriting rules, adding conditions, amounts to fitting yourself to the answer. The closer the fit, the better the curve looks — and future prices will not copy that stretch. A backtest actually worth something usually looks unremarkable: winners and losers, a drawdown of real size, an uncomfortable flat patch in the middle. That is what a real strategy looks like.
Of the five reasons below, the first two are technical errors — the backtest is written wrong — and the last three are sampling errors, where the data was chosen wrong.
Reason one: overfitting, taking noise for a rule
Overfitting can be put plainly: what your strategy learned is not how the market behaves, but what happened to happen in this slice of history.
It rarely arrives in one step; you slide into it. The first version performs so-so, you push the period parameter up a little and it looks better; you add a filter and it looks better again; one drawdown is particularly ugly, so you bolt on a rule that says do not open a position in that situation — and naturally the curve improves. Every step looks reasonable on its own, but strung together you have tailored a garment to this particular history with the benefit of hindsight.
Three checks:
- Out-of-sample testing. Cut the data in two, tune parameters on the first part only, then run the second part once — and only once. If performance drops sharply on the second part, overfitting is close to certain. The hard part is discipline: the moment you look at that result and go back to adjust parameters, it stops being out-of-sample.
- Parameter sensitivity. Nudge the parameters up and down a little and run again. On a healthy strategy small changes produce a gradual change in performance; if a slight move makes returns collapse, what you found is a point on the edge of a cliff, not a plateau.
- Count the parameters. Put a strategy with two or three parameters next to one carrying a dozen switches, thresholds and exception rules, run both on the same slice of data, and the second will produce a pretty curve far too easily. The more parameters, the longer and messier the sample has to be to support them.
Reason two: look-ahead bias, using what you could not have known
Look-ahead bias means the strategy used information that did not yet exist at the moment it made its decision. It is more damaging than overfitting: overfitting at least gives you something real but unreliable, while look-ahead bias empties the whole curve of meaning.
The best-hidden and most common version is using the close of the current candle to decide an entry on that same candle. The code reads: if this candle's close crosses above the moving average, buy on this candle. Nothing about that sentence looks wrong; but in reality you only learn the close once the candle has finished, and by then the candle is behind you — you cannot go back and fill somewhere in the middle of it. Filling that trade at the current candle's price means knowing the outcome in advance, every single time.
There are related patterns: normalising with a mean and standard deviation computed over the entire dataset; defining training labels by whether the following candles made a new high; or shifting a column in the wrong direction so a future row lands on the current one. What they share is that nothing errors. The backtest runs to the end, and it runs beautifully.
There is only one way to check, and it has to be asked signal by signal: at that exact second, did I really have this number in hand? One engineering habit helps a lot — shift every indicator used for a decision by one bar, so the value of the last completed candle decides what happens on this one. Better to be conservative and earn less than to build a conclusion on information you could not have had.
Reason three: survivorship bias, the ones that went to zero are missing
When picking instruments to test, you most likely open the current market list and choose a few familiar, active pairs. The trouble is the word current: whatever is on the list today is what survived.
Everything delisted over the past few years, everything whose liquidity dried up, everything whose team walked away and left it at zero, never enters your sample. So the strategy runs on a set of instruments that demonstrably did not die, and looks robust as a result. In live trading you have no such foreknowledge — what you pick could be exactly the batch that later disappears.
The bias is worst on small-cap coins and lighter on major pairs, but as long as your strategy belongs to the screen-the-whole-market family, it is there.
Historical data that includes delisted instruments is genuinely hard to find in public sources. If you cannot get it, do two things at least: fix the test instruments to a handful of pairs that have existed for a long time with good liquidity, rather than treating a screened best-combination as your conclusion; and write a line into the result stating that this backtest did not include delisted instruments, so that in a few months you have not forgotten the gap yourself.
Reason four: fees and slippage left out
This is the easiest one to verify and the one most often skipped. The profit on paper is frequently the exact size of the cost you did not count.
There are at least three layers of cost: trading fees (maker and taker are usually priced differently), slippage (the difference between the price you wanted and the price you actually got), and holding costs such as funding rates if what you are testing is futures. The higher the trading frequency, the more times those costs get multiplied — a strategy opening and closing a dozen times a day and one that moves twice a month are not in the same league for cost sensitivity.
The approach: put in the fee rate that actually applies to your account, then make a clearly conservative assumption about slippage (how much depends on the liquidity of the pair you are testing and on your own order size — work out your own, do not copy someone else's number), and see how much of the curve is left. If the returns vanish once costs are in, the strategy was never viable, and that has nothing to do with how well it gets executed. For a rough sense of what costs take, the Fee & Discount Calculator on the tools page will do the arithmetic.
Reason five: a sample period that only saw one kind of market
The last reason looks the most harmless and buries the biggest mine: your backtest covered one stretch of market with a single character.
If the test window happens to sit inside a sustained rally, then almost any strategy willing to hold will make money — not because the strategy is good, but because during that period holding anything worked. By the same token, a high win rate produced in a long flat patch can take one stop-loss after another once the market trends. So what matters about a sample period is not how many months it ran, but whether it covered markets with different characters: rising, falling, grinding sideways, ideally with a violent move or two in there.
This is the same coin as the question of where a strategy stops applying: one set of parameters can behave in completely opposite ways depending on the state of the market — which is exactly why a sample containing only one regime tells you nothing about how the strategy holds up in the others.
Five questions to ask before you read the returns
When the backtest finishes, do not look at the return figure first. Go down this table instead. In the other order, the curve convinces you first and you go looking for reasons afterwards.
| Ask yourself | What a failing answer sounds like |
|---|---|
| Which slice of data were the parameters tuned on? Has it since been run on data it never saw? | One slice throughout, with tuning and validation on the same batch |
| For every order decision, did you really have that number at the time? | Entering on the current candle's close; indicators computed over the whole dataset |
| How were the test instruments chosen? | Picking the good performers out of today's market list |
| On what basis were fees and slippage deducted? | Not deducted, or deducted using a number you cannot account for |
| Did the sample period contain a market completely unlike the present one? | One directional stretch and nothing else |
If even one of the five has no answer, the backtest is not yet something to put real money behind. The value of a backtest was never to tell you how much you can earn; it is to discard the clearly unworkable ideas early. It is a sieve, not a crystal ball. And once the sieving is done, demo trading and a small live allocation are two steps that cannot be skipped.
FAQ
A backtest with high returns — does that mean the strategy is good?
Not necessarily, and it is a reason to look harder. The history has already played out, and repeated tuning on it makes a flattering curve easy to produce. What tells you whether a strategy is credible is how it behaves on data that took no part in the tuning, whether small parameter changes make it collapse, and what is left after fees and slippage — not how high the end of the curve sits.
How do I tell whether my strategy is overfitted?
Three moves: split the data in two, tune on the first part only, then run the second part once and only once; nudge the parameters up and down and watch whether performance drifts gently or falls off a cliff; count how many parameters, thresholds and exception rules the strategy carries, since more of them is more suspicious. If two of the three look wrong, treat it as overfitting.
What is look-ahead bias, and why is it so hard to spot?
Look-ahead bias means the strategy used information it could not have had at the moment of the decision. The most common form is entering on the close of the very candle that triggered the entry — the close is only known once that candle has finished, and by then there is no going back inside it to fill. It hides well because nothing errors: the backtest runs to the end, with every trade knowing the answer in advance.
How should fees and slippage be handled in a backtest?
Use the fee rate that actually applies to your own account, bearing in mind that maker and taker are usually priced differently. Slippage has no universal figure: it depends on the liquidity of the pair and on your order size, so make an assumption clearly on the conservative side instead of copying someone else's number. The higher the trading frequency, the more times both are multiplied. A strategy whose profit disappears once costs are deducted was never viable.
How long does the sample period need to be?
The number of months matters less than whether markets of different character appear in it. Rising, falling, grinding sideways, ideally with a violent move or two — a sample like that has actually put the strategy under pressure. A backtest covering one stretch of steady upside proves the market was kind, not that the strategy works.
The backtest passed. What comes next?
Not real money yet. Run it in demo trading at a realistic pace first, to confirm that order placement, reconnection after a dropout and error handling hold up too, then trade a small amount you could lose entirely without it affecting your life, keeping an eye on the gap between live fills and the assumptions in the backtest. A backtest is a sieve: it rules out what clearly does not work, and it does not prove that what is left will make money.
Treat the five as a check-up sheet: run it every time a backtest finishes, before you look at the returns. To shore up the backtesting process itself, go to How to Backtest a Strategy: Pulling Historical OHLCV With ccxt; if quant trading itself is still fuzzy, start with What Is Quant Trading, Really? A Plain-English Intro for Beginners; and the step that comes after a backtest passes is covered in How to Use OKX Demo Trading: Smooth Out Your Strategy with Play Money First.