How to tell if your backtest is lying to you

Your backtest is lying if the results fall apart on a different year, pair or date range. That is overfitting: the strategy has memorised past prices instead of finding a real edge.

Most traders don't lose money because their strategy failed in the market. They lose because it failed in the test and they never noticed.

A backtest can look incredible. A smooth equity curve climbing left to right, a win rate that makes you feel clever. And it can still be worthless, because the strategy never found an edge. It just memorised the past. That's overfitting, and it's the single most common way a 'great' system turns into a losing one the moment real money is on it.

Here is how to catch it before it catches you.

What overfitting actually is

Overfitting is when a strategy is tuned so tightly to one stretch of historical price that it is really just describing what already happened, not finding a pattern that repeats. Give a computer enough rules and enough freedom and it will always find something that worked on the data in front of it. That something usually means nothing.

Four signs your backtest is lying to you

1. It only works on the exact data it was built on. If the results fall apart the moment you test a different year, a different pair, or a slightly different date range, you didn't find an edge. You found a coincidence.

2. Small changes break it. Nudge the stop by a few pips or shift an entry by one candle. If the whole thing collapses, it was balanced on a knife edge that the live market will never respect.

3. It is suspiciously perfect. Very high win rates, almost no drawdown, a curve with no rough patches. Real edges are messy. Perfection is the tell.

4. It has never seen new data. If every number you are looking at comes from the same data the strategy was built on, you have learned nothing about whether it survives. You have only learned that it can memorise.

How to actually test it

The fix is boring, which is exactly why most people skip it.

Hold data back. Build the strategy on one period, then test it on a period it has never seen. This is called out of sample testing, and it is the closest thing to an honest answer you will get without going live.

Walk it forward. Keep retesting on rolling chunks of fresh data, the way time actually moves, instead of judging it on one lucky window.

Stress it. Change the conditions. Shuffle the trades. See what breaks. The strategies worth trading are the ones that bend instead of shatter.

If a strategy still holds up after all that, you have something worth a closer look. If it doesn't, you just saved yourself the tuition the market charges for finding out the hard way.

The shortcut

Doing all of this by hand is slow, and that is why most people don't bother. It is also exactly what we built Nebula to do for you. It generates strategies, tests them on data they have never seen, stress tests them, and gives each one a score from 0 to 100 for how much of the result looks like a genuine edge versus luck. It even shows you the ones that fall apart, because knowing what doesn't hold up is how you learn to read what might.

It is free to run. No account, no card. If you build your own systems, it is worth a look: download Nebula free.