Backtests vs Live Results: Curve Fitting, Look-Ahead Bias, Costs and Demo Accounts

A backtest shows how a set of trading rules would have performed on past data; a live result shows what actually happened with real money at a real broker. Backtests are routinely flattered by curve fitting, look-ahead bias, unrealistic costs and perfect fills, so live results are almost always worse, and often much worse.

In short: treat any backtest as an upper bound, not a forecast. US regulators require registered advisers who show simulated results to warn that they “do not represent actual trading” and are “designed with the benefit of hindsight.” Research on overfitting shows that by trying enough variations, almost anyone can produce an impressive backtest from rules that have no real edge. Demo accounts are closer to live trading but still skip real execution. Only live, complete, verified results tell you what a subscriber would have experienced.

The pitch you have probably seen

“Backtested over 10 years: +3,400%, max drawdown 8%.” Expert advisors (automated trading programs) and signal services often lead with numbers like these. They look precise because they come from software. But a backtest is only as honest as its assumptions, and the person running it controls every one of them.

What regulators say about simulated results

In the US, CFTC Regulation 4.41(b) requires a commodity trading advisor or pool operator that presents simulated or hypothetical performance to include a prescribed disclaimer. It reads in part: “These results are based on simulated or hypothetical performance results that have certain inherent limitations. Unlike the results shown in an actual performance record, these results do not represent actual trading.” It goes on to say that such results “may have under-or over-compensated for the impact, if any, of certain market factors, such as lack of liquidity,” and that simulated programs “are designed with the benefit of hindsight.”

NFA Compliance Rule 2-29, which NFA applies to forex promotional material as well, requires its members to use an even blunter disclaimer, in capitals, which states that “THERE ARE FREQUENTLY SHARP DIFFERENCES BETWEEN HYPOTHETICAL PERFORMANCE RESULTS AND THE ACTUAL RESULTS” later achieved, and that hypothetical trading does not involve financial risk. The CFTC’s advisory on trading systems sold online adds that hypothetical results “may overestimate or underestimate the performance of a system.”

Most signal sellers aren’t registered, so nobody makes them include these warnings. The warnings still apply.

Problem 1: curve fitting (overfitting)

Curve fitting means tuning a strategy’s rules until they match the past closely, so the backtest captures noise rather than a repeatable pattern. Change a moving-average length from 20 to 23, add a filter for Tuesdays, tighten the stop to 17 pips, and keep whatever version scores best. Each tweak is a new trial, and the best of many trials looks good by chance alone.

What the research says

David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu put numbers on this in “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance” (Notices of the American Mathematical Society, 2014). They show that “if only 5 years of data are available, no more than 45 independent model configurations should be tried,” or you are almost certain to find a strategy with an annualised Sharpe ratio of 1 in the backtest and an expected Sharpe ratio of zero afterwards. With only 2 years of data, trying just 7 configurations is enough to produce the same illusion. Their conclusion is that a backtest which doesn’t report how many variations were tried makes it impossible to judge the risk of overfitting. A follow-up paper, “The Probability of Backtest Overfitting” (Journal of Computational Finance, 2017), gives a method for estimating that risk directly. The statistics are covered in depth in overfitting and data snooping.

The same paper warns that the standard disclaimer understates the problem. When the data has memory, they find that “the more we optimize IS [in-sample], the worse is OOS [out-of-sample] performance,” and that when advisers don’t control for overfitting, “good backtest performance is an indicator of negative future results.”

There is also real-world evidence. Antti Suhonen, Matthias Lennkh and Fabrice Perez studied 215 systematic strategies marketed by global investment banks (“Quantifying Backtest Overfitting in Alternative Beta Strategies,” Journal of Portfolio Management, 2017) and reported a median deterioration of about 73% in Sharpe ratios between the backtest period and live trading. These were professionally built products, not social-media signals.

Worked example: the best of many random rules

We reproduced the effect in Python. The expected best Sharpe ratio from N skill-less strategies, each tested on T years of data, is roughly the expected maximum of N standard normal draws divided by the square root of T:

Variations tried	Expected best backtest Sharpe, 2 years of data	5 years of data
7	0.98	0.62
10	1.11	0.70
45	1.58	1.00
100	1.79	1.13
1,000	2.30	1.46

Then we simulated it directly: 200 separate random-walk markets, each with 10 years of daily data. In each market we generated 100 random long-or-short rules (pure coin flips), picked the one with the best Sharpe ratio over the first 5 years, and then ran that “winner” on the next 5 years. The average best in-sample Sharpe ratio was 1.12; the average Sharpe ratio of those same winners out of sample was -0.01. The backtest looked like a respectable strategy. The rules were random.

Problem 2: look-ahead bias

Look-ahead bias is when a backtest uses information that wasn’t available at the moment the trade would have been placed. Common forms:

Using the bar’s close to trade at that bar’s open. A rule like “buy if today closes above the moving average” can’t be filled at today’s opening price.
Intrabar ambiguity. On hourly or daily bars, the test can’t tell whether the stop or the target was hit first, and some software assumes the favourable one.
Revised data. Economic numbers are often revised after release; a news-based backtest that uses revised figures knows things traders didn’t.
Choosing pairs in hindsight. Testing only on the pairs that trended strongly in the period, because you already know which ones did.

Problem 3: costs and fills

Many backtests assume a fixed or zero spread, no slippage, no commission and no overnight financing (swap). For a strategy that trades often, this alone can decide whether it makes money.

Worked example. A scalping system’s backtest shows an average gross gain of 1.5 pips per trade over 1,000 trades a year:

Assumption	Net pips per trade	Net pips per year
Backtest: no costs	+1.5	+1,500
Real average spread of 1.2 pips	+0.3	+300
Plus 0.4 pips average slippage	-0.1	-100

The same rules go from a strong backtest to a small loss without a single trade changing. Spreads also widen around news releases and at the daily rollover, exactly when many short-term systems trade. Transaction costs in retail FX has the full breakdown, including swaps.

Problem 4: demo accounts are not live accounts

Demo results sit between a backtest and a live record. They use real-time prices, so they avoid look-ahead bias. But they can still differ from live trading:

Execution. Demo fills may not reflect real liquidity, requotes or rejected orders, and a live broker’s execution policy can treat large or fast orders differently.
Slippage. Live market orders and stops can fill at worse prices than requested, especially in fast markets.
Resettable. A demo account can be reset or replaced after a bad run, and the record restarted.
Behaviour. Following a stop loss is easier when the money isn’t real.

A provider whose only record is a demo account has shown that their rules ran, not that subscribers could have earned the same result.

A checklist for any backtest

How many variations were tried before this one? If the answer is “lots” or “I don’t know,” assume overfitting.
Was any data held back and tested only once, at the end? Results on data used to design the rules don’t count as evidence.
What spread, slippage, commission and swap were assumed? Compare with your broker’s actual costs at the times the system trades.
What bar size and fill logic? Ask how the test handles a bar where both stop and target could have been hit.
Is there a live record since the backtest was published? That is the only out-of-sample test that matters to you.
How does live compare with backtest? A large gap, in either drawdown or return, tells you which number to believe.

What can go wrong

The backtest was the product. Many “systems” are sold on backtests that were optimised for the sales page.
The live period is too short. A few months of live results can’t confirm a backtest built on years of data.
Live results are themselves selected. Several live accounts, one advertised. See how to verify a signal track record.
Automation hides the details. An expert advisor runs the rules you can’t see, so you can’t check them for look-ahead or overfitting at all.

The short version

A backtest is a best case built with hindsight. Overfitting can turn random rules into an attractive equity curve, look-ahead bias can sneak future information into past trades, and missing costs can flip a profit into a loss. Demo trading avoids some of these problems but not real execution. Regulators require registered advisers to say so plainly; most signal sellers don’t. Ask for live, verified results, and treat the backtest as the most optimistic number you will ever see for that strategy.