Overfitting and Data Snooping: White's Reality Check, Haircut Sharpe Ratios and the Deflated Sharpe Ratio

Overfitting is tuning a trading strategy to the noise in historical data, so that its backtest looks better than its true edge; data snooping is the wider problem of testing many ideas on the same data and reporting the winner. Because the best of many worthless strategies will show a high Sharpe ratio by chance alone, a backtest can only be judged once you know how many versions were tried, which is exactly what White’s Reality Check, Harvey and Liu’s Sharpe ratio haircuts and Bailey and López de Prado’s Deflated Sharpe Ratio correct for.

In short: with five years of daily data, the best of 100 strategies with zero skill has an expected annualized Sharpe ratio of about 1.13, and the best of 1,000 about 1.46. In our simulation, optimizing 131 moving-average crossover rules on pure random walks produced an average best in-sample Sharpe ratio of 0.84 and an average out-of-sample Sharpe ratio of 0.06. Halbert White’s own example found a technical-indicator model that called the S&P 500’s direction 54.7% of the time with a naive p-value of 0.0036; after accounting for the 3,654 models searched, the p-value was 0.204. The number of trials is the single most important fact missing from almost every backtest you will be shown.

This article explains statistical methods with worked examples computed in Python. It does not evaluate any real strategy, provider or product.

A common misconception: “it was optimized on ten years of data and tested out of sample”

Expert advisor sellers and signal services often present an optimized backtest as if long history and an out-of-sample check settle the question. They don’t. If a developer tries 500 parameter sets and keeps the best, ten years of data simply gives the noise more room to produce a lucky winner. And an out-of-sample test only works once. As Bailey and López de Prado put it in “The Deflated Sharpe Ratio” (Journal of Portfolio Management, vol. 40, no. 5, 2014): “If we apply the holdout method enough times (say 20 times for a 95% confidence level), false positives are no longer unlikely: They are expected.” The developer who tries ideas until one survives the holdout has turned the holdout into more in-sample data. (The broader differences between simulated and live results are covered in backtests vs live results.)

The multiple testing problem

A single test at the 5% significance level has a 5% chance of a false positive. Test N independent strategies, none with any edge, and the chance that at least one passes is

1 - (1 - 0.05)^N

which is 40.1% for N = 10, 64.2% for N = 20 and 99.4% for N = 100. Campbell Harvey and Yan Liu give the same N = 10 figure in “Backtesting” (Journal of Portfolio Management, vol. 42, no. 1, 2015): a strategy with a single-test p-value of 0.05, picked from ten, has a multiple-testing p-value of 0.401.

The same logic applies to Sharpe ratios. Under the null hypothesis of no skill, the Sharpe ratios of N backtests scatter around zero, and the maximum of them grows with N. Bailey and López de Prado give an approximation for the expected maximum, using extreme value theory:

E[max SR] = sqrt(V) x ( (1 - gamma) x Z^-1(1 - 1/N) + gamma x Z^-1(1 - 1/(N x e)) )

where V is the variance of the Sharpe ratio estimates across trials, gamma (about 0.5772) is the Euler-Mascheroni constant, Z^-1 is the inverse standard normal distribution and e is Euler’s number.

Worked example 1: the best of N worthless backtests

For five years of daily data (1,250 observations), the standard deviation of an annualized Sharpe ratio estimate under the null is about sqrt(1/5) = 0.447. We compared the approximation with the exact expected maximum of N independent normal estimates (computed by numerical integration) and with a direct simulation of N strategies whose daily returns are pure noise:

Number of independent backtests	Expected best Sharpe (approximation)	Expected best Sharpe (exact)	Average best Sharpe (simulation)
1	0.00	0.00	0.03
10	0.70	0.69	0.67
50	1.02	1.01	1.02
100	1.13	1.12	1.13
1,000	1.46	1.45	1.47
10,000	1.73	1.72	not run

The approximation is accurate to about 0.01 across the range. A Sharpe ratio above 1 over five years looks impressive. It is roughly what you should expect from the best of 50 strategies with no edge at all.

David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu turn this into a rule of thumb in “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance” (Notices of the American Mathematical Society, vol. 61, no. 5, 2014). Their minimum backtest length is the number of years needed so that the best of N skill-less strategies is not expected to show an in-sample Sharpe ratio of 1. In their words, “if only 5 years of data are available, no more than 45 independent model configurations should be tried.” Our calculation from their formula:

Independent configurations tried	Years of data needed so the expected best Sharpe is only 1.0
10	2.5
45	5.0
100	6.4
1,000	10.6

They also give a simple upper bound: minimum backtest length is less than 2 ln(N) divided by the square of the target Sharpe ratio.

White’s Reality Check (2000)

Halbert White, in “A Reality Check for Data Snooping” (Econometrica, vol. 68, no. 5, 2000, pp. 1097-1126), defines the problem precisely: “Data snooping occurs when a given set of data is used more than once for purposes of inference or model selection.” His contribution is a practical test of the null hypothesis that “the best model encountered in a specification search has no predictive superiority over a given benchmark model.”

The idea is to test the maximum, not the winner in isolation. White computes each model’s performance relative to the benchmark, then uses the stationary bootstrap of Politis and Romano (1994) to resample the data many times, recomputing all models’ performance on each resample. The distribution of the best performance across all models under the null gives the Reality Check p-value. Searching more models raises the bar automatically.

His illustration forecasts daily S&P 500 returns one day ahead. From 29 technical indicators (lagged returns, momentum measures, local trends, relative strength indexes and moving-average oscillators), he builds every linear model using exactly three of them: 3,654 models, evaluated from June 3, 1991 to May 31, 1994 against a benchmark containing only a constant.

Performance measure	Best model	Benchmark	Naive p-value (best model alone)	Reality Check p-value
Prediction mean squared error	RMSE 0.006373	RMSE 0.006410	0.1068	0.3674
Direction correct	54.7%	50.8%	0.0036	0.2040

For the directional model, White writes: “This example dramatically illustrates the dangers of data mining. The naive p-value is .0036! Anyone relying on this would be seriously misled.” A 54.7% hit rate on 758 daily forecasts sounds like an edge, and in isolation it would pass any conventional test. Against the backdrop of the 3,653 other models that were also tried, it is unremarkable.

White’s Reality Check was later applied to large universes of technical trading rules by Sullivan, Timmermann and White (“Data Snooping, Technical Trading Rule Performance, and the Bootstrap,” Journal of Finance, 1999); the evidence on technical rules in currency markets specifically is reviewed in do technical trading rules work in FX.

Harvey and Liu: haircut Sharpe ratios

Harvey and Liu start from industry practice: “A common practice in evaluating backtests of trading strategies is to discount the reported Sharpe ratios by 50%.” Their aim is to replace the rule of thumb with a calculation. The steps:

Convert the Sharpe ratio to a t-statistic: for T monthly observations, t = annual SR x sqrt(T / 12).
Compute the single-test p-value.
Adjust it for the number of tests, using Bonferroni, Holm or the Benjamini, Hochberg and Yekutieli (BHY) method.
Convert the adjusted p-value back into a t-statistic and then a haircut Sharpe ratio.

Their simplest example assumes independent tests: with 20 years of monthly returns (T = 240), an annual Sharpe ratio of 0.75 has a single-test p-value of 0.0008. With N = 200 tests, the multiple-testing p-value is 0.15, and the haircut Sharpe ratio is 0.32, a cut of about 60%. Our implementation reproduces this (0.0008, 0.147 and 0.324).

Worked example 2: haircuts by Sharpe ratio and number of tests

Same independent-test method, T = 240 months:

Reported annual Sharpe	Haircut Sharpe, 10 tests	100 tests	200 tests	1,000 tests
0.50	0.27 (46% cut)	0.02 (96%)	0.00 (100%)	0.00 (100%)
0.75	0.59 (21%)	0.40 (47%)	0.32 (57%)	0.13 (82%)
1.00	0.88 (12%)	0.75 (25%)	0.71 (29%)	0.60 (40%)
1.50	1.42 (5%)	1.34 (11%)	1.32 (12%)	1.26 (16%)

The haircut is nonlinear, which is Harvey and Liu’s central finding: “The highest Sharpe ratios are only moderately penalized, while the marginal Sharpe ratios are heavily penalized.” In their full method, which models correlation between tests using the factor-testing record documented by Harvey, Liu and Zhu (at least 316 factors tested to explain equity returns), they report that the haircut is “almost always more than and sometimes much larger than 50%” when the annual Sharpe ratio is below 0.4, and at most 25% when it is above 1.0. They advocate the BHY method, which controls the false discovery rate rather than the probability of any false discovery.

The Deflated Sharpe Ratio

Bailey and López de Prado’s Deflated Sharpe Ratio (DSR) combines two corrections. It starts from their Probabilistic Sharpe Ratio, which gives the probability that the true Sharpe ratio exceeds a threshold after adjusting for sample length, skewness and kurtosis. The DSR sets that threshold equal to the expected maximum Sharpe ratio from N trials (the formula above), rather than zero:

DSR = Z( (SR - SR0) x sqrt(T - 1) / sqrt(1 - g3 x SR + (g4 - 1)/4 x SR^2) )

where SR is the selected strategy’s per-period Sharpe ratio, SR0 is the expected maximum under no skill, T is the number of observations, g3 is skewness and g4 is kurtosis.

Their numerical example: a strategist backtests variations of a Treasury auction seasonality idea and finds one configuration with an annualized Sharpe ratio of 2.5 over five years of daily data. The investor asks for four disclosures: the number of independent trials, the variance of the backtest results, the sample length and the skewness and kurtosis of the returns. The answers are N = 100, a variance of trial Sharpe ratios of 0.5 (annualized), T = 1,250, skewness -3 and kurtosis 10. The paper concludes there is “only a 90% chance that the true SR associated with this strategy is greater than zero,” below the 95% bar, so the investor declines. Our implementation reproduces every number in the example:

Independent trials (N)	Expected best Sharpe under no skill (annualized)	DSR with actual skew and kurtosis	DSR if returns were normal
1	0.00	1.000	1.000
10	1.11	0.994	0.999
46	1.59	0.9505	0.979
88	1.76	0.910	0.9505
100	1.79	0.9004	0.943
1,000	2.30	0.640	0.670

The paper notes that with N = 46 trials the DSR would have been 0.9505 and the investor might have allocated, and that with normal returns the strategy would have passed until N = 88. A Sharpe ratio of 2.5 can be a genuine discovery or a statistical artefact. Which one it is depends on facts the headline number doesn’t contain.

Worked example 3: overfitting a moving-average crossover on random walks

To see the whole process, we simulated what a strategy developer does. Generate a currency-like random walk (daily volatility 0.6%, about 9.5% a year) of 2,016 days, discard a warm-up period, leaving about seven years. Test a grid of 131 moving-average crossover rules (fast average from 2 to 50 days, slow average from 10 to 250 days, long when fast is above slow, short otherwise) on the first half, pick the best, and run it on the second half. Repeat 300 times. In a random walk, no rule has any edge.

Quantity (average over 300 random walks)	Value
Best in-sample Sharpe ratio (about 3.5 years)	0.84
Share of runs where the best in-sample Sharpe exceeded 1.0	32%
Out-of-sample Sharpe ratio of the chosen rule	0.06
Share of runs where the chosen rule lost money out of sample	41%
In-sample Sharpe of the median rule	0.01

The entire in-sample “edge” is selection. In about a third of the random walks, a developer would have found a crossover rule with an in-sample Sharpe ratio above 1, ready to sell as an expert advisor.

One detail matters for applying the DSR. If the 131 rules were independent, the expected best Sharpe ratio over 3.5 years would be about 1.41. The observed 0.84 corresponds to only about 10 independent trials, because rules with similar parameters produce highly correlated returns. The DSR paper addresses this in its appendix on estimating the number of independent trials. Getting N right is part of the method; counting every parameter combination overstates the penalty, and counting only the final rule understates it.

Worked example 4: deflating one of those backtests

Take one run in which the best rule (fast 3, slow 50) earned an in-sample annualized Sharpe ratio of 1.46 over 878 trading days, with a t-statistic of 2.72 and near-normal returns (skew -0.04, kurtosis 3.04). Following the DSR paper, the variance V in the expected-maximum formula is the measured dispersion of the 131 rules’ Sharpe ratios in this run: their standard deviation was 0.42 annualized, below the 0.54 that independent skill-less rules would show over 3.5 years, again because neighbouring rules move together. That is why the thresholds below are lower than the 0.84 and 1.41 quoted above.

Test	Threshold	Result
Probabilistic Sharpe Ratio vs zero	0	0.997
Deflated Sharpe Ratio, N = 10 effective trials	0.66 annualized	0.932
Deflated Sharpe Ratio, N = 131 (all configurations counted as independent)	1.10 annualized	0.749
Actual out-of-sample Sharpe ratio, next 3.5 years		-0.07

On its own, the backtest looks 99.7% likely to be skilled. Deflated for the search, it fails the 95% bar even with the generous count of ten effective trials. Out of sample, it lost money.

The probability of backtest overfitting

Bailey, Borwein, López de Prado and Zhu’s “The Probability of Backtest Overfitting” (circulated from 2013; published in the Journal of Computational Finance, vol. 20, no. 4, 2017) asks a different question: does the selection process itself help? They define the probability of backtest overfitting (PBO) as the probability that the configuration chosen as best in-sample performs below the median of all configurations out of sample.

Their estimator, combinatorially symmetric cross-validation (CSCV), splits the history into S equal blocks, forms every combination of S/2 blocks as the training set with the rest as the test set (for S = 16 that is 12,870 combinations), and records where each in-sample winner ranks out of sample. Every block serves as both training and test data, so no single arbitrary split drives the answer.

Their illustration optimizes a monthly seasonal rule (entry day, holding period, stop loss and side: 8,800 combinations) on 1,000 days of a random walk. The optimum had an annualized Sharpe ratio of 1.27, and its Probabilistic Sharpe Ratio statistic implied under a 1% chance that the true Sharpe was below zero. CSCV found that about 53% of the out-of-sample Sharpe ratios were negative and estimated the PBO at 55%.

We ran CSCV (S = 16) on our moving-average grid for 20 random walks, and for 20 series with a genuine slow-moving trend component in which crossover rules do have an edge:

Data	Average PBO	Range across 20 series	Average in-sample best Sharpe	Average out-of-sample Sharpe of the in-sample best
Random walk (no edge)	0.59	0.22 to 0.91	0.87	-0.07
Persistent trend (real edge)	0.59	0.13 to 0.86	1.15	0.27

The second row is instructive. With a real trend, the rules make money out of sample, yet the PBO is just as high, because picking the in-sample best adds nothing over picking a typical rule: in the matching holdout simulation, the chosen rule’s out-of-sample Sharpe ratio (0.54) was no better than the median rule’s (0.57). The paper anticipates this: a high PBO can occur when “all the N strategies have high but similar Sharpe ratios,” which is overfitting “among many ‘skillful’ strategies.” PBO measures whether optimization adds value, not whether the strategy family has an edge. The authors also warn against using CSCV to steer the search, quoting Strathern: “when a measure becomes a target, it ceases to be a good measure.”

What this means for signals, EAs and “AI” systems

Count the trials, including the invisible ones. A provider running ten channels, or a vendor who tested hundreds of expert advisor settings, has an N far above one. The record or backtest you are shown is the maximum of that set. The same applies across providers: a leaderboard is a list of maxima (see survivorship bias in signal providers).
Ask the DSR questions. How many configurations were tried? How dispersed were their results? How long is the sample? What are the skewness and kurtosis? A seller who cannot answer the first question cannot support a significance claim.
More features, more trials. Machine-learning systems that search over many inputs and architectures are data snooping at industrial scale unless the search is accounted for. Most retail “AI” claims do not say how the model was selected; see AI forex trading claims.
Single-strategy significance is not enough. Passing a t-test (see statistical significance of a track record) is the starting point. The threshold rises with every idea that was tried and discarded.

What can go wrong with the corrections

N is rarely known. The researcher may not have recorded every variant, and earlier researchers’ failed ideas are never visible. Every correction above is only as good as the trial count fed into it.
Trials are correlated. Treating 131 similar rules as 131 independent tests over-penalizes; treating them as one under-penalizes. Bootstrap methods such as White’s handle dependence directly; analytic methods need an estimate of the effective number.
They control false positives, not losses. A strategy that passes a deflated test may still be unprofitable after costs, or may stop working when the regime changes. Bailey et al. note that CSCV cannot detect overfitting to a regime that covers the entire sample.
Power is low. With realistic sample lengths, these tests also reject many genuinely skilled strategies. That is a feature of short, noisy data, not a flaw of the tests; the alternative is accepting far more false discoveries.
Monte Carlo is not a fix. Resampling the trades of an overfitted backtest only produces a confident-looking distribution of fictional results (see Monte Carlo simulation for trading systems).

The short version

When many strategies are tested on the same data, the best one looks skilled by chance: over five years of daily data, the best of 100 skill-less strategies has an expected Sharpe ratio of about 1.13. White’s Reality Check (2000) tests the best model against the bootstrap distribution of the maximum, and in his example it turned a naive p-value of 0.0036 into 0.204. Harvey and Liu (2015) convert the number of tests into a nonlinear Sharpe ratio haircut, heavy for marginal strategies and light for exceptional ones. Bailey and López de Prado’s Deflated Sharpe Ratio (2014) raises the significance threshold to the expected maximum of N trials and adjusts for skew and kurtosis. In our simulation, 131 moving-average rules on random walks produced an average best in-sample Sharpe ratio of 0.84 and an out-of-sample result of essentially zero. A backtest without a trial count cannot be evaluated.