The Backtest Traps That Fool Everyone

A backtest is not a measurement, it is the most optimistic story that can be told about a strategy that already knew it was being tested. Every bug in a backtest points the same way: it makes the number bigger. This guide takes the six failure modes that account for almost all fake alpha and, for each, gives you runnable code that first inflates the result with the bug and then corrects it, so you can see and quantify the lie. It builds directly on the vectorized skeleton from backtest from scratch; if signal → position → returns → costs isn't second nature, start there.

Everything below is plain numpy/pandas on synthetic data, so it runs offline and is fully reproducible. Paste each block into one script (they share the helper) and confirm the numbers yourself.

import numpy as np
import pandas as pd

def sharpe(r, ppy=252):
    r = pd.Series(r).dropna()
    sd = r.std()
    return np.nan if sd == 0 else (r.mean() / sd) * np.sqrt(ppy)

Trap 1, Look-ahead bias (three disguises)

Look-ahead means your position at time t used information that did not exist at t. It has three common disguises. The most brazen is same-bar execution: computing a signal from a bar's close and then "trading" that same close.

rng = np.random.default_rng(0)
n = 1500
ret = pd.Series(rng.normal(0.0002, 0.012, n),
                index=pd.bdate_range("2015-01-01", periods=n))
signal = np.sign(ret)                    # known only at the close of bar t
gross_bug = signal * ret                 # executes bar t using bar t's own return
gross_fix = signal.shift(1) * ret        # executes bar t+1 (honest)
print(sharpe(gross_bug))                 # 21.02  <- physically impossible
print(sharpe(gross_fix))                 # 0.06   <- the truth: no edge

A Sharpe of 21 from pure noise. The bug is signal * ret instead of signal.shift(1) * ret; it earns sign(ret) * ret = |ret|, i.e. money on every single bar. If your backtest posts a Sharpe above ~4, suspect this before anything else.

The second disguise is full-sample normalization, standardizing a feature with statistics computed over the entire history, which smuggles the future into the present. Here the signal is "long when price is below its average"; using the whole-sample mean leaks where price ends up.

rng = np.random.default_rng(11)
price = pd.Series(100*np.exp(np.cumsum(rng.normal(0.0, 0.01, 2000))),
                  index=pd.bdate_range("2013-01-01", periods=2000))
ret = price.pct_change(fill_method=None).fillna(0.0)
pos_bug = (price < price.mean()).astype(float).shift(1).fillna(0.0)              # full-sample
pos_fix = (price < price.expanding(60).mean()).astype(float).shift(1).fillna(0.0)  # point-in-time
print(sharpe(pos_bug*ret))    # 0.81  <- inflated by knowing the final average
print(sharpe(pos_fix*ret))    # 0.53  <- expanding (as-of) mean

The full-sample mean quietly turned a 0.53 into a 0.81, a 50% overstatement from one call to .mean(). Any z-score, min-max scale, PCA, or volatility estimate fit on the whole sample has this bug. Fit on a rolling or expanding window only.

The third disguise is restated data: today's database of "historical" fundamentals contains numbers that were revised after the fact. Ranking on the final, revised value is a leak you can't see in the code.

rng = np.random.default_rng(2)
T, N = 240, 40
final = rng.normal(0, 1, (T, N))                     # fully-revised earnings surprise
fwd_ret = 0.01*final + rng.normal(0, 0.05, (T, N))   # returns driven by the FINAL value
final = pd.DataFrame(final); fwd_ret = pd.DataFrame(fwd_ret)
pit = pd.DataFrame(final.values + rng.normal(0, 1.5, (T, N)))   # noisy as-reported vintage

def long_short(scores, rets):
    out = []
    for t in range(T):
        s, r = scores.iloc[t], rets.iloc[t]
        out.append(r[s >= s.quantile(0.8)].mean() - r[s <= s.quantile(0.2)].mean())
    return pd.Series(out)

print(sharpe(long_short(final, fwd_ret), 12))   # 4.26  <- ranks on restated numbers
print(sharpe(long_short(pit,   fwd_ret), 12))   # 2.04  <- ranks on point-in-time data

Restating the scores doubled the Sharpe (2.04 → 4.26), because the final vintage correlates perfectly with what drove returns. Only a point-in-time database, the number as it was actually reported that day, is honest. See Look-Ahead Bias for the full taxonomy.

Trap 2, Survivorship bias

Backtest a universe of names that exist today and you have silently excluded every company that went to zero. The dead names are exactly the ones that hurt.

rng = np.random.default_rng(3)
T, N = 60, 500
R = rng.normal(rng.normal(0.006, 0.004, N), 0.09, (T, N))
dead = rng.random(N) < 0.30                      # 30% of names will delist
death = rng.integers(6, T, N)
alive = np.ones((T, N), dtype=bool)
for j in np.where(dead)[0]:
    R[death[j], j] = -1.0                        # wiped out that month
    alive[death[j]+1:, j] = False                # gone thereafter
R, alive = pd.DataFrame(R), pd.DataFrame(alive)

port_full = pd.Series([R.iloc[t][alive.iloc[t]].mean() for t in range(T)]).fillna(0)  # honest
port_surv = R.loc[:, alive.iloc[-1]].mean(axis=1)                                     # biased

cagr = lambda r: (1+r).prod()**(12/len(r)) - 1
print(f"{cagr(port_full):.1%}")   # 0.3%  <- includes the names that died
print(f"{cagr(port_surv):.1%}")   # 7.8%  <- survivors only

Restricting to survivors fabricated 7.5% of annual return out of nothing. The honest strategy barely broke even; the biased one looks like a decent equity fund. Whenever you test single stocks, use a universe with delisted tickers and point-in-time constituents. See Survivorship Bias.

Trap 3, Data-snooping (multiple testing)

Try enough strategies and one will look brilliant by chance alone. If you test $N$ independent strategies with zero true edge, the best observed Sharpe Ratio does not stay near zero, its expectation grows like

$\mathbb{E}\!\left[\max_i \widehat{SR}_i\right] \approx \sigma_{SR}\sqrt{2\ln N},$

where $\sigma_{SR}\approx\sqrt{\text{ppy}/T}$ is the standard error of a zero-edge Sharpe. That $\sqrt{2\ln N}$ is the tell: the maximum climbs relentlessly with the number of trials.

rng = np.random.default_rng(4)
years = 4; T = years*252; se = np.sqrt(252/T); reps = 300
for N in [10, 100, 1000, 10000]:
    maxes = []
    for _ in range(reps):
        R = rng.normal(0, 0.01, (N, T))               # N pure-noise strategies, zero edge
        sr = (R.mean(1)/R.std(1)) * np.sqrt(252)       # annualized Sharpe of each
        maxes.append(sr.max())                          # keep only the best one
    print(f"N={N:5d}  E[best Sharpe]={np.mean(maxes):4.2f}  "
          f"E[max]/SE={np.mean(maxes)/se:4.2f}  sqrt(2 lnN)={np.sqrt(2*np.log(N)):4.2f}")

The empirical E[max]/SE tracks the prediction closely:

N (strategies tried)	E[best Sharpe]	E[max]/SE (empirical)	√(2 ln N)	refined
10	0.76	1.53	2.15	1.36
100	1.25	2.50	3.03	2.37
1,000	1.64	3.28	3.72	3.12
10,000	1.95	3.90	4.29	3.74

(The "refined" column is $\sqrt{2\ln N}-\tfrac{\ln\ln N+\ln 4\pi}{2\sqrt{2\ln N}}$; $\sqrt{2\ln N}$ is the leading term and an upper envelope.) Test 1,000 noise strategies and the winner shows an annualized Sharpe of 1.6, a number most people would trade. And it is entirely fake:

rng = np.random.default_rng(5)
R_is, R_oos = rng.normal(0,0.01,(1000,T)), rng.normal(0,0.01,(1000,T))
b = (R_is.mean(1)/R_is.std(1)).argmax()
print((R_is[b].mean()/R_is[b].std())*np.sqrt(252))    # 2.07  in-sample winner
print((R_oos[b].mean()/R_oos[b].std())*np.sqrt(252))   # -0.54 out-of-sample

The champion of 1,000 goes from 2.07 in-sample to −0.54 out-of-sample. This is why a raw Sharpe is meaningless without knowing how many strategies were tried to find it. The fix is to deflate the Sharpe for the number of trials, the The Deflated Sharpe Ratio does exactly this, and to keep a genuinely untouched hold-out. See Data-Snooping Bias and p-values and Multiple Testing.

Trap 4, Ignoring transaction costs

Costs are paid on trades, not holdings, so a signal that flips often can be a paper tiger. Here is a real (synthetic) reversal edge on a mean-reverting series, positive gross, but it turns over the whole book daily.

rng = np.random.default_rng(60)
n = 3000
eps = rng.normal(0, 0.01, n); r = np.zeros(n)
for t in range(1, n):
    r[t] = -0.06*r[t-1] + eps[t]                  # mild negative autocorrelation
ret = pd.Series(r, index=pd.bdate_range("2011-01-01", periods=n))
pos = (-np.sign(ret.shift(1))).fillna(0.0)         # fade yesterday (honest, shifted)
gross = pos*ret
turn = pos.diff().abs().fillna(0.0)
for bps in [0, 2, 3, 5, 10]:
    print(bps, round(sharpe(gross - turn*bps/1e4), 2))
# 0->0.50  2->0.17  3->0.01  5->-0.32  10->-1.14   (turnover ~1.03/day)

Gross Sharpe 0.50, tradeable-looking. But you flip roughly the entire position every day, so the break-even cost is only 3.1 bps per unit of turnover. At a realistic retail 5 bps one-way, the Sharpe is −0.32; at 10 bps it is −1.14. The strategy is dead on arrival, and you would never know if you reported gross. Always charge costs on position.diff().abs(), and remember that real Market Impact grows with size, a strategy that survives at $10k can be uninvestable at scale (Portfolio Capacity). See Transaction Costs.

Trap 5, In-sample overfitting

Sweep a parameter grid, keep the best, and report its backtest, you have fit noise. To see the pure snooping premium, optimize a moving-average crossover in-sample and evaluate the winner out-of-sample, averaged over 30 independent synthetic histories with no real edge.

n = 2000
grid = [(lb, sl) for lb in range(2, 60, 2) for sl in range(20, 220, 10) if lb < sl]  # 530 pairs
is_best, oos_best = [], []
for seed in range(30):
    rng = np.random.default_rng(seed)
    price = pd.Series(100*np.exp(np.cumsum(rng.normal(0.0001, 0.01, n))))
    ret = price.pct_change(fill_method=None).fillna(0.0); split = n//2
    def ss(lb, sl, sub):
        pos = (price.rolling(lb).mean() > price.rolling(sl).mean()).astype(float).shift(1).fillna(0)
        return sharpe((pos*ret)[sub])
    scores = {g: ss(*g, slice(0, split)) for g in grid}
    best = max(scores, key=scores.get)
    is_best.append(scores[best]); oos_best.append(ss(*best, slice(split, n)))
print(round(np.mean(is_best), 2), round(np.mean(oos_best), 2))   # 0.65  0.31

The winner averages 0.65 in-sample but only 0.31 out-of-sample, roughly half the Sharpe was manufactured by the optimization itself, on data with no underlying edge. A single sharp optimum in a parameter sweep is overfitting; a broad plateau is a real effect. Prefer Walk-Forward Analysis evaluation and, for machine-learning pipelines, purged, embargoed cross-validation. See Backtest Overfitting.

Trap 6, P-hacking the train/test split

Even the sacred out-of-sample test can be gamed: try many split points and report the boundary that flatters you. One fixed noise strategy, evaluated over 80 candidate splits:

rng = np.random.default_rng(8)
price = pd.Series(100*np.exp(np.cumsum(rng.normal(0.0001, 0.01, 3000))))
ret = price.pct_change(fill_method=None).fillna(0.0)
strat = (price.rolling(20).mean() > price.rolling(100).mean()).astype(float).shift(1).fillna(0) * ret
oos = pd.Series([sharpe(strat.iloc[cut:]) for cut in range(300, 2700, 30)])
print(round(oos.min(),2), round(oos.median(),2), round(oos.max(),2))   # -0.21  0.10  1.07

The same strategy's out-of-sample Sharpe ranges from −0.21 to 1.07 depending only on where you cut. A p-hacker reports 1.07; the honest number is the median, 0.10. Choose the split once, before looking at results, and never let a hold-out influence a design decision more than once, the moment you iterate against it, it becomes training data.

The through-line

Trap	Bug → Fix	Damage
Same-bar execution	signal*ret → signal.shift(1)*ret	Sharpe 21 → 0
Full-sample normalization	.mean() → .expanding().mean()	Sharpe 0.81 → 0.53
Restated fundamentals	final → point-in-time	Sharpe 4.3 → 2.0
Survivorship	survivors → full universe	+7.5%/yr fabricated
Data-snooping	raw → deflated Sharpe	best-of-1000: 2.1 → −0.5 OOS
Ignored costs	on holdings → on turnover	Sharpe 0.50 → −0.32
In-sample fit	optimize → walk-forward	0.65 → 0.31 OOS
P-hacked split	best cut → fixed cut	1.07 → 0.10

Every fix makes the number worse, and that is the point: honest backtesting is the discipline of removing your own advantages until only the strategy is left. Treat every impressive result as guilty until proven innocent, re-derive it with the fixes above before you believe it, and certainly before you trade it. The natural next step is the gap between even an honest backtest and reality: paper trading, broker APIs, and live execution.