Backtest Design

A backtest is a historical simulation of a trading rule: run the rule over past data, tally the trades, and report the equity curve. It is the single most persuasive artifact in quant research and the single most dangerous, because a backtest is trivially easy to make look good and genuinely hard to make honest. Good design is almost entirely about not fooling yourself, controlling the ways the simulation leaks information from the future, understates costs, or gets confused with a prediction.

A backtest is not a forecast

The deepest conceptual error is treating a backtest as evidence about the future. A backtest evaluates one rule along one realized path of history. The future is a different draw from a partly different distribution. Even a perfectly clean backtest answers only "how would this exact rule have done on this exact history?", a statement about the past. A forecast is a probabilistic statement about an unseen future draw. The bridge between them is stationarity plus out-of-sample validity, and both are fragile: markets are non-stationary, and every time you look at the backtest and adjust the rule, you spend a little of your out-of-sample budget. A backtest with a Sharpe of 3 that was selected from 500 variants is not a forecast of anything, it is the maximum of 500 noisy numbers. See Backtest Overfitting.

Point-in-time data

The simulation must only use information that was actually knowable at each decision time. This is harder than it sounds because most databases store the current, revised view of history, not the as-was view.

• Fundamentals are restated. Earnings, book value, and even revenue are revised for months after the fact. A vendor field like "Q1 EPS" carries the final restated number; a strategy that trades on it in April is using data that did not exist until the following year. Use a database with a report date / point-in-time stamp and lag every fundamental to when it was public. See Look-Ahead Bias.
• Index membership changes. Using today's S&P 500 constituents to backtest a decade of history silently drops every company that went bankrupt or was delisted, Survivorship Bias.
• Corporate actions and identifiers. Splits, spin-offs, and ticker reuse mean a naive price series can splice two different companies together. Adjust with as-of data.

The clean statement: at simulated time $t$, the strategy may consume only the sigma-algebra $\mathcal{F}_t$ of information available at $t$. Any dependence on $\mathcal{F}_{s}$ for $s > t$ is look-ahead.

Realistic costs and slippage

Gross returns are fiction; net returns are the product. Model, per trade:

$$\text{cost} \;=\; \underbrace{c_{\text{comm}}}_{\text{commission}} \;+\; \underbrace{\tfrac{1}{2}\,s}_{\text{half-spread}} \;+\; \underbrace{\kappa\,\sigma\sqrt{\frac{Q}{V}}}_{\text{market impact}} \;+\; \underbrace{c_{\text{borrow}}}_{\text{short financing}},$$

where $s$ is the bid-ask spread, $\sigma$ the volatility, $Q$ the order size, $V$ the average daily volume, and $\kappa$ an impact coefficient. The square-root impact law $\propto \sqrt{Q/V}$ is the standard empirical model: impact grows sublinearly in size but is decisive for large books.

The drag on the strategy is cost per round-trip times turnover:

$$\text{annual drag} \approx 2 \times \text{turnover} \times \text{cost per side}.$$

A rule with 50 bps round-trip cost and 100× annual turnover loses 50% a year to costs, enough to turn most "profitable" high-frequency signals negative. Always report the strategy's turnover alongside its Sharpe, and stress the cost assumption (2× is a standard shock). See Transaction Costs.

Worked example

Suppose a daily mean-reversion signal shows a gross annualized return of 18% with volatility 12%, a gross Sharpe Ratio of 1.5, and it turns over its whole book twice per day (≈500× per year). Assume 3 bps of cost per side for liquid names. Annual cost drag ≈ $2 \times 500 \times 0.0003 = 0.30 = 30\%$. Net return is $18\% - 30\% = -12\%$: a headline Sharpe of 1.5 is actually a money-loser. The gross number was never the point; the cost-adjusted, capacity-adjusted number is. This is the most common way a backtest lies, not through subtle statistics but through an omitted cost line.

Execution and timing assumptions

• Fill realism. Assuming you transact at the closing price, or at the mid, or the day's VWAP, are three different strategies. Trading "on the close" while computing the signal from the close is a one-bar look-ahead; execute at the next available price.
• No shorting where you couldn't short. Hard-to-borrow and short-sale bans matter, especially in crises exactly when the signal fires.
• Capacity. A signal that needs to trade 20% of ADV does not scale; impact swamps it. Report capacity, not just Sharpe.

Failure modes

• Silent look-ahead through restated data, same-bar execution, or full-sample normalization of features.
• Zero or flat costs, or costs that ignore turnover and size, the drag example above.
• Survivorship and delisting omissions inflating returns by 1–4% a year.
• Backtest-and-tweak loops that leak the test set into the design, converting the whole exercise into Data-Snooping Bias.
• One-path overconfidence: reporting a single equity curve with no confidence interval, when the Sharpe estimate itself has a standard error of roughly $\sqrt{(1+\tfrac12\text{SR}^2)/n}$.

In interviews

Be ready to explain, crisply, why a backtest is not a forecast, and to list the three data hygiene problems (look-ahead, survivorship, restatement) and the three cost problems (spread, impact, borrow). A favorite prompt: "Your backtest has a Sharpe of 2.5, what do you check before believing it?" A strong answer walks through point-in-time data, turnover-scaled costs, the number of variants tried (multiple testing), and out-of-sample validity via Walk-Forward Analysis or Purged & Embargoed Cross-Validation, and finishes by deflating the Sharpe for the number of trials. The mindset the interviewer is testing is skepticism toward your own results.