Backtests and overfitting
How convincing historical results are manufactured by accident or design.
A backtest asks how rules would have behaved on historical data. It is useful research, not proof. Flexible rules, repeated testing and unrealistic assumptions can fit noise and create a beautiful past that never existed in practice.
Look-ahead and survivorship bias
Look-ahead bias uses information unavailable at the decision time. Survivorship bias tests only assets that survived, excluding failures and delistings that a real investor could have owned.
Overfitting
Each parameter and filter gives a model another way to match historical noise. If hundreds of variants are tried and only the best is shown, the reported result understates the search process.
Realistic friction
Include spreads, commission, financing, slippage, borrowing constraints, contract rolls and taxes where relevant. Test whether results survive worse assumptions and different periods.
The lesson in three lines.
- 1Historical fit is not future evidence by itself.
- 2Count the full research search, not only the winner.
- 3Stress costs, parameters and market regimes.