Visualizers
Overfitting Visualizer
Fit ever-wigglier curves to noisy data and watch a good model turn into nonsense. The most important lesson in quant research, made obvious.
Overfitting visualizer
Fit a polynomial through noisy data. Turn up the flexibility and watch it start chasing noise instead of signal.
Train error only ever falls. Test error bottoms out at the sweet spot, then explodes as the model memorises noise.
A model flexible enough can pass through every training point exactly, but that usually means it has memorised the random noise rather than the real pattern underneath. That is why the wiggliest curves score a near-perfect train error yet perform terribly on fresh data, where the test error blows up. The sweet spot is a model just flexible enough to capture the signal and no more, and this trap, overfitting, is the single biggest danger in quant research and backtesting.
Learn how it works
Five worked examples. Read a couple before you dive in, try to answer first, then reveal the solution.
A straight line is too simple
Generate wiggly, curved data, then fit a degree-1 polynomial (a straight line). Look at both the train and test error.
Show solution
A straight line can't follow the curves, so it slices right through the middle and misses the shape.
- Train error: high. The line fails to match even the data it was built on.
- Test error: high. It's just as wrong on new data.
This is underfitting, the model is too rigid to capture the real pattern. When both errors are bad, your model is too simple, not too complex.
The sweet spot
Bump the degree up to about 3, matching the natural bendiness of the data. Check the test error now.
Show solution
The curve now follows the true shape of the data, gentle bends in the right places, ignoring the random jitter.
- Train error: low. It fits the data well.
- Test error: low, and this is the lowest it will get.
This is the sweet spot: complex enough to catch the real signal, simple enough to ignore the noise. The goal isn't the fanciest model, it's the one that does best on data it hasn't seen.
Wiggling through every point
Crank the degree up to 12 and watch the curve contort itself through the training dots.
Show solution
The curve now snakes through every single training point, whipping up and down wildly in between them.
- Train error: nearly zero. It nailed every point it was shown.
- But between the points the curve is nonsense, huge swings that no real pattern would ever make.
It has stopped learning the trend and started memorizing the exact dots, noise and all. A perfect score on training data is a warning sign, not a trophy.
The overfitting signature
Slowly increase the degree from 1 to 14 and watch the two error lines, train and test, move together, then split apart.
Show solution
You'll see the classic U-shape:
- Train error falls steadily the whole way, more flexibility always fits the training data better.
- Test error falls at first, bottoms out at the sweet spot, then turns around and explodes upward.
That widening gap, train getting better while test gets worse, is the unmistakable signature of overfitting. The best degree is the bottom of the test-error U, not the far right.
The backtesting trap
Compare the degree-3 fit against the degree-12 fit side by side, and imagine each is a trading strategy fit to past prices.
Show solution
The degree-12 curve fits history perfectly and would look like a dream backtest, yet it's the one that falls apart on new data.
- A model that matches your past data flawlessly has usually memorized noise, not discovered signal.
- Real market signal is faint; a flexible-enough model will always find fake patterns in the randomness.
This is the number-one danger in backtesting: a strategy that looks amazing on history because it was over-tuned to it, then loses money live. Favor the simpler fit that generalizes.
What you'll learn
Why a model that fits your data perfectly is usually worthless out of sample, the difference between capturing signal and memorizing noise.