Long Short-Term Memory: Teaching a Network to Remember What Matters

Financial data is a sequence. What happened yesterday depends on what happened last week, which depends on what happened last month. Volatility clusters over weeks. Trends build over months. Regimes persist for years.

So if you want a neural network to model financial data, you need one that can remember.

The obvious architecture for this is a recurrent neural network: a network that processes a sequence one step at a time, carrying an internal "state" forward from step to step, updating it as new information arrives. In principle it can remember anything. In practice, before 1997, it could remember almost nothing, and the reason it could not was one of the great obstacles in machine learning.

The problem: the vanishing gradient

Here is the failure, and it is worth understanding because it is deep.

A neural network learns by backpropagation: you compute how wrong the network's output was, and then you trace that error backwards through the network, working out how much each internal connection contributed to the mistake, and nudging each one accordingly.

In a recurrent network, "backwards" means backwards in time. If the network made an error at step 100, and the cause of that error was something that happened at step 1, then the error signal has to travel back through 99 steps of the network to reach the connection that needs adjusting.

And at every single step of that journey, the error signal gets multiplied by a number.

If those numbers are slightly less than one, the signal shrinks a little at every step. Multiply by 0.9, ninety-nine times, and the signal that arrives is about one thirty-thousandth of what set out. It has, for practical purposes, vanished. The connection responsible for the mistake receives no meaningful instruction to change. The network is structurally incapable of learning that a cause a hundred steps ago produced an effect today.

And if the numbers are slightly greater than one, the opposite happens: the signal explodes into a meaningless enormous number, and training blows up.

Sepp Hochreiter had diagnosed this in his earlier thesis work. It is called the vanishing gradient problem, and it meant that recurrent networks could only learn dependencies spanning about ten steps. Anything further back was invisible. For financial time series, where the interesting structure spans hundreds of steps, that is a fatal limitation.

The key idea via analogy: a conveyor belt with gates

Hochreiter and Schmidhuber's solution is to redesign the memory itself.

The core insight: the gradient vanishes because it gets multiplied at every step. So build a path through time on which it is not multiplied.

They introduce a memory cell: a piece of internal state that, by default, simply passes itself forward unchanged. Not transformed, not squashed through a non-linearity, not multiplied by a learned weight. Just carried forward, as-is. They called this the constant error carousel, and it is the heart of the paper. Because the cell's value is passed along untouched, the error signal flowing backwards through it is also untouched. It can travel a thousand steps back without shrinking. The vanishing gradient is not mitigated. It is structurally eliminated along that path.

The analogy: think of a conveyor belt running the length of the network's timeline. Information placed on the belt at step 1 rides all the way to step 1000 without degrading.

But a belt that never changes is just as useless as one that forgets everything, because the network needs to be able to write new things and discard stale ones. So the authors add gates: small, learned controllers that decide what happens to the belt at each step.

• The input gate decides: is what I am seeing right now worth writing onto the belt? Most of the time the answer is no, and the belt is left alone.
• The output gate decides: is what is currently on the belt relevant to the decision I have to make right now? The memory can be held without being acted on.
• The forget gate, added a few years later by Gers, Schmidhuber and Cummins, decides: is what is on the belt now stale, and should it be wiped? This turned out to be essential.

These gates are themselves learned from the data. Nobody tells the network what is worth remembering. It works out, from the training signal, which pieces of information are worth carrying forward and for how long.

That is the whole architecture: an undisturbed highway for memory, plus learned gates controlling access to it.

Why it mattered

• It made long-range sequence learning possible. LSTMs could learn dependencies spanning hundreds or thousands of steps, where previous recurrent networks managed about ten. That is not an incremental improvement, it is the difference between the tool being useless and useful.
• It dominated sequence modelling for two decades. Speech recognition, machine translation, handwriting recognition, text generation: from the mid-2000s until the transformer arrived in 2017, LSTMs were the state of the art in essentially every sequence task. If you used a smartphone keyboard's autocomplete or a voice assistant in that era, you were using this paper.
• It is the standard backbone of deep learning on financial time series. The DeepLOB model of Zhang, Zohren and Roberts uses convolutional layers to read the shape of the order book at each instant and then an LSTM to track how that shape evolves over time. That division of labour, convolutions for structure, LSTM for memory, became the default recipe for market data.
• The gating idea outlived the architecture. Gated recurrent units, highway networks, residual connections in very deep networks, and the attention mechanisms in transformers all descend, conceptually, from the insight that you should build an unobstructed path for information (and therefore for gradients) and control it with learned gates.

The honest limitations

• It is inherently sequential, which makes it slow. You cannot process step 500 until you have processed step 499. This means an LSTM cannot exploit parallel hardware efficiently across the time dimension, which is exactly the constraint that transformers were designed to escape, and it is the main reason they displaced LSTMs in language modelling.
• Transformers generally beat it now, given enough data. For most sequence tasks with abundant data, attention-based models outperform LSTMs. LSTMs remain competitive for shorter sequences and smaller datasets, which, notably, describes a lot of financial applications.
• Applying it to financial time series is dangerous in the usual ways. An LSTM has a great many parameters and is extremely flexible. Financial return data is short, non-stationary, and mostly noise. That combination is exactly the recipe for a network that memorizes the training period beautifully and predicts nothing. Every warning in the backtest-overfitting literature applies with double force here.
• It has no concept of the arrow of time in your data pipeline. The architecture respects sequence order internally, but nothing stops you from feeding it a training set that leaks the future. Financial deep learning is littered with LSTM results that turned out to be look-ahead bias.
• It is a black box, and a recurrent one at that. Understanding why an LSTM made a prediction means untangling what got written to a memory cell three hundred steps ago and why it was still there. In practice, nobody can do this.
• Markets are non-stationary in a way that memory does not fix. An LSTM learns what patterns were worth remembering in the training period. If the market's structure changes, the network is faithfully remembering the wrong things.

The one-line takeaway

Hochreiter and Schmidhuber solved the vanishing gradient problem by building a memory cell that carries information forward untouched, like a conveyor belt, with learned gates controlling what gets written to it and what gets read from it, making it possible for a neural network to learn dependencies spanning hundreds of time steps and giving financial deep learning its standard engine for sequences.