Chapter 9: Key Takeaways
-
The vanilla RNN shares parameters across time but suffers from a fundamental mathematical flaw. The gradient of the loss at time $T$ with respect to the hidden state at time $k$ involves a product of $(T - k)$ Jacobian matrices, each bounded by the spectral radius of $\mathbf{W}_{hh}$ times the $\tanh$ derivative. When this spectral radius is less than 1 (which it almost always is), the product shrinks exponentially: a gradient signal at step 100 is attenuated by five or more orders of magnitude by the time it reaches step 1. This is not a training trick problem or a hyperparameter problem — it is a structural consequence of composing contractive functions, and it limits vanilla RNNs to dependencies of roughly 10-20 time steps.
-
The LSTM solves the vanishing gradient problem with additive cell state updates, not architectural complexity. The cell state equation $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$ creates a gradient path where $\frac{\partial \mathbf{c}_t}{\partial \mathbf{c}_{t-1}} = \text{diag}(\mathbf{f}_t)$ — no weight matrix, no $\tanh$ derivative, just the forget gate. When the forget gate is near 1, the gradient flows through the cell state like water through a pipe, regardless of sequence length. The three gates (forget, input, output) are not arbitrary complexity — each solves a specific information flow problem, and the additive update is the critical design choice that makes long-range learning possible.
-
The GRU is a simpler LSTM that usually performs comparably. By merging the cell state and hidden state and coupling the forget and input gates through an interpolation $(1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$, the GRU achieves 75% of the LSTM's parameter count with similar performance on most benchmarks. The LSTM retains an advantage when the task requires independently controlling memory retention and memory writing — the GRU's coupled gate cannot do both simultaneously. In practice, the LSTM-vs-GRU choice matters less than hidden size, number of layers, and learning rate.
-
Attention mechanisms solve the information bottleneck by letting the decoder look at all encoder states, not just the final one. The seq2seq encoder-decoder compresses an entire input sequence into a single fixed-size vector — a lossy operation that degrades with input length. Bahdanau attention computes a dynamic, input-dependent context vector at each decoding step by weighting all encoder hidden states. Luong attention simplifies this with multiplicative scores. The scaled dot-product variant of Luong attention is exactly the attention mechanism in transformers (Chapter 10). Understanding attention as "solving the seq2seq bottleneck" is the key to understanding why transformers generalize it to self-attention.
-
Transformers dominate most sequence tasks, but LSTMs remain the right tool in specific settings. For streaming/online inference (constant per-step cost), edge deployment (no attention matrix computation), and quick baselines (fast to train, stable hyperparameters), LSTMs are practical and sufficient. The choice is not ideological — it is engineering. The right question is always: "Does the transformer's improvement on this task justify its computational cost in this deployment context?"
-
The trajectory from RNN to LSTM to attention to transformer is a single intellectual thread. Each innovation solves a specific, well-defined problem: parameter sharing across time (RNN), vanishing gradients (LSTM), information bottleneck (attention), sequential processing bottleneck (transformer). Understanding this thread — not just the final architecture — gives you the tools to evaluate future innovations and to know when older architectures are still appropriate. This is the "Fundamentals > Frontier" principle in action.