Part II: Deep Learning
"Understanding what is really happening inside a neural network — not just how to call the API — is what separates a senior practitioner from someone who copies tutorials."
Why This Part Exists
Deep learning has transformed machine learning. Transformers power language models, recommendation systems, and scientific computing. Convolutional networks process images and spatial data. Graph neural networks handle relational structures. Generative models create synthetic data and power creative applications. Foundation models have changed the economics of ML: for many tasks, the question is no longer "how do I train a model?" but "how do I adapt a pretrained model?"
This part builds deep learning from first principles. You will implement a neural network from scratch in numpy before touching PyTorch. You will derive backpropagation from the chain rule, not memorize it as a black box. You will trace the evolution from perceptrons through transformers and understand why each architectural innovation solved a specific problem. And you will learn the modern workflow — transfer learning, foundation models, and parameter-efficient adaptation — because this is how deep learning is actually practiced.
Nine chapters cover the full landscape: fundamentals, training craft, CNNs, RNNs, transformers, LLMs, generative models, transfer learning, and graph neural networks.
Chapters in This Part
| Chapter | Focus |
|---|---|
| 6. Neural Networks from Scratch | MLP implementation in numpy and PyTorch, backpropagation derivation |
| 7. Training Deep Networks | Initialization, normalization, regularization, learning rate schedules |
| 8. Convolutional Neural Networks | Convolution derivation, architecture evolution, residual connections |
| 9. Recurrent Networks | RNNs, LSTMs, GRUs, attention mechanisms as transformer precursors |
| 10. The Transformer Architecture | Self-attention derivation, multi-head attention, positional encoding |
| 11. Large Language Models | Architecture, training pipeline, fine-tuning, RAG |
| 12. Generative Models | VAEs, GANs, diffusion models — derivation and implementation |
| 13. Transfer Learning and Foundation Models | Two-tower models, contrastive learning, modern DL workflow |
| 14. Graph Neural Networks | Message passing, GCN, GraphSAGE, GAT, PyTorch Geometric |
Progressive Project Milestones
- M2 (Chapter 6): Build a click-prediction MLP for StreamRec — first in numpy, then PyTorch.
- M3 (Chapter 8): Build content embeddings using 1D CNNs over item descriptions.
- M4 (Chapter 10): Replace the LSTM session model with a transformer.
- M5 (Chapter 13): Build the two-tower retrieval model with pretrained encoders and contrastive learning.
- M6 (Chapter 14): Model the user-item interaction graph with GNN-based collaborative filtering.
Prerequisites
Part I, especially Chapters 1-3 (linear algebra, calculus, probability). Chapter 4 (information theory) is needed for Chapter 12 (generative models).
Chapters in This Part
- Chapter 6: Neural Networks from Scratch
- Chapter 7: Training Deep Networks — Initialization, Batch Normalization, Dropout, Learning Rate Schedules, and the Dark Art of Making It Converge
- Chapter 8: Convolutional Neural Networks — Architecture, Intuition, and Computer Vision Applications
- Chapter 9: Recurrent Networks and Sequence Modeling — RNNs, LSTMs, GRUs, and Their Limitations
- Chapter 10: The Transformer Architecture — Attention Is All You Need (and Why It Changed Everything)
- Chapter 11: Large Language Models — Architecture, Training, Fine-Tuning, RAG, and Practical Applications
- Chapter 12: Generative Models — VAEs, GANs, Diffusion Models, and the Frontier of Data Generation
- Chapter 13: Transfer Learning, Foundation Models, and the Modern Deep Learning Workflow
- Chapter 14: Graph Neural Networks and Geometric Deep Learning — When Your Data Has Structure Beyond Grids and Sequences