Further Reading: Decision Trees and Random Forests

Decision trees are one of the oldest and most studied algorithms in machine learning, and random forests remain among the most widely used. Here are the resources that will deepen your understanding — from intuitive introductions to the original research papers.

Tier 1: Verified Sources

Leo Breiman, Classification and Regression Trees (originally published by Wadsworth, 1984; reprinted by Chapman & Hall/CRC, 1998). This is the book that started it all — the original CART monograph by Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone. It introduced the CART algorithm (the one scikit-learn implements) and established decision trees as a serious statistical tool. More historical and mathematical than practical, but if you want to understand where these ideas came from, this is the primary source.

Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, An Introduction to Statistical Learning with Applications in Python (Springer, 2nd edition, 2023). Chapter 8 covers tree-based methods with exceptional clarity — including bagging, random forests, and boosting. The explanations are accessible to beginners while being mathematically precise. The Python edition (ISLP) is freely available online from the authors. If you read one additional source on tree-based models, make it this one.

Andreas Mueller and Sarah Guido, Introduction to Machine Learning with Python (O'Reilly, 2nd edition, 2024). A hands-on guide to scikit-learn that covers decision trees and random forests with practical examples and excellent visualizations. The code examples are well-designed and directly applicable to real projects. A strong companion for the applied side of what we covered in this chapter.

Christoph Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (self-published, 2nd edition, 2022). If the interpretability theme resonated with you, this book is essential. Molnar covers decision trees, feature importance, partial dependence plots, SHAP values, and other techniques for understanding what models have learned. The book is available free online and is one of the best resources on this increasingly important topic.

Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (O'Reilly, 3rd edition, 2022). Chapter 6 covers decision trees and Chapter 7 covers ensemble methods (including random forests and boosting). Géron's writing is practical and example-driven, with clear explanations of the algorithms' inner workings. An excellent next-level resource once you've mastered the basics from this chapter.

Tier 2: Attributed Resources

Leo Breiman, "Random Forests" (Machine Learning, 2001). The original random forest paper. It's a research paper, so it's more technical than the textbooks above, but Breiman wrote clearly and the key ideas come through. If you want to understand the theoretical justification for why random forests work (and the role of correlation between trees), this is where it all started. Search for "Breiman Random Forests 2001" to find it.

Scikit-learn documentation: Decision Trees and Ensemble Methods. The official scikit-learn documentation includes user guides for DecisionTreeClassifier, RandomForestClassifier, and related classes. The user guides explain the algorithms, and the API references detail every parameter. These are living documents that stay current with the library. Search for "scikit-learn decision trees user guide."

StatQuest with Josh Starmer (YouTube channel). Josh Starmer's videos on decision trees, random forests, and related topics are some of the best free educational content on machine learning. His visual explanations of Gini impurity, information gain, and bagging are particularly clear. Search for "StatQuest decision trees" or "StatQuest random forests" on YouTube.

3Blue1Brown, "But what is a neural network?" and related visual explanations. While not about trees specifically, Grant Sanderson's visual approach to explaining machine learning concepts provides excellent context for understanding how different model families approach the same problem. His channel demonstrates the kind of visual thinking that makes decision trees' geometric interpretation click.

  • If you want to go deeper on the theory: Read Chapter 8 of An Introduction to Statistical Learning. It covers trees, bagging, random forests, and boosting in a unified framework with mathematical precision and clear intuition.

  • If you want more hands-on practice: Work through the decision tree and random forest examples in Mueller and Guido's Introduction to Machine Learning with Python. The exercises are practical and the code is directly transferable to your own projects.

  • If you're interested in model interpretability: Start with Molnar's Interpretable Machine Learning. It goes far beyond feature importance into SHAP values, partial dependence plots, and other tools for understanding what models have learned.

  • If you want to see what comes next (gradient boosting): The same textbooks cover XGBoost and gradient boosting, which build on the ensemble ideas from this chapter but use boosting instead of bagging. These are currently the most competitive algorithms for structured data problems.

  • If you want to understand the bias-variance trade-off more deeply: The discussion in An Introduction to Statistical Learning Chapter 2 is definitive. Understanding this trade-off will help you with every modeling decision for the rest of your career.

  • If you just want to build better models right now: Head straight to Chapter 29. Evaluation metrics will tell you whether your models are actually good — and "good" turns out to be a more complicated concept than you might expect.