Case Study 1: Dependency Management in Production

How Netflix and Spotify Manage Python Dependencies at Scale

When you're working on a class project with three pip-installed packages, dependency management feels straightforward. Create a virtual environment, install your packages, freeze the requirements, done. But what happens when you have thousands of services, hundreds of developers, and millions of lines of code all sharing overlapping sets of dependencies?

That's the reality at companies like Netflix and Spotify, where Python is one part of an enormous software estate. Their solutions to dependency management offer lessons that apply at every scale.

Netflix: The Metaflow Approach

Netflix built Metaflow, a Python framework for data science work, and open-sourced it in December 2019 after using it internally for about two years on hundreds of projects. Data scientists use it to write their workflows as a series of Python steps, which Metaflow can then run on a laptop or on large cloud machines.

Picture the problem a framework like this has to solve. Data scientist A trains a machine learning model using scikit-learn 1.2 and pandas 1.5. The model works. Six months later, data scientist B needs to retrain the same model, but their environment has scikit-learn 1.3 and pandas 2.0. The retrained model produces different results — not because the data changed, but because the library behavior changed between versions.

In production machine learning, this isn't a minor annoyance. When a model's output feeds real decisions, such as what to recommend or which content is worth investing in, an unexplained change in its predictions is a serious problem — and very hard to debug if nobody recorded which library versions produced the original.

Metaflow's answer involves several layers:

1. Per-step environment snapshots. Each step of a workflow can declare exactly which packages it needs, using Metaflow's @pypi or @conda decorators. Metaflow resolves that environment, saves a snapshot of the packages, and reuses the snapshot whenever the step runs again. A rerun months later gets the same libraries even if newer versions have since appeared on PyPI. This goes beyond pip freeze: the environment travels with the workflow.

2. Containerization. When steps run in the cloud, they run inside containers. A container (Docker is the best-known tool) is a lightweight, isolated package that bundles the code, the Python interpreter, all dependencies, and the operating system libraries underneath them. Containers are more isolated than virtual environments because they include that operating system layer, not just Python packages.

Large organizations often add further layers of their own on top of tools like this:

3. Organization-wide version sets. A platform team maintains a curated set of library versions that have been tested together. Teams can experiment with whatever they like, but production deployments use the approved set, which dramatically reduces version conflicts.

4. Automated testing of upgrades. When a new version of a critical library (say, numpy) is released, automated systems install it and run the test suites of the projects that depend on it. If anything breaks, the upgrade is flagged for a human to review.

Spotify: Fleet Management

Spotify faces the problem from a different angle. Its code is spread across thousands of separate GitHub repositories — services, libraries and data pipelines each live in their own, with their own dependency files. That keeps components isolated, but it raises a different question: when a library needs upgrading everywhere, how do you update thousands of repositories without thousands of engineers doing it by hand?

Spotify's answer, which its engineering blog calls fleet management, is to treat the whole collection of components as one fleet and change it with automation:

1. Automated dependency updates. Bots update external and internal library dependencies across Spotify's components every day. Each update arrives as an ordinary pull request.

2. Tests decide; humans review the rest. If a component's automated checks pass, many of these pull requests are merged automatically; the rest wait for an engineer. Engineers still own their code, but detecting the update, editing the files and opening the pull request are fully automated.

3. Fast fleet-wide fixes. The payoff shows when something urgent happens. When the Log4j security vulnerability hit in December 2021, Spotify deployed the fix to 80% of its production backend services within 9 hours.

4. Faster upgrades of shared code. Rolling out a new version of Spotify's internal service framework to 70% of services used to take about 200 days. With fleet-wide automation, it takes under a week.

The Left-Pad Problem at Scale

Large organizations have also learned from incidents across the industry where a single dependency caused widespread breakage. (You'll read about the most famous such incident — the left-pad incident — in Case Study 2.) Their response is to minimize the blast radius:

  • Vendoring critical dependencies. For truly critical libraries, some teams copy the library's source code directly into their project rather than installing it from PyPI. This ensures that even if the library is removed from PyPI, their code continues to work. The trade-off is that vendored code doesn't receive automatic security updates.

  • Lock files. Beyond a hand-maintained requirements.txt, tools like pip-tools and Poetry generate lock files that record the entire dependency tree — every direct and transitive dependency, its exact version and a hash (a fingerprint) of each file. pip can then check that every download is byte-for-byte identical to what was locked, so nobody can slip different code in under the same version number.

  • Package mirrors. Companies run their own mirrors of PyPI so they can install packages even if the main PyPI server is unreachable. Some go further and cache every version of every package they've ever used, so they can rebuild any historical version of their software.

Lessons for Every Developer

You might think this level of dependency management is overkill for your projects. And for a class assignment, it is. But the principles scale down nicely:

  1. Always use virtual environments. This is the single most important habit. Metaflow's snapshotted, containerized environments are the industrial version of what python -m venv gives you for free.

  2. Pin your versions. pip freeze > requirements.txt takes two seconds and saves hours of debugging later. Large organizations pin at the organization level; you pin at the project level. Same principle.

  3. Update deliberately, not accidentally. Don't blindly run pip install --upgrade on everything. Review changelogs. Run your tests. Spotify automates this; you do it manually, but the thought process is the same.

  4. Minimize dependencies. Every library you add is code you don't control. Spotify has entire teams building automation just to keep its dependencies current. You should at least glance at a library's GitHub page before trusting it with your project.

  5. Treat requirements.txt as a first-class part of your project. It's not an afterthought — it's as important as your source code. Without it, your code is only runnable on your machine, by you, right now. With it, anyone can run your code, anywhere, anytime.

The Scale of the Problem

To put this in perspective: by April 2023, Spotify's fleet-management automation had authored and merged more than 300,000 changes across its codebase (in all its languages, not just Python), adding roughly 7,500 a week, with about 75% merged automatically. No team of humans could keep that many components current by hand.

The tools you learned in this chapter — venv, pip, requirements.txt — are the foundation that all of this is built on. The enterprise tools add layers of automation, monitoring, and governance, but the core workflow is the same one you just practiced.

Discussion Questions

  1. Metaflow snapshots each step's environment and runs cloud steps in containers. Why isn't requirements.txt alone sufficient for that use case? What could go wrong if you only pinned Python packages?

  2. Spotify uses automated bots to create upgrade pull requests, and many are merged automatically when tests pass. What are the advantages and risks of automating dependency upgrades? What role should human review play?

  3. "Vendoring" means copying a library's source code into your project instead of installing it as a dependency. Under what circumstances would you consider this, and what trade-offs does it introduce?

  4. Many large companies run their own package mirrors or internal package registries in addition to using PyPI. What benefits does this provide? Could an open-source project do something similar?

  5. The chapter emphasizes "minimize dependencies." But using well-maintained libraries often produces better, more secure code than writing everything yourself. How do you balance these competing concerns?