Case Study 1: Dependency Management in Production

How Netflix and Spotify Manage Python Dependencies at Scale

When you're working on a class project with three pip-installed packages, dependency management feels straightforward. Create a virtual environment, install your packages, freeze the requirements, done. But what happens when you have thousands of Python services, hundreds of developers, and millions of lines of code all sharing overlapping sets of dependencies?

That's the reality at companies like Netflix and Spotify, where Python is a core part of their infrastructure. Their solutions to dependency management offer lessons that apply at every scale.

Netflix: The Metaflow Approach

Netflix runs a staggering amount of Python code. Their data science platform, Metaflow, powers everything from recommendation algorithms to content demand forecasting. Thousands of data scientists and engineers write Python code daily, and each project has its own dependency requirements.

The challenge Netflix faced was familiar: data scientist A trains a machine learning model using scikit-learn version 1.2 and pandas 1.5. The model works. Six months later, data scientist B needs to retrain the same model but their environment has scikit-learn 1.3 and pandas 2.0. The retrained model produces different results — not because the data changed, but because the library behavior changed between versions.

In production machine learning, this isn't a minor annoyance. A model that predicts which shows users will enjoy directly affects what content Netflix invests in. Subtle changes in library behavior can shift predictions in ways that cost millions of dollars.

Netflix's solution involved several layers:

1. Environment snapshots. Every time a model is trained, Metaflow captures the complete environment — not just the requirements.txt, but the exact Python version, operating system, and even hardware architecture. This goes beyond pip freeze; it's a complete record of everything that could affect the model's output.

2. Containerization. Each model runs inside a Docker container — essentially a lightweight virtual machine that bundles the code, the Python interpreter, all dependencies, and the operating system libraries. Containers are more isolated than virtual environments because they include the entire operating system layer, not just Python packages.

3. Dependency pinning at the organization level. Netflix maintains a curated set of "blessed" library versions that have been tested together. Individual teams can use whatever versions they want for experimentation, but production deployments must use the blessed set. This dramatically reduces version conflicts.

4. Automated testing of upgrades. When a new version of a critical library (say, numpy) is released, automated systems install it and run the entire test suite across dozens of projects. If anything breaks, the upgrade is flagged for manual review.

Spotify: The Monorepo Challenge

Spotify takes a different approach. Much of their backend is organized in a monorepo — a single, massive repository containing code for hundreds of services. This creates a unique dependency management challenge.

Imagine a monorepo with 500 Python services. Service A uses requests 2.28. Service B uses requests 2.31. Service C hasn't updated requests in two years and uses 2.22. In a monorepo, all of these services live in the same repository, and ideally, they should all be buildable from the same checkout.

Spotify's engineering team developed internal tooling to handle this:

1. Per-service environments. Each service in the monorepo has its own requirements.txt and its own virtual environment. The build system knows how to create isolated environments for each service, even though they share a repository.

2. Dependency bots. Automated bots scan every service's dependencies weekly. When a security vulnerability is found in a library (say, a critical bug in urllib3), the bot automatically creates pull requests to upgrade every affected service. Engineers review and merge, but the detection and PR creation is fully automated.

3. Internal package registry. In addition to PyPI, Spotify runs an internal package registry for proprietary libraries that are shared across teams. This gives them the same version management capabilities (semantic versioning, dependency resolution, changelogs) for internal code that PyPI provides for open-source code.

4. Gradual rollouts. When upgrading a widely-used dependency, Spotify doesn't update all 500 services at once. They upgrade a handful, monitor for problems in production, then gradually expand. If an issue surfaces, only a small percentage of services are affected.

The Left-Pad Problem at Scale

Both companies learned from incidents across the industry where a single dependency caused widespread breakage. (You'll read about the most famous such incident — the left-pad incident — in Case Study 2.) Their response was to minimize the blast radius:

• Vendoring critical dependencies. For truly critical libraries, some teams copy the library's source code directly into their project rather than installing it from PyPI. This ensures that even if the library is removed from PyPI, their code continues to work. The trade-off is that vendored code doesn't receive automatic security updates.

• Lock files. Beyond requirements.txt, tools like pip-tools and poetry generate lock files that record not just direct dependencies and their versions, but the entire dependency tree including transitive dependencies, their hashes (for integrity verification), and the exact URLs they were downloaded from. This makes builds fully reproducible even if PyPI goes down or a package is yanked.

• Fallback registries. Companies maintain mirrors of PyPI so they can install packages even if the main PyPI server is unreachable. Some go further and cache every version of every package they've ever used, so they can rebuild any historical version of their software.

Lessons for Every Developer

You might think this level of dependency management is overkill for your projects. And for a class assignment, it is. But the principles scale down nicely:

1. Always use virtual environments. This is the single most important habit. Netflix's containerized environments are the industrial version of what python -m venv gives you for free.

2. Pin your versions. pip freeze > requirements.txt takes two seconds and saves hours of debugging later. Netflix pins at the organization level; you pin at the project level. Same principle.

3. Update deliberately, not accidentally. Don't blindly run pip install --upgrade on everything. Review changelogs. Run your tests. Netflix automates this; you do it manually, but the thought process is the same.

4. Minimize dependencies. Every library you add is code you don't control. Netflix and Spotify have dedicated teams to evaluate and manage dependencies. You should at least glance at a library's GitHub page before trusting it with your project.

5. Treat requirements.txt as a first-class part of your project. It's not an afterthought — it's as important as your source code. Without it, your code is only runnable on your machine, by you, right now. With it, anyone can run your code, anywhere, anytime.

The Scale of the Problem

To put this in perspective: Spotify's engineering blog reported that they manage over 10,000 distinct Python dependencies across their services. Netflix's ML platform processes over a trillion events per day using Python pipelines. At that scale, a single version mismatch in a single library can cascade into an outage affecting millions of users.

The tools you learned in this chapter — venv, pip, requirements.txt — are the foundation that all of this is built on. The enterprise tools add layers of automation, monitoring, and governance, but the core workflow is the same one you just practiced.

Discussion Questions

1. Netflix captures the entire environment (OS, Python version, hardware) when training ML models. Why isn't requirements.txt alone sufficient for their use case? What could go wrong if they only pinned Python packages?

2. Spotify uses automated bots to create upgrade pull requests. What are the advantages and risks of automating dependency upgrades? What role should human review play?

3. "Vendoring" means copying a library's source code into your project instead of installing it as a dependency. Under what circumstances would you consider this, and what trade-offs does it introduce?

4. Both companies maintain internal package registries in addition to using PyPI. What benefits does this provide? Could an open-source project do something similar?

5. The chapter emphasizes "minimize dependencies." But using well-maintained libraries often produces better, more secure code than writing everything yourself. How do you balance these competing concerns?