Chapter 20: Secrets and Machine Identity: Service Accounts, API Keys, Certificates, and Securing Non-Human Access

"There is no such thing as a secret that is shared with a computer and a thousand of its closest friends." — A maxim of secrets management (constructed)

Overview

A backup job at Meridian Regional Bank ran every night at 2 a.m. for six years without anyone thinking about it. It connected to the cloud, copied a database snapshot, and exited. To do that, it needed a credential — an Amazon Web Services (AWS) access key — and the engineer who set the job up in 2019 did the obvious, expedient thing: he pasted the key directly into the backup script and committed the script to the bank's internal Git repository. The job worked. He moved teams. The key was never rotated, because no human ever logged in with it and so it never appeared on anyone's password-expiry report. It was not a human identity. It was nobody's, and therefore everybody's.

Then a contractor cloned that repository to their personal laptop to debug an unrelated build problem, and a piece of developer tooling on the laptop quietly uploaded snippets of code to a third-party service for "AI-assisted completion." Now a live key to Meridian's cloud — a key that could read the database the backup job copied — existed on a laptop the bank did not control, inside a service the bank had never assessed, in a repository the bank had assumed was private. Nobody had been phished. No firewall had failed. No human password had leaked. A secret had simply done what unmanaged secrets always eventually do: it spread.

This is the non-human identity problem, and it is the part of identity and access management that most organizations discover the hard way. You have spent the last four chapters securing how people prove who they are and what they may do — passwords and multi-factor authentication 🔗 (Chapter 16), authorization and access control 🔗 (Chapter 17), identity governance for the human lifecycle 🔗 (Chapter 18), and the locked-down privileged accounts that are the keys to the kingdom 🔗 (Chapter 19). But in a modern environment, most of the identities are not people. They are scripts, services, containers, pipelines, and devices, each authenticating with a secret of its own — and they vastly outnumber your employees. If you have hardened human access while leaving machine access as a sprawl of hard-coded keys and twenty-year-old service-account passwords, you have locked the front door and propped the loading dock open with a brick.

This chapter is about closing the loading dock. It is squarely an engineer's chapter, with a strong operations thread, because securing non-human identity is mostly an architecture-and-tooling problem: you replace static, long-lived, scattered secrets with vaulted, short-lived, centrally governed ones, and you build the scanning that catches the secrets that leak anyway.

In this chapter, you will learn to:

• Define a secret precisely and explain why machine identities are harder to govern than human ones, even though the access-control principles are the same.
• Stand up a secrets vault and use dynamic secrets so that credentials are generated on demand, scoped, and expired automatically.
• Secure service accounts and adopt workload identity so a workload proves what it is rather than presenting a password it has.
• Run certificate lifecycle management at scale — issuance, renewal, revocation — and prevent the expired-certificate outage that takes a service down at the worst possible moment.
• Build a secret-scanning capability that finds leaked keys in code, history, and logs, and respond to a confirmed leak with the only step that actually works: rotation.

Learning Paths

This chapter sits at the seam between identity and security operations, so weight it according to where you sit:

🏗️ Security Engineer: This is your chapter end to end. §20.2 (vaults and dynamic secrets), §20.3 (service accounts and workload identity), and §20.4 (certificate lifecycle) are core design material you will implement. The Project Checkpoint extends bluekit with the secret scanner you will wire into a pipeline in Chapter 31. 🛡️ SOC Analyst: Focus on §20.1 (why machine identity is a blind spot), §20.5 (finding and triaging leaked secrets), and the abuse-and-detection passages throughout — a leaked key is an alert you will have to work, and a service account behaving like a person is a hunt you will run. 📋 GRC: Less central, but §20.1 and §20.6 give you the policy spine: who owns non-human identities, how secrets are rotated, and what the auditor will ask. The secrets-management standard in the Project Checkpoint is a policy artifact. 📜 Certification Prep: Security+ touches secrets management, certificate management, and key escrow; CISSP folds this into Identity and Access Management and Security Architecture. The key-takeaways.md file maps the terms to exam domains.

20.1 The non-human identity problem

Let us begin where every security problem should begin — with what goes wrong in the real world when this is missing — and then build the vocabulary.

What goes wrong is the story you just read, and it is not rare. The single most common way large cloud breaches start, year after year, is not an exotic zero-day; it is a credential that should not have existed where it was found. A key in a public code repository. A token printed into a log file that got shipped to a third-party analytics service. A service-account password that was set in 2014, written on a wiki, and has root over the database ever since. These are not failures of human authentication. The humans did everything right. They are failures of machine identity — the identities that programs, not people, use to authenticate to other systems.

A secret is any piece of confidential data that grants access or proves identity to a system — a password, an API key, a token, a private cryptographic key, a database connection string, a TLS certificate's private key, an encryption key. What makes it a secret is not what it is but what it does: possessing it lets the holder act. Secrets are bearer credentials in the worst case — whoever holds the secret is the identity, with no further proof required — which is exactly why a copied key is so dangerous. There is no face to recognize, no second factor to demand. The string is the identity.

A machine identity is the identity a non-human entity uses to authenticate: a server, a script, an application, a container, a function, a device, a continuous-integration job. Where a human identity is anchored to a person (Chapter 18), a machine identity is anchored to a workload or an account that stands in for one. The closely related term workload identity refers specifically to the identity of a running piece of software — a particular container or function — and to the modern pattern of granting access based on what a workload provably is and where it runs rather than on a secret it carries. We will return to that distinction in §20.3, because it is the single most important shift in this field.

The reason machine identity is hard is captured in one statistic that every survey of the field repeats: non-human identities outnumber human identities by a large factor — commonly cited as somewhere between ten-to-one and fifty-to-one in cloud-heavy environments (Tier 2 — the exact multiple varies by source and environment, but the order of magnitude is consistent and is the point). Meridian has roughly 1,800 employees. It has tens of thousands of machine identities: every microservice talking to every other, every scheduled job, every CI pipeline, every device, every integration with a vendor. You ran an access review for the humans in Chapter 18. Did anyone ever run one for the machines?

The properties that make machine identity dangerous, contrasted with human identity:

Read that table as a defender. Two rows are bad news and one is good news. The bad news: machine identities are numerous, unprotected by a second factor, and frequently orphaned, because a script does not show up for an offboarding meeting when the team that built it disbands. The good news is the last row, and we will exploit it hard in §20.5: a machine's behavior should be boring. A backup job runs at 2 a.m., reads one database, and stops. When the credential for that job is suddenly used at 3 p.m. from a new network to list every bucket in the account, that is not a subtle anomaly the way a human logging in from a coffee shop is — it is a workload doing something it has literally never done, and a defender who has baselined machine behavior can catch it cold.

🚪 Threshold Concept: In a modern environment, a secret is an identity, and you have far more identities than you think — most of them non-human, most of them unmanaged, none of them on anyone's roster. Until you can answer "what machine identities do we have, what can each one access, where is each secret stored, and when does it expire?" the way you can already answer those questions for employees, your identity program is only half built. The other half is this chapter.

How these get abused follows directly from the properties above. An attacker who lands anywhere in your environment — through a phished employee, a vulnerable web app, a misconfigured cloud bucket — does not stop at the foothold. They look for secrets, because a secret is a key to somewhere else, and machine secrets are usually over-privileged and never rotated, which makes them the perfect vehicle for lateral movement and persistence. They grep the filesystem for files named .env, credentials, id_rsa, config.json. They read environment variables. They pull the source code and search its history. They check the CI system's stored variables. Finding one hard-coded cloud key can turn a single compromised laptop into total account takeover — which is precisely the path our cold-open backup key would have handed an attacker.

So the defense has a shape, and the rest of this chapter is that shape: stop storing secrets where attackers look (vault them — §20.2), make secrets short-lived so a stolen one expires before it is useful (dynamic secrets and short-lived credentials — §20.2, §20.3), give workloads identities instead of passwords where the platform allows it (workload identity — §20.3), manage certificates as the special, time-bombed secrets they are (§20.4), and assume some secrets will leak anyway and hunt for them (scanning — §20.5). Let us build each piece.

🔄 Check Your Understanding: 1. Define secret and machine identity in one sentence each, and explain why a leaked machine secret is often more dangerous than a leaked human password. 2. The table above lists "no interactive login" as a property of machine identity. Why is that property good news for a defender, even though "no second factor" is bad news?

Answers

1. A secret is any confidential value that grants access or proves identity to a system (password, key, token, certificate); a machine identity is the identity a non-human entity (script, service, container, job) uses to authenticate. A leaked machine secret is often worse because it usually has no second factor to stop a thief, is rarely rotated (so it stays valid for years), and is frequently over-privileged — while a leaked human password may be protected by MFA and will eventually expire. 2. Because a workload's behavior should be perfectly regular and repetitive, any deviation is a stark, high-confidence anomaly — far easier to detect than a human's naturally variable behavior. "No second factor" weakens prevention, but "no interactive login" strengthens detection. Defense in depth (Theme 4) means we lean on the second when the first is weak.

20.2 Secrets vaults and dynamic secrets

The concrete problem is secret sprawl — the same secret copied into many places where it should never live: source code, configuration files, environment variables, CI/CD variables, container images, wiki pages, chat messages, spreadsheets, and the laptops of everyone who ever touched the project. Secret sprawl is the proliferation of secrets across an environment without central control, inventory, or rotation; a secret leak is the specific event in which a secret reaches a place an attacker can read it. Sprawl is the standing condition that makes leaks inevitable: the more copies of a secret exist, the higher the probability that one of them ends up somewhere it should not. Our cold-open backup key was sprawl that became a leak the moment the repository was cloned to an unmanaged laptop.

The architectural answer is to stop scattering secrets and start centralizing them behind an access-controlled service. Secrets management is the discipline of securely storing, distributing, rotating, and auditing secrets across their lifecycle so that no secret is hard-coded, every secret has an owner and an expiry, and every access is logged. The tool that makes this practical is a secrets vault — a hardened, access-controlled service that stores secrets encrypted, releases them only to authenticated and authorized callers, and records every access. HashiCorp Vault is the canonical standalone example; every major cloud provides a managed equivalent (AWS Secrets Manager and AWS Key Management Service, Azure Key Vault, Google Secret Manager and Cloud KMS). The cloud key-management services connect back to the cryptography you learned in Chapters 4 and 5: a vault typically encrypts the secrets it holds under a master key that itself lives in a hardware security module 🔗 (HSM — introduced in Chapter 5; here it is the root of trust for the whole vault).

Here is the shape of a vault and how a workload uses it:

            ┌──────────────────────────────────────────────────────┐
            │                    SECRETS VAULT                      │
            │                                                       │
            │   AuthN/AuthZ        Secret store        Audit log    │
            │   ┌──────────┐     ┌─────────────┐    ┌───────────┐   │
            │   │ identity │     │ encrypted   │    │ who/what  │   │
            │   │ + policy │     │ at rest     │    │ accessed  │   │
            │   │ engine   │     │ (master key │    │ which     │   │
            │   └────┬─────┘     │  in HSM)    │    │ secret    │   │
            │        │           └──────┬──────┘    │ + when    │   │
            │        │                  │           └───────────┘   │
            └────────┼──────────────────┼───────────────────────────┘
                     │ 1. authenticate  │ 3. release secret
                     │    (workload ID, │    (scoped + short-lived,
                     │     not a stored │     leased with a TTL)
                     │     password)    │
                     ▼                  ▼
            ┌─────────────────────────────────────┐
            │   WORKLOAD (service / job / pod)     │
            │   - holds NO long-lived secret       │
            │   - fetches secret at runtime        │
            │   - secret expires when lease ends    │
            └─────────────────────────────────────┘

Figure 20.1 — A secrets vault. The workload authenticates with a platform-provided identity (§20.3), the vault checks policy and releases a scoped secret with a time-to-live (TTL), and every access is logged. The encryption master key lives in an HSM, so even a database thief gets only ciphertext.

Three properties of this design matter, and each one breaks a specific attack:

Centralization means there is exactly one authoritative place a secret lives, and exactly one place to rotate it. When the database password changes, you change it in the vault and every consumer transparently picks up the new value on its next fetch — no hunting through forty config files. This breaks sprawl at the source: workloads fetch at runtime instead of carrying copies.

Access control and audit mean the vault applies the same authorization principles you learned in Chapter 17 to secrets themselves. A workload can read only the secrets its policy permits — least privilege 🔗 (Chapter 3, applied to access in Chapter 17) for machines — and every read is logged. The audit trail is a defender's gift: if a secret leaks, the log tells you exactly which identity fetched it and when, which is the difference between "we may have lost something" and "we know precisely what to rotate."

Dynamic secrets are the property that changes the game. A dynamic secret is a credential the vault generates on demand, scoped to the caller, with a short time-to-live, and then automatically revokes when the lease expires. Instead of a database password that exists forever and is shared by every instance of a service, the vault creates a brand-new database user when a workload asks, hands over credentials valid for, say, one hour, and deletes that user when the hour is up. This is the vault realizing short-lived credentials — credentials deliberately given a brief validity window so that a stolen one is useless almost immediately. A static secret stolen today works until someone notices and rotates it, which historically means years. A dynamic secret stolen today works until its TTL expires, which means minutes to hours. You have not made theft impossible; you have made the stolen goods rot in the thief's hands.

🛡️ Defender's Lens: Short-lived, dynamic secrets convert the secrets problem from a prevention problem you will eventually lose into a time-window problem you can win. The attacker's question changes from "can I find a secret?" (yes, eventually, somewhere) to "can I find and use a secret before it expires, and do so without the vault's audit log lighting up my unusual request?" That is a far harder game for them, and crucially, every secret request now flows through one auditable choke point you can monitor — a single high-value log source instead of a thousand silent files. When you build the SIEM in Chapter 21, vault audit logs are among the first sources you should onboard.

How vaults get abused — because no control is magic, and Theme 4 says we assume each layer fails. The vault becomes the crown jewel: compromise the vault's own root and you have everything. So vaults are hardened disproportionately — the unseal/root keys are split among multiple custodians (so no one person can unlock it alone, an application of separation of duties from Chapter 3), access is tightly scoped, and the audit log is shipped off-box where an attacker who breaches the vault cannot edit it. A subtler abuse: a workload's own identity gets compromised, and the attacker simply asks the vault for that workload's secrets through the front door. The defenses are least privilege (the workload could read only its own scoped secrets, limiting blast radius) and behavioral detection (a workload requesting secrets it has never requested before, or at a rate it never has, is an anomaly — see §20.5). The vault does not eliminate risk; it concentrates it into one well-defended, well-instrumented place, which is exactly the trade a defender wants.

⚠️ Common Pitfall: Putting secrets in environment variables and calling it "secrets management." Environment variables are better than hard-coding a secret in committed source, but they are not a vault. They are readable by any process running as the same user, they frequently get dumped into logs and crash reports and debug endpoints (a single printenv in an error handler can leak everything), they are inherited by child processes, and they have no rotation, no per-access audit, and no expiry. They are a step, not a destination. The destination is: the workload holds no secret at all and fetches a short-lived one from a vault at runtime.

Here is a worked example of a small but real piece of the lifecycle a vault automates — checking whether a certificate is about to expire, which we will need again in §20.4 and which becomes part of bluekit:

# A certificate-expiry check (illustrative; the vault automates this at scale)
from datetime import datetime, timezone

def cert_days_left(not_after: str, now: datetime | None = None) -> int:
    """Days until a certificate expires. not_after is ISO-8601 UTC.
    Negative means already expired."""
    now = now or datetime.now(timezone.utc)
    expiry = datetime.fromisoformat(not_after)
    return (expiry - now).days

# Hand-trace with a fixed 'now' so the result is deterministic:
fixed_now = datetime(2026, 6, 14, tzinfo=timezone.utc)
print(cert_days_left("2026-07-04T00:00:00+00:00", fixed_now))   # 20 days out
print(cert_days_left("2026-06-01T00:00:00+00:00", fixed_now))   # already expired

# Expected output:
# 20
# -13

The arithmetic: from 2026-06-14 to 2026-07-04 is 20 days, so the first call returns 20. From 2026-06-14 back to 2026-06-01 is 13 days in the past, so the second returns -13. A real certificate-management system runs this against every certificate it knows about, every day, and alerts when the number drops below a renewal threshold — the mechanism that prevents the outages we cover in §20.4.

🔄 Check Your Understanding: 1. Explain the difference between a static secret and a dynamic secret, and why dynamic secrets shrink the value of a stolen credential. 2. A teammate says, "We're secure — we moved all our secrets out of code and into environment variables." Give two concrete reasons that is not equivalent to using a vault.

Answers

1. A static secret has a fixed value that persists until someone manually changes it (often for years); a dynamic secret is generated on demand by the vault, scoped to the caller, and given a short TTL after which it is automatically revoked. Dynamic secrets shrink a stolen credential's value because it expires in minutes-to-hours, so a thief has a tiny usable window instead of an open-ended one. 2. Any two of: environment variables are readable by any process running as the same user and are inherited by child processes; they leak easily into logs, crash reports, and debug output; they have no per-access audit trail; they have no automatic rotation or expiry. A vault provides scoped access control, per-access logging, rotation, and short-lived/dynamic secrets — none of which env vars give you.

20.3 Service accounts and workload identity

Now we confront the oldest and most common machine identity: the service account. A service account is an account used by a program, service, or automated process rather than a human — to run a service, execute a scheduled task, or let one application authenticate to another. You met service accounts briefly in Chapter 19 as one of the privileged-account types that PAM must govern 🔗; this chapter owns them in full, because they are where machine identity most often goes wrong.

The classic service account is a disaster waiting to happen, and naming its failure modes is half the defense:

• It has a static password that never changes. Rotating it is scary because nobody is sure which jobs would break, so it is set once and left for a decade. The longer it lives, the more places it has sprawled into.
• It is wildly over-privileged. It was granted domain-admin or cloud-administrator rights "to make it work" during a deadline, and those rights were never trimmed. Now a single service account is a skeleton key.
• It is shared. Multiple services and several humans all use the same account, so the audit log cannot tell you who actually did anything — accountability (the second A of AAA from Chapter 3) is destroyed.
• It has no owner. The team that created it reorganized; nobody knows what it is for, so nobody dares disable it, so it lives forever as an orphan — the machine-identity cousin of the orphaned human account you hunted in Chapter 18.
• It can log in interactively. A service account configured like a person can be used by a person — or an attacker — to log in at a keyboard, which it should never need to do.

📟 War Story: A constructed but representative incident. An attacker who had compromised a single workstation at a mid-size firm ran a standard reconnaissance step and asked the directory for every account whose password was set to never expire and whose name suggested a service (svc-, sql-, backup-). One result was a service account for a reporting tool, created eight years earlier, that had been quietly granted membership in the Domain Admins group "temporarily" in 2017. Its password was on a SharePoint page titled "Server Build Notes." The attacker did not need an exploit. They logged in as the service account and owned the domain in under an hour. No vulnerability was patched-against here because there was no vulnerability in the software sense — only a machine identity that violated every principle in this chapter. The detection that would have caught it: an alert on interactive logon by a service account, which should never happen and is a single, high-fidelity rule.

The defenses for service accounts are direct applications of everything you already know, now pointed at machines:

1. Least privilege (Chapter 3 / Chapter 17), enforced. A service account gets exactly the permissions its workload needs and nothing more. The reporting tool above needed read access to one database; it had administrator of everything. Right-sizing it would have made its compromise a contained nuisance instead of a catastrophe.
2. No interactive logon. Configure service accounts so they cannot be used at a keyboard (in Active Directory, deny interactive and remote-interactive logon rights; in cloud, service identities simply have no console-login capability). This makes the §20.5 detection — "service account logged in interactively" — a bright line that should never be crossed.
3. Rotation, automated. The secret rotates on a schedule without a human touching it. Modern directories support managed service accounts (in Windows, group Managed Service Accounts — gMSA — where the platform rotates the password automatically and no human ever knows it). A password no human knows cannot be written on a wiki.
4. A named owner and a lifecycle. Every service account is tied to a team and a purpose, and is reviewed and retired like a human identity. This is identity governance (Chapter 18) extended to non-humans — and it is the step most organizations skip.

But the deepest fix is to stop using service-account passwords at all, where the platform allows it, and move to workload identity. The idea is a genuine shift in how authentication works, so it earns careful explanation. With a traditional service account, a workload proves who it is by presenting something it has — a stored secret. With workload identity, a workload proves who it is by what it provably is and where it runs, attested by the platform it runs on. The cloud or orchestrator issues the workload a short-lived, automatically-rotated credential based on the workload's verified identity, and no long-lived secret is ever stored anywhere.

Concretely:

• In AWS, an IAM role attached to a compute instance, container, or function means the workload retrieves temporary, auto-rotating credentials from the platform's metadata service. There is no access key to leak — our cold-open backup job, rebuilt this way, would have had no key to hard-code in the first place.
• In Kubernetes, a service account token bound to a specific pod, federated to the cloud's identity system, lets the pod authenticate as itself with a short-lived token the platform mints and rotates.
• Across clouds and on-prem, the SPIFFE/SPIRE framework issues each workload a cryptographically verifiable identity document (a SPIFFE Verifiable Identity Document, or SVID — typically an X.509 certificate, tying this directly to §20.4) based on attestation of what the workload is and where it runs.

The payoff is that the most dangerous secret — the long-lived, hard-coded, over-privileged service-account credential — ceases to exist. You cannot leak a key you never created. This is the single highest-leverage move in machine-identity security: not "store the secret better" but "don't have a static secret."

🔗 Connection: Workload identity is where this chapter reaches forward to the architecture of Part VII. Mutual TLS 🔗 (mTLS — introduced in Chapter 5; applied here) is how two workloads with workload identities authenticate each other: each presents a certificate proving its identity, and each verifies the other, so neither trusts the other merely because of network position. That "verify identity, never trust the network" principle is the heart of the zero-trust architecture you will design in Chapter 32, and securing the build pipeline's own machine identities is central to the DevSecOps work of Chapter 31. Workload identity is the mechanism that makes zero trust real for non-human actors.

How workload identity gets abused — Theme 4 again. The credentials are short-lived, but they are retrievable by anything that can reach the platform's metadata endpoint. The classic attack is server-side request forgery (SSRF — a web flaw you will study in Chapter 13) used to trick a vulnerable application into fetching its own instance's credentials from the metadata service and returning them to the attacker. The defenses are layered: require the newer, session-token-protected version of the metadata service (which defeats naive SSRF), scope the workload's role to least privilege so stolen temporary credentials can do little, and — because the credentials are short-lived — rely on the narrow time window plus behavioral detection. Notice the pattern: even when an attacker defeats one layer, short-lived credentials and least privilege ensure the breach is small and brief. That is defense in depth working as designed.

🔄 Check Your Understanding: 1. List three failure modes of a classic service account, and give the specific defense for each. 2. Explain workload identity and why it is more secure than giving a service a stored API key. What is the one secret it eliminates?

Answers

1. Any three of: static never-changing password → automated rotation / managed service accounts (gMSA); over-privileged → least privilege, right-sized to the workload's actual need; shared across services/people → one identity per workload to preserve accountability; no owner/lifecycle → assign an owner and review/retire like a human identity (IGA for machines); interactive logon allowed → deny interactive logon and alert if it ever occurs. 2. Workload identity lets a workload authenticate based on what it provably is and where it runs (attested by the platform), receiving short-lived, auto-rotating credentials instead of carrying a stored secret. It is more secure because there is no long-lived, hard-codeable key to leak, the credentials rotate automatically, and they expire quickly. It eliminates the long-lived, static service-account credential / API key entirely.

20.4 Certificate lifecycle at scale

There is one kind of secret that is special enough to deserve its own section, because it has a property no password has: it expires on a fixed date, and when it does, things break loudly. That secret is the digital certificate.

You met the X.509 certificate and public key infrastructure 🔗 in Chapter 4, and you saw certificates do their job in the TLS handshake and saw their lifecycle introduced in Chapter 5. Here we treat the operational discipline of managing them across an organization: certificate lifecycle management — the end-to-end process of issuing, deploying, monitoring, renewing, and revoking digital certificates so that every certificate is valid, trusted, and replaced before it expires. It is the part of cryptography that is not math at all. It is logistics, and it is where organizations with flawless cryptography still take themselves down.

The lifecycle has distinct phases, each with its own failure mode:

   ENROLLMENT ──► ISSUANCE ──► DEPLOYMENT ──► MONITORING ──► RENEWAL
   (request,       (a CA signs   (install on    (track expiry,  (re-issue
    prove           the cert)     every server   validate        BEFORE
    identity,                     that needs it)  the chain)      expiry)
    generate key)                                      │             │
        ▲                                              │             │
        │                                              ▼             │
        │                                         REVOCATION         │
        │                                        (if compromised:    │
        │                                         CRL / OCSP says    │
        └────────────────── replace ───────────── "do not trust")◄──┘

Figure 20.2 — The certificate lifecycle. Every arrow is a place a real organization has caused an outage or a trust failure. The loop must close — renewal must happen before expiry — and revocation must be reachable when a key is compromised.

Walk the phases as a defender:

Enrollment and issuance. A requester proves its identity, generates a key pair (keeping the private key — itself a secret, ideally generated and stored in an HSM 🔗 so it never leaves hardware), and a certificate authority (CA — Chapter 4) signs a certificate binding the identity to the public key. The internal CA that signs Meridian's service-to-service certificates is itself a crown-jewel secret: if an attacker can make the CA sign certificates, they can impersonate anything. So the CA's private key lives in an HSM, issuance is tightly controlled and logged, and short-lived certificates are preferred so that the damage from any single mis-issuance is time-bounded.

Deployment. The certificate and its private key are installed on every endpoint that needs them. The failure mode here is sprawl again — the same private key copied to dozens of servers, so that compromising any one server compromises the identity everywhere. The fix is automation that places a distinct key on each host, and never emails private keys around.

Monitoring. Someone — or, properly, something — must track every certificate's expiry. This is where cert_days_left from §20.2 lives in production: a daily sweep across the entire inventory, alerting well before the threshold.

Renewal. The certificate is re-issued and deployed before it expires. When this fails, you get the single most common, most embarrassing, and most preventable outage in all of operations: the expired-certificate outage. A certificate lapses, every client that validates it refuses to connect, and a service goes hard-down at the exact moment — its expiry date — that you could have seen coming for ninety days. It has taken down payment systems, authentication providers, government services, and major platforms, repeatedly, for one reason: a human was supposed to remember, and humans do not remember dates ninety days out.

⚠️ Common Pitfall: Tracking certificate expiry in a spreadsheet maintained by one person. This is the default state of most organizations and it is a self-inflicted outage scheduled for whenever that person is on vacation. Certificates expire on a deadline you cannot negotiate with — there is no grace period, no "it still mostly works." The only reliable answer is automation: an inventory that discovers certificates (you cannot renew what you do not know exists — the §20.1 visibility problem again), a monitor that alerts on approaching expiry, and, ideally, automated renewal. The ACME protocol (the automated certificate-management standard popularized by Let's Encrypt) renews public-web certificates with no human in the loop; internal certificate-management platforms do the same inside the enterprise. Short-lived certificates make this mandatory and therefore reliable — if a certificate is valid for only days, manual renewal is impossible, so automation is forced, and an automated process does not forget.

Revocation. Sometimes a certificate must be invalidated before its natural expiry — because the private key was stolen, or the certificate was mis-issued, or a workload was decommissioned. Revocation is published through a certificate revocation list (CRL — a signed list of revoked serial numbers) or queried in real time through the Online Certificate Status Protocol (OCSP). Both connect to Chapter 4's PKI. The defender's hard truth about revocation is that it is unreliable in practice: clients that cannot reach the CRL/OCSP endpoint often "fail open" and trust the certificate anyway, so revocation may not actually stop a determined attacker who has the private key. This is the deepest argument for short-lived certificates: a certificate valid for 24 hours barely needs revocation, because it revokes itself by expiring before the slow, unreliable revocation machinery would have mattered. Short lifetimes turn a fragile control (revocation) into an automatic one (expiry).

How certificates get abused — three ways, each with a defense:

• A stolen private key lets an attacker impersonate the certificate's identity (decrypt traffic, or pose as a trusted service). Defense: keep private keys in HSMs so they cannot be exfiltrated; rotate to short lifetimes; revoke and re-issue on suspicion.
• A rogue or mis-issued certificate — an attacker convinces a CA to issue a certificate for a name they do not own, or compromises a CA. Defense: Certificate Transparency (public, append-only logs of every certificate a CA issues, which you can monitor to catch a certificate issued for your domain that you did not request) and tight, logged internal-CA controls.
• Expiry as a self-inflicted denial of service — not an attacker at all, but the availability leg of the CIA triad falling over on its own. Defense: the monitoring and automated renewal above.

🛡️ Defender's Lens: Treat certificate expiry as a managed risk with a known date, not a surprise. Because the expiry date is knowable in advance for every certificate you own, the expired-certificate outage is one of the only catastrophic failures in all of security that is completely preventable with a calendar and a script. If you build nothing else from this chapter, build the inventory-and-alert sweep: discover every certificate, run cert_days_left daily, and page someone at 30 days and again at 7. That single control prevents an entire category of outage — and an outage is a security incident, because availability is security (Chapter 1).

🧩 Try It in the Lab: In your own lab or on a domain you control, use a command-line tool to read a public certificate's validity dates — for example, openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates. This prints the notBefore and notAfter fields. Feed the notAfter value into your own cert_days_left and confirm the math. Then think about how you would do this for ten thousand certificates you did not know you had — that scaling problem is the whole discipline of certificate lifecycle management.

🔄 Check Your Understanding: 1. Why is an expired-certificate outage considered one of the most preventable failures in operations, and what single control prevents it? 2. Give two reasons short-lived certificates are more secure operationally than long-lived ones. (Hint: think about renewal and revocation.)

Answers

1. Because every certificate's expiry date is known in advance, so the failure is a deadline you can see coming for months; the control that prevents it is automated discovery + expiry monitoring + (ideally) automated renewal — an inventory plus a daily cert_days_left-style sweep with alerts well before expiry. 2. (a) Renewal: short lifetimes force automation (manual renewal is impossible at a multi-day cadence), and automation does not forget, so it is more reliable than a human-maintained spreadsheet. (b) Revocation: a short-lived certificate effectively revokes itself by expiring quickly, so it does not depend on the slow, often fail-open CRL/OCSP machinery to stop a compromised key — the exposure window from a stolen key is bounded by the short lifetime regardless of whether revocation reaches every client.

20.5 Finding leaked secrets: secret scanning

Defense in depth says assume the previous layers fail. You will vault secrets, adopt workload identity, and automate certificates — and a developer will still, someday, paste a key into a commit, print a token into a log, or push a .env file. So the final control is to assume secrets leak and go find them. This is secret scanning — automatically searching code, configuration, commit history, logs, and other artifacts for patterns that look like credentials, so a leaked secret is caught and rotated before an attacker uses it.

The mechanism is pattern recognition. Many secrets have recognizable, structured formats, which is a gift to defenders even though it is also a gift to attackers grepping a stolen filesystem. An AWS access key ID begins with AKIA followed by sixteen uppercase alphanumeric characters. A GitHub personal access token begins with ghp_. A Google API key begins with AIza. A Slack token begins with xoxb- or xoxp-. A private key is bracketed by -----BEGIN ... PRIVATE KEY-----. A scanner encodes these as regular expressions and flags any match. Real tools (git-secrets, truffleHog, gitleaks, and the secret-scanning built into code-hosting platforms) ship with libraries of hundreds of such patterns, plus high-entropy detection for secrets with no fixed format.

Here is the core of a secret scanner, the function that becomes part of bluekit:

import re

# Each pattern targets a known credential format. Defensive use only.
SECRET_PATTERNS = {
    "aws_access_key_id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
    "github_pat":        re.compile(r"\bghp_[0-9A-Za-z]{36}\b"),
    "google_api_key":    re.compile(r"\bAIza[0-9A-Za-z_\-]{35}\b"),
    "slack_token":       re.compile(r"\bxox[baprs]-[0-9A-Za-z\-]{10,}\b"),
    "private_key_block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
}

def scan_secrets(text: str) -> list[tuple[str, str]]:
    """Return (kind, matched_value) for every suspected secret in text."""
    findings = []
    for kind, pattern in SECRET_PATTERNS.items():
        for match in pattern.findall(text):
            findings.append((kind, match))
    return findings

sample = """
    aws_key = "AKIAIOSFODNN7EXAMPLE"     # obviously-fake example value
    token   = "ghp_EXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPL"
    note    = "no secret on this line"
"""
for kind, value in scan_secrets(sample):
    print(f"{kind}: {value}")

# Expected output:
# aws_access_key_id: AKIAIOSFODNN7EXAMPLE
# github_pat: ghp_EXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPL

Hand-tracing the match: AKIAIOSFODNN7EXAMPLE is AKIA followed by exactly sixteen uppercase alphanumeric characters (IOSFODNN7EXAMPLE is 16), so the AWS pattern fires. ghp_ followed by 36 characters (EXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPL — count them, 36) matches the GitHub pattern. The third line contains no credential-shaped token, so nothing matches it. The two findings print; the comment line and the prose do not. (Note: AKIAIOSFODNN7EXAMPLE is the well-known documentation placeholder AWS itself uses in examples — never a live key; we use only such obviously-fake values, per the book's rules.)

Where you run the scanner is as important as the scanner itself, and there are three placements, each catching leaks at a different stage:

1. Pre-commit, on the developer's machine. A git hook scans staged changes and refuses the commit if it finds a secret. This is the best outcome — the secret never enters history at all. It is also bypassable (a developer can skip the hook), so it cannot be your only layer.
2. In the pipeline, on every push (Chapter 31). The continuous-integration system scans each change and fails the build on a finding. This is enforceable because it runs on infrastructure the developer does not control. We wire scan_secrets into exactly such a gate in Chapter 31's DevSecOps work.
3. Across history and running systems, continuously. Scan the entire repository history (because a secret committed and later "deleted" is still in the git history forever), and scan logs and artifacts where secrets get printed. This catches what slipped through before scanning existed and what leaks at runtime.

🛡️ Defender's Lens — and the one rule that matters most: When a secret is confirmed leaked, rotate it. This is the single most important sentence in the chapter. Deleting the commit does not help — the secret is in git history, in clones on other machines, in CI caches, possibly already scraped by an attacker's bot that watches public repositories in real time (secrets pushed to public GitHub are routinely abused within minutes). The exposed value must be treated as compromised the instant it leaks, and the only response that actually closes the exposure is to invalidate the old secret and issue a new one. This is exactly why the vault and short-lived credentials of §20.2 matter so much: if rotation is automated and secrets are short-lived, "rotate the leaked secret" is a fast, low-drama action instead of a multi-day fire drill of hunting down every consumer. The leak is the alert; rotation is the response; everything else is theater.

How attackers use leaks, so you know what you are racing: criminals run automated scanners against every public commit pushed to code-hosting platforms, continuously, worldwide. The window between a secret being pushed publicly and being used to spin up cryptomining or pivot into an account is measured in minutes. This is the same "automation of attack" asymmetry from Chapter 1 — they scan everything because it is cheap. Your defense is to scan first and faster, on the way in (pre-commit and pipeline) so most secrets never go public, and to make any secret that does escape expire fast and rotate easily.

Beyond pattern-matching for new leaks, the SOC also hunts for use of leaked or misused secrets, and this is where §20.1's "machine behavior is boring" insight pays off. Useful detections for a SOC analyst (these become SIEM use cases in Chapter 21):

• A service account logging in interactively — it never should; one event is enough to investigate.
• A workload using a credential from a new network, geography, or at a new time — the 2 a.m. backup job suddenly active at 3 p.m. from a foreign IP.
• A spike in vault secret requests, or requests for secrets a workload has never asked for — lateral movement using a compromised workload identity.
• An access key used from many source IPs in a short window — a leaked key being shared among an attacker's infrastructure.
• First-ever use of a long-dormant credential — an old, forgotten secret suddenly waking up.

🔄 Check Your Understanding: 1. A developer realizes they committed an API key, so they delete the file and push a "fix." Why is the secret still compromised, and what is the only effective response? 2. Give two reasons it is better to catch a secret with a pre-commit hook and a pipeline scan than to rely only on scanning public repositories after the fact.

Answers

1. The secret remains in the git history (and in any clones, CI caches, and possibly already-scraped copies), so deleting the current file does not remove it; an attacker's automated scanner may have grabbed it within minutes of the push. The only effective response is to rotate the secret — invalidate the leaked value and issue a new one — because you cannot un-expose it. 2. Any two of: pre-commit/pipeline scanning catches the secret before it becomes public, so it never reaches attackers' scanners at all; pipeline scanning runs on infrastructure the developer cannot bypass, making it enforceable; catching it early avoids the costly history-rewrite and full rotation fire drill; relying on after-the-fact public scanning means you are always racing attackers who scan continuously and may already have used the key. (Pre-commit alone is bypassable, which is why the enforceable pipeline gate is the safety net.)

20.6 Meridian's API keys and service accounts

Time to make this concrete at the bank. Sam Whitfield, the security engineer, was handed an uncomfortable mandate after the cold-open incident — the live AWS key discovered on a contractor's laptop. Dana, the CISO, framed it precisely: "We just learned that we have no idea how many secrets we have or where they are. Find out, then fix it so this can't happen again." This is the machine-identity equivalent of the human access review from Chapter 18, and Sam ran it in the same spirit.

Discovery first — because you cannot govern what you cannot see (§20.1). Sam's team scanned every Meridian Git repository, current state and full history, with scan_secrets-style tooling. The results were the normal, sobering findings of a first scan: the live AWS backup key (already being rotated as incident response — the only response that works, §20.5); two database connection strings with embedded passwords in older commits; a handful of expired API keys for vendors Meridian no longer used; and a long-forgotten private key for an internal service, sitting in history since 2020. In parallel, they enumerated service accounts across Active Directory and the AWS environment: 312 service accounts in AD, of which 47 had passwords set to never expire and 11 held administrative rights they almost certainly did not need; in AWS, 38 long-lived IAM access keys, several older than three years and used from more source addresses than any single workload should explain.

📟 War Story: A constructed Meridian finding. One AD service account, svc-statements, generated customers' monthly account-statement PDFs. It had been a member of an administrative group since a 2019 incident when someone "needed it to work for the close" and never trimmed the rights afterward. Its password — unchanged for four years — was findable in an old runbook in the wiki. By itself it ran a boring nightly job. In an attacker's hands it was domain-wide compromise with a paper trail leading nowhere, because three different teams shared it. It is the cold-open's twin: a non-human identity that quietly violated least privilege, rotation, ownership, and accountability all at once. Catching the use of an account like this is one detection — interactive logon by svc-statements — that Marcus's SOC added to the watch list that week.

Then the standard — the program increment. Sam drafted Meridian's secrets-management standard, the artifact that turns these one-off fixes into policy, with the rule set this chapter has built:

Sam sequenced the rollout by risk, because all of it at once was impossible: rotate the live key and the worst service accounts now (incident response); stand up the vault and migrate the highest-value secrets this quarter; pilot workload identity on the next new cloud service so the bank stops creating new static keys; turn on pipeline secret scanning immediately (it is cheap and prevents tomorrow's leak); and build the certificate inventory before the next surprise expiry. Elena, the GRC analyst, mapped the standard to Meridian's obligations — secrets and key management touch the PCI-DSS requirements around protecting stored cryptographic keys and restricting access — so the work doubled as audit evidence (full compliance crosswalking comes in Chapter 28).

⚖️ Authorization & Ethics: When Sam's team scanned the repositories, they found secrets — which means they momentarily held live credentials they were not the intended users of. Handle discovered secrets like the toxic material they are: do not paste a found key into a chat to show a colleague (you have just leaked it again), record only that a secret of a given type was found at a given location (not its value), rotate it through the proper owner, and document the chain. Scanning your own organization's repositories is authorized and expected; the same tooling pointed at someone else's code without permission is not. The rule from Chapter 1 holds for secrets as for everything else: your own systems, or explicit authorization, and nothing else.

This is identity governance (Chapter 18) and privileged access management (Chapter 19) extended to the machines — the same principles, a far larger and more invisible population. Meridian now has, for the first time, an answer to the question it could not answer in the cold open: what machine identities do we have, what can each access, where is each secret, and when does it expire?

Project Checkpoint

Program increment — the secrets-management standard. Meridian's security program gains the secrets-management standard from §20.6: the nine-rule table above, the discovery results that justify it, and the risk-sequenced rollout plan. File it alongside the access-control policy (Chapter 17), the identity-governance process (Chapter 18), and the PAM standard (Chapter 19); together these four are the bank's complete identity-and-access management program for both human and non-human identities. This is the artifact an auditor asks for when they say "show me how you protect your keys and service accounts," and it is one section closer to the board deck you assemble in Chapter 38.

bluekit increment — secrets.py. We add the chapter's two workhorse functions to the toolkit: scan_secrets, which finds leaked credentials by pattern, and cert_days_left, which flags certificates before they expire. Together they cover two of the chapter's defenses — finding leaks (§20.5) and preventing expiry outages (§20.4). As always, the code is illustrative and is never executed during authoring; the expected output is hand-traced in a comment.

# bluekit/secrets.py  — Chapter 20 increment
"""Find leaked secrets and flag soon-to-expire certificates.
Defensive use only: scan systems you own or are authorized to scan.
Uses ONLY obviously-fake placeholder secret values in examples."""
import re
from datetime import datetime, timezone

SECRET_PATTERNS = {
    "aws_access_key_id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
    "github_pat":        re.compile(r"\bghp_[0-9A-Za-z]{36}\b"),
    "private_key_block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
}

def scan_secrets(text: str) -> list[tuple[str, str]]:
    """Return (kind, value) for each suspected secret found in text."""
    out = []
    for kind, pat in SECRET_PATTERNS.items():
        out.extend((kind, m) for m in pat.findall(text))
    return out

def cert_days_left(not_after: str, now: datetime | None = None) -> int:
    """Days until an ISO-8601 UTC expiry; negative means already expired."""
    now = now or datetime.now(timezone.utc)
    return (datetime.fromisoformat(not_after) - now).days

if __name__ == "__main__":
    code = 'key = "AKIAIOSFODNN7EXAMPLE"\nok = "not a secret"'
    for kind, value in scan_secrets(code):
        print(f"LEAK {kind}: {value}")
    fixed_now = datetime(2026, 6, 14, tzinfo=timezone.utc)
    print("cert days left:", cert_days_left("2026-07-04T00:00:00+00:00", fixed_now))

# Expected output:
# LEAK aws_access_key_id: AKIAIOSFODNN7EXAMPLE
# cert days left: 20

Trace it: the scanner finds AKIAIOSFODNN7EXAMPLE (AKIA + 16 uppercase chars) and prints one LEAK line; the second source line has no credential pattern. cert_days_left computes 2026-07-04 minus 2026-06-14 = 20 days. Two of the bank's nine standard rules — "scan for leaks" and "manage certificates" — now have running code behind them. In Chapter 31 you will lift scan_secrets straight into a CI gate so a leak fails the build before it ever ships.

Summary

This chapter extended identity and access management from people to machines — the larger, more invisible, and more dangerous half of the identity problem.

• A secret is an identity. A secret is any confidential value (password, API key, token, private key, certificate) that grants access; machine identity is the identity non-human entities use; workload identity grants access based on what a workload provably is rather than a secret it carries. Machine identities outnumber human ones by roughly 10–50×, usually have no second factor, no lifecycle, and no owner.
• Secret sprawl (secrets copied everywhere) makes secret leaks inevitable. The fix is secrets management: a secrets vault (HashiCorp Vault, cloud KMS/Secrets Manager, HSM-backed) that centralizes, scopes, audits, and — via dynamic secrets — issues short-lived credentials that expire before a thief can use them.
• Service accounts classically fail five ways: static password, over-privilege, sharing, no owner, interactive logon. Fix each with least privilege, no interactive logon, automated rotation (managed service accounts), named ownership, and — best of all — workload identity (IAM roles, federated/Kubernetes tokens, SPIFFE/SVID, mTLS between workloads) so the dangerous long-lived secret never exists.
• Certificate lifecycle management — enroll, issue, deploy, monitor, renew, revoke — is logistics, not math. The expired-certificate outage is the most preventable failure in operations; defeat it with automated discovery, expiry monitoring (cert_days_left), and automated renewal (ACME). Short-lived certificates make renewal automatic and make unreliable revocation nearly irrelevant.
• Secret scanning assumes leaks happen and finds them by pattern (AKIA…, ghp_…, -----BEGIN … PRIVATE KEY-----) at three placements: pre-commit, pipeline (Chapter 31), and across history/logs. When a secret leaks, the only real response is to rotate it — deleting the commit does nothing. Detect use of misused secrets via the "machine behavior is boring" anomalies (service account logging in interactively; a workload active from a new place or time).
• Meridian discovered its sprawl (a live key on a contractor's laptop; over-privileged, never-rotated service accounts), rotated the worst immediately, and adopted a nine-rule secrets-management standard plus bluekit's secrets.py (scan_secrets, cert_days_left).

Spaced Review

Retrieval practice over a recent and an older chapter. Answer before expanding.

1. (Ch.19) Privileged access management vaults and rotates human privileged credentials and grants them just-in-time. Name two ways the secrets vault and dynamic secrets of this chapter apply the same ideas to machine credentials, and one way the machine problem is harder.
2. (Ch.4) A certificate authority signs an X.509 certificate that binds an identity to a public key. Using that, explain in one or two sentences what a "rogue certificate" is and why Certificate Transparency helps a defender detect one.
3. (Ch.19) Why is "a service account that can log in interactively" a problem, and how does that connect to the privileged-access principle that admin paths should be tightly constrained?

What's Next

You have now secured identity from end to end — who people are (Chapter 16), what they may do (Chapter 17), how their accounts are governed (Chapter 18), how privileged human access is locked down (Chapter 19), and, in this chapter, how the machines, services, and secrets that vastly outnumber your people are governed too. Identity is the modern perimeter, and you have now built that perimeter for humans and non-humans alike.

But a perimeter you cannot see through is a wall in the dark. Every control in Part IV — every failed login, every vault secret request, every service account that logs in where it should not, every certificate about to expire — produces telemetry, and that telemetry is worthless until something collects it, normalizes it, and correlates it into an alert a human can act on. Part V is Security Operations, and it opens with Chapter 21 on the Security Information and Event Management (SIEM) platform: the system that ingests the logs from everything you have built and turns the background radiation of your environment into detections. The machine-identity anomalies you just learned to look for become some of your very first SIEM use cases. You have built the locks; now you build the room full of screens that watches them.