Chapter 4: Cryptography Fundamentals: Encryption, Hashing, Digital Signatures, and PKI

"There are two kinds of cryptography in this world: cryptography that will stop your kid sister from reading your files, and cryptography that will stop major governments from reading your files." — Bruce Schneier

Overview

When attackers finally got into a mid-size retailer's network — and in the breaches that defined the last decade, they always eventually got in — the question that decided whether it was an embarrassing week or a company-ending year was almost never "did they get in?" It was "what did they find when they did?" In the worst cases they found a database of customer card numbers sitting in plaintext, and the breach became a hundred-million-dollar event: card reissuance, fraud losses, fines from the card brands, class actions, a CISO testifying to Congress. In the cases that stayed merely bad, the attacker exfiltrated the same database and got a useless blob of ciphertext, because someone had encrypted it correctly and kept the keys somewhere the attacker never reached. Same intrusion. Same stolen file. Wildly different outcome — decided entirely by cryptography that had been put in place long before the attacker arrived.

That is the defender's relationship with cryptography, and it is not the relationship the movies sold you. You are almost never going to break a cipher. The mathematics of modern encryption is, for practical purposes, unbreakable — a correctly chosen algorithm with a large enough key would outlast the heat death of the sun against brute force. Cryptography fails in the real world for completely different reasons: a key left in a source-code repository, a password "encrypted" with a hashing scheme from 1995, a random number that was not actually random, a certificate nobody noticed had expired, a developer who rolled their own cipher because the library looked complicated. Your job is not to be a cryptographer. Your job is to use the cryptographers' work correctly — to know which tool guarantees what, to recognize the handful of mistakes that account for nearly every real crypto failure, and to build the encryption standard that protects Meridian's customers whether or not the network around it holds.

This chapter is the bedrock for a great deal of what follows. The applied side of all this — TLS handshakes, VPNs, disk encryption — is Chapter 5's job; here we build the conceptual machinery underneath it. We will keep the mathematics honest but minimal: enough modular arithmetic to see why asymmetric encryption works, never more than a defender needs. And we will hand-trace every line of code rather than run it, because the point is to understand the guarantees, not to produce a hash.

In this chapter, you will learn to:

• Name precisely what each cryptographic primitive guarantees — confidentiality, integrity, authenticity, non-repudiation — and, just as important, what it does not.
• Choose between symmetric and asymmetric encryption, understand why real systems use both together, and solve the key-distribution problem that asymmetric cryptography exists to solve.
• Use a hash function correctly: for integrity, and — with a salt and a slow algorithm — for storing passwords so that a stolen database is not a catastrophe.
• Explain digital signatures, HMAC, and how a public key infrastructure (PKI) lets a stranger's certificate be trusted.
• Reason about key entropy and key sizes well enough to defend a "this is strong enough" decision to an auditor — and to spot the crypto mistakes that turn encryption into theater.

Learning Paths

Cryptography underpins every track, but it is the load-bearing chapter for engineers and certification candidates. Weight your reading accordingly:

🏗️ Security Engineer: This is your chapter. Read all of it, but linger on §4.2 (modes — where engineers actually go wrong), §4.6 (PKI — you will operate certificates for the rest of your career), and §4.7 (the failure catalog). The Meridian encryption standard in the checkpoint is a template you will reuse. 📜 Certification Prep: Every term here is on Security+ and CISSP. Master the symmetric-vs-asymmetric distinction, the hash-vs-encryption distinction, key-size rules of thumb, and the PKI components — these generate a disproportionate share of exam questions. The key-takeaways.md maps each to a domain. 🛡️ SOC Analyst: You will not design crypto, but you must recognize failed crypto in the wild — a certificate error, a downgrade, a hash you can identify in a leaked dump. Focus on §4.4 (password storage, so you understand credential-dump severity) and §4.7. 📋 GRC: Focus on §4.1 (what crypto does and doesn't guarantee, for policy language), §4.6 (PKI governance), and the Meridian encryption standard — the artifact you will hand an auditor against PCI-DSS and GLBA.

4.1 What cryptography guarantees — and what it doesn't

Before a single algorithm, get the vocabulary exact, because the most expensive crypto mistakes come from confusing what a primitive provides. People say "we encrypted it" and believe they have solved a problem they have not touched.

Start with the two words everything else hangs from. Plaintext is the original, readable data — a card number, a message, a file — before any cryptographic transformation. Ciphertext is the scrambled, unintelligible output after encryption. The transformation between them is governed by a key: a secret value (a string of bits) that parameterizes the algorithm so that the same public algorithm produces different output for different keys. This is the single most important idea in the whole field, and it is counterintuitive to newcomers: the security lives in the key, not in the secrecy of the algorithm. The algorithms — AES, RSA, SHA-2 — are public, published, scrutinized by thousands of researchers for decades. They are strong because they are public and have survived that scrutiny. What you keep secret is the key.

🚪 Threshold Concept: Kerckhoffs's principle — a cryptosystem should remain secure even if everything about it except the key is public knowledge. The corollary is the most practical rule in this chapter and one you will repeat to developers for the rest of your career: do not roll your own crypto, and do not rely on secret algorithms. A "proprietary encryption" with a hidden algorithm is a red flag every single time. Real security comes from a public algorithm whose only secret is a well-protected key. Once you internalize that the algorithm is supposed to be public, "security through obscurity" stops feeling clever and starts looking like a vulnerability.

Now, the four guarantees. Different primitives provide different ones, and a defender must ask, for any control, which of these it actually delivers:

• Confidentiality — keeping data secret from those who should not read it. This is what encryption provides. It is the guarantee everyone thinks of first, and the only one many people realize they need.
• Integrity — detecting whether data has been altered. This is what hash functions and message authentication codes provide. Encryption alone does not guarantee integrity — a subtle point we will return to, because assuming it does is a classic and dangerous mistake.
• Authenticity — proving who created or sent something. This is what HMAC (with a shared secret) and digital signatures (with key pairs) provide.
• Non-repudiation — the property that a signer cannot later credibly deny having signed. Only digital signatures provide this, because only they bind an action to a key that one party alone holds. (We met confidentiality, integrity, and availability as the CIA triad in §1.5; cryptography is one of the primary tools for the first two of those legs.)

Here is the trap, stated plainly because it has caused real breaches: encryption provides confidentiality, not integrity. If you encrypt a message and an attacker who cannot read it flips some bits of the ciphertext, you will decrypt it to garbage — or, with certain modes, to a different but valid-looking message the attacker partially controls. Encryption hid the data; it did not promise that what you decrypt is what was originally encrypted. Real systems therefore combine encryption with an integrity check, and the modern answer — authenticated encryption, which we meet in §4.2 — does both in one operation precisely so that engineers stop getting this wrong.

Equally important is what cryptography does not do, because over-trusting it is its own failure mode:

• Cryptography does not protect data in use. While your application is decrypting and processing a card number, it sits in memory as plaintext, where malware or a memory-scraping attacker (the technique behind several large retail breaches) can read it.
• Cryptography does not fix access control. If the attacker compromises an account that is authorized to decrypt the data, encryption does nothing — the system happily decrypts for them. Encryption protects against someone who steals the ciphertext, not against someone who steals the credentials that unlock it. This is why §4.7 insists that key management, not algorithm choice, is where you win or lose.
• Cryptography does not make a system trustworthy end to end. A signed software update from a compromised build pipeline is a perfectly valid signature on malware. The math was flawless; the trust was misplaced.

⚠️ Common Pitfall: "It's encrypted, so we're compliant / safe." This sentence has preceded many breaches. Encrypted how? With what algorithm? Where is the key? Who can decrypt? Is integrity protected? "Encrypted" is not a security property until those questions are answered. An auditor who hears "it's encrypted" and stops asking is not auditing; a defender who says it and stops thinking is not defending.

🔄 Check Your Understanding: 1. A developer says, "I encrypted the config file, so the database password in it is now tamper-proof." What is wrong with that claim? 2. Which single guarantee — confidentiality, integrity, authenticity, or non-repudiation — is provided by a digital signature but not by an HMAC, and why?

Answers

1. Encryption provides confidentiality (the password is now unreadable to someone who steals the file) but not integrity — without an integrity check, an attacker could alter the ciphertext, and depending on the mode you might not detect it. "Tamper-proof" requires an integrity/authenticity mechanism, not encryption. 2. Non-repudiation. A digital signature uses a private key only the signer holds, so the signer cannot deny it; an HMAC uses a secret shared by both parties, so either could have produced it — there is no way to prove which one did.

4.2 Symmetric encryption and modes of operation

The phishing email from Chapter 1 nearly reached Meridian's customer data. Suppose it had: the attacker lands on a file server holding an export of the cardholder data environment. Whether that export is a breach or a non-event comes down to one question — is it encrypted, and with what? The workhorse answer for data like this is symmetric encryption.

Symmetric encryption uses a single shared key for both encryption and decryption: the same secret that scrambles the plaintext unscrambles the ciphertext. Its defining property is speed — symmetric algorithms are extraordinarily fast, fast enough to encrypt every disk write, every database field, every packet on a busy network without anyone noticing. Its defining problem is the key-distribution problem: both parties must already share the secret key, and getting that key to the other party securely — without an eavesdropper grabbing it in transit — is genuinely hard. We solve that problem in §4.3; for now, assume both ends have the key.

The algorithm you will use, essentially always, is AES — the Advanced Encryption Standard, a symmetric block cipher standardized by NIST in FIPS 197. "Block cipher" means it encrypts data in fixed-size chunks (AES uses 128-bit blocks). AES comes in three key sizes — 128, 192, and 256 bits — and AES-128 and AES-256 are both considered secure for the foreseeable future; AES-256 is the common default for sensitive data and is what regulators and standards bodies expect to see. AES replaced the older DES (Data Encryption Standard), whose 56-bit key is now brute-forceable in hours and which survives only as a cautionary tale and in the slightly-less-dead form 3DES (Triple DES), now also deprecated. The rule for a defender is short: AES, 128- or 256-bit, full stop. You will almost never have a good reason to use anything else for symmetric encryption.

But here is where engineers actually go wrong, and where a defender must pay attention — not the algorithm, but the mode of operation. A block cipher only knows how to encrypt one 128-bit block. To encrypt anything larger (which is everything), you need a mode that defines how the blocks chain together. The mode matters enormously, and the wrong one can hand the attacker your plaintext through an unbroken cipher.

The cautionary mode is ECB (Electronic Codebook), the naive approach: chop the data into blocks and encrypt each one independently with the same key. The fatal flaw is that identical plaintext blocks produce identical ciphertext blocks. Patterns in the data survive encryption. The famous demonstration is the "ECB penguin": encrypt a bitmap image in ECB mode and you can still see the penguin, because regions of identical color encrypt to identical ciphertext. ECB leaks structure, and structure is often enough to recover meaning. Never use ECB for anything real. If you see ECB in a code review, that is a finding.

The fix is to introduce randomness so identical plaintext does not produce identical ciphertext. That randomness comes from an IV (initialization variable, almost always called the initialization vector) or a nonce ("number used once") — a value combined with the key so that encrypting the same plaintext twice yields different ciphertext. The critical rules: an IV/nonce does not need to be secret, but it must be unpredictable (for modes like CBC) or never reused with the same key (for modes like CTR and GCM). Reusing a nonce with the same key in certain modes is catastrophic — it can leak the relationship between the two plaintexts and, in GCM, can even let an attacker forge messages. This is one of the most consequential and least-known crypto mistakes, and we flag it again in §4.7.

Modern practice has converged on authenticated encryption with associated data (AEAD) — a mode that provides confidentiality and integrity and authenticity in a single operation, so the engineer cannot accidentally get the "encryption without integrity" trap from §4.1. The dominant AEAD mode is AES-GCM (Galois/Counter Mode); ChaCha20-Poly1305 is a widely used alternative, favored where hardware AES acceleration is absent. AEAD produces, alongside the ciphertext, an authentication tag — a short value that the recipient verifies on decryption; if even one bit of the ciphertext (or the associated data) was altered, the tag check fails and decryption is refused. This is the property §4.1 said you needed: tampering is detected, not silently decrypted into garbage.

🛡️ Defender's Lens: When you review how an application encrypts data, you are not checking "did they use AES?" — almost everyone does. You are checking the mode and the IV handling. The questions that find real bugs: Is it ECB? (instant finding). Is it an authenticated mode like GCM, or unauthenticated like CBC with no separate integrity check? Where does the IV/nonce come from — a cryptographic random source, and is it guaranteed unique per key? Is the same key used to encrypt an unbounded amount of data? A breach-grade vulnerability is far more likely to live in those answers than in the cipher itself.

Here is the contrast that organizes this section, as a labeled diagram:

        SYMMETRIC ENCRYPTION (one shared secret)

   PLAINTEXT ──► [ ENCRYPT ] ──► CIPHERTEXT ──► [ DECRYPT ] ──► PLAINTEXT
   "PAN: 4111…"      ▲             (unreadable)      ▲           "PAN: 4111…"
                     │                               │
                  KEY  K  ◄─────── same secret ──────┘
                 (e.g. AES-256, 32 random bytes)

   FAST. The problem: how do both ends get K without an eavesdropper
   grabbing it?  ──►  that is the key-distribution problem (§4.3).

   MODE matters as much as the cipher:
     ECB   identical plaintext -> identical ciphertext   ✗ leaks patterns
     CBC   chains blocks, needs unpredictable IV         ~ no built-in integrity
     GCM   counter mode + auth TAG (AEAD)                ✓ confidentiality + integrity

Figure 4.1 — Symmetric encryption. One key both ways; blazing fast; the hard parts are distributing the key (§4.3) and choosing a safe mode (prefer authenticated GCM).

🧩 Try It in the Lab: In your own environment, take a small text file and encrypt it twice with AES-256-GCM using your platform's standard crypto library (for example, the cryptography package in Python), supplying a fresh random nonce each time. Observe that the two ciphertexts differ even though the input was identical — that is the IV/nonce doing its job. Then read your library's documentation for what happens if you reuse a nonce; note that good libraries make the safe path the default. Do this only on data and systems you own.

🔄 Check Your Understanding: 1. Why is ECB mode dangerous even though it uses the same strong AES cipher as GCM? 2. An engineer hard-codes a single fixed IV ("to keep things deterministic") and reuses it for every message under the same key. Name the property they have destroyed.

Answers

1. Because ECB encrypts each block independently, identical plaintext blocks become identical ciphertext blocks, leaking the structure/patterns of the data regardless of how strong AES is. The weakness is in the mode, not the cipher. 2. They have destroyed the unpredictability/uniqueness the IV exists to provide: with a fixed IV, identical plaintexts again produce identical ciphertexts, and depending on the mode this leaks plaintext relationships (and in GCM can enable forgery). The IV must be unique (and for some modes unpredictable) per encryption.

4.3 Asymmetric encryption and the key-distribution problem

Symmetric encryption has a chicken-and-egg problem we deferred: to send someone an encrypted message, you both need the same secret key — but how do you agree on that key if you have never met and every channel between you might be wiretapped? You cannot just email the key; an eavesdropper reads the email. For decades this was cryptography's central unsolved problem, and it is the problem asymmetric encryption was invented to solve.

Asymmetric encryption (also called public-key cryptography) uses a mathematically linked pair of keys instead of one shared secret: a public key that can be published to the entire world, and a private key that is kept secret by its owner. The two are bound by a hard mathematical relationship, and the magic is the asymmetry in the name: data encrypted with the public key can only be decrypted with the matching private key. So anyone can encrypt a message to you using your published public key, and only you — holder of the private key — can read it. The eavesdropper sees the public key (it is public, after all) and the ciphertext, and can do nothing with them. The key-distribution problem dissolves: there is no secret to distribute, because the encryption key is meant to be public.

The two algorithms you must know are RSA and ECC. RSA (named for Rivest, Shamir, and Adleman) is the classic, and its security rests on a beautifully simple-to-state hardness: multiplying two large prime numbers is easy, but factoring their product back into those primes is, for large enough numbers, computationally infeasible. Schematically, RSA encryption is

$$c = m^e \bmod n$$

where $m$ is the message (as a number), $c$ is the ciphertext, $n$ is the public modulus (the product of two secret primes), and $e$ is the public exponent; decryption recovers the message with the private exponent $d$ via $m = c^d \bmod n$. You do not need to perform this arithmetic — libraries do — but seeing it demystifies the field: there is no magic, only modular exponentiation and the fact that recovering $d$ requires factoring $n$, which is hard. The catch is key size. Because factoring has gotten steadily easier with better algorithms and faster hardware, RSA needs large keys: 2048 bits is today's practical minimum, and 3072 or 4096 bits is recommended for data that must stay secret for a long time. A 1024-bit RSA key is considered too weak to deploy.

ECC — elliptic-curve cryptography — achieves the same goals with dramatically smaller keys by basing its hardness on a different problem (the elliptic-curve discrete logarithm). The headline number every defender should remember: a 256-bit ECC key provides security roughly comparable to a 3072-bit RSA key. Smaller keys mean faster operations and less data to move, which is why ECC dominates in mobile, IoT, and modern TLS. The trade-off is that ECC is more sensitive to implementation mistakes (a bad random number during signing can leak the private key — see §4.7), so it must be done with vetted libraries and never by hand.

Here is the crucial practical point, the one that ties this section back to the last: asymmetric encryption is slow — orders of magnitude slower than AES — and it can only encrypt small amounts of data (roughly, no more than the key size). You do not bulk-encrypt a database with RSA. So real systems use hybrid encryption, which combines both worlds and is exactly how TLS protects every "https" page you load (we walk the full handshake in Chapter 5):

1. Generate a fresh random symmetric key (a "session key") — say, an AES-256 key.
2. Encrypt the actual data with that fast symmetric key.
3. Encrypt only the small symmetric key with the recipient's public key, and send it alongside the ciphertext.
4. The recipient uses their private key to recover the symmetric key, then uses that to decrypt the bulk data.

You get the speed of symmetric encryption and the key-distribution solution of asymmetric encryption, each used for the job it is good at. This pattern — asymmetric to exchange a key, symmetric to do the work — is everywhere once you learn to see it.

     ASYMMETRIC: a key PAIR; what one key locks, only the other unlocks

   Alice wants to send Bob a secret.  Bob has a key pair.

        Bob's PUBLIC key  (published to the world)
                 │ encrypt
   PLAINTEXT ────┘                          ┌──── decrypt
       │                                    │   Bob's PRIVATE key
       └────────►  CIPHERTEXT  ─────────────┘        (secret)
                      ▲
            an eavesdropper sees the public key AND the ciphertext
            and still cannot read it.  No shared secret to steal.

   But asymmetric is SLOW + size-limited, so real systems go HYBRID:
     ┌─ RSA/ECC encrypts a small random AES key  (solves distribution)
     └─ AES encrypts the actual data             (solves speed)

Figure 4.2 — Asymmetric encryption solves key distribution; hybrid encryption marries it to symmetric speed. This is the architecture under TLS (Chapter 5).

🔗 Connection: The same key pair has a second superpower we will use in §4.5: reverse the roles. If Bob encrypts (signs) something with his private key, anyone can verify it with his public key — proving it came from Bob, since only he holds the private key. Public-key cryptography thus underlies both confidentiality (encrypt to a public key) and authenticity/non-repudiation (sign with a private key). One mechanism, two of the four guarantees from §4.1.

🔄 Check Your Understanding: 1. Why can a brand-new contact publish their public key on a website with no security risk, while their private key must be guarded absolutely? 2. Why do real systems almost never encrypt a large file directly with RSA, and what do they do instead?

Answers

1. The public key can only encrypt to the owner (and verify their signatures); it cannot decrypt or sign, so publishing it grants no power to an attacker. The private key is the only thing that can decrypt messages sent to the owner and create their signatures, so its compromise breaks everything. 2. RSA is slow and can only encrypt data smaller than its key size. Real systems use hybrid encryption: a fast symmetric key (AES) encrypts the file, and RSA/ECC encrypts only that small symmetric key.

4.4 Hashing, salting, and password storage

So far, cryptography has been reversible: encrypt, then decrypt. Now we need the opposite — a one-way transformation — and it solves a different family of problems, starting with the one that fills the news: stolen passwords.

A hash function takes input of any size and produces a fixed-size output (a hash, or digest) with three defining properties. First, it is deterministic: the same input always yields the same digest. Second, it is one-way (preimage-resistant): given a digest, you cannot feasibly compute an input that produces it — the function cannot be run backward. Third, it is collision-resistant: you cannot feasibly find two different inputs that produce the same digest. A small change to the input changes the output completely and unpredictably (the "avalanche effect"). Critically, a hash uses no key — it is a public function anyone can compute — which is exactly why it provides integrity (anyone can verify data is unaltered by recomputing the hash) but not confidentiality (it is not encryption, and there is nothing to "decrypt").

The algorithm family you should use is SHA-2 (Secure Hash Algorithm 2, standardized in NIST FIPS 180-4), most commonly SHA-256 (a 256-bit digest). SHA-3 (FIPS 202) is a newer, structurally different family standardized as an alternative — equally fine, less common in the field. What you must not use for security: MD5 and SHA-1. Both are broken — researchers can deliberately construct collisions (two different inputs with the same digest), which destroys the collision-resistance the security relies on. A collision is the event that should be infeasible: two distinct inputs hashing to the same value. For MD5, collisions can be produced in seconds on a laptop; for SHA-1, a collision was demonstrated publicly in 2017. Seeing MD5 or SHA-1 used to verify integrity or signatures is a finding. (You may still see MD5 used as a non-security checksum against accidental corruption; that narrow use is defensible, but never trust it against a deliberate attacker.)

Where hashing meets the headlines is password storage, and here the naive application of everything above is a disaster — so this is worth getting exactly right. You must never store passwords in plaintext (a database breach then hands the attacker every credential directly). The first instinct — "store the hash of the password, and at login hash what the user types and compare" — is correct in shape but, done naively, still fails badly against two attacks:

• Precomputed lookups (rainbow tables). Because a plain hash is unsalted and deterministic, an attacker can precompute the hashes of billions of common passwords once and simply look up every stolen hash. 5f4dcc3b... is password to anyone with such a table. The hash provided no protection against a guessing attacker.
• Speed. General-purpose hashes like SHA-256 are designed to be fast — billions of guesses per second on a GPU. Against a stolen hash database, the attacker just hashes guesses at enormous speed until matches appear. Fast is a feature for integrity and a catastrophe for passwords.

The defenses are two ideas you must pair. The first is a salt: a unique, random value generated per password and stored alongside the hash; you hash the salt plus the password together. The salt does not need to be secret — it lives right next to the hash — but because it is unique per user, it defeats precomputation entirely: a rainbow table would have to be rebuilt for every single salt, which is infeasible, and two users with the identical password now have completely different stored hashes. The second is to use a deliberately slow, password-specific hashing algorithm — bcrypt, scrypt, or Argon2 (the current recommendation) — which are engineered with a tunable work factor (cost) so that a single hash takes a meaningful fraction of a second. That is invisible to a legitimate user logging in once, but it slows an attacker's billions-per-second to a crawl, turning a feasible offline attack into an infeasible one. Modern algorithms like Argon2 and scrypt are additionally memory-hard, deliberately consuming large amounts of memory to blunt the GPU and custom-hardware advantage attackers rely on.

🛡️ Defender's Lens: When a "breach of N million passwords" hits the news, your first question as a defender — and the question that determines how bad it is — is how were they stored? Plaintext or unsalted MD5/SHA-1: assume every password is recovered; force a global reset; expect credential-stuffing against your other systems (Chapter 16). Salted bcrypt/Argon2 with a high work factor: most strong passwords are safe, weak ones will still fall slowly, and you have time to respond. The algorithm choice made years earlier sets the severity of the incident. This is Theme 4 in miniature — you assume the database will be stolen and design the storage so that theft is survivable.

⚠️ Common Pitfall: "We hash our passwords, so we're fine." Hashing with a fast, unsalted algorithm (plain SHA-256, or worse MD5) is barely better than plaintext against a real attacker. The phrase that matters is "salted, with a slow/memory-hard, password-specific algorithm (Argon2/bcrypt/scrypt) at an appropriate work factor." If a system hashes passwords with raw SHA-256 and no salt, write the finding. The canonical, deeper treatment of credential attacks and defenses is Chapter 16; what you need here is the principle: salt, and be slow on purpose.

A short hand-traced illustration of the shape of correct password storage (the algorithm here is shown schematically; in real code you call a library like Argon2, never a bare hash):

   REGISTER a user
     salt      = random 16 bytes        -> "9f1c…a7"  (unique per user, stored in clear)
     stored    = slow_hash(salt + password, work_factor=high)
                                          -> "$argon2id$v=19$…$<digest>"   (stored)
     # plaintext password is never stored

   LOGIN attempt
     candidate = slow_hash(salt + typed_password, work_factor=high)
     grant access  IFF  candidate == stored

   Two users, same password "Spring2026!", DIFFERENT salts
     -> completely different stored digests  -> rainbow tables useless

Figure 4.3 — Correct password storage: per-user salt plus a deliberately slow, memory-hard algorithm. The salt defeats precomputation; the slowness defeats brute force. Use a library (Argon2/bcrypt), never a bare fast hash.

🔄 Check Your Understanding: 1. A salt is stored in plaintext right next to the password hash. If it is not secret, what attack does it actually prevent, and how? 2. Why is SHA-256 an excellent choice for verifying a downloaded file's integrity but a poor choice for storing passwords?

Answers

1. A salt defeats precomputation (rainbow tables) and makes identical passwords hash differently. Because the salt is unique per user, an attacker cannot use a precomputed table; they would have to brute-force each password against its specific salt individually. Its value comes from uniqueness, not secrecy. 2. For file integrity you want speed and only need one-wayness and collision-resistance — SHA-256 is perfect. For passwords, speed is the enemy: SHA-256's billions-of-hashes-per-second lets an attacker brute-force a stolen database. Passwords need a deliberately slow, salted, memory-hard algorithm (Argon2/bcrypt/scrypt).

4.5 HMAC and digital signatures

We can now encrypt (confidentiality) and hash (integrity). But two gaps remain from §4.1: how do you prove a message was not just unaltered but came from a specific sender (authenticity), and how do you achieve the stronger property that the sender cannot later deny it (non-repudiation)? Two mechanisms close these gaps, and the difference between them — shared secret versus key pair — is a favorite exam distinction and a real design decision.

The first is the HMAC — Hash-based Message Authentication Code (standardized in FIPS 198-1). Recall that a bare hash provides integrity but not authenticity: anyone can recompute a hash, so a hash alone proves only that data is unaltered, not who produced it — an attacker who alters the message can simply recompute the hash to match. An HMAC fixes this by mixing a shared secret key into the hashing in a specific, carefully designed way: $\text{HMAC}(K, m)$ combines the key $K$ and the message $m$ so that only someone who holds $K$ can produce or verify the correct code. Now the tag proves two things at once: the message is unaltered (integrity) and it was produced by someone holding the shared key (authenticity). HMAC is fast, simple, and ubiquitous — it authenticates API requests, protects session tokens, and forms part of protocols throughout the stack. Its limitation is the limitation of all shared secrets: both parties hold $K$, so an HMAC proves the message came from one of them but cannot prove which — there is no non-repudiation, because either party could have produced it (and could plausibly blame the other).

The second mechanism is the digital signature, and it provides the property HMAC cannot: non-repudiation. A digital signature uses the asymmetric key pair from §4.3, run in reverse. To sign, you hash the message and then transform that hash with your private key; the result is the signature, attached to the message. To verify, anyone uses your public key to check that the signature corresponds to the message's hash. Because only you hold the private key, a valid signature proves the message came from you specifically and that it has not been altered — and, crucially, you cannot later deny it, because no one else could have produced it. That is non-repudiation, and it is why digital signatures, not HMACs, underpin software-update signing, signed legal documents, code signing, and the certificates in §4.6.

Note the role of hashing inside signing: you do not sign the whole (possibly huge) message with slow asymmetric math — you sign the hash, a small fixed-size digest. This is why a broken hash breaks signatures: if an attacker can find a collision (§4.4), they can get you to sign a benign document whose hash matches a malicious one, and your signature on the benign document is now also a valid signature on the malicious one. This is precisely why SHA-1's collisions retired it from signing. The common signing algorithms are RSA signatures and ECDSA (the elliptic-curve digital signature algorithm); both are standardized under NIST FIPS 186 (the Digital Signature Standard).

Here is the side-by-side that makes the choice concrete:

   AUTHENTICITY: two ways to prove who produced a message

   HMAC  (shared secret)                  DIGITAL SIGNATURE (key pair)
   ─────────────────────                  ────────────────────────────
   key K is SHARED by both ends           signer holds PRIVATE key alone
   tag = HMAC(K, message)                 sig = sign(PRIVATE, hash(message))
   verify with the SAME key K             verify with the signer's PUBLIC key
   ✓ integrity + authenticity             ✓ integrity + authenticity
   ✗ NO non-repudiation                   ✓ NON-REPUDIATION
       (either holder could make it)         (only the private-key holder could)
   fast, symmetric, great for             slower, asymmetric, for signing
   API calls / session tokens             updates, certificates, documents

Figure 4.4 — HMAC vs. digital signature. Both prove authenticity; only the signature — built on a private key one party alone holds — provides non-repudiation. Choose HMAC for fast mutual authentication, signatures when you must prove a single, undeniable origin.

🛡️ Defender's Lens: Signatures are where "trust the math" meets "trust the operation." A signature verifies the key that signed, not the intent behind it. When attackers stole a software vendor's code-signing private key (or compromised the build system that wielded it), they shipped malware with perfectly valid signatures — every verification passed, because the key was real. The defensive lessons are downstream of the math: protect signing keys like crown jewels (hardware-backed storage, tight access — the territory of Chapters 5 and 20), and treat signature verification as necessary but not sufficient. A valid signature tells you "this key signed this"; it cannot tell you the key was wielded by someone you trust.

🔄 Check Your Understanding: 1. Two services share a secret key and authenticate every request with an HMAC. Service A later claims it never sent a particular request. Can the HMAC settle the dispute? Why or why not? 2. Why does signing a message actually mean signing the message's hash, and what attack does this make possible if the hash function is weak?

Answers

1. No. Because the HMAC key is shared, either A or B could have produced the tag — the HMAC proves the request came from one of the two but not which, so it cannot disprove A's denial. Settling it requires a digital signature (A's private key) for non-repudiation. 2. Asymmetric operations are slow and size-limited, so you sign a small fixed-size hash of the message rather than the whole thing. If the hash is weak (collision-prone, like SHA-1), an attacker can craft two messages with the same hash, get the victim to sign the benign one, and reuse that signature on the malicious one — a chosen-prefix collision attack.

4.6 PKI, certificate authorities, and X.509 certificates

Public-key cryptography from §4.3 has a gaping hole we glossed over: when you download a public key claiming to belong to meridianbank.example, how do you know it really does? An attacker performing a man-in-the-middle attack can hand you their public key, claim it is the bank's, and now reads everything you "encrypt to the bank." Public keys solve key distribution but create an authentication problem: binding a key to a real-world identity. The system that solves it is public key infrastructure, and you will operate pieces of it for the rest of your career.

A public key infrastructure (PKI) is the whole framework of policies, roles, and technologies that creates, distributes, validates, and revokes digital certificates — binding public keys to verified identities. Its beating heart is the certificate authority (CA): a trusted third party that vouches for the binding between a public key and an identity by issuing a digitally signed certificate. The CA's signature (a digital signature, §4.5) is the vouching: "I, this CA, have verified that this public key belongs to meridianbank.example, and I stake my reputation on it." The whole system reduces the impossible problem of "trust millions of strangers' keys" to the tractable one of "trust a small number of well-known CAs," whose own keys are pre-installed in your operating system and browser as trusted root certificates.

The certificate itself follows the X.509 standard — the near-universal format for a public-key certificate. An X.509 certificate is essentially a structured document containing: the subject (whose identity this is — e.g., the domain name), the subject's public key, the issuer (which CA signed it), a validity period (not-before and not-after dates), a serial number, the allowed key usages (e.g., "server authentication"), and — making the whole thing tamper-evident — the CA's digital signature over all of the above. Anyone can verify that signature using the CA's public key; if it checks out and the CA is trusted, the binding is trusted.

Trust flows through a chain of trust (the certificate chain), and understanding it is what separates someone who can actually debug a TLS error from someone who just clicks "proceed anyway." A root CA almost never signs end-entity certificates directly; instead it signs intermediate CA certificates, which in turn sign the end-entity (leaf) certificate for a server. Verification walks the chain upward: the server's certificate is signed by an intermediate, the intermediate is signed by the root, and the root is one your system already trusts. Each link is a digital signature you can verify. If any link is missing, expired, or untrusted, the chain breaks and validation fails.

   THE CHAIN OF TRUST (how a stranger's certificate becomes trustworthy)

   ┌─────────────────────────┐
   │  ROOT CA certificate    │  pre-installed & trusted by your OS/browser
   │  (self-signed)          │  — the anchor of trust
   └───────────┬─────────────┘
               │ signs
   ┌───────────▼─────────────┐
   │  INTERMEDIATE CA cert   │  signed by the root
   └───────────┬─────────────┘
               │ signs
   ┌───────────▼─────────────┐
   │  LEAF (server) cert     │  subject: meridianbank.example
   │  subject pubkey + CA sig│  contains the bank's public key
   └─────────────────────────┘

   Verify each ▲ signature up to a root you already trust.
   Break ANY link (expired, untrusted issuer, name mismatch) -> validation FAILS.

Figure 4.5 — The certificate chain. Trust in millions of servers reduces to trust in a handful of pre-installed roots, transmitted downward by digital signatures.

Two more pieces complete the picture, because certificates do not just get issued — they expire and sometimes go bad. Revocation is the mechanism to declare a certificate invalid before its expiry, used when a private key is compromised or a certificate was mis-issued. The two revocation mechanisms are CRLs (Certificate Revocation Lists — published lists of revoked serial numbers) and OCSP (the Online Certificate Status Protocol — a live query to ask "is this specific certificate still good?"). Revocation is famously imperfect (clients do not always check, and checking has privacy and availability costs), which is one reason the industry has shifted toward short-lived certificates that simply expire before revocation would matter. The full certificate lifecycle — issuance, renewal, revocation, and the operational discipline to never let a production certificate expire unnoticed — is Chapter 5's territory; the lesson to carry from here is that PKI is not "set up once" but an ongoing operational responsibility.

⚠️ Common Pitfall: The single most common self-inflicted PKI outage is the expired certificate. A certificate nobody was tracking reaches its not-after date, and suddenly a critical service throws security errors and customers cannot connect — a self-inflicted denial of service (an availability failure, the third leg of the CIA triad from §1.5). Major outages at large companies have come down to exactly this. The defense is unglamorous and essential: an inventory of every certificate, automated renewal, and alerting well before expiry. We build a cert_days_left helper for exactly this purpose in Chapter 20.

📟 War Story: A constructed but representative incident. Meridian's online-banking certificate was issued for one year and renewed manually by whoever happened to remember. One year the person who remembered had left the bank. At 00:00 on the not-after date, the certificate expired; by the time the first customer-service calls arrived ("my browser says the bank is unsafe!"), the mobile app — which pinned the certificate — was failing for everyone, and the on-call engineer spent two hours discovering that nothing was hacked at all. The cause was an operational gap, not an attacker. Dana's after-action note was one line: "Cryptography we fail to operate is a vulnerability we created ourselves." Certificate lifecycle management went into the program that quarter.

🔄 Check Your Understanding: 1. What specific problem does a certificate authority solve that raw public-key cryptography (§4.3) leaves open? 2. Your browser trusts a website's certificate even though it has never seen that website's CA before — only an intermediate, which it also has never seen. How does trust still reach it?

Answers

1. It solves identity binding/authentication: raw public-key crypto lets anyone publish a key, but provides no way to know the key actually belongs to the claimed identity. A CA vouches for that binding with its digital signature, preventing a man-in-the-middle from substituting their own key. 2. Through the chain of trust: the leaf certificate is signed by the intermediate CA, and the intermediate is signed (directly or via further intermediates) by a root CA that is pre-installed and trusted in the browser. The browser verifies each signature up the chain to that trusted root.

4.7 Crypto failures to avoid

Here is the chapter's most practical section, and the one a defender returns to most. Modern cryptographic algorithms are, for all intents, unbreakable — so essentially every real-world crypto failure is a failure of implementation, configuration, or operation. If you internalize this catalog, you will catch the overwhelming majority of crypto problems you will ever face, none of which require breaking any math.

1. Rolling your own crypto. The cardinal sin (§4.1). Cryptography is breathtakingly hard to implement correctly; subtle flaws that would never occur to a competent engineer are routinely fatal. Custom ciphers, homemade key-exchange schemes, and "we XOR with a secret string" are vulnerabilities, not features. Use vetted, standard libraries (your platform's crypto library, libsodium, the cryptography package) and standard algorithms. The only acceptable amount of homemade cryptography in production is none.

2. Weak or deprecated algorithms. Using MD5 or SHA-1 where collision-resistance matters; DES/3DES or RC4 for encryption; RSA-1024; ECB mode. Each of these is broken or badly weakened and each is a finding. The defender's quick-reference (and the table in key-takeaways.md) exists so you can recognize them on sight.

3. Bad randomness. This one is subtle and devastating. Cryptography depends utterly on unpredictable random numbers — for keys, IVs/nonces, and salts. If the random number generator is predictable (using a non-cryptographic PRNG like a language's default random(), or seeding from a guessable value like the current time, or a hardware source with too little entropy at boot), an attacker can predict the "secret" values and the whole system collapses while every algorithm remains "unbroken." Real catastrophes have come from exactly this: predictable keys generated by devices with insufficient entropy, and — for ECDSA specifically — a reused or predictable nonce during signing leaks the private key outright. Always use a cryptographically secure random source (secrets in Python, /dev/urandom, your platform's CSPRNG), never the general-purpose one. This is the practical motivation for the entropy_bits function in this chapter's checkpoint.

4. Nonce/IV reuse. Reusing a nonce with the same key (§4.2) — especially in GCM and other counter modes — leaks plaintext relationships and can enable message forgery. Unique IV/nonce per encryption is not optional.

5. Key management failures — where defenders actually lose. This is the big one, and it is operational, not mathematical. The strongest AES-256 in the world is worthless if: - the key is hard-coded in source code or a config file that lands in a Git repository (a leak we hunt for in Chapter 20); - the key is stored next to the data it protects, so stealing the data steals the key; - the key is never rotated, so one compromise exposes years of data; - everyone has access to the key, so a single compromised account decrypts everything. The discipline of generating, storing, distributing, rotating, and destroying keys safely — key management — is where real programs succeed or fail, and it is why Chapter 5 covers hardware-protected key storage and Chapter 20 covers secrets management. Cryptography moves the security problem from "protect the data" to "protect the key"; if you do not then protect the key, you have moved the problem, not solved it.

6. Encryption without integrity. Using an unauthenticated mode (CBC alone) without a separate integrity check, leaving ciphertext malleable (§4.1, §4.2). Prefer authenticated encryption (AES-GCM).

7. Trusting a valid signature too much. A valid signature proves a key signed something, not that the intent was legitimate (§4.5). Compromised signing keys produce valid signatures on malware. Verification is necessary, not sufficient.

🛡️ Defender's Lens: Almost none of these are "the crypto was broken." They are "the crypto was used wrong." That is good news for a defender, because it means your highest-leverage crypto work is not advanced mathematics — it is code review and configuration audit: grep for MD5, DES, ECB, and random() in a non-crypto context; hunt for hard-coded keys; confirm authenticated modes; confirm a CSPRNG; confirm salts and slow password hashing; confirm certificates are inventoried and not expiring. Every one of those is a defensive skill you already have, pointed at the right targets.

⚖️ Authorization & Ethics: The same understanding that lets you audit crypto could, misapplied, let you attack weak crypto on systems that are not yours — and attacking cryptographic protections on systems you do not own or are not authorized to test can violate computer-crime law. As throughout this book: apply these skills to your own systems or with explicit written authorization. Studying how crypto fails is how a defender prevents those failures, not a license to exploit them elsewhere.

🔄 Check Your Understanding: 1. Of the seven failure categories above, which one is operational rather than algorithmic, and why is it described as "where defenders actually lose"? 2. A code review finds keys generated with the language's default random() function seeded from the current timestamp. Which failure category is this, and what is the consequence?

Answers

1. Key management (category 5). The algorithm can be perfect, but if the key is hard-coded, stored next to the data, never rotated, or broadly accessible, the protection fails entirely — and these are the mistakes real organizations make most, because they are mundane operational lapses rather than exotic math. 2. Bad randomness (category 3). A non-cryptographic PRNG seeded from a guessable value (the time) produces predictable keys; an attacker who can guess or narrow the seed can reproduce the keys, defeating the encryption while the cipher itself remains "unbroken." Use a CSPRNG.

Project Checkpoint

This chapter advances Meridian's program with the artifact every regulated organization eventually needs and every auditor asks for, and adds the cryptographic core of bluekit.

Program increment — Meridian's encryption standard. After the near-miss, and with a PCI-DSS assessment on the horizon, Dana asks Sam (the security engineer) to write Meridian's encryption standard: the one-page policy that says, in approved language, which algorithms and key sizes are permitted, and how keys are handled. This is exactly the document §4.7 implies — it pre-decides the right answers so engineers cannot drift into weak ones. The standard Sam drafts says, in essence:

• Data at rest (databases, backups, the cardholder data environment): AES-256, authenticated mode (GCM) where the platform supports it; transparent database/disk encryption acceptable for breadth, with field-level encryption for the most sensitive elements (full card numbers).
• Data in transit: TLS with strong cipher suites only (the specifics, and the handshake, are Chapter 5's — this standard simply points there and forbids deprecated protocols).
• Hashing: SHA-256 (or SHA-3) for integrity; never MD5 or SHA-1 for security. Passwords: salted Argon2 (or bcrypt) at an approved work factor — never a bare fast hash (the authentication standard in Chapter 16 elaborates).
• Asymmetric: RSA-3072+ or ECC P-256+; RSA-1024 prohibited.
• Randomness: cryptographically secure sources only.
• Key management: no keys in source code or config; keys stored separately from the data they protect; defined rotation; least-privilege access to keys. (Hardware-protected key storage and full key lifecycle are specified in Chapter 5; secrets handling in Chapter 20.)

This standard is the data-protection backbone of Meridian's PCI-DSS posture (PCI-DSS Requirement 3 governs protection of stored cardholder data, and Requirement 4 governs transmission — we map the full compliance picture in Chapter 28). It is the directly testable answer to "how does Meridian protect card data at rest?" — the question this chapter's anchor exists to answer.

bluekit increment — cryptutil.py. We add the cryptographic helpers a defender reaches for: a correct hash wrapper, an HMAC for message authenticity, and an entropy calculator to judge whether a key or password has enough randomness. As everywhere in this book, the code is illustrative and never executed during authoring — the expected output is hand-traced into a comment using clearly-illustrative values.

# bluekit/cryptutil.py  — Chapter 4 increment
"""Defensive crypto helpers built on the standard library's vetted primitives.
We do NOT implement cryptography ourselves — we call hashlib/hmac/secrets.
All outputs below are hand-traced illustrations; nothing is run at authoring.
"""
import hashlib, hmac, math

def sha256_hex(data: bytes) -> str:
    """Hex SHA-256 digest of bytes — for INTEGRITY, never for passwords."""
    return hashlib.sha256(data).hexdigest()

def hmac_sign(key: bytes, msg: bytes) -> str:
    """HMAC-SHA256 tag (authenticity + integrity) for a shared-key message."""
    return hmac.new(key, msg, hashlib.sha256).hexdigest()

def entropy_bits(charset_size: int, length: int) -> float:
    """Approx. entropy (bits) of a random secret: length * log2(charset)."""
    if charset_size < 2 or length < 1:
        raise ValueError("charset_size >= 2 and length >= 1 required")
    return round(length * math.log2(charset_size), 1)

if __name__ == "__main__":
    print(sha256_hex(b"meridian"))             # integrity digest (illustrative)
    print(hmac_sign(b"shared-key", b"transfer:500"))  # auth tag (illustrative)
    print(entropy_bits(94, 12))                # 12-char full-keyboard password
    print(entropy_bits(2, 128))                # a 128-bit random key

# Expected output (illustrative — hashes shown truncated/fake, NOT real digests):
# 9c1185a5c5e9fc54612808977ee8f548b2258d31...   <- 64 hex chars in reality
# 7f9a2b41c0d3e6f5a8b1c2d3e4f5061728394a5b...   <- HMAC tag, 64 hex chars
# 78.7     <- 12 chars * log2(94) ~= 12 * 6.555 = 78.66 -> rounds to 78.7 bits
# 128.0    <- 128 chars * log2(2) = 128 * 1 = 128 bits (a full AES-128 key)

Hand-trace the two entropy_bits lines, because the arithmetic is the teachable part and the hashes are not (a hash digest cannot be predicted by hand — that is the whole point of §4.4, which is why those two lines are shown truncated and explicitly labeled fake). For a 12-character password drawn from the ~94 printable-ASCII characters: $12 \times \log_2 94 = 12 \times 6.555 \approx 78.7$ bits — strong. For a 128-bit key, each bit is one of two values, so entropy is $128 \times \log_2 2 = 128 \times 1 = 128$ bits, by definition. The function turns "is this strong enough?" into a number you can defend: a useful rule of thumb is that 128 bits of entropy is the modern floor for a key or a high-value secret, and entropy_bits lets you check a password policy or a token generator against it. With these three functions, bluekit can now verify integrity, authenticate a message, and grade a secret's strength — the practical crypto a defender uses most.

Summary

This chapter built the cryptographic foundation the rest of the book stands on — focused, as a defender must be, on using crypto correctly rather than breaking it.

• The guarantees, kept distinct: encryption provides confidentiality; hashing and MACs provide integrity; HMAC and digital signatures provide authenticity; only digital signatures provide non-repudiation. Encryption does not provide integrity — use authenticated encryption.
• Symmetric encryption (one shared key — AES-128/256) is fast and does the bulk work; the mode matters as much as the cipher — never ECB, prefer authenticated AES-GCM, and never reuse a nonce/IV with the same key.
• Asymmetric encryption (a public/private key pair — RSA or ECC) solves the key-distribution problem but is slow, so real systems use hybrid encryption: asymmetric to exchange a symmetric key, symmetric to encrypt the data. Key-size floors: RSA-3072+ (2048 minimum) or ECC P-256+.
• Hash functions (use SHA-2/SHA-256 or SHA-3; never MD5/SHA-1 for security) are one-way and collision-resistant. For passwords, add a per-user salt and a deliberately slow, memory-hard algorithm (Argon2/bcrypt/scrypt) — never a bare fast hash. The storage choice sets the severity of a future breach.
• PKI binds keys to identities: a certificate authority signs X.509 certificates, and trust flows down a chain of trust from pre-installed roots. Operating certificates (renewal, revocation, not letting them expire) is an ongoing responsibility, not a one-time setup.
• Crypto failures are almost never broken math — they are rolling your own, weak algorithms, bad randomness, nonce reuse, key-management lapses, encryption without integrity, and over-trusting a valid signature. The defender's leverage is code review and configuration audit, not cryptanalysis.
• Meridian gained an encryption standard (the PCI-DSS data-protection backbone) and bluekit gained cryptutil.py — sha256_hex, hmac_sign, and entropy_bits.

Spaced Review

Open your memory before moving on. Without scrolling up, answer these — two from this chapter's recent neighbors (Chapters 2 and 3), reinforcing the spaced-repetition habit:

1. (Ch.3) The cardholder data export is encrypted with AES-256. Classify this control by function (preventive / detective / corrective) and by type (administrative / technical / physical). Which leg(s) of the CIA triad does it serve?
2. (Ch.3) Meridian's encryption standard says "no keys in source code; least-privilege access to keys." Which security principle from Chapter 3 does the second clause apply, and how does it limit the blast radius of one compromised account?
3. (Ch.2) A digital signature on a software update was valid, yet the update was malware because the vendor's build pipeline was compromised. Where in the cyber kill chain does weaponizing a trusted update channel sit, and why does this make supply-chain attacks so effective?
4. (Ch.4) State the difference between an HMAC and a digital signature in one sentence, and name the one guarantee only the signature provides.

What's Next

You now have the cryptographic primitives and — more important for a defender — the judgment about how they fail. But primitives are not yet a working defense. Chapter 5 takes everything here and puts it to work in the systems you actually operate: the TLS 1.3 handshake that protects Meridian's online banking (you will finally see hybrid encryption, signatures, and certificates cooperate in one protocol), VPNs and IPsec for data in transit, full-disk and database encryption for data at rest, the certificate and key lifecycle (including hardware-protected key storage), and how to scan for weak TLS configurations in your own environment. The math you learned here is about to become a handshake you can read packet by packet — and the encryption standard you drafted for Meridian is about to acquire the operational detail that makes it real.