Exercises: Security Metrics, Measurement, and Reporting to the Board

These exercises move from computing metrics to exercising judgment about which metrics to report and how to frame them for a board. Difficulty is marked ⭐ (recall/application), ⭐⭐ (analysis), and ⭐⭐⭐ (synthesis/open-ended). A dagger (†) marks problems with a full worked solution in Appendix: Answers to Selected Exercises — try every problem before you read one.

Where a problem asks you to "build a slide" or "choose metrics," there is rarely one perfect answer; the reasoning — and your ability to defend each number against "what decision does this change?" — matters more than the exact wording. Use documentation values only; all figures here are illustrative.

Part A — The vocabulary of measurement ⭐

1.† In one sentence each, define security metric, vanity metric, key performance indicator (KPI), and key risk indicator (KRI). Then write one sentence that correctly uses all four in the context of Meridian's patching program.

2. Classify each as a likely vanity metric or useful metric, and give the one-phrase reason: (a) "firewall blocked 2.4M packets this quarter"; (b) "94% of endpoints run EDR, target 100%"; (c) "total SIEM alerts generated"; (d) "median days to patch critical vulns, trended 6 months"; (e) "number of security awareness emails sent"; (f) "percent of critical systems sending logs."

3. Explain the difference between a KPI and a KRI using one Meridian example of each. Then explain why a board cares more about the KRI than the KPI.

4.† Define control coverage and explain — with a concrete example — why the denominator is the part most likely to make a coverage metric lie. What single capability from earlier in the book must be solid for any coverage metric to be trustworthy?

5. Match each metric to the NIST CSF function it best evidences (Govern, Identify, Protect, Detect, Respond, Recover): (a) MTTD; (b) EDR coverage; (c) MTTR-to-containment; (d) asset-inventory completeness; (e) backup-restore success rate; (f) percent of policies reviewed on schedule.

Part B — Compute it (MTTD / MTTR / coverage) ⭐⭐

6.† Compute MTTD and MTTR. Five (illustrative) incidents this quarter, times in hours from a common origin:

(a) Compute MTTD. (b) Compute MTTR (detection → containment). (c) Compute the median detection time. (d) The mean MTTD and the median differ sharply — explain why, and state which you would put in front of a board and why. (e) Which single incident is the program's biggest improvement opportunity?

7. Compute coverage. Meridian has 240 servers, 52 privileged accounts, and 70 critical systems. EDR runs on 228 servers; MFA covers 52 privileged accounts; 59 critical systems send logs. (a) Compute each coverage percentage. (b) Which figure is most reassuring, and why should you still interrogate it? (c) Which figure points at the most urgent, most actionable task, and what is that task?

8.† A team is told to "improve MTTR by 30% this quarter." Describe two ways a team could make the MTTR number drop without actually responding faster or better — and for each, name the guardrail metric that would expose the gaming. What does this illustrate about setting any single metric as a target?

9. An analyst computes MTTD by measuring from "the time the SIEM first alerted" to "the time an analyst acknowledged it." Explain precisely why this systematically understates true MTTD, and what it should measure instead. Reference the role of forensics.

10.† Analyze this metric. A monthly report shows: "Vulnerabilities found: 2,100 (down from 2,400). Vulnerabilities remediated: 1,800. Critical vulns open past SLA: 47 (up from 31)." (a) Which of these is the KRI a board should see, and why? (b) The first two numbers are trending in a "good" direction while the third worsens — what story is actually being told, and why would reporting only the first two mislead? (c) Recommend the single sentence you would lead the board slide with.

Part C — Analyze this report (judgment & framing) ⭐⭐

11.† Pick the right metric (vs. vanity). A CISO's draft board slide lists six numbers: (1) "12M attacks blocked", (2) "MTTD 5.5 h, down from 9 h", (3) "1,400 vulnerabilities", (4) "security risk: 1 of 5 dimensions above appetite", (5) "98% training completion", (6) "MTTR 6.9 h vs. ~12 h peer". Keep the three most board-worthy, cut the three weakest, and justify every keep and cut in one phrase each. Then write the one-sentence headline the kept three support.

12. A maturity report claims the program is at "Level 2.47." Identify the two distinct problems with that figure (one about the decimals, one about how the score was likely derived) and rewrite the claim as you would present it, including what evidence you would attach.

13.† Reframe for the board. Rewrite each technical statement as a board-legible, risk-focused one: (a) "We deployed 140 ATT&CK-mapped detection rules." (b) "We patched the Log4Shell instances in 6 days." (c) "We have 9 critical systems not yet logging." (d) "Maturity rose from 2.0 to 2.5."

14. A director asks: "You show four green metrics and one amber. Is the amber really the only thing that worries you?" Explain why the honest answer matters more than the reassuring one here, and what an all-green deck signals to an experienced board member. Tie your answer to credibility as a currency.

15.† A peer benchmark on a board slide reads "Our MTTD (5 h) beats the industry average (8 h)." Critique this from a citation-honesty standpoint: what must be true for this to be a fair claim, how should it be sourced and labeled, and how could a fabricated or cherry-picked benchmark backfire in the room?

Part D — Build it / design it ⭐⭐–⭐⭐⭐

16.† Build a board slide. Using the four Meridian incidents from §36.3 of the chapter (MTTD 5.5 h, MTTR 6.9 h, median MTTD 1.7 h) and the coverage figures (EDR 95%, MFA 100%, logging 85%), draft the text of a single executive "Response & Coverage" slide. It must: lead with one risk-focused sentence, present no more than five numbers, include the median beside the mean to handle the outlier, and pre-empt one likely director question in a footnote.

17. Design a dashboard. Design (in words or ASCII) the operational dashboard Marcus's SOC would use daily. List 6–8 operational metrics, state the decision each one drives, and explain why none of them belong on the board deck. Contrast it explicitly with the executive scorecard in Figure 36.3.

18. Design a dashboard (executive). For a different organization — a 5,000-bed hospital system — design a one-screen board scorecard. Choose the top risks (remember the CIA re-weighting toward availability and safety), pick the 5–7 metrics, and state which board question each answers. How does it differ from Meridian's bank scorecard, and why?

19.† Write the metric definition. Write a precise, gaming-resistant definition of "Mean Time to Respond" for Meridian's IR program: state the start event, the end event, what is included/excluded (e.g., business-hours vs. clock time, which severities count), and the paired guardrail metric. The goal is a definition two different analysts would compute identically.

20. Build the maturity story. Given Meridian's maturity table (overall 2.0 → 2.5, target 3.0; third-party risk the laggard at 2.0), draft the three-sentence narrative Dana would speak to the board while that slide is up — leading with the trajectory, naming the laggard as the priority, and connecting the target to a funded ask.

21. ⭐⭐⭐ Design the whole pack. Outline (slide-by-slide, 6–7 slides) a board metrics pack for an organization you know. For each slide name its purpose, the board question it answers, and the single metric or visual it carries. Then state the one-sentence headline the whole pack supports and the one "watch" item you would put on the honesty slide.

Part E — Respond to this (governance scenario) ⭐⭐

22.† Respond to this situation. Two days before the board meeting, Theo discovers that the "85% critical-system logging coverage" figure on the draft deck is wrong: the denominator excluded 14 systems that a recent acquisition added, and true coverage is closer to 68%. The meeting is set; the slide is "done." Walk through, in steps, what Dana should do — about the number, about the deck, and about what (if anything) to say in the room — and justify it in terms of credibility and the integrity-of-measurement principle.

23. A new board member, a retired CFO, says: "I don't understand any of these security metrics. Just tell me — are we going to get breached?" Draft Dana's two-to-three-sentence response that is honest (you cannot promise "no"), useful (gives the director something to govern with), and reframes the question toward risk, trend, and appetite.

Part F — CTF-style challenge ⭐⭐⭐

24.† The dashboard that hid the breach. You are handed a glowing quarterly security report from a company that was breached the following month. Every metric is green: "10M attacks blocked," "99.5% uptime," "100% antivirus coverage," "98% training completion," "zero incidents this quarter." Using only this report, identify (a) at least three reasons "zero incidents" is more alarming than reassuring given the other numbers; (b) which crucial metrics are conspicuously absent (name at least four — think MTTD/MTTR, coverage denominators, risk-vs-appetite, detection coverage); and (c) the one question you would have asked this CISO that the all-green deck was structured to prevent. Write it up as a half-page "what this report didn't tell you" memo. (Case Study 2 is the long-form version of this exercise.)

Part G — Interleaved & forward-looking ⭐⭐

25. (Interleaved — Chapters 27, 21, 23.) Trace one metric down through three earlier chapters: take "percent of critical vulnerabilities remediated within SLA" and explain (a) how Chapter 23 produces the raw data, (b) how it connects to the Chapter 27 risk register and risk appetite, and (c) how the Chapter 21 SIEM/logging stack would help you verify a related coverage claim. Then state whether this metric is a KPI, a KRI, or both, and for whom.

26. (Interleaved — Chapter 24.) MTTR measures response time, and Chapter 24 built the IR plan and playbooks that response runs on. Explain how a blameless postmortem both improves future MTTR and protects the honesty of the MTTR metric (consider what pressure to "look good on MTTR" does to a team's willingness to report the slow, ugly incidents).

27. ⭐⭐⭐ Open reflection — forward to Chapter 38. The chapter says this metrics pack is the part of the capstone that answers "is it working?" Write half a page predicting which other checkpoints from the book (risk register, architecture, IAM, SOC, compliance) will supply which slides of the full Chapter 38 board presentation, and what the single hardest thing to measure honestly across the whole program will be.

Solutions to daggered (†) problems are in the Answers appendix. The remaining problems are deliberately open — bring them to a study group, your instructor, or a willing CISO.