Key Takeaways: AI and Machine Learning in Security
A one-page reference. Reread before an exam or before moving on. Dense by design.
When to use ML at all (the default is "write the rule")
| If… | Then use… | Because |
|---|---|---|
| You can state the rule exactly | Deterministic rule | Explainable, cheap, no training data, can't be poisoned |
| You have many clean, labeled examples of both classes | Supervised ML | Accurate on the known (spam, malware, phishing); blind to the novel |
| "Different from this entity's own past" is a useful signal, no labels | Unsupervised / anomaly detection + UEBA | Surfaces the unseen; cannot say if it's malicious |
| None of the above | Better logging, an allowlist, or a person | ML adds cost/opacity/fragility for no gain |
Threshold concept: Anomalous ≠ malicious, and malicious ≠ anomalous. A backup job is anomalous and harmless; a "low and slow" attacker using valid credentials is malicious and statistically normal.
Supervised vs. unsupervised detection
| Supervised | Unsupervised (anomaly) | |
|---|---|---|
| Learns from | Labeled examples | Unlabeled data (learns "normal") |
| Output | A predicted label ("phishing") | "Unusual" + a score (not "why") |
| Strong at | Catching things like training data | Surfacing never-before-seen deviation |
| Blind to | Genuinely novel attacks | Attackers who stay statistically normal |
| Needs | Large, clean, labeled set (rare in security) | A stable, uncontaminated baseline |
Anomaly detection: the z-score (do it by hand)
$$\mu = \tfrac{1}{n}\sum x_i \qquad \sigma = \sqrt{\tfrac{1}{n}\sum (x_i-\mu)^2} \qquad z = \frac{x-\mu}{\sigma}$$
Flag when $\lvert z\rvert \ge$ threshold (commonly 3 — but that's a business decision, not a law).
Worked anchor: baseline $[2,3,2,4,3,2,3,4,2,5] \Rightarrow \mu=3.0,\ \sigma=1.0$; tonight $=9 \Rightarrow z=(9-3)/1=6.0 \Rightarrow$ anomaly.
Where the z-score breaks (say it out loud):
- Assumes a stable, single-peaked baseline (multi-modal data → baseline per context).
- Contaminated baseline → attacker already inside looks normal (this is poisoning, §34.4).
- A few extreme points inflate $\sigma$ → desensitizes the detector (use median/MAD — robust stats).
- Says nothing about why (the explainability tax).
- Zero-variance baseline → $z$ undefined; the function should raise, not return a number.
UEBA (the multi-signal generalization)
Build per-entity baselines across many features; fuse weak per-feature anomalies into one risk score. Catches account takeover and slow insider drift that no single feature or signature would.
| Design element | Example ("compromised teller account") |
|---|---|
| Entity | Individual user accounts |
| Features | Login hour, source geo/IP, accounts viewed, data exported, first-time access |
| Baseline | Per-user, 30 days, business vs. off-hours, + peer group |
| Scoring | Weighted sum of per-feature z-scores; tuned combined threshold |
Cost of fusion: harder to explain, more tuning (feature weights + combined threshold).
False-positive economics (the math that decides everything)
The base rate: attacks are rare, so even a great detector yields a mostly-false queue.
Anchor: 1,000,000 logins/day, 100 malicious, detector 99% TP-rate & 1% FP-rate → TP $=99$, FP $\approx 9{,}999$, queue $\approx 10{,}098$, precision $\approx 1\%$ (99 of 100 alerts false).
| Demand this | Not this |
|---|---|
| Precision (alerts that are real) + recall (attacks caught) + alert volume at the threshold | "Accuracy" (a do-nothing model is 99.99% accurate on rare events) |
Levers to make a detector workable: (1) raise threshold to fit SOC capacity (cost: more false negatives — document them); (2) narrow scope to shrink the denominator (best lever); (3) stack independent signals; (4) enrich before alerting; (5) risk-rank, don't binary-alert.
Adversarial ML (the model is a target — MITRE ATLAS)
| Attack | What it does | When | Defenses |
|---|---|---|---|
| Data poisoning (availability) | Inject bad samples → accuracy collapses | Training | Curate/validate data; monitor accuracy; held-out clean test set |
| Data poisoning (backdoor/targeted) | Model accurate except on a secret trigger → benign | Training | Data provenance; watch for trigger clusters |
| Gradual baseline poisoning | Attacker already inside; baseline drifts to embrace them | Online | Slow, vetted, human-reviewed baselines; robust stats; peer check |
| Model evasion | Craft input misclassified (malware scores "clean") | Inference | Don't be the only gate; limit/log queries; adversarial training; ensembles; human + context |
Unifying defense = Theme 4 (assume the layer fails): put the model behind deterministic rules and in front of human review, with validated inputs and monitored queries.
AI-enabled attacks + LLMs in the SOC
AI-enabled attack: uses AI to scale/sharpen existing attacks — it rarely invents new ones.
| Threat | Defense (mostly old fundamentals) |
|---|---|
| Machine-generated spear-phishing / BEC | Retire "spot the typo"; teach urgency+authority+secrecy; DMARC (Ch.9) |
| Deepfake voice/video fraud (wire authorization) | Out-of-band verification + dual authorization — a deepfake can't answer a callback to a number it doesn't control; the channel is the control, not the content |
| AI-assisted malware/recon | Defense in depth; behavioral detection downstream (Ch.35 deepens) |
LLMs in the SOC — useful (summarize, draft rules, explain commands) but introduce prompt injection (esp. indirect: malicious instructions hidden in data the model ingests). Reference: OWASP Top 10 for LLM Applications.
- Least privilege / constrained agency (Ch.3, 17): give the model minimal tools/data — a summarizer needs no send-mail.
- Untrusted input, unverified output (Ch.12, 13): never let raw LLM output trigger a privileged action.
- Human-in-the-loop for consequential actions; the LLM drafts, a human approves.
- No secrets in prompts — anything in context can be exfiltrated by a successful injection.
Certification crosswalk
| Concept | CompTIA Security+ | (ISC)² CISSP domain |
|---|---|---|
| Anomaly-based / behavioral detection, UEBA | 2.0/4.0 (monitoring, analytics) | Security Operations |
| AI/ML-enabled attacks, deepfakes | 2.0 Threats, Vulnerabilities & Mitigations | Security Operations; Security & Risk Mgmt |
| Adversarial ML (poisoning, evasion) | 2.0 (emerging threats) | Security Architecture & Engineering |
| LLM/prompt-injection, secure AI use | 2.0/4.0 (secure operations) | Software Development Security |
| Base rate / precision-recall reasoning | 4.0 (alerting, tuning) | Security Operations |
Project additions this chapter
- Meridian program: Analytics and Behavioral Detection section — narrow anomaly-detection pilot (service + privileged accounts), human-reviewed baseline (anti-poisoning), enrichment-before-alert, lead-generator-not-oracle guardrail, success measured as analyst-experienced precision; AI-risk note (deepfake out-of-band verification; LLM governance vs. OWASP LLM Top 10 + NIST AI RMF).
bluekittoolkit:mlsec.py—zscore_anomaly(series, threshold=3.0)→(z, is_anomaly); raises on too-short or zero-variance input.
Common pitfalls
- Comparing detectors by accuracy on a rare-event problem (use precision + recall + volume).
- Running a detector on everything instead of narrowing scope first (base-rate suicide).
- Letting an anomaly baseline self-tune continuously from attacker-reachable data (invites poisoning).
- Treating an anomaly as a verdict ("anomalous ≠ malicious").
- Wiring a corruptible model to an automatic action (self-inflicted DoS + adversarial attack surface).
- Buying "AI" to replace a missing control rather than scale a working one.
- Fighting deepfakes with content detection instead of channel-based verification.
- Giving an LLM consequential tools it doesn't need, then feeding it untrusted text.