Chapter 10: Network Monitoring and Traffic Analysis: Seeing the Invisible with Wireshark, Zeek, and Flow Data

"You can't defend what you can't see." — A maxim repeated in every SOC

Overview

For three weeks, a single workstation inside Meridian Regional Bank had been talking to a server it had no business talking to. Every 3,600 seconds, give or take a few, it sent a small, encrypted HTTPS request to an address in a cloud-hosting range, received a small reply, and went quiet again. No alert fired. The endpoint agent saw an ordinary browser process making an ordinary TLS connection. The firewall saw outbound port 443 — the same port that carries every legitimate web request in the building — and allowed it, as it allows millions of others a day. The proxy logged the destination as an unremarkable domain registered six weeks earlier. Nothing in any single event was wrong. The machine was, by every device's local judgment, behaving normally.

It was not behaving normally. It was beaconing — checking in with an attacker's command-and-control server on a fixed schedule, waiting for instructions. The thing that gave it away was not any one connection but the pattern of all of them: the same internal host, the same external destination, the same tiny payload size, at almost exactly the same interval, hour after hour, day after day, including at 3 a.m. on a Sunday when no human was at the keyboard. No endpoint and no firewall is built to notice a rhythm that plays out over weeks. But a defender watching the network — watching the connections themselves as data, summarized into flow records and connection logs — can see that rhythm as clearly as a heartbeat on a monitor. Theo Brandt found it in an afternoon, not because he was looking at any one packet, but because he asked the network a question no single device could answer: which internal host is talking to the same outside address on a metronome?

This chapter is about acquiring that vision. The network is where attacks travel — every phishing payload, every stolen file, every command from an attacker to the malware they planted has to cross a wire or a radio link to get where it is going. The endpoint can be blinded by a rootkit; the application log can be turned off by an attacker with enough access; but the network carries the traffic regardless, and if you are watching the network, the attacker has to move through a medium you control and instrument. That is the defender's home-field advantage made concrete. In Chapters 6 and 7 you learned how networks are built and where controls like firewalls and intrusion detection sit. Now you learn to see what crosses them — to turn raw packets into evidence, at three different altitudes: the surgical detail of a single packet (Wireshark), the structured summary of every connection (Zeek), and the bird's-eye census of all traffic (flow data). By the end, you will have designed the network-visibility layer for Meridian that feeds everything in Part V.

In this chapter, you will learn to:

• Explain why network telemetry is some of the most trustworthy evidence a defender has, and where its blind spots are.
• Capture packets and read a protocol exchange in Wireshark, and decide when full packet capture is worth its cost.
• Use Zeek to turn a stream of packets into structured, searchable connection logs — the workhorse of network security monitoring.
• Use NetFlow/IPFIX flow records to watch enormous volumes of traffic cheaply, and find the conversations that matter.
• Recognize the network signatures of beaconing, data exfiltration, and lateral movement, and write the logic that detects them.
• Design where to place taps and sensors so that the traffic you need actually reaches your tools — and hand that telemetry to the SIEM.

Learning Paths

Network monitoring is the bridge between the network you built in Part II and the detection you will run in Part V. It serves two paths most directly.

🛡️ SOC Analyst: This is core curriculum. Read every section. §10.2 (Wireshark) is your microscope for deep investigation; §10.3 (Zeek) and §10.5 (detecting beaconing/exfil/lateral movement) are what you will use daily; §10.6 shows how this telemetry becomes the alerts in your queue. The Meridian beaconing hunt in Case Study 1 is the skill in miniature. 🏗️ Security Engineer: Focus on §10.4 (flow data at scale — the architecture decision of full-packet vs flow), §10.6 (sensor placement, taps vs SPAN, feeding the SIEM), and the Project Checkpoint (designing the visibility layer). You decide where the sensors go and what the SOC can see; get that wrong and the analysts are blind. 📋 GRC / 📜 Cert: Network monitoring underpins continuous-monitoring requirements in nearly every framework (NIST CSF Detect, PCI-DSS network monitoring, CIS Control 13). Security+ tests packet capture, flow analysis, and NDR concepts; the key-takeaways.md maps them.

10.1 You can't defend what you can't see

Begin with a question that sounds rhetorical but is not: how would you know if an attacker were inside your network right now? Not "could you prevent it" — assume prevention has already failed, because that is the defender's standing assumption (Theme 4, and the reason Chapters 6 and 7 are about layered network controls rather than a single wall). The attacker is in. They have a foothold on one machine. What would betray them?

The honest answer, for most organizations, is almost nothing — unless someone is watching the network. The attacker's foothold looks, from the endpoint's perspective, like an ordinary running program. Their lateral movement to a second machine looks like an ordinary login. Their theft of a database looks like an ordinary file read. Every individual action can be made to resemble legitimate activity, and a sufficiently careful attacker will make sure it does. But all of these actions share one inescapable property: they generate network traffic. To control the foothold, the attacker must communicate with it from outside. To move laterally, they must connect from one machine to another. To steal data, they must move it across the network to somewhere they can retrieve it. The network is the one place where the attacker cannot be a ghost, because the network is the medium they are using to act.

🚪 Threshold Concept: The network is ground truth. An attacker who has compromised a host can lie to you through that host — disable its logging, hide their processes, tamper with its records — because they control it. But they generally do not control the switches, routers, and sensors that carry their traffic, and traffic on the wire is what it is. This is why mature defenders treat network telemetry as a primary, relatively trustworthy source of evidence, and why "visibility" is not a buzzword but the precondition for every detection in this book. You cannot detect, hunt, respond to, or investigate what you never recorded.

This is the principle that gives the chapter its title and the SOC its first commandment: you can't defend what you can't see. Detection (Chapter 22), incident response (Chapter 24), and the SIEM that ties them together (Chapter 21) all depend on telemetry that someone, in advance, decided to collect. The most common and most painful failure in a real breach investigation is not that the analysts lacked skill; it is that the data they needed was never recorded, so the question "how did the attacker get in, and what did they touch?" simply has no answer. Visibility is a decision you make before the incident, not during it.

But "watch the network" is not one activity — it is a spectrum of three, distinguished by how much detail they keep and how much they cost. Picture them as three altitudes over the same terrain:

   ALTITUDE          TOOL              KEEPS                          COST / RETENTION
  ───────────────────────────────────────────────────────────────────────────────────
   GROUND LEVEL      Wireshark /       every byte of every packet     HUGE storage;
   (microscope)      full PCAP         (payloads, headers, timing)    hours-to-days only
                          │
                          │  "what exactly was in this conversation?"
                          ▼
   ROOFTOP           Zeek (NSM)        one structured record per       MODERATE; weeks-
   (street map)                        connection: who, when, how      to-months; the
                                       long, how much, which protocol  SOC workhorse
                          │
                          │  "what connections happened, and were any of them odd?"
                          ▼
   SATELLITE         NetFlow /         a tiny summary per flow:         TINY per record;
   (census)          IPFIX             src, dst, ports, bytes, packets  months-to-years
                                       — NO payload at all              across the WHOLE net
                          │
                          │  "who is talking to whom, how much, everywhere, over time?"

Figure 10.1 — The three altitudes of network visibility. Full packet capture (Wireshark) keeps everything but is too expensive to retain for long or deploy everywhere. Flow data (NetFlow/IPFIX) keeps almost nothing per record but can cover the entire network for years. Zeek logs sit in the middle and are, for most detection work, the sweet spot.

You will use all three, and a recurring skill of this chapter is choosing the right altitude for the question you are asking. To find the content of a suspicious exchange, you drop to the packet level. To detect a behavior across thousands of connections, you rise to flow or Zeek. The beaconing at the start of this chapter was invisible at ground level — each packet looked fine — and obvious from the satellite, where the rhythm of the whole pattern came into focus.

The remaining sections build this capability from the ground up: the microscope first (§10.2), then the street map (§10.3), then the census (§10.4), then the detections that combine them (§10.5), and finally how all of it flows into the SIEM (§10.6).

🔄 Check Your Understanding: 1. An attacker has full control of a compromised laptop, including its local logs and endpoint agent. Why is network telemetry still likely to reveal their activity even though the laptop's own records cannot be trusted? 2. You need to answer two questions: (a) "exactly what credentials were sent in this one suspicious login?" and (b) "has any internal host contacted this malicious IP in the last six months?" Which altitude — packet, Zeek, or flow — fits each, and why?

Answers

1. The attacker controls the host but not the network infrastructure carrying their traffic; to act (command the foothold, move laterally, exfiltrate data) they must generate traffic across switches, routers, and sensors they do not control, and that traffic on the wire is trustworthy evidence regardless of what the host claims. 2. (a) Packet capture (Wireshark/full PCAP) — only full payload shows the actual content of a single exchange. (b) Flow data — it is the only altitude cheap enough to retain for six months across the whole network, and "did host X ever contact IP Y" needs only the summary, not payloads.

10.2 Packet capture with Wireshark

Start at ground level, because everything above it is a summary of what lives here. A packet (defined in Chapter 6 as the unit of data the network moves) is, on the wire, just a sequence of bytes: some header fields that say where it is going and how to handle it, and then a payload — the actual content. Packet capture (PCAP) is the act of recording those bytes exactly as they cross a network interface, byte for byte, so that you can examine them later. The recorded file is also called a PCAP (after the libpcap library and .pcap file format that nearly every tool uses). When you capture a packet, you are taking a perfect photograph of a moment of network traffic: nothing summarized, nothing lost.

Wireshark is the standard tool for capturing and, above all, reading packets. It is a graphical protocol analyzer that takes a stream of raw bytes and decodes it into human-readable structure — it knows the format of thousands of protocols, so it can take an opaque blob and tell you "this is a DNS query for meridianbank.example," "this is a TLS Client Hello offering these cipher suites," "this is an HTTP GET for /login with this User-Agent." (Its command-line sibling, tshark, does the same without the GUI, which matters for automation.) Wireshark does not change traffic and does not attack anything; it is a read-only microscope. Its companion, tcpdump, is the lightweight capture tool you will run on a server to grab packets that you then open in Wireshark to read.

⚖️ Authorization & Ethics: Capturing packets means recording other people's traffic, which can include credentials, personal data, and confidential content. Capture only on networks you own or are explicitly authorized to monitor, capture only what you need, and protect and dispose of the captures like the sensitive data they are. On a regulated network like Meridian's, packet captures that touch the cardholder data environment are themselves in PCI-DSS scope. The rule from Chapter 1 holds: your own lab, or written authorization — nothing else.

What a capture actually shows you

A single TCP connection, captured, lays its whole life bare. Recall the three-way handshake from Chapter 6 — SYN, SYN-ACK, ACK. In a capture it looks like this (an illustrative, annotated view of what Wireshark's packet list shows, using documentation addresses):

No.  Time      Source           Destination      Proto  Length  Info
  1  0.000000  10.20.4.55       192.0.2.80       TCP    74      49832 → 443 [SYN] Seq=0 Win=64240
  2  0.000412  192.0.2.80       10.20.4.55       TCP    74      443 → 49832 [SYN, ACK] Seq=0 Ack=1
  3  0.000455  10.20.4.55       192.0.2.80       TCP    66      49832 → 443 [ACK] Seq=1 Ack=1
  4  0.000900  10.20.4.55       192.0.2.80       TLS    517     Client Hello (SNI=updates.example)
  5  0.041203  192.0.2.80       10.20.4.55       TLS    1414    Server Hello, Certificate, ...
  6  0.043110  10.20.4.55       192.0.2.80       TLS    93      Client Key Exchange, ...
  7  0.044001  10.20.4.55       192.0.2.80       TLSv1.3 ...    Application Data (encrypted)

Figure 10.2 — A captured TLS connection. Packets 1–3 are the TCP handshake; packets 4–6 are the TLS handshake; packet 7 onward is encrypted application data. Even though the content is encrypted, the capture still reveals the source, destination, ports, timing, sizes, and — crucially — the Server Name Indication (SNI), the hostname the client is connecting to, which is sent in the clear in the Client Hello.

Read that figure carefully, because it teaches the single most important lesson about packet analysis in the age of encryption. Encryption hides the payload, not the metadata. The bytes of packet 7 are gibberish to you and to any eavesdropper — that is what TLS is for (Chapter 5). But everything around the payload is still visible: who is talking to whom (the IP addresses), over what service (the ports), how much (the lengths), when and in what rhythm (the timestamps), and often what hostname the client asked for (the SNI, and the certificate the server presents). A defender who internalizes this stops mourning the fact that traffic is encrypted and starts mining the metadata that encryption leaves untouched. The beaconing at the start of this chapter was entirely encrypted; we caught it on metadata — destination, size, and timing — alone.

What an attack looks like in a capture, and how you read it

Suppose Theo is handed a capture from a host suspected of being compromised and asked, "is this thing exfiltrating data?" At the packet level, he is not going to read terabytes by hand. He uses Wireshark's filters — its query language for slicing a capture down to what matters. A few he reaches for constantly:

• ip.addr == 10.20.4.55 — show only packets to or from the suspect host.
• dns — show only DNS traffic (a goldmine; see §10.5 and Chapter 9 on DNS abuse).
• http.request — show only HTTP requests, with their methods and URIs.
• tcp.flags.syn == 1 && tcp.flags.ack == 0 — show only connection attempts (bare SYNs), which reveals scanning and connection sprays.
• frame.len > 1400 — show only large frames, a crude first pass for bulk transfer.

Wireshark's Statistics → Conversations view then summarizes the capture into a table of who-talked-to-whom and how many bytes each way — which is, in effect, Wireshark computing flow records on the fly (foreshadowing §10.4). If Theo sees that the suspect host sent 4 GB to a single external address over a few hours while receiving almost nothing, that asymmetry — lots of bytes out, few in, to one destination — is the classic shape of exfiltration, and he has his answer without reading a single payload byte.

🛡️ Defender's Lens: Attackers know Wireshark exists and that defenders use it, so they encrypt their command-and-control and exfiltration to defeat payload inspection. Do not let that discourage you — turn it around. When payloads are opaque, behavioral and metadata analysis become your primary tools, and those are exactly what scale to Zeek and flow data. An encrypted channel that beacons on a fixed interval or that moves gigabytes outbound is more conspicuous in metadata, not less, because the attacker has removed your ability to read the content but not your ability to see the shape. The shape is often enough.

⚠️ Common Pitfall: Treating full packet capture as a network-wide, always-on solution. A busy network can push tens of gigabits per second; capturing every byte of that produces petabytes and overwhelms any storage budget within days. Teams that try to "capture everything everywhere forever" either go broke or quietly drop packets and get incomplete captures that betray them during an investigation. Full PCAP is a targeted, short-retention tool — turned up on a specific segment when you have a reason, or kept in a rolling buffer of only the last few hours. For continuous, network-wide visibility, you climb to the next altitude: Zeek.

🧩 Try It in the Lab: Install Wireshark on your own machine, start a capture on your active interface, and visit a single HTTPS website. Stop the capture and find your connection. Confirm three things: (1) you can see the TCP handshake; (2) you can read the SNI hostname in the Client Hello even though the page is encrypted; (3) the application data itself is unreadable. Then open Statistics → Conversations and see your traffic summarized into flow-like rows. You have just operated all three altitudes on your own traffic. (Your own machine and network only.)

10.3 Zeek: turning packets into logs

Full packet capture answers "what exactly was in this conversation?" but it is too heavy to run everywhere all the time, and reading raw packets does not scale to detection across thousands of hosts. What a SOC actually lives on is a tool that watches all the traffic in real time and writes down, for every connection, a compact structured summary of what happened — without keeping every byte. That tool is Zeek.

Zeek (formerly called Bro) is an open-source network security monitoring (NSM) platform. Point it at a copy of your network traffic and, instead of storing packets, it produces a set of richly structured logs: one entry per network connection, plus protocol-specific logs for DNS queries, HTTP requests, TLS handshakes, SSH sessions, files transferred, and much more. Zeek is, in essence, an interpreter that turns the raw, low-level event of "these packets crossed the wire" into the high-level, analyst-friendly fact of "host A made an HTTPS connection to host B at this time, it lasted 12 seconds, A sent 2 KB and received 40 KB, and the server's certificate was for updates.example." That transformation — from packets to logs — is what makes network data usable at scale, and it is why Zeek (or a commercial equivalent) sits at the heart of nearly every serious network-monitoring deployment.

Reading a Zeek log line

The workhorse is Zeek's connection log, conn.log. Each line is one connection. Here is an illustrative entry (Zeek logs are tab-separated; shown here field-by-field for readability, with documentation addresses):

# conn.log  (one connection, fields labeled)
ts        = 1718374512.512    # timestamp (epoch seconds)
uid       = CwjjJp2NrFvVQ3     # unique connection ID (joins to other Zeek logs)
id.orig_h = 10.20.4.55         # originator (who initiated) — the internal host
id.orig_p = 49832              # originator port (ephemeral)
id.resp_h = 192.0.2.80         # responder (who was contacted) — the external server
id.resp_p = 443                # responder port → HTTPS
proto     = tcp
service   = ssl                # Zeek identified the app-layer protocol as TLS/SSL
duration  = 0.273              # connection lasted 0.273 seconds
orig_bytes= 517                # bytes the internal host SENT
resp_bytes= 1664               # bytes it RECEIVED
conn_state= SF                 # normal establishment and teardown (SYN/FIN seen)

Figure 10.3 — A single Zeek conn.log record. This one connection, summarized in a few dozen bytes, captures who talked to whom, when, for how long, in which direction the data flowed, how much, and what protocol — everything you need for most detection, without storing a single payload byte. The uid lets you pivot to the matching ssl.log entry to see the SNI and certificate, or to dns.log to see what name resolved to that IP.

Notice how much detection power lives in those fields. id.resp_p = 443 with service = ssl is a normal web request. But the same structure makes anomalies pop: a connection where id.resp_p = 4444 and service is empty (an unrecognized service on a suspicious port) is worth a look; a connection with a huge orig_bytes and tiny resp_bytes is outbound bulk transfer (exfiltration's shape); thousands of connections from one id.orig_h to many distinct id.resp_h on the same port in seconds is scanning or worm-like spread. And because every Zeek log shares the uid, you can follow a single connection across logs — from conn.log (it happened) to dns.log (the name it resolved) to ssl.log (the certificate it presented) to files.log (the file it carried). This pivoting is the daily craft of network monitoring.

Zeek also ships a notice.log for things it is configured to consider noteworthy, and — critically — it is programmable: you can write Zeek scripts that watch for specific conditions and raise a notice (for example, "a host that has never used SMB suddenly initiated SMB to twenty internal hosts"). This is where Zeek crosses from passive logging into active detection, and it foreshadows the detection-engineering work of Chapter 22.

🔗 Connection: Zeek's protocol logs are the natural place to operationalize the DNS and email defenses of Chapter 9. The dns.log records every query and answer, which is exactly where you spot the DNS tunneling and DGA (domain-generation algorithm) patterns Chapter 9 described — a host making hundreds of queries for long, random-looking subdomains of one domain is using DNS as a covert channel. Zeek gives you the data; the dga_score logic previewed in this chapter's toolkit gives you a way to flag it.

What an attack looks like in Zeek, and how you detect it

Return to the chapter's opening beacon. In conn.log, it does not appear as one alarming line — it appears as many ordinary lines that are too similar to each other. The internal host 10.20.4.55 shows up over and over, always to 192.0.2.80:443, always with orig_bytes near 500 and resp_bytes near 1,200, with duration under a second, spaced almost exactly an hour apart, around the clock. No single line is suspicious. The aggregate is screaming. Detection at the Zeek level is therefore usually a query over many records — "group connections by (source, destination, port); for each group, look at the timing of the connections" — rather than a rule that fires on one event. That grouping-and-timing analysis is precisely what the beacon_score function in this chapter's pktflow.py performs, and what Case Study 1 walks through end to end.

📟 War Story (constructed, Tier 3): A mid-size insurer ran excellent endpoint detection and a well-tuned firewall and considered itself well-instrumented. It had no network security monitoring. An attacker who got in through a contractor's laptop ran their command-and-control over HTTPS to a cloud-hosted server, and lived in the network for four months. Every endpoint check passed; every firewall rule was satisfied; the traffic was encrypted and the destination had a clean reputation. When the intrusion was finally discovered (the attacker got greedy and triggered a data-loss alert on the way out), the responders asked the obvious question — how long, and what did they touch? — and found they could not answer it, because no one had been recording the connections. The post-incident review's first recommendation was a single line: deploy Zeek and retain flow data. The lesson is that prevention and endpoint visibility are necessary and not sufficient; without network monitoring, you can be thoroughly breached and never know, and you certainly cannot scope it afterward.

10.4 Flow data at scale (NetFlow/IPFIX)

Zeek is the sweet spot for detection, but even Zeek logs grow large on a big network, and they require deploying sensors that see full traffic. There is one more altitude, the highest and cheapest, and it has a magical property: you may already have it, generated for free by hardware you already own. That altitude is flow data.

A flow is one conversation between two endpoints, summarized: it records the source and destination IP addresses, the source and destination ports, the protocol, the start and end times, and counts of bytes and packets — and nothing else. No payload, no headers beyond the five-tuple, just the metadata of "these two endpoints exchanged this much traffic, this way, during this window." NetFlow is Cisco's protocol for emitting these records (now standardized, with vendor-neutral successors, as IPFIX — IP Flow Information Export — and seen as sFlow and J-Flow on other gear). The crucial operational fact is that routers and switches can generate flow records as a built-in feature: the network hardware that is already forwarding your traffic can, with a few lines of configuration, also report a summary of every conversation to a flow collector. You get a census of all traffic across your entire network without deploying a single new sensor in many cases.

🚪 Threshold Concept: Full packet capture vs flow is the central architectural tradeoff of network visibility, and understanding it changes how you design monitoring. Full packet capture keeps everything about some traffic for a short time; flow keeps almost nothing about all traffic for a long time. You cannot afford full capture everywhere, and flow alone cannot show you content — so a mature program uses both, deliberately: flow (and Zeek) for broad, long-retention, behavior-level visibility across the whole network, and targeted full capture for deep, short-retention inspection where a specific investigation needs the bytes. The question is never "which one?" It is "what does each cover, and where are my gaps?"

Reading a flow record

A flow record is austere — and that austerity is its power, because a record this small can be kept for years across millions of conversations. Here is an illustrative flow export, shown as a table a collector would present (documentation addresses; bytes/packets are per-flow totals):

start        duration  proto  src_ip:port        dst_ip:port        packets   bytes   dir
─────────────────────────────────────────────────────────────────────────────────────────
13:00:12.4   0.27 s    TCP    10.20.4.55:49832   192.0.2.80:443     14        2181    →out
14:00:14.1   0.26 s    TCP    10.20.4.55:50341   192.0.2.80:443     14        2180    →out
15:00:11.9   0.28 s    TCP    10.20.4.55:50455   192.0.2.80:443     15        2195    →out
16:00:13.0   0.27 s    TCP    10.20.4.55:50611   192.0.2.80:443     14        2179    →out
   ...        (one nearly identical flow every hour, around the clock, for weeks)   ...

Figure 10.4 — Four flow records from the chapter's beacon. Individually, each is an unremarkable short HTTPS conversation. Stacked, the pattern is unmistakable: same internal host, same external destination and port, near-identical tiny byte counts, at a fixed one-hour cadence, including overnight. Flow data has discarded every payload byte and kept exactly the four facts — who, whom, how much, when — that expose the beacon.

This is the altitude at which the chapter's opening beacon was finally caught, and it shows why. The thing that betrays a beacon is the regularity of its timing and the constancy of its size, and those two properties survive perfectly into flow records even though everything else has been thrown away. You do not need payloads to see a heartbeat. You need only the start times and the byte counts — which is precisely, and almost only, what flow keeps.

What attacks look like in flow, and how you detect them

Three of the most important attacker behaviors leave clear flow signatures, and §10.5 turns each into a detection:

• Beaconing — many small flows from one internal host to one external destination, at a regular interval. Detect by grouping flows by (source, destination) and measuring the regularity of the inter-arrival times.
• Exfiltration — a large volume of orig_bytes (outbound) to one destination, often far exceeding that host's normal outbound baseline. Detect by summing outbound bytes per (source, destination) and comparing to a baseline.
• Lateral movement / scanning — one internal host suddenly making connections to many internal hosts, especially on administrative ports (SMB 445, RDP 3389, WinRM 5985). Detect by counting distinct internal destinations per source over a short window.

Notice that all three detections are aggregations over many flows, not rules on a single packet — which is exactly why flow data, despite keeping so little per record, is so powerful for detection. And all three depend on knowing what normal looks like, which is the next idea.

The network baseline

A network baseline is a model of what normal traffic looks like for your environment — which hosts talk to which, on what ports, in what volumes, at what times. It is the reference against which anomalies are judged, because almost every network detection is some form of "this is different from normal." A baseline answers questions like: does this server normally talk to the internet at all? (a domain controller that suddenly makes outbound HTTPS is alarming; a web server doing so may be routine); how much does this workstation normally send outbound in a day? (so that a 50× spike stands out); what is the normal set of internal destinations for this host? (so that contacting twenty new ones is visible as lateral movement). Building a baseline is mostly a matter of collecting flow data over a representative period and summarizing it — and it is the unglamorous prerequisite that makes the glamorous detections possible. Without a baseline, every alert is a guess about whether "unusual" is actually unusual.

⚠️ Common Pitfall: Baselining during an already-compromised period and enshrining the attacker's traffic as "normal." If you build your baseline from a window in which the beacon was already running, the beacon becomes part of "normal" and your anomaly detection will never flag it. Baselines must be built thoughtfully, refreshed, and sanity-checked against known-good expectations (e.g., "should this asset ever talk to the internet?"), not just learned blindly from whatever traffic happened to be present. Automated baselining is helpful but never a substitute for a defender who knows what the network is supposed to do.

🔄 Check Your Understanding: 1. Flow records contain no payload. Explain how it is still possible to detect command-and-control beaconing from flow data alone. Which two flow fields carry the signal? 2. Your CFO's laptop normally sends about 200 MB outbound per day. Today it sent 80 GB to a single cloud-storage IP overnight. Which attacker behavior does this match, and which flow-based detection (and which part of the baseline) caught it?

Answers

1. Beaconing is defined by regular timing and constant size, both of which survive into flow records; the signal is carried by the flow start times (regular inter-arrival intervals) and the byte counts (near-constant per flow). 2. Data exfiltration; the detection is "sum outbound bytes per (source, destination) and compare to baseline," and the part of the baseline it relied on was the laptop's normal daily outbound volume (~200 MB), against which 80 GB is an obvious anomaly.

10.5 Detecting beaconing, exfiltration, and lateral movement

You now have the three altitudes and the idea of a baseline. This section is the payoff: turning telemetry into detections for the three attacker behaviors that network monitoring catches best. These three correspond, not coincidentally, to three of the most consequential stages of an intrusion — establishing command-and-control, achieving the objective (data theft), and spreading through the environment. Catching any one of them can stop a breach; the network is often the only place you can catch them at all.

Detecting beaconing

Beaconing is the regular, automated check-in of compromised malware to its command-and-control (C2) server. After an attacker establishes a foothold, the implant must periodically ask the C2 "any instructions?" — because the attacker is not at the keyboard every second, and because outbound connections from inside a network are far more likely to be allowed than inbound ones. That periodicity is the implant's tell. A human browsing the web produces irregular, bursty, varied traffic; an implant beaconing produces metronomic traffic — the same destination at the same interval, regardless of whether anyone is using the machine.

The detection follows directly. Group network records (flows or Zeek connections) by the pair (internal source, external destination). For each group with enough connections, look at the inter-arrival times — the gaps between consecutive check-ins. If those gaps are tightly clustered around a single value (say, all within a few seconds of 3,600), the traffic is suspiciously regular: a low variance in inter-arrival time is the signature of automation. Real implants often add jitter — a small random wobble to the interval — to evade exactly this detection, so good beacon scoring looks for low relative variance rather than perfect equality, and weighs the number of check-ins (a regular pattern that repeats fifty times is far more convincing than one that repeats three times). This is exactly what beacon_score computes in the Project Checkpoint: a number that is high when the gaps between connections are consistent and the pattern is long-lived.

   HUMAN WEB BROWSING (irregular):
   |--gap--|---gap---|-gap-|------gap------|--gap--|---gap---|  intervals vary wildly
   09:01   09:14     09:33  09:38           10:20   10:31     ...   → LOW beacon score

   C2 BEACON, 1h interval with small jitter (regular):
   |----gap----|----gap----|----gap----|----gap----|----gap----|  intervals ~constant
   09:00:12    10:00:31    11:00:05    12:00:48    13:00:19      ...   → HIGH beacon score

Figure 10.5 — A beaconing timeline. Human-generated traffic to a destination has wildly varying gaps between connections; a beacon has near-constant gaps even when jittered. The detection is not "what was sent" — it is "how regular is the rhythm," measured as the variance of the inter-arrival times.

🛡️ Defender's Lens: Attackers fight beacon detection with longer intervals (a beacon that checks in once a day is far harder to spot than one every minute) and heavier jitter (randomizing the interval to muddy the regularity). This is why long-retention flow data matters so much: a once-a-day beacon is invisible in a capture that only goes back a few hours, but stands out plainly in flow data retained for months, where you can see the same daily check-in repeat thirty times. Your retention window is a weapon — the longer you keep flow data, the slower and stealthier a beacon has to be to hide from you, and slow beacons constrain the attacker's tempo. This is also why §10.6's bridge to the SIEM matters: correlating a weak beacon signal with other telemetry turns a maybe into a finding.

Detecting data exfiltration

Data exfiltration (network) is the unauthorized transfer of data out of an organization across the network — the moment a breach turns into a loss. Its network signature is usually some form of outbound volume anomaly: a host sending far more data out than it normally does, to a destination that is not part of its normal pattern. The detection is the one named in §10.4: sum outbound bytes per (source, destination) over a window, and flag totals that exceed the source's baseline by a large factor, especially to new or external destinations.

But attackers complicate this in instructive ways, and a good defender anticipates each:

• Slow exfiltration ("low and slow"). Rather than one giant transfer, the attacker trickles data out a little at a time to stay under volume thresholds. Counter it with cumulative baselines (total outbound to a destination over days, not just per hour) and by watching for new long-lived low-volume channels.
• DNS and other covert channels. Data can be smuggled inside DNS queries (encoding stolen bytes into the subdomains of attacker-controlled domains), inside ICMP, or inside seemingly normal HTTPS. This is where Zeek's dns.log earns its keep: a host making an abnormal volume of DNS queries, or queries for unusually long or high-entropy names, is a tunneling suspect (the Chapter 9 connection, and the dga_score-style logic in this chapter's toolkit).
• Blending with normal services. Exfiltration to a legitimate cloud-storage or file-sharing provider can hide in traffic that looks routine. Counter it with baselines that are per-host and per-destination, not just global — the CFO's laptop sending 80 GB to cloud storage overnight is anomalous for that host even though cloud storage is a normal destination for the organization as a whole.

The unifying principle: exfiltration is detected by comparison to a baseline of normal outbound behavior, sliced finely enough (per host, per destination, cumulative over time) that the attacker cannot easily hide in an aggregate.

Detecting lateral movement

Lateral movement is an attacker's spread from their initial foothold to other systems inside the network — the step between getting in and reaching the actual target. Because most attackers land on a low-value machine (a clicked-on workstation) but want a high-value one (a database, a domain controller), lateral movement is nearly universal in serious intrusions, and the network is the ideal place to see it, because moving from machine A to machine B is a network connection by definition.

Its signature is a host reaching out to many internal destinations it does not normally contact, frequently on administrative protocols: SMB (port 445, for file shares and remote execution), RDP (3389, remote desktop), WinRM (5985/5986), and SSH (22). A workstation that normally talks to a handful of internal servers but suddenly connects to forty internal hosts on port 445 in five minutes is almost certainly scanning for shares or spreading — and that is an east-west traffic pattern (internal-to-internal, the term from Chapter 6), which is exactly the traffic a perimeter firewall never sees and which therefore makes internal network monitoring indispensable. The detection: count distinct internal destinations per internal source over a short window, on sensitive ports, and flag sources whose count jumps far above their baseline.

🔗 Connection: Lateral movement is where network monitoring and identity monitoring reinforce each other. The network sees the connections (host A reached host B on 445); identity and endpoint telemetry see the authentications (an account logged into host B from host A, perhaps using a stolen credential or a technique like pass-the-hash). Neither alone tells the whole story; correlated in the SIEM (Chapter 21), they turn "an unusual connection" plus "an unusual login" into a high-confidence lateral-movement detection. This correlation across data sources is the entire reason the network telemetry you build here flows into the SIEM, and it previews the threat-hunting tradecraft of Chapter 22.

🔄 Check Your Understanding: 1. Why is east-west (internal-to-internal) traffic monitoring essential for catching lateral movement, and why does a traditional perimeter firewall typically miss it? 2. An attacker exfiltrates 60 GB but spreads it as 1 GB per hour over 60 hours to avoid a "more than 5 GB/hour" alert. Name the baseline strategy that would still catch this, and the Zeek log you would also check in case the data left via DNS instead.

Answers

1. Lateral movement is internal-to-internal connections, which by definition never cross the perimeter, so a firewall placed at the network edge never sees them; catching them requires sensors (Zeek, flow from internal switches) positioned to observe east-west traffic inside the network. 2. A cumulative baseline — total outbound bytes to a destination over a multi-day window, not just per hour — would flag 60 GB to one destination even though no single hour exceeded the threshold; and dns.log would reveal exfiltration via DNS tunneling (abnormal query volume or long, high-entropy names).

10.6 Feeding the SIEM: from sensors to detections

You have learned to capture packets, generate Zeek logs, and collect flow data, and to read each for attacker behavior. The final question is architectural: how does this telemetry get to a place where it can be correlated, alerted on, and retained? The answer is that network monitoring is not an island — it is a feeder. Its job is to produce high-quality network telemetry and deliver it into the SIEM (Security Information and Event Management), the central system that ingests logs from every source, correlates them, and raises the alerts a SOC analyst sees. Chapter 21 is devoted to the SIEM; this section builds the network-visibility layer that supplies it, and that requires two design decisions: where the sensors get their traffic and what flows downstream.

Getting traffic to your sensors: SPAN vs tap

A sensor — a server running Zeek, or a full-capture appliance — can only analyze traffic it actually receives. But switches, by design, send traffic only to its intended destination, not to a monitoring box. So you must deliberately deliver a copy of the traffic to the sensor, and there are two ways to do it, known together as span/tap:

• A SPAN port (Switched Port Analyzer; also called port mirroring) is a switch feature that copies traffic from one or more ports (or a whole VLAN) to a designated monitor port, where your sensor listens. It is cheap and needs no new hardware — just configuration. Its weaknesses: a busy switch under load may drop mirrored packets before your monitor (the switch prioritizes forwarding real traffic over copying it), it can miss certain errors and physical-layer details, and it competes for the switch's resources.
• A network tap (Test Access Point) is a dedicated hardware device inserted inline on a link that passively copies every bit passing through it to a monitor port, with no impact on the live traffic and no packet loss. Taps are the gold standard for fidelity — they see everything, even at full line rate — at the cost of being physical devices you must buy and install at each monitoring point.

        SPAN / PORT MIRROR (cheap, lossy under load):
                 ┌──────────────┐
   real traffic  │              │  copy ──► [ Zeek / capture sensor ]
   ════════════► │   SWITCH     │ (mirror)        (may miss packets
                 │              │                  when switch is busy)
                 └──────────────┘

        NETWORK TAP (full fidelity, dedicated hardware):
                        ┌─────┐
   ══════════════════►  │ TAP │  ════════════════►   (live traffic, untouched)
        link A───B      │     │── copy ──► [ Zeek / capture sensor ]
                        └─────┘            (sees 100%, even at line rate)

Figure 10.6 — SPAN versus tap. A SPAN port mirrors traffic via switch configuration — free but lossy when the switch is loaded. A network tap is dedicated inline hardware that copies every bit with no loss — higher fidelity, at the cost of physical devices. Use taps where completeness is critical (e.g., monitoring the link to the cardholder data environment); SPAN where convenience and cost dominate.

🚪 Threshold Concept: Sensor placement determines what you can detect — full stop. A detection you cannot feed with data does not exist. If your only sensor is at the internet perimeter, you can see traffic entering and leaving the building but you are blind to all east-west movement between internal hosts — meaning you cannot catch lateral movement no matter how good your tools are. Visibility is a coverage problem before it is a tooling problem: you must place sensors so that the traffic you need to analyze actually crosses one of them. Mapping "what do I need to see?" to "where must a sensor sit to see it?" is the core of network-monitoring design, and it is the work of this chapter's Project Checkpoint.

What flows downstream, and why not everything

Not all three altitudes feed the SIEM equally, and the reason is volume. Raw packets are far too voluminous to ship into a SIEM — they stay in the capture appliance, retrieved on demand during an investigation. What flows into the SIEM is the structured, summarized telemetry: Zeek logs (every connection, with rich protocol fields) and flow records (the lightweight census), plus the notices and alerts your sensors generate. This is the same volume logic as Figure 10.1, applied to the pipeline: the higher, cheaper altitudes feed the central system continuously; the heavy ground-level data is kept locally and pulled only when a specific investigation needs the bytes.

Once in the SIEM, network telemetry becomes far more powerful than it is alone, because it can be correlated with everything else. A Zeek conn.log showing host A connecting to host B on 445 is mildly interesting; correlated with an authentication log showing a privileged login to host B from A at the same instant, and an endpoint alert on A, it becomes a high-confidence lateral-movement detection that fires an alert. That correlation — turning multiple weak signals from different sources into one strong one — is the entire premise of the SIEM and the reason the visibility layer you build here is the foundation of the security operations you will build in Part V.

🔗 Connection: This section is the explicit bridge from Part II to Part V. The network telemetry designed here is consumed by the SIEM (Chapter 21), mined by detection engineering and threat hunting (Chapter 22), and relied upon during incident response (Chapter 24) to scope what an attacker touched. When a responder in Chapter 24 asks "how long was the attacker here and what did they reach?", the answer comes from exactly the Zeek and flow retention you are about to design for Meridian. Build the visibility layer well now, and the later chapters have ground truth to work from; build it poorly, and they are guessing.

🔄 Check Your Understanding: 1. Why do raw packet captures generally not get shipped into the SIEM, while Zeek logs and flow records do? What stays behind, and how is it accessed? 2. A SOC complains it cannot detect lateral movement even though it runs excellent Zeek software. The only sensor is a tap on the internet uplink. Diagnose the problem in terms of sensor placement, and state the fix.

Answers

1. Raw packets are far too voluminous to ingest and retain centrally, so they stay in the capture appliance and are pulled on demand during a specific investigation; the SIEM ingests the lightweight, structured summaries (Zeek logs, flow records) and the alerts/notices, which are small enough to retain and correlate at scale. 2. The sensor sees only north-south (internet) traffic, so east-west internal-to-internal connections — where lateral movement lives — never reach it; the tool is irrelevant if the data never arrives. The fix is to add internal visibility: enable flow export on internal switches and/or place sensors (taps/SPAN) on internal segments so east-west traffic is observed.

Project Checkpoint

This chapter contributes Meridian's network-monitoring design to the security program and adds the pktflow.py module to bluekit. Together they answer the question Dana posed after reading the post-incident notes from a peer bank's breach: "If something like that were happening inside us right now, would we even see it?"

Program increment — the network-monitoring design. Sam Whitfield drafts a one-page visibility plan that maps Meridian's needs to sensor placement, using the §10.6 principle that coverage precedes tooling. The plan specifies: (1) flow export (NetFlow/IPFIX) enabled on all core and distribution switches and the internet edge routers — cheap, network-wide, long retention, the census layer; (2) Zeek sensors fed by taps on the highest-value links: the internet uplink (north-south), the link into the cardholder data environment (full-fidelity tap, because PCI-DSS demands it and a SPAN drop there is unacceptable), and a sensor watching east-west traffic between the branch/user zones and the data-center zone so lateral movement is visible; (3) targeted full packet capture as a rolling 72-hour buffer on the CDE and internet-edge sensors only, retrieved on demand; (4) retention tiers — flow for 13 months (to satisfy a year-plus of look-back for slow beacons), Zeek logs for 90 days, full PCAP for 72 hours; (5) every sensor's logs, flows, and notices forwarded to the SIEM (Chapter 21). The plan explicitly notes the gap it does not yet close — encrypted payloads — and points to baselining and correlation as the compensating strategy.

bluekit increment — pktflow.py. We implement the three functions the chapter has been building toward: summarize_flows (turn raw flow rows into per-host/destination summaries — the baseline-building primitive), top_talkers (rank conversations by volume — the exfiltration first-look), and beacon_score (measure the regularity of a host's check-ins — the beacon detector). As always, the code is illustrative and never executed during authoring; the expected output is hand-traced in a comment.

# bluekit/pktflow.py  — Chapter 10 increment
"""Flow analysis for defenders: summarize, rank, and score for beaconing.
Input rows are simple dicts (as a flow collector or Zeek conn.log would yield).
Illustrative and hand-traced; not executed at authoring."""
from collections import defaultdict


def summarize_flows(rows):
    """Aggregate flows by (src, dst) -> total bytes, flow count. Baseline primitive."""
    agg = defaultdict(lambda: {"bytes": 0, "flows": 0})
    for r in rows:
        key = (r["src"], r["dst"])
        agg[key]["bytes"] += r["bytes"]
        agg[key]["flows"] += 1
    return dict(agg)


def top_talkers(flows, n=3):
    """Return the n (src, dst) pairs with the most bytes — the exfil first-look."""
    items = [(k, v["bytes"]) for k, v in summarize_flows(flows).items()]
    return sorted(items, key=lambda kv: kv[1], reverse=True)[:n]


def beacon_score(timestamps):
    """Regularity of check-ins in [0,1]; high = beacon-like (low relative variance)."""
    ts = sorted(timestamps)
    if len(ts) < 3:
        return 0.0                                   # too few to judge a rhythm
    gaps = [b - a for a, b in zip(ts, ts[1:])]
    mean = sum(gaps) / len(gaps)
    if mean == 0:
        return 0.0
    var = sum((g - mean) ** 2 for g in gaps) / len(gaps)
    cv = (var ** 0.5) / mean                          # coefficient of variation
    return round(max(0.0, 1.0 - cv), 3)               # low spread -> score near 1.0


if __name__ == "__main__":
    flows = [
        {"src": "10.20.4.55", "dst": "192.0.2.80",  "bytes": 2181},
        {"src": "10.20.4.55", "dst": "192.0.2.80",  "bytes": 2180},
        {"src": "10.20.4.55", "dst": "192.0.2.80",  "bytes": 2195},
        {"src": "10.20.9.10", "dst": "198.51.100.7", "bytes": 9_000_000},
    ]
    print("top talker:", top_talkers(flows, n=1)[0])
    # hourly check-ins (epoch secs) with small jitter -> regular -> high score
    beacon = [3600, 7201, 10799, 14402, 18000]
    print("beacon score:", beacon_score(beacon))
    # bursty human browsing -> irregular -> low score
    human = [0, 700, 2000, 2700, 4000]
    print("human score: ", beacon_score(human))

# Expected output:
# top talker: (('10.20.9.10', '198.51.100.7'), 9000000)
# beacon score: 0.999
# human score:  0.7

Trace the logic to see why the numbers fall where they do. top_talkers sums bytes per pair and ranks: the host moving 9 MB to one destination dwarfs the three ~2 KB flows, so it surfaces first — exactly the outbound-volume signal of exfiltration. For beacon_score, the hourly beacon's gaps are [3601, 3598, 3603, 3598] — a mean of 3,600 with tiny spread (variance 4.5, standard deviation ≈ 2.1), so the coefficient of variation is about 0.0006 and the score is ~0.999, screaming "regular." The human browsing's gaps are [700, 1300, 700, 1300] — a mean of 1,000 with a standard deviation of 300, so the coefficient of variation is 0.30 and the score drops to 0.7. (Real beacon hunting would set a threshold — say, 0.9 — and weight by the number of check-ins, since a regular pattern seen fifty times is far more convincing than one seen three times; this is the core idea, and Case Study 1 applies it to Meridian's actual telemetry.) You now hold the beacon detector that opened this chapter — the tool that turns "I have a hunch this host is beaconing" into a defensible, repeatable score.

Summary

This chapter built the network-visibility layer — the precondition for everything in Part V.

• You can't defend what you can't see. The network is ground truth: an attacker who compromises a host can lie through it, but cannot avoid generating traffic across infrastructure they do not control. Visibility is a decision made before an incident, not during it.
• Network visibility comes at three altitudes, traded off by detail vs cost/retention:
• Full packet capture (PCAP) via Wireshark/tcpdump — every byte; a microscope for deep, targeted, short-retention investigation. Encryption hides the payload, not the metadata (IPs, ports, sizes, timing, SNI).
• Zeek — network security monitoring that turns packets into one structured connection log per connection (plus protocol logs); the SOC's daily workhorse, with uid-based pivoting across logs.
• NetFlow/IPFIX flow data — a tiny per-conversation summary (5-tuple + bytes/packets/time), often generated free by routers/switches; cheap enough to retain network-wide for years.
• Full packet capture vs flow is the central architecture tradeoff: keep everything about some traffic briefly, or almost nothing about all traffic for a long time. Mature programs use both deliberately, plus Zeek in the middle.
• A network baseline (who talks to whom, on what ports, in what volumes, when) is the reference for nearly every detection; build it from flow data and sanity-check it — never baseline during a compromised window.
• The three behaviors network monitoring catches best, each detected by aggregation over many records, not a single-packet rule:
• Beaconing — regular C2 check-ins; detect by low variance of inter-arrival times (beacon_score); defeated by long intervals and jitter, countered by long flow retention.
• Data exfiltration (network) — outbound volume anomaly vs baseline; counter "low-and-slow" with cumulative baselines and watch dns.log for tunneling.
• Lateral movement — one host contacting many internal hosts on admin ports (SMB/RDP/WinRM/SSH); an east-west pattern a perimeter firewall never sees.
• Sensor placement determines what you can detect. Deliver traffic to sensors via a SPAN port (cheap, lossy under load) or a network tap (full fidelity, dedicated hardware). The structured telemetry (Zeek logs, flow) feeds the SIEM (Chapter 21); raw PCAP stays local and is pulled on demand. Correlation in the SIEM turns weak network signals into strong detections.
• Network detection and response (NDR) is the discipline of doing all of the above as an operational capability — continuously monitoring network telemetry to detect, investigate, and respond to threats.
• Project: Meridian's network-monitoring design (flow everywhere, Zeek taps on high-value and east-west links, targeted PCAP, retention tiers, all into the SIEM) and pktflow.py (summarize_flows, top_talkers, beacon_score).

Spaced Review

Retrieval practice over this chapter and its prerequisites. Answer before checking.

1. (This chapter) State the central tradeoff between full packet capture and flow data in one sentence, and give one question that each is uniquely suited to answer.
2. (Chapter 7) Network monitoring is often described as a detective control, while a firewall blocking a connection is preventive. Using the control-type vocabulary, explain the difference and why a defense-in-depth design wants both. How does an IPS blur the line between the two?
3. (Chapter 6) Lateral movement shows up as east-west traffic. Define east-west versus north-south traffic, and explain why a firewall placed only at the network perimeter cannot see lateral movement.
4. (This chapter) Why does a beacon that checks in once per day argue for long flow-data retention, and how does that retention turn the attacker's stealth into a constraint on their tempo?

What's Next

You can now see the network — capture its packets, log its connections, census its flows, and pick out the beacons, exfiltration, and lateral movement hiding in the traffic. But a stream of Zeek logs and flow records from sensors across Meridian is not yet a detection capability; it is raw material. The next leap is to collect telemetry from every source — network, endpoint, identity, cloud, application — into one place where it can be normalized, correlated, and turned into prioritized alerts a SOC analyst can act on. That system is the SIEM, and Chapter 21 (the start of Part V — Security Operations) builds it, beginning with the very network telemetry you designed here. The visibility layer is laid; now we build the brain that watches it.