Appendix L: Performance Counters and Measurement Reference

This appendix covers the tools and event names for measuring CPU and memory performance, as used in Part VI of this book.

perf stat — Essential Events

perf stat runs a program and reports hardware performance counters. It requires either root access or perf_event_paranoid set to 0 or 1.

perf stat ./program
perf stat -e event1,event2,... ./program
perf stat -r 5 ./program          # repeat 5 times and average
perf stat -p PID                  # attach to running process

Default Events (always available)

These events are available on all Linux systems without root:

Derived Metrics

From default perf stat output, compute:

perf stat — Hardware-Specific Events

These require root access or appropriate permissions. Available events vary by CPU model.

Memory Hierarchy Events

perf stat -e L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores \
             L2_rqsts.all_demand_data_rd,L2_rqsts.demand_data_rd_miss \
             LLC-loads,LLC-load-misses \
             ./program

Pipeline Events

perf stat -e cycles,instructions,stalled-cycles-frontend,stalled-cycles-backend ./program

Intel-Specific PMU Events

For Intel CPUs, more specific events are available using raw event codes or the intel_pt driver. Use perf list to see all available events on your system.

Commonly Used Intel Events

# Retire stall cycles:
perf stat -e cpu/event=0xc4,umask=0x00/  # BR_INST_RETIRED.ALL_BRANCHES

# Detailed cache breakdown:
perf stat -e '{cpu/event=0xd1,umask=0x01/,  # MEM_LOAD_RETIRED.L1_HIT
               cpu/event=0xd1,umask=0x02/,  # MEM_LOAD_RETIRED.L2_HIT
               cpu/event=0xd1,umask=0x04/,  # MEM_LOAD_RETIRED.L3_HIT
               cpu/event=0xd1,umask=0x20/}' # MEM_LOAD_RETIRED.L3_MISS

Key Intel Skylake Events

perf record and perf report

perf record samples the program at regular intervals and records which code was executing.

# Record at default frequency (4000 Hz):
perf record ./program
perf report                     # interactive TUI

# Record at higher frequency:
perf record -F 99999 ./program

# Record with call graph (requires frame pointers or DWARF):
perf record -g ./program
perf report --call-graph callee

# Record specific events:
perf record -e cache-misses:u ./program    # :u = user space only

# Annotate specific function:
perf annotate function_name

Interpreting perf report

The report shows percentages of samples in each function. A function consuming 80% of samples is not necessarily slow itself — it may be waiting for memory. Use:

• High cycles % but low LLC-miss % → compute-bound (the function is slow itself)
• High LLC-miss % → memory-bound (the function is waiting for data from DRAM)
• High stalled-cycles-backend → memory or dependency stall
• High stalled-cycles-frontend → instruction cache miss or branch misprediction

RDTSC and RDTSCP

The rdtsc (Read Time-Stamp Counter) instruction reads the processor's cycle counter.

Correct RDTSC Usage

; Serialize before reading (prevent out-of-order execution from moving
; code across the measurement boundary):
lfence
rdtsc               ; reads TSC: high 32 bits → EDX, low 32 bits → EAX
shl     rdx, 32
or      rax, rdx    ; combine into RAX (full 64-bit TSC value)
mov     [start_tsc], rax

; ... code to measure ...

lfence
rdtsc
shl     rdx, 32
or      rax, rdx
mov     [end_tsc], rax

mov     rax, [end_tsc]
sub     rax, [start_tsc]   ; elapsed cycles in RAX

RDTSCP (with CPU ID)

rdtscp additionally reads the processor ID into ECX, allowing detection of TSC measurement across CPU migrations:

rdtscp              ; TSC → EDX:EAX, processor ID → ECX
shl     rdx, 32
or      rax, rdx
; Check ECX matches the same CPU between start and end

Notes on RDTSC

• The counter increments at the reference frequency (e.g., 3.0 GHz nominal), not the boosted frequency
• On modern CPUs (Skylake+), the reference frequency is reported in CPUID leaf 0x15
• rdtsc is not privileged in x86-64 (unlike in 32-bit mode); it can be used in user space
• The lfence before and after is necessary for accurate measurement because out-of-order execution can reorder the rdtsc relative to the measured code
• Minimum overhead: approximately 20-40 cycles per measurement pair

C Wrapper

#include <stdint.h>

static inline uint64_t rdtsc_start(void) {
    uint32_t lo, hi;
    __asm__ volatile (
        "lfence\n\t"
        "rdtsc\n\t"
        "lfence"
        : "=a"(lo), "=d"(hi)
    );
    return ((uint64_t)hi << 32) | lo;
}

static inline uint64_t rdtsc_end(void) {
    uint32_t lo, hi;
    __asm__ volatile (
        "lfence\n\t"
        "rdtsc"
        : "=a"(lo), "=d"(hi)
    );
    return ((uint64_t)hi << 32) | lo;
}

// Usage:
uint64_t start = rdtsc_start();
// ... measured code ...
uint64_t elapsed = rdtsc_end() - start;

Memory Bandwidth and Latency Reference

Approximate Memory Hierarchy Latencies (Intel Skylake / AMD Zen 3)

Note: These are rough approximations. Actual values depend on memory frequency, interleaving, prefetcher behavior, and workload patterns. Use perf stat and a bandwidth benchmark (like STREAM) for your specific hardware.

Cache Line Size

On all modern x86-64 and ARM64 processors: 64 bytes. This means: - Any access within a 64-byte aligned block pulls the entire block into cache - False sharing occurs when two threads write to different variables in the same cache line - Padding structures to 64 bytes eliminates false sharing: alignas(64) in C++

TLB Capacity (Approximate, Skylake)

TLB thrashing occurs when the working set exceeds TLB coverage, causing frequent page walks. Use huge pages (2 MB or 1 GB) to extend TLB coverage for large working sets.

Agner Fog Instruction Tables Summary

Agner Fog maintains detailed per-instruction latency and throughput tables for every microarchitecture since the Pentium. Available at: https://agner.org/optimize/

How to Read the Tables

• Latency: cycles from input available to output ready (dependency chain cost)
• Reciprocal throughput: one instruction every N cycles (parallelism limit)
• Execution ports: which CPU execution units can run this instruction

Example (Skylake):

Critical Path Optimization

For a loop with dependent instructions:

; Dependency chain (serial — bad):
vmulps  ymm0, ymm1, ymm2    ; latency 4
vaddps  ymm0, ymm0, ymm3    ; latency 4; depends on prev instruction
; Total: 8 cycles for 16 floats

; Unrolled to break dependency chain (parallel — good):
vmulps  ymm0, ymm1, ymm2    ; starts cycle 0
vmulps  ymm4, ymm5, ymm6    ; starts cycle 0 (no dependency)
vaddps  ymm0, ymm0, ymm3    ; starts cycle 4
vaddps  ymm4, ymm4, ymm7    ; starts cycle 4 (no dependency)
; Total: ~4 cycles for 32 floats (2× throughput improvement)

Profiling Strategy Summary