Case Study 5.1: GDB Deep Dive — Watching a System Call Happen

A complete GDB session that watches the sys_write syscall from user space, through the kernel transition, and back

Overview

The syscall instruction is one of the most important instructions in assembly programming, yet it's also one of the most opaque: you execute it, and something happens in the kernel, and you get a return value. This case study uses GDB to observe the complete syscall mechanism — the register setup, the mode transition, and the return — at the assembly level.

We'll trace a sys_write call through all visible steps: the register setup before syscall, the instruction itself, the kernel's return path via sysret, and what RAX contains after it's all done. We'll also watch what happens to RCX and R11 during the transition.

The Setup

We use the hello world program from Chapter 5, built with debug symbols:

nasm -f elf64 -g -F dwarf hello.asm -o hello.o
ld hello.o -o hello
gdb hello

We set a breakpoint right before the sys_write syscall:

(gdb) break _start
Breakpoint 1 at 0x401000: file hello.asm, line 11.
(gdb) run
Starting program: /path/to/hello

Breakpoint 1, _start () at hello.asm:11
11          mov     rax, 1

(gdb) display /x $rax
(gdb) display /x $rdi
(gdb) display /x $rsi
(gdb) display /x $rdx
(gdb) display /x $rcx
(gdb) display /x $r11
(gdb) display /x $rflags

We include RCX and R11 specifically because the syscall instruction saves RIP to RCX and RFLAGS to R11.

Phase 1: Register Setup

Step through the four mov instructions:

(gdb) stepi    ; mov rax, 1
12          mov     rdi, 1
4: /x $rax = 0x1        ← syscall number: sys_write
1: /x $rdi = 0x0
2: /x $rsi = 0x0
3: /x $rdx = 0x0
5: /x $rcx = 0x0        ← RCX will be clobbered by SYSCALL
6: /x $r11 = 0x0        ← R11 will be clobbered by SYSCALL
7: /x $rflags = 0x202   ← RFLAGS before syscall (IF=1, reserved=1)

(gdb) stepi    ; mov rdi, 1
13          mov     rsi, msg
1: /x $rdi = 0x1        ← file descriptor: stdout

(gdb) stepi    ; mov rsi, msg
14          mov     rdx, len
2: /x $rsi = 0x402000   ← address of "Hello, Assembly!\n"

(gdb) stepi    ; mov rdx, len
15          syscall
3: /x $rdx = 0x11       ← length: 17 bytes

At this point, all four syscall arguments are set: - RAX = 1 (sys_write) - RDI = 1 (stdout) - RSI = 0x402000 (message address) - RDX = 0x11 = 17 (byte count)

Phase 2: The syscall Instruction

Now we execute syscall:

(gdb) stepi    ; syscall instruction
Hello, Assembly!
16          mov     rax, 60
4: /x $rax = 0x11       ← return value: 17 bytes written
1: /x $rdi = 0x1        ← RDI unchanged
2: /x $rsi = 0x402000   ← RSI unchanged
3: /x $rdx = 0x11       ← RDX unchanged
5: /x $rcx = 0x40101e   ← RCX NOW HOLDS THE RETURN ADDRESS (next instruction after syscall)
6: /x $r11 = 0x302      ← R11 NOW HOLDS RFLAGS AT TIME OF SYSCALL
7: /x $rflags = 0x246   ← RFLAGS changed (different bits set on return)

Several critical things happened in that one stepi:

1. The program output appeared. The kernel executed sys_write and the string was written to stdout.

2. RAX changed from 1 to 17 (0x11). The syscall return value is placed in RAX. sys_write returns the number of bytes written — 17.

3. RCX was overwritten with 0x40101e. This is the address of the instruction immediately after the syscall instruction. The SYSCALL mechanism works like this: before jumping to the kernel, the CPU saves RIP (the return address) into RCX so it knows where to return. If your code had an important value in RCX before the syscall, it's gone.

4. R11 was overwritten with 0x302. This is the value of RFLAGS at the time the syscall was executed. The SYSRET instruction (used by the kernel to return to user space) restores RFLAGS from R11. This is why R11 is clobbered.

5. RFLAGS changed. The flags after the syscall return may differ from what they were before, because the kernel's return path (SYSRET) restores RFLAGS from R11, which was the pre-syscall value... but with potential modifications.

What Actually Happened Inside the Kernel (Educational Overview)

GDB can't easily trace into kernel code from user space (it would require kernel debugging setup). But we can describe what happened:

1. SYSCALL executes: The CPU saves RIP to RCX, saves RFLAGS to R11, switches to kernel privilege level (CPL 0), loads the kernel stack pointer from the per-CPU GS-based structure, and jumps to the syscall entry point (the address in the IA32_LSTAR MSR).

2. Kernel syscall entry: The Linux kernel's entry_SYSCALL_64 (in arch/x86/entry/entry_64.S) saves all registers to the kernel stack, sets up the kernel's C environment, and calls sys_call_table[rax] — which for RAX=1 is __x64_sys_write.

3. sys_write executes: The VFS layer finds the file descriptor 1 (stdout), calls the terminal driver's write function, which copies the bytes to the terminal's output buffer and wakes up any process waiting to display them.

4. Return path: The kernel restores saved registers, places the return value in RAX, executes SYSRET (which restores RIP from RCX and RFLAGS from R11 — which the kernel modified as needed), and returns to user space.

5. Back in user space: Execution continues at the instruction after syscall (address 0x40101e), with RAX containing the return value (17).

Phase 3: After the Syscall

(gdb) stepi    ; mov rax, 60
17          xor     rdi, rdi
4: /x $rax = 0x3c       ← sys_exit syscall number (60 = 0x3c)
5: /x $rcx = 0x40101e   ← RCX still has the old return address from sys_write
6: /x $r11 = 0x302      ← R11 still has the old RFLAGS value

(gdb) stepi    ; xor rdi, rdi
18          syscall
1: /x $rdi = 0x0        ← exit status: 0
4: /x $rax = 0x3c

(gdb) stepi    ; syscall (sys_exit — program terminates)
[Inferior 1 (process 12345) exited normally]

The second syscall (sys_exit = 60, 0x3c) terminates the program. GDB reports "exited normally."

Key Observations Summary

What SYSCALL Does to Registers

The Practical Rule

If you have important values in RCX or R11 that you need after a syscall, save them before the syscall:

push rcx        ; save RCX
push r11        ; save R11
; set up syscall arguments
syscall
pop  r11        ; restore R11
pop  rcx        ; restore RCX

Why 17 Bytes Were Written

sys_write(1, msg, 17) wrote the 17 bytes of "Hello, Assembly!\n" (16 ASCII characters + newline). The return value 17 confirms all 17 bytes were written successfully. If fewer bytes were written (possible with some I/O), the return value would be less than 17, and a production program would retry.

Extending the Investigation

Some GDB tricks for deeper syscall investigation:

; Break specifically ON the syscall instruction:
(gdb) break *0x40101c    ; address of the syscall instruction

; Examine what's at the message address:
(gdb) x/s $rsi           ; print the string that will be written

; Watch for changes to a specific register:
(gdb) watch $rcx         ; stop whenever RCX changes
; (Warning: hardware watchpoints are limited; this may slow execution)

; Examine the syscall table address (if you have kernel symbols):
; (requires kernel debug info -- usually not available in user-mode GDB)

; Count syscalls in a longer program:
; Run with strace instead of GDB:
strace -c ./hello        ; count and time each syscall type

This case study demonstrates that syscall is not magic — it's a precisely specified instruction with documented register effects. Understanding those effects prevents an entire category of hard-to-diagnose bugs where values in RCX or R11 are silently corrupted by syscalls that the programmer didn't realize would clobber those registers.