Case Study 36-1: Stack Canary in Action — GDB Trace Through Protected Code

Introduction

Reading about stack canaries is one thing. Watching one fire in a debugger is another. This case study traces through a complete GDB session: compiling a vulnerable function with -fstack-protector, disassembling to see the canary check assembly, running the program normally to see the canary stored and checked, and then triggering the stack overflow to see the protection respond.

The goal is to make the canary mechanism concrete and observable, not abstract. Security defenses only inspire confidence when you understand exactly what they do.

🔐 Security Note: This lab uses an intentionally vulnerable function compiled in a controlled educational environment. The procedure shown here is how security engineers verify that mitigations are working and understand their limits.

Setup: The Vulnerable Function

/* canary_demo.c */
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

void vulnerable(const char *input) {
    char buf[32];
    strcpy(buf, input);
    printf("Processed: %s\n", buf);
}

int main(int argc, char **argv) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <input>\n", argv[0]);
        return 1;
    }
    vulnerable(argv[1]);
    return 0;
}

Compile WITH stack protection (default on modern GCC):

gcc -g -fstack-protector-strong -o canary_demo canary_demo.c
checksec --file=canary_demo
# Stack: Canary found ✓

Phase 1: Disassembling the Protected Function

$ gdb -q canary_demo
(gdb) set disassembly-flavor intel
(gdb) disassemble vulnerable
Dump of assembler code for function vulnerable:
   0x401196 <+0>:   push   rbp
   0x401197 <+1>:   mov    rbp,rsp
   0x40119a <+4>:   sub    rsp,0x40
   ; === CANARY PROLOGUE BEGIN ===
   0x40119e <+8>:   mov    rax,QWORD PTR fs:0x28      ; read canary from TLS
   0x4011a7 <+17>:  mov    QWORD PTR [rbp-0x8],rax    ; store at rbp-8
   0x4011ab <+21>:  xor    eax,eax                    ; clear rax (security)
   ; === CANARY PROLOGUE END ===
   0x4011ad <+23>:  mov    QWORD PTR [rbp-0x38],rdi   ; save input arg
   0x4011b1 <+27>:  mov    rax,QWORD PTR [rbp-0x38]   ; reload
   0x4011b5 <+31>:  mov    rsi,rax                    ; strcpy src
   0x4011b8 <+34>:  lea    rax,[rbp-0x28]             ; buf at rbp-40 (size 32)
   0x4011bc <+38>:  mov    rdi,rax                    ; strcpy dst
   0x4011bf <+41>:  call   strcpy@plt
   0x4011c4 <+46>:  lea    rax,[rbp-0x28]
   0x4011c8 <+50>:  mov    rsi,rax
   0x4011cb <+53>:  lea    rdi,[rip+0xe36]            ; "Processed: %s\n"
   0x4011d2 <+60>:  mov    eax,0x0
   0x4011d7 <+65>:  call   printf@plt
   ; === CANARY EPILOGUE BEGIN ===
   0x4011dc <+70>:  mov    rax,QWORD PTR [rbp-0x8]    ; reload canary from stack
   0x4011e0 <+74>:  xor    rax,QWORD PTR fs:0x28      ; XOR with original
   0x4011e9 <+83>:  je     0x4011f0 <vulnerable+90>   ; if zero (ok), proceed
   0x4011eb <+85>:  call   __stack_chk_fail@plt        ; canary changed!
   ; === CANARY EPILOGUE END ===
   0x4011f0 <+90>:  leave
   0x4011f1 <+91>:  ret

Phase 2: Observing the Canary Value at Runtime

(gdb) break *0x4011a7    ; breakpoint just before canary store
(gdb) run HELLO

Breakpoint 1, 0x00000000004011a7 in vulnerable ()
(gdb) info registers rax
rax  0x4cd8b2e5f70a2300    ; ← This IS the canary value for this run

(gdb) # Note the LOW BYTE: 0x00. Always zero (null terminator protection)
(gdb) # The remaining 7 bytes are random from /dev/urandom at process start

(gdb) stepi    ; execute: mov [rbp-0x8], rax
(gdb) x/8xb ($rbp - 8)
0x7ffd...b7b8:  0x00 0x23 0x0a 0xf7 0xe5 0xb2 0xd8 0x4c
; ^ null byte ^ random bytes (little-endian of 0x4cd8b2e5f70a2300)

Phase 3: Verifying Normal Operation

Continue to the epilogue check:

(gdb) break *0x4011dc    ; epilogue canary load
(gdb) continue

Breakpoint 2, 0x00000000004011dc in vulnerable ()
(gdb) x/8xb ($rbp - 8)
0x7ffd...b7b8:  0x00 0x23 0x0a 0xf7 0xe5 0xb2 0xd8 0x4c
; Matches what we stored — normal operation

(gdb) stepi    ; load canary into rax
(gdb) stepi    ; xor with fs:0x28
(gdb) info registers rax
rax  0x0                   ; ← zero! XOR of identical values = 0

(gdb) stepi    ; je: jumps (ZF=1, values matched)
(gdb) # Next is leave; ret — normal return

The canary is intact. Normal flow proceeds.

Phase 4: Triggering the Canary

Now run with an input large enough to overwrite the canary:

Stack layout for vulnerable():
- buf: rbp-0x28 (40 bytes below rbp)
- buf size: 32 bytes
- gap (alignment): 8 bytes
- canary: rbp-0x8 (8 bytes below rbp)
- saved RBP: rbp
- return addr: rbp+8

Offset from buf[0] to canary:
  40 - 8 = 32 bytes of buf
  + 8 bytes of gap
  = 40 bytes to canary

Input needed to reach canary: 40+ bytes

(gdb) delete breakpoints
(gdb) run $(python3 -c "print('A' * 40 + 'B' * 8)")

*** stack smashing detected ***: terminated
Program received signal SIGABRT

The canary was overwritten with BBBBBBBB (0x4242424242424242). The epilogue check:

(gdb) break *0x4011dc    ; epilogue
(gdb) run $(python3 -c "print('A'*40 + 'B'*8)")

Breakpoint, 0x4011dc in vulnerable ()
(gdb) x/8xb ($rbp - 8)
0x7ffd...b7b8:  0x42 0x42 0x42 0x42 0x42 0x42 0x42 0x42
; ← Canary overwritten with 'B' bytes

(gdb) stepi    ; mov rax, [rbp-8]
(gdb) info registers rax
rax  0x4242424242424242   ; ← overwritten canary

(gdb) stepi    ; xor rax, [fs:0x28]
(gdb) info registers rax
rax  0xe9a0e47ab1c8a142   ; ← non-zero! Canary mismatch

(gdb) stepi    ; je FAILS (ZF=0)
(gdb) # Next instruction: call __stack_chk_fail
(gdb) stepi    ; call __stack_chk_fail

*** stack smashing detected ***: ./canary_demo terminated

Phase 5: What __stack_chk_fail Does

__stack_chk_fail is a function in glibc (libc.so):

/* glibc source: sysdeps/x86_64/stack_chk_fail.S (simplified) */
void __stack_chk_fail(void) {
    __fortify_fail("stack smashing detected");
}

void __fortify_fail(const char *msg) {
    write(2, "*** ", 4);
    write(2, msg, strlen(msg));
    write(2, " ***: ", 6);
    write(2, program_invocation_short_name, ...);
    write(2, " terminated\n", 12);
    abort();     /* raises SIGABRT, cannot be caught */
}

The function: 1. Writes the error message to stderr 2. Calls abort() which sends SIGABRT to itself 3. SIGABRT cannot be caught (the process terminates unconditionally)

Key Observations

  1. The canary low byte is always null: visible in the stored value 0x...00. This means string functions that terminate on null cannot read or copy the full canary. A leak via puts(canary_address) would only show an empty string.

  2. The canary is random per-process: running the demo twice shows different canary values. An attacker who exploited the program yesterday and noted the canary value cannot use it today.

  3. The canary position is deterministic: the compiler places it at rbp-8, just below the saved RBP. This layout is consistent across compilations.

  4. Abort is unconditional: SIGABRT from abort() cannot be intercepted by a signal handler in the same process. The OS terminates the process.

  5. The overhead is tiny: two instructions in the prologue (load + store) and three in the epilogue (load + xor + branch). This is why -fstack-protector-strong is enabled by default — the performance cost is negligible.

The canary is not perfect (format string leaks can reveal it) but it is extremely effective against naive overflow-to-return-address attacks. It raises the exploitation bar from "overflow and redirect" to "overflow, leak canary, overflow again with correct canary" — a significantly more complex requirement.