Case Study 11.2: A Stack Buffer Overflow — The Memory Layout

Purpose and Framing

This case study is defensive analysis, not an exploitation tutorial. The goal is to understand precisely what happens in memory during a stack buffer overflow, using GDB to examine the actual bytes. This knowledge is the foundation of both exploit writing (Chapters 35-37) and defense: you cannot design effective mitigations for a vulnerability you do not understand at the byte level.

The Vulnerable Function

// vulnerable.c — compiled with: gcc -O0 -fno-stack-protector -g -o vulnerable
#include <string.h>
#include <stdio.h>

void safe_function() {
    puts("This should never print if we corrupt the return address.");
}

void vulnerable_function(char *input) {
    char buffer[64];
    strcpy(buffer, input);   // no bounds check
    printf("Buffer contents: %s\n", buffer);
}

int main(int argc, char *argv[]) {
    if (argc < 2) {
        puts("Usage: ./vulnerable <input>");
        return 1;
    }
    vulnerable_function(argv[1]);
    return 0;
}

We compile with -fno-stack-protector to disable stack canaries (so we can see what the vulnerable memory looks like without the defense in place). This is for educational purposes.

The Stack Layout

The function vulnerable_function has this stack frame:

Higher addresses:
┌───────────────────────────────────────────────────┐
│ main's stack frame                                │
│   ...                                             │
├───────────────────────────────────────────────────┤  ← RSP before CALL
│ return address to main (8 bytes)                  │  ← [RBP + 8]
├───────────────────────────────────────────────────┤  ← RBP → [RBP + 0]
│ saved RBP (main's frame pointer, 8 bytes)         │
├───────────────────────────────────────────────────┤  ← [RBP - 8]
│ char buffer[64] — HIGH end of buffer              │  64 bytes
│   buffer[56..63]                                  │  ← [RBP - 8]  to [RBP - 1]
│   buffer[48..55]                                  │  ← [RBP - 16] to [RBP - 9]
│   buffer[40..47]                                  │
│   buffer[32..39]                                  │
│   buffer[24..31]                                  │
│   buffer[16..23]                                  │
│   buffer[8..15]                                   │
│   buffer[0..7]  — LOW end of buffer               │  ← [RBP - 72]
└───────────────────────────────────────────────────┘  ← RSP
Lower addresses

The total layout from buffer[0] to the return address: - buffer[0] starts at [RBP - 72] (64 bytes = 8 × 8-byte slots, but GCC may add padding; verify with GDB) - Return address is at [RBP + 8]

Distance from buffer[0] to return address: 72 + 8 = 80 bytes.

GDB Session: Examining the Stack

$ gcc -O0 -fno-stack-protector -g -o vulnerable vulnerable.c
$ gdb ./vulnerable

(gdb) break vulnerable_function
Breakpoint 1 at 0x401142: file vulnerable.c, line 9.
(gdb) run AAAA
Starting program: ./vulnerable AAAA

Breakpoint 1, vulnerable_function (input=0x7fffffffe5a3 "AAAA") at vulnerable.c:9
9           char buffer[64];
(gdb) next
10          strcpy(buffer, input);
(gdb) info frame
Stack level 0, frame at 0x7fffffffe290:
 rip = 0x401142 in vulnerable_function (vulnerable.c:10)
 called by frame at 0x7fffffffe2c0
 source language c.
 Arglist at 0x7fffffffe280, args: input=0x7fffffffe5a3 "AAAA"
 Locals at 0x7fffffffe280, Previous frame's sp is 0x7fffffffe290
 Saved registers:
  rbp at 0x7fffffffe280, rip at 0x7fffffffe288
(gdb) p &buffer
$1 = (char (*)[64]) 0x7fffffffe240
(gdb) p $rbp
$2 = (void *) 0x7fffffffe280

From GDB: - buffer starts at 0x7fffffffe240 - rbp is 0x7fffffffe280 - Offset from buffer to RBP: 0x7fffffffe280 - 0x7fffffffe240 = 0x40 = 64 - Return address is at RBP+8: 0x7fffffffe288 - Offset from buffer to return address: 64 + 8 = 72 bytes in this compilation

The exact offset varies by compiler version and optimization level. Always verify with GDB.

(gdb) next
11          printf("Buffer contents: %s\n", buffer);
(gdb) x/20gx 0x7fffffffe240
0x7fffffffe240: 0x4141414100000000  0x0000000000000000
0x7fffffffe250: 0x0000000000000000  0x0000000000000000
0x7fffffffe260: 0x0000000000000000  0x0000000000000000
0x7fffffffe270: 0x0000000000000000  0x0000000000000000
0x7fffffffe280: 0x00007fffffffe2c0  ← saved RBP
0x7fffffffe288: 0x0000000000401196  ← return address (back to main)

We can see: "AAAA\0" written into buffer (0x41414141 = 'AAAA'), then zeroes for the rest of the buffer, then the saved RBP, then the return address.

What an Overflow Looks Like

Now run with an 80-byte input (72 bytes to reach return address + 8 bytes to overwrite it):

$ python3 -c "print('A'*72 + 'BBBBBBBB', end='')" | xargs ./vulnerable

Or in GDB, examine what happens when strcpy writes 80 bytes:

(gdb) run $(python3 -c "import sys; sys.stdout.buffer.write(b'A'*72 + b'B'*8)")
(gdb) # After strcpy, examine the stack:
(gdb) x/20gx 0x7fffffffe240
0x7fffffffe240: 0x4141414141414141  0x4141414141414141  ← buffer: 'AAAA...'
0x7fffffffe250: 0x4141414141414141  0x4141414141414141
0x7fffffffe260: 0x4141414141414141  0x4141414141414141
0x7fffffffe270: 0x4141414141414141  0x4141414141414141
0x7fffffffe280: 0x4141414141414141  ← saved RBP OVERWRITTEN with 'AAAAAAAA'
0x7fffffffe288: 0x4242424242424242  ← return address OVERWRITTEN with 'BBBBBBBB'

The return address is now 0x4242424242424242. When ret executes, it will try to jump to address 0x4242424242424242. This is not a valid mapped address, so the OS delivers SIGSEGV:

(gdb) continue
Program received signal SIGSEGV, Segmentation fault.
0x0000000000401168 in vulnerable_function ()

Wait — SIGSEGV happens during the function's ret instruction, which tries to jump to the overwritten address.

What a Stack Canary Prevents

Recompile with stack protection:

$ gcc -O0 -fstack-protector-all -g -o protected vulnerable.c

The compiler inserts:

vulnerable_function:
    push   rbp
    mov    rbp, rsp
    sub    rsp, 80            ; extra space for canary

    ; Read canary from Thread Local Storage (FS:[0x28])
    mov    rax, qword fs:[0x28]
    mov    qword [rbp - 8], rax   ; store canary between locals and saved RBP

    ; ... function body (strcpy, etc.) ...

    ; Check canary before return:
    mov    rdx, qword [rbp - 8]   ; reload canary
    xor    rdx, qword fs:[0x28]   ; compare with original
    jne    .stack_chk_fail        ; if different, abort
    leave
    ret
.stack_chk_fail:
    call   __stack_chk_fail       ; terminates the program

The stack layout with canary:

[RBP + 8]  = return address
[RBP + 0]  = saved RBP
[RBP - 8]  = CANARY (random 8 bytes, set at function entry)
[RBP - 72] = char buffer[64]

Now buffer[0] is at [RBP - 72], the canary is at [RBP - 8]. Any overflow that reaches the return address must first overwrite the canary. The canary check before ret detects the corruption and aborts before the overwritten return address can be used.

The Distance: A Summary

Exact offsets depend on compiler, optimization level, and alignment decisions. Always measure with GDB or objdump on the actual binary.

Defenses at a Glance

All four together make exploitation significantly harder. Chapters 35-37 show how determined attackers bypass these defenses with techniques like ROP (Return-Oriented Programming).