Case Study 34-1: Reverse Engineering a Crackme

Introduction

Crackmes are small programs designed as reverse engineering practice challenges. They simulate a class of real-world software: programs with license key validation, serial number checking, or authentication routines that you must understand without access to source code. In professional security work, the same skills apply to analyzing commercial software, validating binary integrity, and understanding obfuscated malware.

This case study walks through a complete crackme analysis: loading the binary, finding the validation routine, understanding the comparison logic, and extracting the correct password — all without running the program with the correct input.

🔐 Security Note: The techniques demonstrated here are used by: security researchers auditing software for vulnerabilities, malware analysts understanding malicious software, CTF competitors solving reverse engineering challenges, and software engineers debugging their own compiled output. These skills do not authorize circumventing software protections on software you do not own or have permission to analyze.

The Target Binary

Our crackme, crackme01, is a 64-bit ELF binary. It prompts for a password, validates it, and prints either "Access granted" or "Access denied."

$ file crackme01
crackme01: ELF 64-bit LSB executable, x86-64, dynamically linked, stripped

$ ./crackme01
Enter password: hello
Access denied.

$ ./crackme01
Enter password: test
Access denied.

The binary is stripped — no symbols. We cannot simply run objdump -t and find a function named check_password.

Phase 1: Intelligence Gathering

Start with non-execution analysis:

# What libraries does it use?
$ ldd crackme01
        linux-vdso.so.1
        libc.so.6

# What functions does it import?
$ objdump -T crackme01
DYNAMIC SYMBOL TABLE:
00000000 w   D  *UND* printf
00000000 w   D  *UND* fgets
00000000 w   D  *UND* strcmp
00000000 w   D  *UND* strlen
00000000 w   D  *UND* puts
00000000 w   D  *UND* __stack_chk_fail

# What interesting strings exist?
$ strings crackme01
Enter password:
Access granted. Welcome!
Access denied.
/bin/sh

The dynamic symbol table tells us a lot: strcmp means there is a direct string comparison somewhere. strlen means it checks length. fgets is used to read input (safer than gets). __stack_chk_fail means the binary was compiled with stack canaries.

The strings are also informative: "Enter password:", "Access granted", "Access denied" are the expected UI strings. But what is /bin/sh doing in a crackme? That is suspicious — we will keep it in mind.

Phase 2: Finding Main

Open in GDB and find main:

$ gdb -q crackme01
(gdb) set disassembly-flavor intel
(gdb) break *0x401080    ; entry point (from readelf -h crackme01)
(gdb) run
Breakpoint 1, 0x0000000000401080 in ?? ()

(gdb) disassemble $rip, +80
   0x401080: endbr64
   0x401084: xor    ebp, ebp
   0x401086: mov    r9, rdx
   0x401089: pop    rsi
   0x40108a: mov    rdx, rsp
   0x40108d: and    rsp, 0xfffffffffffffff0
   0x401091: push   rax
   0x401092: push   rsp
   0x401093: lea    r8, [rip+0x2b5]       ; __libc_csu_fini
   0x40109a: lea    rcx, [rip+0x24e]      ; __libc_csu_init
   0x4010a1: lea    rdi, [rip+0x2c8]      ; main — RIP at 0x4010a8, +0x2c8 = 0x401370
   0x4010a8: call   0x401060 <__libc_start_main@plt>

(gdb) # main is at 0x4010a8 + 7 + (let's calculate)
(gdb) # RIP after lea rdi is 0x4010a8. 0x4010a8 + 0x2c8 = 0x401370
(gdb) break *0x401370
(gdb) continue
Breakpoint 2, 0x0000000000401370 in ?? ()

Phase 3: Static Analysis of main

(gdb) disassemble 0x401370, 0x401450
   0x401370: push   rbp
   0x401371: mov    rbp, rsp
   0x401374: sub    rsp, 0x50           ; 80 bytes local storage
   0x401378: mov    rax, [fs:0x28]      ; canary — this explains __stack_chk_fail
   0x401381: mov    [rbp-0x8], rax      ; store canary at rbp-8
   0x401385: xor    eax, eax

   ; Print "Enter password: "
   0x401387: lea    rdi, [rip+0xc76]    ; "Enter password: "
   0x40138e: mov    eax, 0x0
   0x401393: call   printf@plt

   ; Read input: fgets(buf, 64, stdin)
   0x401398: mov    rdx, [rip+0x2cc1]   ; stdin
   0x40139f: lea    rax, [rbp-0x48]     ; buf at rbp-72 (72 bytes from rbp)
   0x4013a3: mov    esi, 0x40           ; 64 bytes
   0x4013a8: mov    rdi, rax
   0x4013ab: call   fgets@plt

   ; Strip trailing newline: buf[strlen(buf)-1] = '\0'
   0x4013b0: lea    rax, [rbp-0x48]
   0x4013b4: mov    rdi, rax
   0x4013b7: call   strlen@plt
   0x4013bc: sub    rax, 0x1
   0x4013c0: mov    BYTE [rbp-0x48+rax], 0x0

   ; Call validate function
   0x4013c6: lea    rax, [rbp-0x48]
   0x4013ca: mov    rdi, rax
   0x4013cd: call   0x4011a0            ; <-- the validation function!

   ; Check return value
   0x4013d2: test   eax, eax
   0x4013d4: je     0x4013e5            ; 0 = fail
   0x4013d6: lea    rdi, [rip+0xc83]    ; "Access granted. Welcome!"
   0x4013dd: call   puts@plt
   0x4013e2: jmp    0x4013f1
   0x4013e5: lea    rdi, [rip+0xc7e]    ; "Access denied."
   0x4013ec: call   puts@plt
   ; Canary check
   0x4013f1: mov    rax, [rbp-0x8]
   0x4013f5: xor    rax, [fs:0x28]
   0x4013fe: je     0x401405
   0x401400: call   __stack_chk_fail@plt
   0x401405: leave
   0x401406: ret

We now know: 1. Input is read into a 64-byte buffer at rbp-0x48 2. The validation function is at 0x4011a0 3. It returns non-zero for success

Phase 4: Analyzing the Validation Function

(gdb) disassemble 0x4011a0, 0x401196
   0x4011a0: push   rbp
   0x4011a1: mov    rbp, rsp
   0x4011a4: sub    rsp, 0x10
   0x4011a8: mov    [rbp-0x8], rdi    ; save input pointer

   ; Phase 1: Check length == 8
   0x4011ac: mov    rdi, [rbp-0x8]
   0x4011b0: call   strlen@plt
   0x4011b5: cmp    rax, 0x8
   0x4011b9: jne    0x401210          ; wrong length → fail

   ; Phase 2: XOR transform and check
   ; Load 8-byte input as a 64-bit integer
   0x4011bf: mov    rax, [rbp-0x8]
   0x4011c3: mov    rax, QWORD [rax]   ; load 8 bytes of input
   0x4011c6: xor    rax, 0x2a2a2a2a2a2a2a2a  ; XOR with 0x2a2a2a2a2a2a2a2a
   0x4011d0: cmp    rax, 0x534f534f534f4f53   ; compare with this constant
   0x4011da: jne    0x401210           ; mismatch → fail

   ; Phase 3: Check last byte is 0x21 ('!')
   ; Wait — if length == 8 and we check 8 bytes as a word, there's no "last byte" check
   ; Let's re-examine... actually 0x4011d0 checks the XOR'd 8 bytes against a constant

   ; Success path
   0x4011dc: mov    eax, 0x1
   0x4011e1: jmp    0x401215

   ; Fail path
   0x401210: xor    eax, eax

   0x401215: leave
   0x401216: ret

Phase 5: Solving the Challenge

The validation logic: 1. Input must be exactly 8 characters long 2. The 8 input bytes, loaded as a little-endian 64-bit integer and XOR'd with 0x2a2a2a2a2a2a2a2a, must equal 0x534f534f534f4f53

To find the correct input, reverse the XOR:

correct_bytes = target XOR key
correct_bytes = 0x534f534f534f4f53 XOR 0x2a2a2a2a2a2a2a2a

Compute byte by byte (little-endian order, so the first byte of input is the least significant byte):

The password is yeeyeyey... let us verify by checking: 0x79 XOR 0x2a = 0x53 ✓, 0x65 XOR 0x2a = 0x4f ✓.

Actually, re-reading the bytes: target is 0x534f534f534f4f53 in little-endian. In memory, the bytes are stored as 53 4f 4f 53 4f 53 4f 53. Let's redo:

Byte 0 (lowest) of the 64-bit value in little-endian: 0x53 → XOR 0x2a = 0x79 = y Byte 1: 0x4f XOR 0x2a = 0x65 = e

The password is yeeyseys — or more precisely, whatever the 8-byte XOR inverse produces. The key insight is that the algorithm is fully reversed analytically.

Phase 6: Verification with GDB

Set a breakpoint at the comparison and check our derived input:

(gdb) break *0x4011d0
(gdb) run
Enter password: yeeyeyey
Breakpoint 3, 0x00000000004011d0 in ?? ()

(gdb) info registers rax
rax  0x534f534f534f4f53

(gdb) print/x 0x534f534f534f4f53 == 0x534f534f534f4f53
$1 = 0x1    ; true — they match!

(gdb) continue
Access granted. Welcome!

What About /bin/sh?

The string /bin/sh in the binary is suspicious for a simple crackme. Reverse engineering further reveals a dead code path — a function that is never called from main — that contains a execve("/bin/sh", ...) sequence. This is a common CTF trick: include a "free gift" in the binary that becomes the target of a more advanced challenge (a ROP chain to redirect execution to it). For now, we note it and move on.

Lessons Learned

1. String cross-references guide analysis: "Enter password" found the main validation flow; the suspicious /bin/sh flagged a secondary challenge.
2. Dynamic symbol table reveals algorithm: strcmp suggests comparison; XOR against a constant is a weaker but common obfuscation.
3. Work backward from the comparison: find the final cmp, understand what both sides are, reverse the transformation.
4. No need to run the binary to solve it: pure static analysis found the password completely.
5. Canary detection: the presence of __stack_chk_fail told us the binary was compiled with -fstack-protector.

The skills applied here — finding function boundaries, reading library calls, understanding XOR obfuscation, reversing a simple transform — apply directly to real-world license key validation, simple malware obfuscation, and CTF challenges.