Chapter 11 Exercises: The Stack and Function Calls

Section A: Stack Mechanics

Exercise 1. Given the following sequence, what are the values of RSP and the stack contents (as a list from top to bottom) after each instruction? Initial state: RSP = 0x7FFFFFE000, RAX = 0x1111, RBX = 0x2222, RCX = 0x3333.

push rax
push rbx
push rcx
pop  rdx
pop  rsi

Draw a stack diagram showing the state after each instruction.

Exercise 2. A function is called with RSP = 0x7FFFFFFFE000. After the function's prologue (push rbp; mov rbp, rsp; sub rsp, 64), what are the values of:

(a) RSP (b) RBP (c) The address of [rbp - 8] (first local variable) (d) The address of [rbp - 64] (last local variable, 8 bytes) (e) The address of [rbp + 8] (return address as seen from inside the function)

Exercise 3. True or false, and explain:

(a) After a function returns, the memory it used on the stack is immediately zeroed. (b) POP loads a value from memory and increments RSP. (c) CALL is exactly equivalent to push rip; jmp target. (d) If a function pushes RBX in its prologue and pops RBX in its epilogue, the caller's RBX is guaranteed to be unchanged after the call. (e) The 16-byte stack alignment requirement means the stack must always contain a multiple of 16 bytes.

Exercise 4. What is wrong with the following function epilogue?

my_function:
    push rbp
    mov  rbp, rsp
    push rbx              ; save rbx
    sub  rsp, 32          ; 32 bytes of locals

    ; ... function body ...

    add  rsp, 32          ; deallocate locals
    ret                   ; BUG: what did we forget?

Section B: ABI and Calling Convention

Exercise 5. Write the NASM code to call the C function memcpy(dst, src, 64). Assume: - dst address is in R12 (callee-saved, so it is preserved across calls) - src address is in R13 - You are already inside a function with a valid stack frame

After the call, restore the stack to its pre-call state.

Exercise 6. The following C function has the System V ABI signature:

int64_t compute(int64_t a, int64_t b, int64_t c, int64_t d,
                int64_t e, int64_t f, int64_t g, int64_t h);

Write the NASM code to call compute(1, 2, 3, 4, 5, 6, 7, 8). Include the stack alignment fix.

Exercise 7. Identify which registers are caller-saved and which are callee-saved in the System V AMD64 ABI:

Exercise 8. The following function violates the ABI. Find all violations:

bad_function:
    ; Computes: return a + b (a=RDI, b=RSI)
    push rbp
    mov  rbp, rsp

    mov  rbx, rdi         ; use rbx to hold 'a'
    add  rbx, rsi         ; rbx = a + b
    mov  rax, rbx         ; return value

    pop  rbp
    ret
    ; (rbx was never saved/restored)

Exercise 9. Write a complete NASM function implementing int64_t power(int64_t base, int64_t exp) using recursion. The function should compute base^exp for exp >= 0. Use the fast exponentiation algorithm: - If exp == 0: return 1 - If exp is even: return power(base, exp/2)^2 - If exp is odd: return base * power(base, exp-1)

Include complete prologue, epilogue, and callee-save/restore for any registers you use beyond the argument registers.

Section C: Stack Frame Analysis

Exercise 10. Given the following disassembly of a function (Intel syntax), reconstruct the C function signature and identify all local variables, their types (based on size), and what each register holds:

mystery_func:
    push   rbp
    mov    rbp, rsp
    sub    rsp, 48
    mov    [rbp - 8],  rdi       ; save arg1
    mov    [rbp - 16], rsi       ; save arg2
    mov    [rbp - 24], rdx       ; save arg3
    mov    dword [rbp - 28], ecx ; save arg4 (32-bit)
    ; function body...
    mov    eax, dword [rbp - 28]
    add    rax, [rbp - 8]
    leave
    ret

Exercise 11. Use GDB to examine a real function's stack frame. Write the following C code, compile it with gcc -O0 -g, and run it under GDB:

#include <stdio.h>

long triple(long x) {
    long result = x * 3;
    return result;
}

int main() {
    long y = triple(7);
    printf("%ld\n", y);
    return 0;
}

In GDB, set a breakpoint at triple. When it hits, use: - info registers to see all register values - x/4gx $rsp to dump 4 quadwords from RSP - backtrace to see the call stack

Answer: What is at [rsp], [rsp+8], [rsp+16]?

Section D: Implementing Functions

Exercise 12. Write a NASM function void reverse_array(int64_t *arr, int64_t n) that reverses an array in-place. The function must: - Save/restore all callee-saved registers it uses - Maintain 16-byte stack alignment - Not use any external function calls

Exercise 13. Write a NASM function int64_t sum_recursive(int64_t *arr, int64_t n) that recursively sums an array: - Base case: if n == 0, return 0 - Recursive case: return arr[0] + sum_recursive(arr+1, n-1)

This function is intentionally inefficient (to practice recursion in assembly). Include the full stack frame with frame pointer.

Exercise 14. A common pattern in kernel code is "save all registers, do work, restore all". Write a NASM macro SAVE_ALL that pushes all 15 general-purpose registers (excluding RSP) in a consistent order, and RESTORE_ALL that pops them in reverse order. Then write a function wrapper that uses these macros to call a function do_work(uint64_t context_ptr) with all registers saved.

Section E: Stack Alignment and Calling

Exercise 15. A function currently has this structure:

my_function:
    push rbp
    mov  rbp, rsp
    push rbx          ; saves 8 bytes → RSP % 16 == 0 after this
    push r12          ; saves 8 bytes → RSP % 16 == 8 after this
    ; ... need to call external_func() here ...

What instruction must be inserted before call external_func to ensure 16-byte alignment? Explain why.

Exercise 16. Compute the correct sub rsp, N value for each of the following local variable requirements. Assume the function starts with push rbp; mov rbp, rsp and pushes the indicated callee-saved registers:

(a) 3 int64_t locals (24 bytes), pushes RBX (8 bytes) as only callee-saved register (b) 5 int32_t locals (20 bytes), pushes RBX, R12, R13 (24 bytes) (c) 7 int64_t locals (56 bytes), pushes RBX (8 bytes)

For each, state the final RSP alignment before any nested call.

Exercise 17. Write a complete NASM program (with _start) that: 1. Calls a function hello_world which calls write (syscall 1) to print "Hello, World!\n" to stdout 2. hello_world takes no arguments and returns nothing 3. After hello_world returns, the program calls exit (syscall 60) with status 0

Include correct prologue/epilogue in hello_world. Use the Linux syscall convention (syscall number in RAX, args in RDI, RSI, RDX).

Exercise 18 (Buffer Overflow Preview). Consider this stack layout in a vulnerable function:

[rbp + 8]  = return address (placed by CALL)
[rbp + 0]  = saved RBP
[rbp - 8]  = int64_t counter (= 0)
[rbp - 72] = char buffer[64]  (64-byte buffer starting at rbp-72)

(a) What is the offset from the start of buffer to the return address? (b) If an attacker writes N bytes to buffer without bounds checking, starting at rbp-72, what is the minimum N to reach and overwrite the return address? (c) If the attacker wants to place the value 0x4141414141414141 in the return address position, exactly what bytes at what offset must they write?

Exercise 19 (Challenge: Fibonacci with Memoization). Implement int64_t fib_memo(int64_t n, int64_t *memo) in NASM, where: - n is the Fibonacci index (RDI) - memo is an array of at least n+1 int64_t values, initialized to -1 (RSI) - If memo[n] != -1, return memo[n] (memoized result) - Otherwise, compute fib_memo(n-1, memo) + fib_memo(n-2, memo), store in memo[n], return it

This requires careful management of callee-saved registers across recursive calls.

Exercise 20 (Challenge: Implementing printf("%ld\n", value) without libc). Write a NASM function print_int64(int64_t value) that: 1. Converts a signed 64-bit integer to its decimal string representation 2. Writes it to stdout using the write syscall 3. Follows the full System V ABI (can be called from C)

Hint: divide by 10 repeatedly to extract digits in reverse order, then print in reverse.