Chapter 11 Further Reading: The Stack and Function Calls

ABI Specification

System V Application Binary Interface, AMD64 Architecture Processor Supplement github.com/hjl-tools/x86-psABI/wiki/X86-psABI The definitive specification. Chapter 3 covers calling conventions, register usage, stack frame layout, and the red zone. Appendix A has the DWARF unwind table format. This is the document GCC and Clang implement; everything in this chapter derives from it.

Windows x64 Calling Convention learn.microsoft.com/en-us/cpp/build/x64-calling-convention Windows uses a different calling convention: only 4 register arguments (RCX, RDX, R8, R9), different callee-saved set, and a mandatory 32-byte "shadow space" above the return address. Code that must work on both Linux and Windows needs to abstract over this difference.

Stack Frame Internals

DWARF Debugging Standard, Section 6.4: Call Frame Information dwarfstd.org DWARF Call Frame Information (CFI) is how debuggers walk the call stack without frame pointers. Each function has entries in .eh_frame describing how to find the previous frame's registers at any given PC. Understanding this explains how backtrace in GDB works even with -fomit-frame-pointer.

"Function Call Conventions and Stack Frame Layout" — Bryan Cantrill (now Oxide Computer) YouTube lecture. Clear explanation of the System V ABI with animated stack diagrams. Covers the 16-byte alignment requirement and its historical context (alignment needed for FXSAVE before SSE was common).

Buffer Overflows and Mitigations

"Smashing The Stack For Fun And Profit" — Aleph One (Elias Levy), Phrack Magazine #49, 1996 phrack.org/issues/49/14.html The paper that defined modern stack overflow exploitation. Still readable and technically accurate for the basic technique. The stack layout diagrams and shellcode injection methodology are foundational.

"Stack Smashing Protection" — Hiroaki Etoh, IBM Research The original description of the GCC stack canary implementation (then called ProPolice, now -fstack-protector). Explains the canary placement strategy and why local variables are reordered to put arrays near the canary.

GCC Documentation: Stack Protection gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html Documents -fstack-protector, -fstack-protector-strong, -fstack-protector-all, and related options. The "strong" variant protects more function types (those taking the address of local variables, for example).

Performance

"Function Call Overhead" — Agner Fog, optimizing_assembly.pdf (agner.org) Chapter 14 covers the cost of function calls: CALL/RET, push/pop overhead, and how to minimize it (leaf functions, inlining, tail call optimization). The concrete cycle counts for function call overhead versus inline code are useful for justifying when to write assembly functions versus inlining C.

"Tail Call Optimization" — GCC wiki gcc.gnu.org/wiki/TailCalls Explains when GCC transforms return func(args) into a jmp instead of call + ret, eliminating the stack frame growth for recursive calls at the cost of losing the frame in backtraces. Relevant to the recursive factorial example: factorial(n-1) is not a tail call because we multiply after it, so TCO cannot apply.