Chapter 2: Numbers in the Machine

Everything Is a Bit Pattern

The CPU does not know what your bits mean. It knows how to manipulate bit patterns according to rules — and different instructions apply different rules to the same bit pattern. The number 0xFF stored in a byte might be the unsigned integer 255, the signed integer -1, the ASCII character ÿ, or the least significant byte of some larger value. The meaning is imposed by the instruction that operates on it, not by the bit pattern itself.

This chapter is about the rules. If you understand binary representation, two's complement, IEEE 754, and the flags register, you can predict the result of every arithmetic instruction in x86-64 assembly. If you don't, arithmetic in assembly will produce results that seem random until the bugs accumulate enough to find.

This is not optional background material. This is the machine's type system.

Bits, Bytes, and Data Widths

The fundamental unit of x86-64 memory is the byte — 8 bits. The architecture processes data in four sizes:

The NASM size suffixes appear on memory operands: BYTE [rax], WORD [rbp-2], DWORD [rbp-4], QWORD [rbp-8]. When the size is ambiguous from context, you must specify it.

The x86-64 instruction mnemonics also carry size implications: al, bl, cl, dl are byte registers; ax, bx are word registers; eax, ebx are dword registers; rax, rbx are qword registers. The instruction mov al, [rdi] loads a byte; mov rax, [rdi] loads a quadword.

Hexadecimal as Binary Shorthand

Binary is the machine's native language, but writing 64 bits in binary is unwieldy. Hexadecimal (base 16) is the solution: each hex digit corresponds to exactly 4 binary bits, making the conversion trivial.

Binary:  0000 0001 0010 0011 0100 0101 0110 0111
Hex:       0    1    2    3    4    5    6    7

Binary:  1000 1001 1010 1011 1100 1101 1110 1111
Hex:       8    9    A    B    C    D    E    F

In NASM, hex literals are written as 0x prefix (0xFF) or h suffix (0FFh — the leading 0 is required when starting with A-F to distinguish from identifiers). In GDB output and objdump output, hex is standard.

Key conversions to memorize:

Unsigned Integers: Range and Overflow

An n-bit unsigned integer represents values from 0 to 2^n − 1. The bit pattern is the value in base 2.

Unsigned arithmetic wraps around at 2^n:

mov al, 255     ; al = 0xFF = 11111111b
add al, 1       ; al = 0x00 = 00000000b  (wrapped to 0!)
                ; Carry Flag (CF) = 1    (overflow out of the top)

This wraparound is defined behavior, not an error. The processor signals it by setting the Carry Flag (CF). If you're tracking unsigned overflow, check CF after the operation.

Range check example:

; Check if rax is a valid array index (unsigned, must be < 100)
cmp  rax, 100       ; compare rax with 100 (unsigned comparison)
jae  out_of_bounds  ; jump if rax >= 100 (unsigned: JAE = Jump if Above or Equal)
; rax is in range [0, 99]

Note: jae for unsigned "above or equal"; jge for signed "greater or equal". These are different instructions operating on different flag combinations.

Two's Complement: Signed Integers

Two's complement is the system x86-64 uses to represent signed integers. It is brilliant for one reason: addition and subtraction work identically whether the operands are signed or unsigned. The hardware doesn't need to know the sign — the programmer decides the interpretation.

The Representation

For an n-bit two's complement number: - The most significant bit (MSB) has weight −2^(n−1) instead of +2^(n−1) - All other bits have their normal positive weights

For an 8-bit value b7 b6 b5 b4 b3 b2 b1 b0:

value = -128×b7 + 64×b6 + 32×b5 + 16×b4 + 8×b3 + 4×b2 + 2×b1 + 1×b0

Example: 10110100 binary

= -128×1 + 64×0 + 32×1 + 16×1 + 8×0 + 4×1 + 2×0 + 1×0
= -128 + 32 + 16 + 4
= -76

The bit pattern 0x80 (10000000) represents −128. The bit pattern 0xFF (11111111) represents −1. The bit pattern 0x7F (01111111) represents +127.

The Flip-and-Add-1 Rule (and Why It Works)

To negate a two's complement number: flip all bits, then add 1.

Why? For any bit pattern P, its bitwise complement ~P satisfies P + ~P = 0xFF...FF (all ones, which is −1 in two's complement). Therefore:

P + ~P = -1
P + ~P + 1 = 0
~P + 1 = -P

So ~P + 1 is the negative of P. This is not a trick — it's a theorem.

Example: negate 5 in 8-bit two's complement:

5  = 00000101
~5 = 11111010  (flip all bits)
+1 = 11111011  = -5  ✓

Verify: 5 + (-5) = 00000101 + 11111011 = 100000000
                                          ^-- 9th bit: Carry, discarded in 8-bit arithmetic
                                              Result: 00000000 = 0  ✓

In x86-64 assembly, the NEG instruction performs two's complement negation:

mov rax, 5
neg rax          ; rax = -5 (0xFFFFFFFFFFFFFFFB)

Signed Arithmetic Overflow vs. Carry

This is one of the most confused concepts in assembly. Two flags govern arithmetic errors:

• Carry Flag (CF): Set when an arithmetic result cannot fit in the destination as an unsigned value. Unsigned overflow.
• Overflow Flag (OF): Set when an arithmetic result cannot fit in the destination as a signed value. Signed overflow.

They are independently set and independently checked:

; Example 1: CF set, OF clear (unsigned overflow, no signed overflow)
mov al, 200     ; 200 = 0xC8 (interpreted as signed: -56)
add al, 100     ; 200 + 100 = 300 (doesn't fit in 8-bit unsigned: wraps to 44)
                ; Result: al = 44, CF = 1, OF = 0
                ; -56 + 100 = 44 (fits in signed 8-bit: no signed overflow)

; Example 2: OF set, CF clear (signed overflow, no unsigned overflow)
mov al, 100     ; 100 = 0x64 (fits as both signed and unsigned)
add al, 100     ; 100 + 100 = 200 (fits in unsigned 8-bit)
                ; Result: al = 200 = 0xC8, CF = 0, OF = 1
                ; Signed: 100 + 100 = 200, but max signed 8-bit is 127: overflow!

; Example 3: Both CF and OF set
mov al, 200     ; 0xC8 (unsigned 200, signed -56)
add al, 200     ; Unsigned: 200+200=400, doesn't fit: CF=1, result=144
                ; Signed: -56 + (-56) = -112, fits (-128 to 127): OF = 0
                ; Wait -- let me recalculate with actual hardware behavior:
                ; al = 0xC8 + 0xC8 = 0x190, lower 8 bits = 0x90 = 144
                ; Signed: 0x90 = -112 (bit 7 set). -56 + -56 = -112: fits! OF = 0

; Example 4: True both-set case
mov al, 0x80    ; -128 (signed), 128 (unsigned)
add al, 0x80    ; 0x80 + 0x80 = 0x100, lower 8 bits = 0x00
                ; CF = 1 (unsigned 128+128=256, doesn't fit in 8 bits)
                ; OF = 1 (signed -128 + -128 = -256, doesn't fit in signed 8-bit)

The rule for signed overflow: OF is set when two positive numbers add to a negative result, or two negative numbers add to a positive result. Equivalently: OF = XOR of the carry into the MSB and the carry out of the MSB.

Register Trace: Arithmetic with Flags

Note: After add eax, 1 in the last row, the full RAX is 0x0000000000000000 because writing EAX zeros the upper 32 bits.

Sign Extension

When moving a smaller signed value into a larger register, you must sign extend — fill the upper bits with copies of the sign bit.

movsx  rax, al      ; sign-extend  8-bit AL  to 64-bit RAX
movsx  rax, ax      ; sign-extend 16-bit AX  to 64-bit RAX
movsx  rax, eax     ; sign-extend 32-bit EAX to 64-bit RAX (also: movsxd)
movzx  rax, al      ; zero-extend  8-bit AL  to 64-bit RAX (unsigned extension)
movzx  rax, ax      ; zero-extend 16-bit AX  to 64-bit RAX

; Example:
mov  al, -5         ; al  = 0xFB = 11111011
movsx rax, al       ; rax = 0xFFFFFFFFFFFFFFFB = -5 in 64-bit
movzx rax, al       ; rax = 0x00000000000000FB = 251 in 64-bit (different!)

Sign extension matters when calling functions that expect 64-bit arguments but you're computing with narrower values. If you compute a loop index in eax and then use it in a memory address, the compiler uses movsx rax, eax (or lets the hardware do it implicitly if the index fits) to ensure the full 64-bit address is computed correctly.

The CDQ instruction sign-extends EAX into EDX:EAX (32-bit → 64-bit), and CQO sign-extends RAX into RDX:RAX (64-bit → 128-bit). Both are used to prepare the dividend for IDIV.

The RFLAGS Register

RFLAGS is a 64-bit register (the lower 32 bits are EFLAGS, the lower 16 bits are FLAGS). Most programs only touch a few of its bits:

Bit  Symbol  Name              Set when...
0    CF      Carry Flag        Unsigned overflow; also used by shift/rotate, ADCX
2    PF      Parity Flag       Low 8 bits of result have even number of 1s
4    AF      Adjust Flag       BCD arithmetic carry from bit 3 to bit 4
6    ZF      Zero Flag         Result is zero
7    SF      Sign Flag         Result has bit 7 (or MSB) set (negative if signed)
8    TF      Trap Flag         (System) single-step mode; triggers debug exception per instruction
9    IF      Interrupt Flag    (System) enables hardware interrupts; set by STI, cleared by CLI
10   DF      Direction Flag    String ops increment (DF=0) or decrement (DF=1) SI/DI
11   OF      Overflow Flag     Signed overflow
12-13 IOPL  I/O Privilege Level (System) minimum CPL for I/O instructions
14   NT      Nested Task       (System) set if current task was called with CALL
16   RF      Resume Flag       (System) suppresses debug fault for one instruction
17   VM      Virtual-8086 Mode (System) sets V86 mode
18   AC      Alignment Check   Triggers alignment exception if set and CPL=3
21   ID      ID Flag           Can be toggled if CPUID is supported

For user-space assembly, the flags you care about are CF, ZF, SF, OF, DF, and PF (occasionally).

Reading RFLAGS

You cannot mov rax, rflags — RFLAGS is not a general-purpose register. To read it:

pushfq          ; push RFLAGS onto the stack (8 bytes)
pop rax         ; rax now contains RFLAGS

To modify specific bits:

pushfq
pop  rax
or   rax, (1 << 8)   ; set bit 8 (TF — trap flag)
push rax
popfq                ; restore RFLAGS with modification

How Flags Enable Conditional Jumps

The conditional jump instructions (jz, jne, jl, jge, etc.) test one or more flags:

The distinction between jl (signed less-than) and jb (unsigned below) is critical. Using the wrong one on an unsigned comparison produces silently wrong results.

Fixed-Width Arithmetic: The Hardware Reality

Every arithmetic instruction in x86-64 operates on a fixed-width operand: 8, 16, 32, or 64 bits. The hardware computes the mathematical result and then truncates it to the operand width. The flags record whether the truncation lost information.

This is different from how most programmers think about arithmetic. In mathematics, 127 + 1 = 128. In 8-bit two's complement arithmetic, 127 + 1 = -128 (with OF=1, indicating signed overflow). The hardware is correct within the rules of fixed-width arithmetic; the programmer's interpretation determines whether the result is meaningful.

; 64-bit addition: no overflow for reasonable values
mov  rax, 9000000000000000000   ; fits in 64-bit signed
add  rax, 9000000000000000000   ; result: -446744073709551616 (overflow! OF=1)

; But unsigned? 9e18 + 9e18 = 1.8e19, max 64-bit unsigned = 1.84e19...
; Actually this still fits unsigned.

; Maximum 64-bit signed overflow:
mov rax, 0x7FFFFFFFFFFFFFFF    ; INT64_MAX
add rax, 1                     ; rax = 0x8000000000000000 = INT64_MIN, OF=1

The INTO instruction triggers an overflow exception when OF=1, but it doesn't exist in 64-bit mode. Instead, check OF with jo after arithmetic.

IEEE 754 Floating Point

Integer arithmetic is exact (within its range). Floating-point arithmetic is approximate, with rules so subtle that entire books have been written about them.

The Representation

IEEE 754 single precision (32-bit float):

31  30           23 22                    0
┌─┬─────────────┬──────────────────────────┐
│S│   Exponent   │         Mantissa         │
│1│    8 bits    │         23 bits          │
└─┴─────────────┴──────────────────────────┘

Value = (-1)^S × 2^(E-127) × (1 + M/2^23)

IEEE 754 double precision (64-bit double):

63  62                  52 51                               0
┌─┬────────────────────┬─────────────────────────────────────┐
│S│      Exponent       │              Mantissa               │
│1│      11 bits        │              52 bits                │
└─┴────────────────────┴─────────────────────────────────────┘

Value = (-1)^S × 2^(E-1023) × (1 + M/2^52)

The value is: sign × 2^(exponent - bias) × (1.mantissa in binary).

The implicit leading 1 is the key: the mantissa always starts with 1. (which is not stored, saving one bit). This is called the "implicit leading bit" or "hidden bit."

Why 0.1 + 0.2 ≠ 0.3

The decimal number 0.1 cannot be represented exactly in binary. Here's why:

0.1 in binary = 0.0001100110011001100110011... (repeating)

Just like 1/3 = 0.33333... repeats in decimal, 1/10 = 0.0001100110011... repeats in binary. The 23-bit mantissa of a float can only store a finite number of these bits.

The closest float to 0.1 is:

0.1 exactly: 0.1000000000000000055511151231257827021181583404541015625
Stored float: the nearest representable value

When you add two approximate values and compare to a third approximate value, the accumulated rounding errors mean the comparison fails.

In assembly terms:

; XMM registers hold floating-point values
; ADDSS = Add Scalar Single-precision
; SUBSS = Subtract Scalar Single-precision
; COMISS = Compare ordered Scalar Single-precision

section .data
    val1    dd 0.1       ; NASM converts 0.1 to the nearest float
    val2    dd 0.2
    val3    dd 0.3
    ; Note: val1 + val2 ≠ val3 due to rounding

section .text
    movss   xmm0, [val1]
    addss   xmm0, [val2]    ; xmm0 = 0.1 + 0.2 (approximately 0.30000000000000004)
    movss   xmm1, [val3]
    comiss  xmm0, xmm1      ; compare: xmm0 == xmm1?
    je      they_are_equal  ; this jump is NOT taken
    ; We end up here: 0.1 + 0.2 ≠ 0.3 in float!

Special Values

IEEE 754 reserves special bit patterns for non-numeric states:

NaN has a critical property: NaN ≠ NaN. A comparison of NaN with any value (including itself) returns "unordered" (false for ordered comparisons). This is how you detect NaN: x != x is true only if x is NaN.

Denormal (subnormal) numbers cause significant performance problems. When a floating-point result would underflow to zero with a normal representation, the CPU produces a denormal value instead — a number with no implicit leading 1 bit. Computing with denormals is handled in software on many CPUs and can be 10-100x slower than normal floating-point. The MXCSR register has bits to flush denormals to zero (FTZ) and treat denormals as zero (DAZ) for performance-critical code.

Exact Representation

Which numbers CAN be exactly represented?

All integers up to 2^53 can be exactly represented as double-precision floats (the mantissa has 52 explicit bits plus the implicit 1). Beyond that, consecutive integers begin to be separated by gaps larger than 1.

All fractions of the form p/2^n can be exactly represented if the result has at most 52 significant binary digits. So 0.5, 0.25, 0.125, 0.0625 are all exact. But 0.1, 0.2, 0.3, 0.4, 0.6, 0.7, 0.8, 0.9 are not.

This is fundamental: floating-point is exact only for powers-of-2 fractions and small integers.

Endianness: Little-Endian x86

When a multi-byte value is stored in memory, the bytes must be arranged in some order. x86-64 is little-endian: the least significant byte is stored at the lowest address.

; Store the value 0x01020304 at address [buffer]
section .data
    buffer: dd 0x01020304    ; stored in memory as: 04 03 02 01 (little-endian!)

; In memory at buffer's address:
; [buffer+0] = 0x04  (least significant byte)
; [buffer+1] = 0x03
; [buffer+2] = 0x02
; [buffer+3] = 0x01  (most significant byte)

This is counterintuitive when you first encounter it — the bytes are "backwards" compared to how you write the number. But it means that the least significant byte is always at the base address, which simplifies some operations.

Network byte order is big-endian. Internet protocols (TCP/IP, DNS, HTTP/2 binary framing) use big-endian byte order. When reading network data on an x86-64 machine, you must byte-swap multi-byte fields.

The BSWAP instruction reverses the byte order of a 32-bit or 64-bit register:

; Convert a 32-bit value from network order (big-endian) to host order (little-endian)
mov   eax, [network_data]  ; load big-endian value
bswap eax                   ; byte-reverse: 0x01020304 → 0x04030201
; eax now contains the host-order value

For 16-bit values, BSWAP is not directly available; instead use XCHG al, ah.

Why little-endian? Little-endian has one practical advantage: you can read a multi-byte integer at its base address regardless of its size. The byte at address A is always the least significant byte, whether you're reading a byte, word, dword, or qword. This simplifies certain protocols and data structures. Big-endian has the advantage that the hexdump of a binary file "reads" correctly left-to-right, which some argue is more natural for humans.

Neither is universally better. x86 was little-endian from its earliest days; ARM and MIPS can operate in either mode; network protocols historically chose big-endian. Live with it.

Putting It Together: A Complete Arithmetic Example

Here is a complete program that exercises the concepts from this chapter: it reads two 64-bit integers, adds them, checks for overflow, and prints the result.

; arith.asm — demonstrates integer arithmetic, flags, and overflow detection
; nasm -f elf64 arith.asm -o arith.o && ld arith.o -o arith

section .data
    overflow_msg    db "Overflow detected!", 10
    overflow_len    equ $ - overflow_msg
    result_msg      db "Result: "
    result_len      equ $ - result_msg

section .bss
    result_buf      resb 20    ; buffer for number-to-string conversion

section .text
    global _start

; Helper: convert RAX to decimal string in result_buf
; Returns: RCX = length
int_to_string:
    mov  rdi, result_buf
    add  rdi, 19           ; point to end of buffer
    mov  BYTE [rdi], 0     ; null terminator
    mov  rbx, 10           ; divisor
    xor  rcx, rcx          ; character count
.loop:
    xor  rdx, rdx          ; zero RDX before division (required!)
    div  rbx               ; rax = rax/10, rdx = rax%10
    add  dl, '0'           ; convert digit to ASCII
    dec  rdi
    mov  [rdi], dl
    inc  rcx
    test rax, rax          ; is quotient zero?
    jnz  .loop             ; if not, continue
    ; rdi points to start of string, rcx = length
    mov  rsi, rdi          ; for sys_write
    ret

_start:
    ; Compute 0x7FFFFFFFFFFFFFFF + 1 (max signed 64-bit + 1 = signed overflow)
    mov  rax, 0x7FFFFFFFFFFFFFFF
    add  rax, 1            ; OF=1 (signed overflow), SF=1, ZF=0, CF=0

    ; Check for signed overflow
    jo   .overflow         ; jump if OF=1

    ; No overflow: print the result
    push rax               ; save result
    mov  rax, 1            ; sys_write
    mov  rdi, 1            ; stdout
    mov  rsi, result_msg
    mov  rdx, result_len
    syscall
    pop  rax               ; restore result

    call int_to_string
    mov  rax, 1
    mov  rdi, 1
    ; rsi already set by int_to_string
    mov  rdx, rcx
    syscall
    jmp  .exit

.overflow:
    mov  rax, 1            ; sys_write
    mov  rdi, 1            ; stdout
    mov  rsi, overflow_msg
    mov  rdx, overflow_len
    syscall

.exit:
    mov  rax, 60           ; sys_exit
    xor  rdi, rdi          ; exit code 0
    syscall

Register trace for the key arithmetic section:

The result 0x8000000000000000 is −9,223,372,036,854,775,808 interpreted as signed (INT64_MIN), which is correct: we overflowed from the maximum positive to the minimum negative value. The OF flag tells us this happened. The jo instruction detects it.

Summary

The machine's number system has rules, and those rules are precise:

• Unsigned arithmetic wraps with CF; signed arithmetic overflows with OF
• Two's complement is mathematically principled, not just a convention
• Sign extension is mandatory when widening signed values; zero extension for unsigned
• Floating-point is approximate; only powers-of-2 fractions and small integers are exact
• x86-64 is little-endian; network protocols are big-endian; bswap converts between them
• RFLAGS records arithmetic outcomes; conditional jumps test flags

Every assembly bug involving arithmetic can be diagnosed using these rules. When a value is wrong, ask: which instruction set the wrong value? What did the flags say? Was it signed or unsigned arithmetic? Was there an overflow that wasn't checked? The rules answer the question.

🔄 Check Your Understanding: What is the value of rax after these two instructions? nasm mov rax, 0x0000000100000000 add eax, eax

Answer

The instruction add eax, eax adds the lower 32 bits of RAX to themselves. EAX before the add is 0x00000000 (the lower 32 bits of 0x0000000100000000). 0 + 0 = 0. The result written to EAX is 0, which also zeros the upper 32 bits. So rax = 0x0000000000000000.

This is a classic gotcha: the upper 32 bits of RAX (which contained 0x00000001) are destroyed by the 32-bit write. If the intent was to double the 64-bit value, the instruction should have been add rax, rax.