Why Assembly ?

Most people have conflicting opinions about learning Assembly. They often ask: “Why learn Assembly when modern compilers can generate much better optimized code?”

It’s a fair question. Compilers today are incredibly smart. So why bother?

The simple answer is this, Assembly helps you see what the computer is actually doing. Compilers write the code, but Assembly reveals the truth behind it.

It makes debugging easier because you can understand the real machine instructions when optimized code looks confusing.

In cybersecurity , it’s even more valuable. Reverse engineering malware, analyzing binaries, and building exploits all depend on reading and writing Assembly.

Even if you rarely write it, learning Assembly removes the mystery of how programs really work.

The most important part? You mainly need to learn how to read Assembly, not how to write it perfectly. This skill alone makes debugging easier and unlocks reverse engineering.

What is NASM?

NASM, or the Netwide Assembler, is an 80x86 and x86-64 assembler designed for portability. It works by translating assembly language source files into machine-readable object files, which are then linked into executable programs.

NASM is chosen for educational purposes because it supports a wide range of output formats, allowing the same source code to be compiled for Windows, Linux, or macOS with minimal changes.

CPU Basics

Before we begin, we must learn the basics of the CPU. We will cover essential concepts such as: What are registers? What is the stack? What are flags? and much more.

Registers

A register is small, high-speed storage located directly inside the CPU. Unlike your computer RAM which are external to the CPU. Registers built into the hardware of the CPU itself. In NASM we can interact with these registers by name such as eax, ebx, rcx to perform every operation in the program.

Best understood through example

At the first line, we place the number 10 into the EAX register. On the second line, we place the number 20 into the EBX register. Finally, the third line adds the value in EBX to the value already in EAX. Because 10 plus 20 is 30, the EAX register now holds the final result of 30.

The Register size

When we talk about the size of a register we mean how much data it can hold at one time.

There have been a few generations of sizes:

• 8-bit (e.g., Intel 8080): Held 1 byte (e.g., 0xFF).
• 16-bit (e.g., Intel 8086): Held 2 bytes (e.g., 0xFFFF).
• 32-bit (e.g., Intel 80386): Held 4 bytes (e.g., 0xFFFFFFFF).
• 64-bit (Modern x86-64): Holds 8 bytes (e.g., 0xFFFFFFFFFFFFFFFF).

The "Backward Compatibility" Trick:
Modern 64-bit CPUs are backward compatible. This means a 64-bit register (like RAX) can be partially used as a 32-bit (EAX), 16-bit (AX), or 8-bit (AL / AH) register. This is crucial for running older software.

For example, as we can see in the above image, we have the RAX register which is 64-bit (8 bytes). But here's the trick it's divided into smaller pieces, all sitting on the right side:

• RAX (64-bit) → The full register.
• EAX (32-bit) → The right half of RAX.
• AX (16-bit) → The right quarter of EAX.
• AL (8-bit) → The far-right byte of AX.
• AH (8-bit) → The byte just to the left of AL.

It is Important to note: If you modify EAX, the upper 32 bits of RAX get zeroed out. But if you modify AL, only that last byte changes—the rest of RAX stays untouched! So think of it like a variable.

The General-Purpose Registers (GPRs)

There are some of these registers you can use them for almost everything but historically they each have "favorite" job.

• Low bits = the least significant bits (rightmost bits, small values)
• High bits = the most significant bits (leftmost bits, large values)

Here is the master lists:

The Instruction Pointer (RIP / EIP)

RIP (or EIP in 32-bit) holds the memory address of the next instruction to be executed. The CPU automatically updates it after each instruction. You never write to it directly instead, you modify it using jump, call, and ret instructions (which we'll cover later).

The Stack Pointers (RBP and RSP)

RSP (Stack Pointer) points to the top of the stack. When you push data, RSP goes down; when you pop, it goes up. It automatically adjusts with push and pop instructions. We'll discuss the stack in detail later.

RBP (Base Pointer) acts as a fixed reference for the current function's stack frame. It helps you access local variables and parameters at consistent offsets. You'll often see it used with instructions like push rbp and mov rbp, rsp at the start of a function. We will dive deeper into the stack too.

Flags (RFLAGS / EFLAGS)

The flags register isn't like the other registers it does not hold single number. Instead it's a collection of boolean values called flags. these flags are automatically set or cleared by arithmetic and comparison operations let's see the most important ones you'll encounter.

CF (Carry Flag)

CF = 1 if an unsigned addition exceeds the maximum value that can fit in the register.

For example, AL (8-bits) can hold a maximum unsigned value of 255:

Here we place 100 in AL. On the second line, we add 200 to AL, which becomes 300. Since AL cannot hold 300, the CF flag will be set to 1.

PF (Parity Flag)

The Parity Flag is simple it just counts the number of 1s in the lowest 8 bits of the result.

• PF = 1 if there's an even number of 1s.
• PF = 0 if there's an odd number of 1s.

Important: PF only looks at the lowest 8 bits the rest of the register is ignored!

First we set AL to have 5 value which if we convert to binary will look like this 00000101 as we can see we have 2 bits are set to 1 which are even so the PF flag is set to 1.

OF (Overflow Flag)

The Overflow Flag is used for signed arithmetic. It's set to 1 when a signed operation produces a result too big or too small to fit in the register.

Think of it like this: signed numbers have a range for 8-bit, that's -128 to +127. If you go beyond that, OF tells you something went wrong.

Example: Overflow!

Here we place 127 in AL, the maximum value for 8-bit signed integers. Adding 1 should give 128, but that's outside the signed range (-128 to +127). So AL wraps to -128, and the CPU sets OF = 1 to flag the signed overflow. CF = 0 since 128 fits in unsigned range, SF = 1 because the result is negative, and ZF = 0 since it's not zero.

ZF (Zero Flag)

ZF = 1 if the result of an operation is zero. ZF = 0 if the result is anything else.

Here we place 5 in AL. On the next line, we subtract 5 from AL, which gives us 0. Since the result is zero, the CPU sets ZF = 1.

SF (Sign Flag)

SF = 1 if the result is negative. SF = 0 if the result is positive.

Here we place 10 in AL. On the next line, we subtract 20 from AL, which gives us -10. Since the result is negative, the CPU sets SF = 1.

Stack

Stack is a region of memory that works like LIFO (Last in, First out). The CPU use the stack for storing return address when function is called, passing arguments to function, and storing local variables.

The Stack Grows Downward

Here's the tricky part: the stack grows downward in memory. This means when you add something to the stack, the stack pointer (RSP) moves to a lower memory address.

RSP points to the top of the stack. So when we push data, RSP decreases (moves down). When we pop data, RSP increases (moves up).

Example:

Let's say RSP starts at 0x1000

1. First line: We do push rax. RSP starts at 0x1000, so it updates to 0x1000 - 8 = 0x0FF8. The value of RAX is stored at address 0x0FF8.
2. Second line: We do push rbx. RSP is at 0x0FF8, so it updates to 0x0FF8 - 8 = 0x0FF0. The value of RBX is stored at address 0x0FF0.
3. Third line: We do pop rcx. RSP is at 0x0FF0, so it loads the value from 0x0FF0 into RCX, then RSP updates to 0x0FF0 + 8 = 0x0FF8.
4. Fourth line: We do pop rdx. RSP is at 0x0FF8, so it loads the value from 0x0FF8 into RDX, then RSP updates to 0x0FF8 + 8 = 0x1000 (back to where we started).

Function Call Mechanics

This is where the magic happens:

call instruction, it does two things:

1. Pushes the return address (the address of the next instruction) onto the stack
2. Jumps to the function

ret instruction, it does two things:

1. Pops the return address off the stack
2. Jumps to that address (puts it into RIP)

Example CALL and return

When first start the program, we start at 0x1000 address it will execute what ever and go to the next line which is in the RIP

Now we are in the line 0x2000 and will execute what ever in this address then go to next line base on RIP, notice how the RIP get automatically update to the next line

Now we are in line 0x3000 and in this line we hit `CALL` instruction which will do two things:

1. push the next instruction to the stack, so the CPU will push the `RIP` value to stack
2. jump to that function.

Now we are in my_function in line 0xA000 notice the RIP changed as well
By the CPU. we execute whatever in this address and go to next.

We hit the line 0xB000 we execute what ever and go to next

Now we are in line `0xC000` which have Ret instruction, it will do the following:

1. it will pop the value from the stack in another word it will remove the value 0x4000 from the stack.
2. change the current RIP with the value from the stack so RIP will change from
   0xd000 to 0x4000
3. jump back to the RIP address

After we pop the return value from the stack the execution flow return to normal.

Function Prologue and Epilogue

When a function is called, it needs to set up its own stack frame. This is done through the prologue (setup) and epilogue (cleanup).

Prologue (Function Setup)

At the start of a function, we save the current base pointer (RBP) and set it to the current stack pointer (RSP). This creates a fixed reference point for accessing local variables and parameters.

What's happening:

1. push rbp – Save the caller's RBP on the stack (so we can restore it later)
2. mov rbp, rsp – Set RBP to point to the current stack top (our new frame base)
3. sub rsp, 32 – Reserve space for local variables (grow stack down)

Epilogue (Function Cleanup)

Before returning, we restore the stack and base pointer to their original state.

What's happening:

1. mov rsp, rbp – Set RSP back to where RBP is (effectively deallocating local variables)
2. pop rbp – Restore the caller's RBP from the stack
3. ret – Pop return address and jump back

We need the prologue and epilogue

RSP moves constantly as we push and pop data, making it hard to access local variables. RBP acts as a fixed reference point that stays constant during the function, so we can always access variables at predictable offsets like [rbp-8]. We also save the caller's RBP at the start and restore it at the end so we don't corrupt the caller's stack frame. This creates a chain that allows nested functions to return properly and helps debuggers trace the call stack.

Instructions You Must Know

There are tons of instructions you might encounter, but I'll explain the most common ones you'll see when disassembling programs or writing simple assembly code. For a full list, check out this.

Mov instruction

mov is the basic data-movement instruction. It copies data from a source to a destination. It does not do math, it does not change the source.

Syntax

mov destination,source

1. Destination cannot be memory if source is memory
   mov [rbx], [rcx] is invalid
2. Size of source and destination must match mov eax, bl ; invalid: 32-bit ← 8-bit

Examples

In last line, take the value stored at memory address 0x1000 and copy it into RAX. like dereference pointer in c or c++

Add/sub instruction

ADD and SUB are the basic arithmetic instructions in x86. I’ll explain with examples and comments for clarity.

Syntax

add destination, source

this mean destination = destination + source

• destination can be a register or memory.
• source can be a register, memory, or immediate (constant).
• The CPU updates flags:
  • CF = carry (unsigned overflow)
  • OF = overflow (signed overflow)
  • ZF = zero flag
  • SF = sign flag
  • PF = parity

The same rules for sub

LEA Instruction

LEA (Load Effective Address) calculates an address and stores it in a register. It does NOT access memory it just does the math.
Syntax

lea destination, source

This means destination = address calculated from source.

• Destination must be a register.
• Source is a memory operand (with brackets).
• does not effect flags.

Examples:

mov will copy the actual value from memory or register to the destination. lea just calculates the address it doesn't access memory at all.

XOR Instruction

XOR performs a bitwise exclusive OR operation. It's very important in shellcode because it can zero registers without creating null bytes.

Syntax

xor destination, source

This means destination = destination ^ source (bitwise XOR).

• Destination can be a register or memory.
• Source can be a register, memory, or immediate.

Flags affected:

• ZF = 1 if result is zero
• SF = 1 if result is negative
• PF = 1 if even number of 1s in low byte
• CF = 0 (always cleared)
• OF = 0 (always cleared)

Examples

TEST Instruction

TEST performs a bitwise AND between two operands but does NOT store the result. It only sets flags.

Syntax

test operand1, operand2

This means operand1 & operand2 (bitwise AND) but the result is discarded.

Flags affected:

• ZF = 1 if result is zero
• SF = 1 if result is negative
• PF = 1 if even number of 1s in low byte
• CF = 0 (always cleared)
• OF = 0 (always cleared)

Examples:

CMP Instruction

CMP compares two values by subtracting them behind the scenes. It sets flags but doesn't store the result.

Syntax

cmp operand1, operand2

This means operand1 - operand2 (subtraction) but the result is discarded. Operands are unchanged.

• Both operands must be the same size.
• Can compare register, memory, or immediate.

Flags affected:

• ZF = 1 if operands are equal
• SF = 1 if result is negative (operand1 < operand2 signed)
• CF = 1 if unsigned borrow (operand1 < operand2 unsigned)
• OF = 1 if signed overflow
• PF = Parity flag

Examples:

JMP Instruction

JMP is the unconditional jump instruction. It transfers control to another location in the program.

Syntax

jmp destination

This means RIP = destination (jump to that address).

• Destination can be a label or memory address.

Flags affected: None.

Examples:

Conditional Jumps (JCC)

Conditional jumps check flags (set by CMP or TEST) and jump if the condition is true.

Syntax

Common JCC instructions:

x64 Calling Convention

When writing functions in x64 assembly, you need to follow specific rules so your code works correctly with other functions (and with C/C++ code). This is called the calling convention.

A calling convention is a set of rules that defines:

• Where arguments are passed (registers or stack)
• Who cleans up the stack (caller or callee)
• Which registers must be preserved

Functions arguments

Linux (System V AMD64 ABI)

When you call a function in Linux x86-64, arguments are passed using specific registers. The first argument goes into RDI, the second into RSI, the third into RDX, the fourth into RCX, the fifth into R8, and the sixth into R9. Any additional arguments beyond the sixth are passed on the stack in right-to-left order. This calling convention is designed to minimize memory access and keep most function parameters in fast CPU registers.

Windows (Microsoft x64 ABI)

On Windows x86-64, the calling convention is different. The first argument is placed in RCX, the second in RDX, the third in R8, and the fourth in R9. If there are more arguments, they are passed on the stack in right-to-left order. The caller must always reserve 32 bytes of shadow space on the stack before making a function call, even if fewer arguments are used. This space is used by the callee for register spilling and maintains a consistent calling environment.

Example:

Before call function you must put arguments in the correct register before calling. for both windows and linux.

Return Value (Callee's Job)

the function returns a value in RAX for example:

When the caller executes call my_function, the function runs and places the value 42 in RAX. After ret returns control to the caller, the caller can access the result in RAX.

Note For system calls we put the syscall number in RAX register we will see it later.

Preserved vs Scratch Registers

This is the most important part! Some registers must be preserved, others can be freely modified.

Preserved Registers (Callee Save) , RBP, RBX and R12-R15. If you modify these registers you MUST save them at the start and restore them before returning from the function.

Scratch Registers (Caller Saved): RAX,RCX,RDX,RSI,RDI,R8-11,RSP. you can modify these as you want without saving them. the caller knows they may by changed.

Stack Alignment

The stack must be 16-byte aligned before a call instruction. This means RSP must be a multiple of 16. Why? Some SSE/AVX instructions require 16-byte alignment. Before you call a function, make sure that is true. just make sure Before calling a function, ensure you've done an EVEN number of pushes (or equivalent sub rsp, 8n).

Syscalls (Linux focus)

Syscalls are how your program communicates with the operating system kernel. When you need to read a file, write to the console, or exit your program, you use a syscall.

Before a syscall executes, place:

• RAX – The syscall number (which operation you want)
• RDI – First argument
• RSI – Second argument
• RDX – Third argument
• R9 – Fourth argument (rarely used)

Then execute syscall instruction. The kernel runs the operation and returns the result in the RCX.

Common Syscall Numbers (x86-64 Linux)

• 0 – read (read from file descriptor)
• 1 – write (write to file descriptor)
• 2 – open (open a file)
• 3 – close (close a file descriptor)
• 60 – exit (exit the program)
• 9 – mmap (map memory)
• 57 – fork (create new process)

Let's see an example:

First, the program places 1 in RAX (write syscall), 1 in RDI (stdout), the address of msg in RSI, and the length in RDX. When syscall executes, the kernel writes "Hello, World!" to the console. Then the program places 60 in RAX (exit syscall) and 0 in RDI (success code). The second syscall terminates the program.

NASM Sections

In NASM assembly, the program is divided into sections. Each section tells the assembler and linker where to place different parts of your code and data in the final executable.

.data Section - Initialized Data

This section used for Initialized data have read/write permissions.

Directives in .data:

• db – Define Byte (1 byte)
• dw – Define Word (2 bytes)
• dd – Define Double Word (4 bytes)
• dq – Define Quad Word (8 bytes)
• dt – Define Ten bytes (10 bytes)
• equ – Define a constant (no memory allocated)

.bss Section – Uninitialized Data

This section reserves space for variables that will be set at runtime. The linker initializes this section to zeros. have read/write permissions

Usage: Buffers, input storage, temporary data.

• resb – Reserve Byte (1 byte)
• resw – Reserve Word (2 bytes)
• resd – Reserve Double Word (4 bytes)
• resq – Reserve Quad Word (8 bytes)
• rest – Reserve Ten bytes (10 bytes)

.text Section – Executable Code

This section contains the actual machine code instructions. It's marked as read-only and executable.

Usage: Your program code.

.rodata Section – Read-Only Data

The .rodata section stores read-only data—constants that your program can read but never modify.

Usage: Strings, constant values, lookup tables, format strings.

How to Compile and Run NASM Code (Linux)

To actually run the assembly code examples, you need to compile them. Here's how to do it on Linux.

Step 1: Install Required Tools

First, make sure you have NASM (the assembler) and ld (the linker) installed:

sudo apt install nasm binutil this install both tools you need.

Step 2: Assemble the Code

The assembler converts your assembly code into machine code (object file):

nasm -f elf64 add.asm -o add.o the -f elf64 output format for 64-bit linux. and the add.asm source file, and -o add.o output object file.

Step 3: Link the Code

If you have a standalone program like Example 5 (syscall), link it:

ld add.o -o add This creates an executable file called add and you can just run it!

Code examples

Add Two number and return result

Compare Two Numbers and Return Larger

Count from 1 to N (Loop)

Save and Restore Preserved Registers

Simple Syscall - Print to Screen

What next?

This was a brief explanation of how assembly works and the basics you need to know. If you're interested in learning more and diving deeper into assembly programming, here are some great resources, the best one is the art of assembly book, you have also Felix Cloutier's x86 Reference. Good luck!