Buffer Overflow and Arbitrary Code Execution: Causes & Defenses
Arbitrary Code Execution (ACE)
Arbitrary Code Execution (ACE): A vulnerability where an attacker supplies input to a program such that the CPU interprets that input as executable instructions and runs it. The CPU cannot distinguish between legitimate program code and malicious injected code. Once ACE happens, the attacker controls instruction flow; security boundaries are effectively gone and the system executes whatever the attacker wants.
Buffer Overflow (BO)
BO (Buffer Overflow): All buffer overflows (BO) can be a method of achieving ACE. A buffer overflow occurs when a program writes more data to a buffer than it was allocated to hold, causing adjacent memory to be overwritten. This commonly happens because languages like C do not perform bounds checking and trust the programmer completely; the CPU blindly follows memory addresses.
Adjacent memory may contain:
- Function return address
- Saved registers
- Other control data which can redirect execution
A buffer overflow is dangerous not because data is overwritten, but because control data is overwritten.
Privilege Escalation
Privilege escalation is when an attacker starts execution with low privileges and ends with high privileges (root/admin). Root access grants full system control: read/write files, install malware, and disable defenses.
Low-Level Exploitation and Stack Focus
BO exploits use low-level control of compiled languages to place executable code directly into memory segments, bypassing normal operating system checks. The core attack often focuses on the stack frame.
Historical Example: Morris Worm (1988)
Morris Worm (1988): One of the first widespread buffer overflow attacks. It spread automatically via networked services, shut down early internet hosts, and proved that small memory bugs could cause global failure.
Process and Memory
Process is a running instance of a program. Each process has its own virtual address space, its own stack and heap, a Process ID (PID), and its own execution context (registers, instruction pointer).
The OS loads the program into memory and the program’s stack is created. The instruction pointer (IP) moves through instructions sequentially. When functions are called, the return address is pushed onto the stack and execution jumps to the function; when the function ends the return address is popped and execution continues at that address. This return mechanism is what buffer overflows abuse.
The fork() system call creates a child process by copying the parent’s memory. The xeyes demo illustrated how a child process launched from a terminal depends on the parent for control flow: when the parent process finishes or the terminal closes, the child can be affected. Process finish and return via the return address; closing the terminal can kill the child process.
How Stack Buffer Overflows Work
The stack is a memory region used by a process to store local variables, function parameters, saved frame pointers, and the return address. When a function returns, the CPU uses the return address to determine where execution should continue. If a buffer overflow overwrites this return address, execution can be redirected.
Shellcode and NOP Sleds
Shellcode is attacker-supplied machine code injected into memory during an overflow. Although it often opens a shell, it can perform any operation the attacker desires. Shellcode is architecture- and operating system-specific, typically represented in hexadecimal or binary form as an executable payload.
NOP sled is a sequence of no-operation instructions placed before shellcode. Its purpose is to increase exploit reliability by allowing the CPU to “slide” into the shellcode even if the jump address is not precise. A NOP sled increases exploit reliability by giving a larger target to land execution into before reaching the payload.
Defenses Against Buffer Overflows
Defenses are divided into compile-time and run-time protections. Each defense mitigates risk but does not completely eliminate buffer overflow vulnerabilities.
Compile-time Defenses
- Use modern programming languages with bounds checking.
- Adopt safe coding practices and safe libraries.
- Stack canaries: place known values near the return address and check them before function return; terminate the program if they are modified.
Run-time Defenses
- Address Space Layout Randomization (ASLR) to randomize memory locations.
- Guard pages that detect illegal memory access.
- Non-executable stack regions (NX bit) to prevent execution of injected shellcode on the stack.
Heap Overflow
Heap overflow is similar to a stack overflow, but the attack uses the heap memory segment to execute shellcode or corrupt control structures. Heap attacks often manipulate allocator metadata or function pointers stored on the heap.
Ocarina of Time (OOT) and SRM
OOT (Ocarina of Time): A real-world example of ACE without modifying hardware or software. The exploit relied on memory behavior rather than a gameplay logic bug. It demonstrates that CPU execution depends only on memory contents.
Stale Reference Manipulation (SRM)
Stale Reference Manipulation (SRM) occurs when a program frees an object from memory but continues to use a pointer that references it. This creates a dangling pointer, allowing later memory allocations to occupy the same location while still being accessed through the old reference.
In OOT, manipulating the actor heap caused freed rock memory to overlap with wonder item code, allowing partial instruction overwrites that redirected execution to controller input memory. TASBot then sent controller inputs that were executed as machine code, achieving full ACE and proving that buffer overflows are not required for exploitation and that the CPU blindly executes memory.
Why Buffer Overflows Still Exist
BO vulnerabilities still exist because modern operating systems and many critical systems rely heavily on C and C-derived languages, which prioritize performance and hardware control over memory safety. These languages do not enforce bounds checking, placing responsibility entirely on the programmer.
Additional reasons include:
- Large amounts of legacy code written decades ago make rewriting expensive and risky.
- Human factors: many programmers lack training in secure coding practices.
- Technical and economic constraints prevent the widespread elimination of buffer overflow vulnerabilities.
Key Takeaways
- Arbitrary Code Execution occurs when memory contents are interpreted as instructions.
- Buffer overflows can overwrite control data like return addresses and saved registers to redirect execution.
- Shellcode and NOP sleds are common exploitation techniques; defenses like ASLR, NX, and stack canaries reduce risk.
- Real-world examples (Morris Worm, OOT/SRM) show the practical impact of memory vulnerabilities.
- Memory safety requires both safe languages and disciplined secure coding, but legacy systems and economic factors maintain exposure.