How CPU Caches Work
Modern CPUs do not get their speed from clocks alone. They get it from a hierarchy of tiny, fast memory buffers that keep the core from waiting on much slower DRAM. Understanding L1, L2, and L3 caches, cache lines, and hit rates explains why some code is fast and some is slow.
A modern CPU executes hundreds of millions of instructions every second, but its main memory, the DRAM in your sticks of RAM, is far too slow to keep up. A single DRAM read takes roughly 100 nanoseconds or more, which means the core would sit idle for tens of cycles every time it needs a value that is not already handy. The architecture solution is a hierarchy of small, fast memory layers that live inside or next to the chip, each larger and slower than the one above it: L1 cache, L2 cache, and L3 cache, sitting on top of main memory.
The key insight is locality. Programs do not touch memory randomly. Temporal locality means a value that was just used is likely to be used again in the next few instructions. Spatial locality means memory locations near a just-used location are likely to be used soon, which is why code executes almost sequentially and data structures are scanned in order. Caches exploit both: they keep a copy of the most recently and most neighborhood-relevant data close to the core, so the vast majority of memory accesses hit a fast layer instead of reaching DRAM.
Caches do not copy bytes one at a time. They move data in blocks called cache lines, typically 64 bytes on x86 hardware. When the core needs one value that is not in the cache, the memory subsystem fetches the entire line containing that value and stores it in the cache. If the program then reads a neighboring value, it is already there for free. This is spatial locality being cashed in, and it is why reading a struct field to field is fast while reading scattered, stride-heavy data is not.
When an application feels slow, the bottleneck is often not the CPU but the memory hierarchy. Tight loops over contiguous arrays run near full speed; loops that hop through pointers scattered across DRAM miss the cache on nearly every access and stall waiting for memory. Profilers that report cache miss rates, or metrics like L3 miss latency, point directly at this problem.
Each cache level has a distinct role. L1 is the smallest and fastest, typically 32 to 64 kilobytes per core, and is so close to the execution units that it can serve data in one or a few cycles. Many cores split L1 into a data cache and an instruction cache. L2 is larger, often several hundred kilobytes to a couple of megabytes per core, at a cost of a few more cycles. L3 is the largest, shared by multiple cores, and can total many tens of megabytes on server chips. Each level down trades capacity for latency, so a lookup that misses L3 still lands in DRAM only a few cycles behind what is realistic.
The efficiency of the whole hierarchy is measured by hit rate: the fraction of memory accesses satisfied by a given level without going lower. A well-behaved program hits L1 for most accesses, L2 for the rest, and reaches DRAM only rarely. The cost model is the weighted sum of each level's latency, scaled by how often access reaches that level. This is why doubling the L1 hit rate has more impact than doubling the L3 size: the latency savings happen on the path taken every cycle, not the rare miss path.
Caches also track state with simple protocols. Each line is marked valid, indicating it holds live data, and on write-capable caches, modified, indicating it holds changes not yet written back to memory. When the core writes a value, the write lands in the L1 data cache first, and the copy in DRAM is updated later, either when the line is evicted or by an explicit flush. This write-through-versus-write-back choice is a classic tradeoff: write-back saves DRAM bandwidth while write-through keeps the slow layer current at a cost.
In multi-core systems, every core owns its own L1 and L2, so the same variable can exist in several caches at once, with possibly different values. Keeping them coherent, all cores seeing a consistent view of memory, is handled by a cache coherency protocol. When one core writes a line, the protocol invalidates or updates the stale copies in other cores before any one of them reads it again. That background traffic is invisible in single-threaded programs but becomes a real cost when threads contend on the shared data, which is one reason lock-heavy code is slow even when the critical section is tiny.
The same principles shape how compilers and operating systems squeeze performance. Loop unrolling and block size tuning aim to fit working sets inside L1 or L2. Prefetching hardware predicts which lines a stream will touch next and starts fetching them before the core asks, hiding latency that would otherwise cost real cycles. NUMA servers add another twist: memory physically attached to a different socket takes longer to reach, so pinning threads and data near their local memory node matters. In every case the underlying model is identical, a fast small layer shielding a larger, slower one.
Understanding cache hierarchy explains a lot that feels arbitrary in performance work. A program that does ten times the arithmetic but touches data in a poor order can run slower than a simpler one. A database that reorders its row layout improves query throughput without changing the query. A game streams textures in chunks that match cache and TLB behavior. The cache is not a detail, it is the shape of every modern CPU. When you can reason about hit rates, working sets, and coherency, a large share of performance questions stop being mysterious and become arithmetic.
Sources: Intel - Introduction to Intel 64 and IA-32 Architectures (chapter on memory types); University of Wisconsin Madison - How to Write High Performance Software (memory hierarchy).
Corrections: Report an error at corrections@wireshaman.com.