Storage Hardware: HDD, SSD, and NVMe¶

Why the hardware shapes the software¶

B+Trees were designed in the 1970s for spinning disks where a random read cost 10+ ms. LSM trees became popular because they turn random writes into sequential ones. Modern SSDs change the cost model again. You cannot reason about why a storage engine is built the way it is without knowing what the device underneath rewards and punishes.

Hard disk drives (HDD)¶

An HDD is a stack of spinning magnetic platters with one read/write head per surface on a moving arm.

A random read pays three costs:

Worked example: reading 1,000 random 4 KB pages costs about 1,000 × (5 + 4 + 0.03) ms ≈ 9 seconds. Reading the same 4 MB sequentially costs one seek plus 4 MB / 150 MB/s ≈ 35 ms. That ~250x gap between random and sequential access is the entire historical motivation for write-ahead logs, LSM trees, and large B+Tree pages: anything that converts random I/O into sequential I/O wins enormously on an HDD.

An HDD delivers roughly 100–200 random IOPS (I/O operations per second) regardless of how small the read is.

Solid-state drives (SSD)¶

An SSD has no moving parts. Data lives in NAND flash cells, and the physics of flash imposes rules that ripple all the way up into database design.

The three rules of NAND flash¶

1. Read and program (write) happen in pages — typically 4–16 KB.
2. A page cannot be overwritten in place. It must be erased first.
3. Erase happens only in whole blocks — a block is hundreds of pages (several MB) — and each block survives only a limited number of erase cycles (roughly 1,000–3,000 for consumer TLC/QLC, more for enterprise drives).

The Flash Translation Layer (FTL)¶

Because pages cannot be overwritten, the SSD's controller runs a Flash Translation Layer: a mapping from the logical block address the OS writes to a physical page. An "overwrite" of logical block 42 actually writes to a fresh physical page and marks the old one stale.

Stale pages accumulate, so the FTL runs garbage collection: pick a block with many stale pages, copy its still-valid pages elsewhere, erase the block. Those copies are writes the host never asked for — device-level write amplification. It stacks multiplicatively on top of the application's own write amplification: an LSM tree with 10x amplification on an SSD with 2x internal amplification wears the flash at 20x the user's write rate.

The FTL also explains several practical rules:

• Over-provisioning: SSDs reserve hidden spare capacity so GC always has free blocks. A nearly full SSD has little spare space, GC must copy more valid data per erased block, and write throughput and latency degrade sharply.
• TRIM / discard: when a file is deleted, the OS tells the SSD which logical blocks are now unused, so GC does not waste effort copying dead data. Deleting obsolete SSTables is exactly this case.
• Sequential, large, aligned writes are still better — not because of seeks, but because they fill whole erase blocks with data that will die together, leaving GC little to copy. An LSM tree that writes large immutable files and deletes them whole is naturally SSD-friendly.

Latency and parallelism¶

An SSD's IOPS figure is only reached with many requests in flight at once. Internally the drive has many independent flash chips and channels; one request at a time (queue depth 1) leaves most of them idle. A single thread issuing blocking 4 KB reads one after another gets maybe 10–20K IOPS from a drive rated for 1M. This is why high-performance engines use many concurrent readers or asynchronous I/O — see Async I/O and io_uring.

NVMe¶

NVMe is the protocol modern SSDs speak over PCIe, replacing the SATA/AHCI interface designed for spinning disks. AHCI has one command queue of 32 entries; NVMe supports up to 64K queues of up to 64K entries each, typically one queue per CPU core, so cores submit I/O without contending on a shared lock. The result is that on NVMe, the software stack (system calls, the kernel block layer, context switches) becomes a significant fraction of each I/O's latency — the driving reason io_uring and kernel-bypass frameworks like SPDK exist.

Power loss and the volatile write cache¶

Most drives have a DRAM write cache. A write can be acknowledged once it reaches that cache, before it is on flash or platter. If power fails, data in the cache is lost — unless the drive has power-loss protection (capacitors that flush the cache), common on enterprise SSDs and rare on consumer ones.

This is why fsync must do two things: push data from the OS page cache to the device, and send a cache-flush command (or use Force Unit Access writes) so the device commits it to stable media. See Durability and fsync.

Sector atomicity: drives generally guarantee that a single sector (512 bytes or 4 KB) is written entirely or not at all on power loss, but nothing larger. A 16 KB database page written during a power cut can end up half old, half new — a torn write. Databases defend against it with checksums on every page and log record, and some with double-write buffers (InnoDB) or full-page images in the WAL (PostgreSQL).

Measuring a device: fio¶

# Random 4 KB reads, 32 in flight, bypassing the page cache, for 30 seconds
fio --name=randread --filename=/data/fio.test --size=4G \
    --rw=randread --bs=4k --iodepth=32 --ioengine=io_uring \
    --direct=1 --runtime=30 --time_based --group_reporting

# fsync latency: how long does a durable 4 KB append take?
fio --name=synclat --filename=/data/fio.test --size=256M \
    --rw=write --bs=4k --fsync=1 --runtime=30 --time_based

Run the second one before designing your WAL: the fsync latency of the target disk (hundreds of microseconds on a good NVMe, several milliseconds on many cloud volumes, ~10 ms on an HDD) directly bounds how many durable commits per second a single writer can achieve — which is why group commit exists.

Cloud block storage¶

Cloud volumes (AWS EBS, GCP Persistent Disk) are network-attached: every I/O crosses the network to a replicated storage service. Expect ~0.5–2 ms latency instead of ~50 µs, and IOPS/throughput limits set by the volume type you pay for, not by the hardware. Instance-local NVMe ("instance store") is much faster but disappears when the instance stops — so systems that use it must replicate data themselves.

Common pitfall¶

Benchmarking a storage engine on a laptop SSD and assuming production will look the same is a frequent mistake. A laptop NVMe with a fast volatile write cache may report fsync in 50 µs (or ignore flushes entirely), while the production cloud volume takes 1–2 ms per fsync and caps IOPS at the provisioned level. The engine's design point — batch size, group commit window, number of compaction threads — has to be chosen against the production device's numbers, measured with fio, not the development machine's.