Linux I/O Stack and the Page Cache¶

What happens when you call read()¶

1. The system call switches to kernel mode.
2. The VFS (virtual file system) finds the file's in-memory inode from the file descriptor.
3. The kernel checks the page cache — an in-memory cache of file contents, indexed by (inode, page offset). On a hit, it copies the page to your buffer and returns. No disk I/O at all.
4. On a miss, the file system translates the file offset to a disk block, the block layer queues a request, the driver submits it, the device DMAs the data into a freshly allocated page-cache page, and the process sleeps until the interrupt signals completion.
5. The page is copied into your buffer.

The key consequence: with ordinary ("buffered") I/O, every byte of file data passes through the page cache. Reads may never reach the disk; writes almost never reach it immediately.

The page cache uses all your free memory¶

The kernel uses otherwise unused RAM as page cache, and evicts it when applications need memory. This is why free -h on a database server shows little "free" memory but plenty "available":

               total        used        free      shared  buff/cache   available
Mem:            62Gi        12Gi       1.1Gi       0.2Gi        49Gi        49Gi

Here 49 GB of file data is cached. That memory is reclaimable, so it counts toward "available." Many databases (PostgreSQL in particular) deliberately rely on the page cache as a second layer of caching on top of their own buffer pool.

Writes: dirty pages and writeback¶

n, err := f.Write(data) // returns once the data is in the page cache

A buffered write() copies data into page-cache pages, marks them dirty, and returns — usually in microseconds. The data is not on disk. Kernel flusher threads write dirty pages back later, triggered by:

Two consequences matter for storage engines:

1. After a crash or power loss, up to ~30 seconds of "successful" writes can vanish. Only fsync/fdatasync make data durable — see Durability and fsync.
2. A burst of writes that pushes dirty memory past dirty_ratio makes write() itself suddenly block for a long time — a classic source of unexplained latency spikes in write-heavy services. Engines that write large files (compaction output) often call fdatasync or sync_file_range periodically while writing, to keep dirty pages bounded (RocksDB's bytes_per_sync option does exactly this).

Readahead¶

When the kernel sees sequential reads on a file, it reads ahead — fetching the next pages before they are requested. This makes sequential scans fast but wastes I/O and cache space for random access. Applications can hint their pattern:

import "golang.org/x/sys/unix" // Linux

fd := int(f.Fd())
unix.Fadvise(fd, 0, 0, unix.FADV_RANDOM)       // disable readahead (point lookups)
unix.Fadvise(fd, 0, 0, unix.FADV_SEQUENTIAL)   // aggressive readahead (compaction input)
unix.Fadvise(fd, off, n, unix.FADV_DONTNEED)   // drop these pages from the cache

POSIX_FADV_DONTNEED after reading compaction inputs prevents a one-time scan of gigabytes of old SSTables from evicting the hot data that foreground reads depend on.

Positional I/O: pread and pwrite¶

n, err := f.ReadAt(buf, blockOffset) // pread(2)

pread reads at an explicit offset without moving the file position. That makes it safe for many goroutines to read the same file concurrently — no Seek + Read race. Every SSTable reader uses ReadAt on a shared, read-only *os.File.

Short reads are legal at the system-call level: pread can return fewer bytes than requested (at end of file, or when interrupted by a signal). (*os.File).ReadAt already loops until buf is full, retries on EINTR, and returns io.EOF if the file ends first — the io.ReaderAt contract requires a non-nil error whenever n < len(buf). Code that calls unix.Pread directly must write that loop itself:

func readFully(fd int, buf []byte, off int64) error {
    for len(buf) > 0 {
        n, err := unix.Pread(fd, buf, off)
        if err == unix.EINTR {
            continue
        }
        if err != nil {
            return err
        }
        if n == 0 {
            return io.ErrUnexpectedEOF // the file is shorter than its metadata says
        }
        buf, off = buf[n:], off+int64(n)
    }
    return nil
}

The io.Reader interface has the weaker contract — a single Read may return fewer bytes than asked even when more are coming — which is why framing code uses io.ReadFull (Designing a Binary Protocol).

Direct I/O: bypassing the page cache¶

f, err := os.OpenFile(path, os.O_RDONLY|unix.O_DIRECT, 0) // Linux

O_DIRECT transfers data straight between the device and your buffer, skipping the page cache. Databases with their own buffer pool (InnoDB, ScyllaDB, RocksDB optionally) use it for three reasons:

• No double caching. Without it, a hot page lives once in the buffer pool and again in the page cache — wasting half the memory.
• Predictable memory and latency. The kernel's eviction and writeback decisions no longer interfere with the engine's own.
• Control. The engine decides exactly what is cached and when writes happen.

The cost is strict rules: the buffer address, file offset, and length must all be aligned to the device's logical block size (typically 512 B or 4 KB) — otherwise the call fails with EINVAL. You also lose readahead and must implement your own caching, which is a lot of work to do better than the kernel. That is why LevelDB, and RocksDB by default, use buffered I/O plus their own block cache of uncompressed blocks, and treat O_DIRECT as an option.

// Go's allocator does not guarantee 4 KB alignment: over-allocate and
// slice from the first aligned address.
func alignedBuf(size, align int) []byte {
    b := make([]byte, size+align)
    off := (align - int(uintptr(unsafe.Pointer(&b[0]))&uintptr(align-1))) % align
    return b[off : off+size : off+size]
}

buf := alignedBuf(64<<10, 4096) // aligned buffer for O_DIRECT

An anonymous unix.Mmap is another way to get page-aligned memory; packages such as github.com/ncw/directio wrap both steps. On macOS there is no O_DIRECT; fcntl(F_NOCACHE) (unix.FcntlInt(fd, unix.F_NOCACHE, 1)) disables caching for a file instead, which is how the random-read measurements in Async I/O and io_uring bypassed the cache.

O_DIRECT does not imply durability: the device's volatile write cache can still hold the data. You still need fdatasync (or O_DSYNC).

Memory-mapped files: mmap¶

data, err := unix.Mmap(int(f.Fd()), 0, int(size), unix.PROT_READ, unix.MAP_SHARED)
// data is a []byte backed by the file: data[off] may page-fault and load it.
defer unix.Munmap(data)

mmap maps a file into the address space, so reading the file is just reading memory; the kernel loads pages on page faults. It is attractive — zero copies, no read calls, the kernel manages caching — and LMDB is built entirely on it. It also has serious problems for a general database:

The 2022 CIDR paper "Are You Sure You Want to Use MMAP in Your Database Management System?" (Crotty, Leis, Pavlo) works through these in detail. A reasonable rule: mmap is acceptable for read-only immutable files that fit comfortably in memory (LevelDB optionally mmaps SSTables on 64-bit systems); use pread and an explicit cache otherwise.

Observing the I/O stack¶

Common pitfall¶

Measuring read performance right after writing a dataset reports page cache speed, not disk speed — the data just written is still cached. A benchmark that writes 2 GB and then reads it back on a machine with 16 GB of RAM never touches the disk on the read side. Either use a dataset several times larger than memory, open files with O_DIRECT for the measurement, or drop the cache between phases (sync; echo 3 | sudo tee /proc/sys/vm/drop_caches on a test machine).