Files and Durability in Go¶

The gap between "written" and "safe"¶

f.Write(b) returning nil means the bytes were copied into the kernel's page cache. It does not mean they are on disk, and after a power loss they may be gone — or, worse, partially there. Every database, queue, and configuration store built in Go has to close this gap explicitly, with the right syscalls in the right order. The OS and hardware side is in Linux I/O Stack and Page Cache and Durability and fsync; this page is how to do it correctly from Go, with measurements.

Measurements: Go 1.26.4, Apple M1 Pro, internal NVMe SSD, APFS.

How Go does file I/O¶

• os.File wraps a file descriptor. Read, Write, ReadAt (pread), WriteAt (pwrite) each make one system call per call — there is no hidden buffering.
• Regular files cannot use the netpoller (the kernel reports them as always ready), so file I/O is a blocking syscall: the goroutine holds its thread for the duration, and many concurrent slow disk operations make the runtime create many threads (Runtime Scheduler). Bound concurrent disk I/O with a semaphore.
• bufio.Writer/Reader batch small operations into large syscalls. Writing 128-byte records with one Write each cost 3.2 µs per record here (310,000 writes/s) — almost all of it syscall overhead.
• io.Copy from an *os.File to a socket or another file uses zero-copy kernel paths on Linux (sendfile, splice, copy_file_range) through the ReaderFrom/WriterTo interfaces. Wrapping either side (e.g. in a bufio or a custom io.Reader) hides the interface and silently falls back to a user-space copy.
• os.ReadFile/os.WriteFile are convenient but WriteFile does not fsync.

What fsync costs¶

Appending 128-byte records, measured:

Durability per record	Latency per record	Records/s
Write only (page cache)	3.2 µs	310,000
Write + raw fsync(2)	31.6 µs	31,600
Write + fcntl(F_BARRIERFSYNC)	225 µs	4,450
Write + f.Sync() (Go's choice: fcntl(F_FULLFSYNC))	4,455 µs	224

This table is macOS-specific and instructive. On macOS, fsync(2) only pushes data to the drive, not through the drive's volatile write cache — fast, and not durable against power loss. F_FULLFSYNC asks the drive to flush its cache, which is what durability actually requires, and Go's os.File.Sync uses it on Darwin (see internal/poll/fd_fsync_darwin.go). On Linux, fsync/fdatasync already issue a cache flush to the device; typical costs are tens of microseconds on datacenter NVMe with power-loss protection and milliseconds on consumer SSDs and network disks. Measure on the hardware you deploy to.

The ratio is the design constraint: a durable write is 100–1,000x more expensive than a buffered one, so durable systems amortize fsync over many records (group commit), which is how the storage engine in this handbook turns 224 synced writes/s into thousands (Write-Ahead Log).

The rules of durable writes¶

1. Sync the file after writing data you must not lose (f.Sync(); on Linux, unix.Fdatasync skips flushing unchanged metadata such as mtime and can be cheaper).
2. Sync the directory after creating, renaming, or deleting a file. A file's name lives in its directory; without syncing the directory, a crash can lose a fully synced file because its directory entry never reached disk.
3. Check every error, including from Close — on some filesystems (NFS, and when the kernel defers errors) a failed write is first reported at close.
4. Treat a failed fsync as fatal for that file. After an fsync error, Linux may mark the dirty pages clean and drop them; retrying fsync can then succeed without the data ever being written (PostgreSQL's "fsyncgate", 2018). The safe response is to stop using the file and recover from the log or a replica.

func syncDir(dir string) error {
    d, err := os.Open(dir)
    if err != nil {
        return err
    }
    defer d.Close()
    return d.Sync()
}

Atomic file replacement¶

To replace a configuration file, a manifest pointer, or a snapshot so that a crash leaves either the old or the new version — never a truncated mix:

// WriteFileAtomic replaces path with data. After it returns nil, the new
// content survives a crash; if it fails or crashes midway, the old file
// is intact.
func WriteFileAtomic(path string, data []byte, perm os.FileMode) error {
    dir := filepath.Dir(path)
    tmp, err := os.CreateTemp(dir, ".tmp-"+filepath.Base(path)+"-*") // same directory = same filesystem
    if err != nil {
        return err
    }
    defer os.Remove(tmp.Name()) // no-op after a successful rename

    if _, err := tmp.Write(data); err != nil {
        tmp.Close()
        return err
    }
    if err := tmp.Chmod(perm); err != nil {
        tmp.Close()
        return err
    }
    if err := tmp.Sync(); err != nil { // 1. data durable before it gets the name
        tmp.Close()
        return err
    }
    if err := tmp.Close(); err != nil {
        return err
    }
    if err := os.Rename(tmp.Name(), path); err != nil { // 2. atomic name swap (POSIX rename)
        return err
    }
    return syncDir(dir) // 3. the rename itself is durable
}

Each step defends against a specific crash:

Crash after...	Without the step	With it
Write, before Sync	Rename could become durable before the data: the new name points to an empty or partial file	Data is on disk before the name changes
Rename, before directory Sync	After reboot the old file may be back (the rename was lost) — acceptable only if callers did not act on the new version	Rename persisted
Any point	—	Old or new content, never a mix

The temporary file must be in the same directory (rename is only atomic within one filesystem) and the pattern is how the storage engine updates its CURRENT file (Compaction and Recovery).

Append-only logs¶

Appending is the friendliest pattern for durability: existing bytes never change, so a crash can only damage the tail. A log record format needs:

• a length so a reader knows where each record ends;
• a checksum (CRC32-C — hardware accelerated, hash/crc32 with crc32.Castagnoli) so a torn or corrupted tail is detected instead of parsed as garbage;
• a recovery rule: a bad record at the end of the last log is an unacknowledged write that was torn by the crash — truncate it and continue; a bad record in the middle is real corruption — stop and report.

The full design (32 KB blocks, fragment types, group commit) is the Write-Ahead Log page. Encoding details are in Binary Encoding and Checksums.

Preallocation, direct I/O, and locking¶

• Preallocate log and data files (unix.Fallocate on Linux) so appends do not have to allocate blocks and update metadata on every write — fdatasync on a preallocated file then only flushes data.
• O_DIRECT (unix.O_DIRECT with os.OpenFile) bypasses the page cache; buffers must be aligned to the device's logical block size (typically 4 KB), which Go's allocator does not guarantee — allocate a larger slice and offset into it, or use mmap-backed buffers. Databases use it to manage caching themselves; most Go programs should not.
• Exclusive access: unix.Flock(fd, LOCK_EX|LOCK_NB) on a LOCK file prevents two processes from opening the same data directory — the lock is released automatically if the process dies.
• mmap for read-mostly data is covered, with its Go-specific hazards, in unsafe, cgo, and System Calls.

Testing durability¶

Unit tests never crash the machine, so durability bugs survive them. Techniques, cheapest first:

1. Simulated crash by copying: copy the data directory while the process is running (as if it died at that instant), truncate the last file by a few bytes, reopen the copy, and check invariants. The storage engine's test suite does this (Testing and Benchmarking).
2. A fault-injecting filesystem interface: route all file operations through an interface (like Pebble's vfs.FS) so tests can make writes fail, drop unsynced data on "crash", or reorder operations.
3. Real power-cut and kill -9 testing in a VM, for the final word.

Common pitfall¶

Writing a state file with os.WriteFile and trusting it. A service stores its last processed Kafka offset with os.WriteFile("offset", ...) after each batch. After an unclean host reboot, the file exists but is empty: WriteFile truncated it, the new contents were still in the page cache, and the truncation's metadata reached disk first. The service restarts from offset 0 and reprocesses a day of events. Two independent mistakes: no fsync (the data was never durable), and in-place overwrite (there was a window in which neither the old nor the new content existed). The atomic-replace function above fixes both, at the cost of two fsyncs — which is why such state is usually written every N records or every few seconds rather than after every record.

Review questions¶

1. What exactly is guaranteed when f.Write returns nil?
2. Why is raw fsync(2) on macOS 140x faster than Go's f.Sync(), and which one is durable?
3. Why must you fsync the directory after renaming a file into place?
4. Why is retrying a failed fsync dangerous on Linux?
5. In a checksummed append-only log, how do you tell a torn write from corruption, and why do they need different handling?