Durability and fsync¶

What "durable" means¶

An operation is durable when its effects survive a crash of the process, the kernel, or the power supply. For a storage engine, the contract is precise: once Put() returns OK to the caller (with sync enabled), the value must be readable after any crash. Everything on this page is about what it takes to keep that promise, because the operating system's defaults do not.

Three kinds of crash, three levels of safety¶

A process crash loses only the leftmost box: a successful write() already survives kill -9. Power loss can lose everything left of "stable media." Testing with kill -9 therefore proves very little about durability — see Testing and Benchmarking a Storage Engine.

The system calls¶

For an append-only log, the standard choice is write() + fdatasync(). Preallocating the log file (fallocate) and then overwriting it makes fdatasync cheaper still, because the file size no longer changes on each append.

macOS caveat: fsync() on macOS does not flush the drive's write cache. Real durability needs fcntl(fd, F_FULLFSYNC), which is much slower. Go's (*os.File).Sync already uses F_FULLFSYNC on macOS — measured at 4.4 ms per call against 32 µs for a raw fsync(2) in Files and Durability in Go; LevelDB does the same, and SQLite exposes it as PRAGMA fullfsync.

Rule 1: a new file needs its directory synced¶

A file's name is an entry in its parent directory, and the directory is a separate file with its own data blocks. fsync on the new file makes the file's contents durable, but not necessarily the directory entry that points to it.

f, err := os.OpenFile("/db/000123.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
if err != nil {
    return err
}
if _, err := f.Write(record); err != nil {
    return err
}
if err := f.Sync(); err != nil { // the file's contents are durable
    return err
}
// Crash here: on some file systems, after reboot /db/000123.log may not
// exist, even though its data blocks were written.

d, err := os.Open("/db")
if err != nil {
    return err
}
defer d.Close()
return d.Sync() // now the name is durable too

Every time an engine creates, renames, or deletes a file it depends on after a crash — a new WAL segment, a new SSTable, a new manifest — it must fsync the containing directory.

Rule 2: atomic replace = write temp, fsync, rename, fsync dir¶

rename() is atomic on POSIX file systems: after a crash, the target name refers either to the old file or the new one, never a mix. That makes it the standard way to update a small file such as LevelDB's CURRENT (which names the active manifest):

func setCurrent(dir, manifest string) error {
    tmp := filepath.Join(dir, "CURRENT.tmp")
    if err := writeFileSync(tmp, []byte(manifest+"\n")); err != nil { // write + fsync tmp
        return err
    }
    if err := os.Rename(tmp, filepath.Join(dir, "CURRENT")); err != nil { // atomic swap
        return err
    }
    return syncDir(dir) // make the rename durable
}

This is the function the Go storage engine uses; a general WriteFileAtomic with every error path is in Files and Durability in Go.

Skip any step and a crash can break it:

Rule 3: a failed fsync cannot be retried ("fsyncgate")¶

In 2018, PostgreSQL developers discovered that on Linux, if fsync returns an error (EIO, for example from a failing disk or a full thin-provisioned volume), the kernel may mark the dirty pages clean and drop the error. A second fsync then returns success — but the data was never written. PostgreSQL had been retrying failed fsyncs, believing the retry made the data durable.

The correct response to an fsync failure in a storage engine is to treat the database as unable to guarantee durability: stop accepting writes, surface the error, and recover from the WAL on restart. PostgreSQL now PANICs on fsync failure. LevelDB and RocksDB put the DB into a sticky background-error state where all later writes fail.

if err := db.log.Sync(); err != nil {
    // Do NOT retry and continue. The kernel may already have discarded the data.
    db.setBackgroundError(err) // every subsequent write now returns this error
    return err
}

Rule 4: assume torn and reordered writes¶

Without fsync between them, writes can reach the disk in any order, and a write larger than one sector can be partially applied. After power loss, a log file may end with:

• a complete final record,
• a partial record (torn write),
• garbage bytes (a size update persisted before the data),
• a valid record after a missing one, if the writes were reordered.

The defense is the same everywhere: every record carries a length and a checksum (Binary Encoding and Checksums), and recovery stops at the first record that does not verify. The WAL page shows the record format.

The cost of durability, and group commit¶

fdatasync on an NVMe drive with power-loss protection can take ~20–100 µs; on a consumer SSD that honors cache flushes, often ~0.5–2 ms; on a network volume, ~1–3 ms; on an HDD, ~10 ms. A single writer that syncs every commit can therefore do only a few hundred to a few tens of thousands of commits per second.

Group commit amortizes it: concurrent writers queue up, one leader appends all their records and issues a single fdatasync, and then all of them return. 50 writers waiting for one 2 ms sync = 25,000 commits/s instead of 500. See WAL for the general idea and Write-Ahead Log for the implementation.

Many engines also offer a non-sync mode (LevelDB's WriteOptions::sync = false, the default): writes survive process crashes but may lose the last few seconds on power loss. That is a legitimate choice for some data, as long as it is made explicitly.

File system behavior matters¶

Crash guarantees differ between file systems and mount options. The 2014 OSDI paper All File Systems Are Not Created Equal (Pillai et al.) tested real applications — including LevelDB, SQLite, Git, and PostgreSQL — against file-system crash behavior and found bugs in almost all of them, typically from assuming an ordering or atomicity the file system did not guarantee. The safe approach is to rely only on what POSIX guarantees (rename atomicity, fsync semantics) plus checksums, and never on a particular file system's convenient behavior. ext4 (default data=ordered) and XFS are the common choices for databases on Linux.

Common pitfall¶

Calling bufio.Writer.Flush() and believing the data is on disk. Flush only moves data from the user-space buffer into the kernel with write() — the first arrow in the diagram. It survives a process crash but not a power loss. Durability requires (*os.File).Sync() on the underlying file after the flush, plus a directory sync if the file is new. The same confusion exists one level down: os.WriteFile and (*os.File).Close do not sync either. Know which arrow each call moves your bytes across.