Memory Model and Atomics¶

Why mutexes are not the whole story¶

Concurrency Primitives and Race Conditions cover mutexes and critical sections. A mutex is the right default. But the hottest paths of a storage engine — a memtable read that must not block on a writer, a sequence number read by every query, a reference count touched on every lookup — use atomic operations directly. Using them correctly requires understanding a fact that surprises almost everyone at first: without synchronization, other cores may see your writes in a different order than you made them, and your reads may return values older than ones another core already wrote.

The Go side of the rules (happens-before, channels, sync) is in Channels, Mutexes, and the Memory Model; this page is about the hardware underneath and the atomic patterns built on it. Measurements: Go 1.26.4, Apple M1 Pro (arm64).

Reordering happens at three levels¶

1. The compiler may reorder independent memory accesses, or keep a value in a register so a loop never re-reads memory.
2. The CPU executes out of order and buffers stores in a store buffer before they reach the cache; a core's own later loads can complete before its earlier stores are visible to others.
3. The cache hierarchy may make writes visible to different cores at different times.

x86 is relatively strict (total store order: it never reorders two stores, or two loads, but does let a load pass an earlier store). ARM — including Apple silicon and AWS Graviton — is much weaker and reorders stores with stores and loads with loads too. Code that is only tested on x86 can be broken on ARM.

Measured: two litmus tests¶

Two goroutines, each locked to its own OS thread, run a tiny racy program 2 million times, released together by a spin barrier. The racy accesses use plain variables in separate cache lines; the disassembly confirms the compiler emitted the loads and stores in program order, so any reordering observed is the CPU's.

Store buffering — can both goroutines miss each other's store?

// Goroutine 1            // Goroutine 2
x = 1                     y = 1
r1 = y                    r2 = x
// Is r1 == 0 && r2 == 0 possible?

Message passing — can a reader see the flag but not the data?

// Writer                 // Reader
data = 42                 if ready == 1 {
ready = 1                     r = data // can this be 0?
                          }

Store buffering happens on every mainstream CPU, x86 included: each core's store sits in its store buffer while its next load reads the old value from cache. The message-passing failure is ARM-specific: the two stores (or the two loads) were reordered in hardware. On x86 that second test would show zero failures with plain variables — which is exactly how such bugs survive testing on developer laptops and x86 CI, then fail on ARM servers.

With sync/atomic, both tests show zero violations: Go's atomics emit the barrier-carrying instructions (LDAR/STLR on arm64) that forbid these reorderings.

Go's atomics: one ordering, sequentially consistent¶

The Go memory model defines all sync/atomic operations as sequentially consistent: they behave as if executed in one global order that every goroutine agrees on, and an atomic store observed by an atomic load creates a happens-before edge — everything written before the store is visible after the load.

var ready atomic.Bool
var data int

// Writer
data = 42
ready.Store(true) // everything before this store...

// Reader
for !ready.Load() { // ...is visible after a load that sees it
}
fmt.Println(data) // guaranteed 42

Languages such as C, C++, and Rust let each atomic operation choose a weaker ordering (relaxed, acquire, release). Go deliberately offers only the strongest one: there is no ordering to choose and get wrong. The cost is small — on arm64 a sequentially consistent load or store compiles to a load-acquire or store-release instruction, and on x86 an atomic load is a plain MOV while a store is an XCHG. What remains expensive is contention, not the ordering: measured in Channels, Mutexes, and the Memory Model, an uncontended atomic.Int64.Add costs 6.9 ns and the same add from 8 goroutines on one counter costs 71 ns.

Always use the typed atomics (atomic.Int64, atomic.Bool, atomic.Pointer[T]): they make non-atomic access to the same variable impossible to write by accident, and guarantee 64-bit alignment on 32-bit platforms.

Pattern 1: counters¶

type Stats struct {
    bytesWritten atomic.Uint64
}

func (s *Stats) RecordWrite(n int) { s.bytesWritten.Add(uint64(n)) }

Each increment is atomic, so none are lost. Under heavy contention, every Add on one shared counter bounces the cache line between cores. For counters on the hot path of every request, shard them: one counter per worker or per CPU, each padded to its own cache line, summed on read (2.0 ns per increment instead of 71 ns in the measurement above). Without the padding the shards share a line and it is worse than one counter — false sharing, measured at 45x in CPU and Cache Efficiency.

Pattern 2: publishing an object¶

Fully initialize an object, then make it reachable through an atomic pointer. Readers that load the pointer see the fully initialized object. This is how a skip list lets readers traverse while a single writer inserts, with no lock on the read path:

type node struct {
    key  []byte
    next atomic.Pointer[node]
}

// Writer (writers are serialized among themselves):
func insertAfter(prev, x *node) {
    x.next.Store(prev.next.Load()) // 1. initialize x completely; no reader can reach it yet
    prev.next.Store(x)             // 2. publish: readers that see x also see x.key and x.next
}

// Readers (any number, concurrently, no lock):
func next(n *node) *node { return n.next.Load() }

If step 2 were a plain assignment, a reader on ARM could follow the new pointer and see x.key not yet written — the message-passing failure measured above. The full lock-free skip list is in Memtable and Skip List.

The same pattern at a larger scale is copy-on-write configuration: build a new immutable *Config, then cfg.Store(newCfg); every request does cfg.Load() with no lock and always sees a complete config (Copy-on-write).

Pattern 3: reference counting¶

type tableFile struct {
    refs atomic.Int32
    f    *os.File
}

func (t *tableFile) ref() { t.refs.Add(1) }

func (t *tableFile) unref() {
    if t.refs.Add(-1) == 0 {
        t.f.Close() // last reference: release the resource
    }
}

In Go the garbage collector frees memory, so reference counting is only needed for resources whose release must happen at a known time: file descriptors, files to delete from disk, pooled buffers. The storage engine uses exactly this for SSTables, so compaction can delete a file while an older iterator still reads it (Compaction and Recovery). The thread that drops the last reference must see every write other goroutines made before dropping theirs; because each Add is sequentially consistent, it does.

Pattern 4: compare-and-swap¶

var maxSeen atomic.Uint64

func updateMax(v uint64) {
    for {
        cur := maxSeen.Load()
        if v <= cur || maxSeen.CompareAndSwap(cur, v) {
            return
        }
        // Another goroutine changed it between Load and CAS: retry.
    }
}

CompareAndSwap(old, new) sets the value to new only if it still equals old, atomically, and reports whether it did. It is the building block of every lock-free algorithm: read, compute, and commit only if nothing changed in between.

Lock-free is not automatically faster¶

Lock-free structures avoid blocking, but they are hard to get right — the ABA problem, and safe memory reclamation: when may a removed node be reused if a reader might still hold a pointer to it? (Go's GC solves reclamation for memory, which makes lock-free structures much easier in Go than in languages without one; see Lock-Free Programming and Memory Reclamation.) And an uncontended sync.Mutex costs only about 13 ns. Storage engines use lock-free techniques narrowly:

• Reads of an immutable or append-only structure (skip list, SSTable index) while one writer appends.
• Reference counts, sequence numbers, published snapshots.
• Everything else: a mutex, held briefly.

LevelDB protects its whole database object with one mutex, released around slow work like I/O; the Go storage engine in this handbook does the same.

Checking your work¶

The race detector (go test -race) detects data races — any non-atomic variable accessed by two goroutines without a happens-before edge, at least one access a write. The litmus tests above are exactly the kind of program it reports immediately. Run every concurrent test under -race (Concurrency Bugs). It cannot prove a lock-free protocol correct, but it catches the most common mistake: forgetting that a variable is shared at all.

Common pitfall¶

"Fixing" a racy flag by reading it in a loop and hoping the write shows up. A shutdown flag is a plain bool written by one goroutine and polled by a worker's for !stop { ... }. With optimizations on, the Go compiler may load stop once and keep it in a register, so the loop never exits; and even when it does reload, on ARM the worker can see stop == true before it sees the writes made just before setting it (the 0.7% case above). Neither failure shows up reliably in tests. Use atomic.Bool — or, more idiomatically in Go, close a channel or cancel a context, which also lets the worker block instead of spin.

Review questions¶

1. Name the three levels at which memory accesses can be reordered.
2. Why did the store-buffering test fail 32% of the time even though each goroutine's code was in program order?
3. Why would the message-passing test show zero failures on x86 with plain variables, and what does that mean for code tested only on x86?
4. What ordering do Go's atomics provide, and what does Go give up by not offering weaker ones?
5. In Go, when is reference counting needed at all, given the garbage collector?