Concurrency Bugs¶

Three families¶

Concurrency bugs in Go fall into three families, each with its own detection tool:

Family	What goes wrong	Primary tool
Data races	Two goroutines access the same memory without synchronization, at least one writing	The race detector (-race)
Logic races	Every access is synchronized, but the interleaving still produces a wrong result (check-then-act, lost wakeups, wrong flush conditions)	Stress/load tests, model checks, careful review
Liveness bugs	Deadlocks, leaks, starvation: something never happens	Goroutine dumps, leak detectors, timeouts in tests

The memory-model background is in Channels, Mutexes, and the Memory Model.

Outputs are real (Go 1.26.4, darwin/arm64), paths shortened.

The race detector¶

go test -race ./...
go run -race ./cmd/server
go build -race -o server-race ./cmd/server   # a race-enabled binary for staging/canaries

The detector instruments every memory access and synchronization operation and tracks happens-before relations with vector clocks (ThreadSanitizer). It reports a race when two accesses to the same address, at least one a write, are not ordered by any happens-before edge. Cost: typically 5–10x slower and 5–10x more memory.

type Stats struct{ hits map[string]int }

func main() {
    s := &Stats{hits: map[string]int{}}
    var wg sync.WaitGroup
    for _, p := range []string{"/a", "/b"} {
        wg.Add(1)
        go func() {
            defer wg.Done()
            s.hits[p]++ // two goroutines, one map, no lock
        }()
    }
    wg.Wait()
}

==================
WARNING: DATA RACE
Read at 0x00c00009a090 by goroutine 8:
  runtime.mapassign_fast64ptr()
      internal/runtime/maps/runtime_fast64.go:374 +0x38c
  main.main.func1()
      race/main.go:19 +0x9c

Previous write at 0x00c00009a090 by goroutine 7:
  runtime.mapaccess2_faststr()
      internal/runtime/maps/runtime_faststr.go:161 +0x2ac
  main.main.func1()
      race/main.go:19 +0xc8

Goroutine 8 (running) created at:
  main.main()
      race/main.go:17 +0xe4

Goroutine 7 (finished) created at:
  main.main()
      race/main.go:17 +0xe4
==================
Found 1 data race(s)
exit status 66

Read it in four parts: the two accesses (which goroutine, read or write, which line — main.go:19 both times), and where each goroutine was created (main.go:17). The runtime frames (mapassign...) only tell you it was a map. The process exits with status 66 when races were found, which is what fails CI.

What the race detector cannot do:

• It only finds races that happen during the run. A race in a code path that the tests don't execute concurrently is invisible. Tests must actually run things in parallel (t.Parallel(), many goroutines, realistic load).
• It does not find logic races — the protocol server's flush deadlock had perfectly synchronized memory accesses and was invisible to it.
• It does not find deadlocks.

Run -race in CI on every test, and run a race-enabled build under realistic load in staging occasionally: production traffic exercises interleavings that tests never do.

Logic races: check-then-act¶

The most common logic race is a check and an action that are each atomic but not atomic together:

if _, ok := cache.Load(key); !ok { // check
    v := compute(key)                // other goroutines also get here
    cache.Store(key, v)              // act: compute ran N times
}

if balance.Load() >= amount {        // check
    balance.Add(-amount)             // act: two withdrawals both pass the check → negative balance
}

No data race exists — sync.Map and atomics are synchronized — but the invariant breaks. Fixes: put the check and the act in one critical section (mu.Lock() around both), use an atomic primitive that does both (LoadOrStore, CompareAndSwap in a retry loop), or deduplicate with singleflight. In databases, the same bug is the lost update and write skew (Iterators, Snapshots, and Transactions).

Deadlocks¶

The four conditions — mutual exclusion, hold and wait, no preemption, circular wait (Deadlocks) — all hold for Go mutexes. Go-specific shapes:

Shape	Example
Lock ordering (AB-BA)	transfer(a, b) and transfer(b, a) concurrently — the dump in Panics, Stack Dumps, and Delve
Recursive locking	A method holding mu calls another method that also locks mu; Go mutexes are not reentrant
RWMutex recursive read lock	RLock held, a writer queues, the same goroutine RLocks again → blocked behind the writer, which waits for the first RLock
Channel cycles	Goroutine A sends to B and waits for B's reply; B is blocked sending to A
Unbuffered send with no receiver while holding a lock	The would-be receiver needs the same lock
WaitGroup.Wait inside a goroutine counted by the same WaitGroup	Waits for itself

Prevention:

• Global lock order: always lock accounts by ascending ID; document the order for any code that holds two locks.
• Never hold a lock while calling out (callbacks, channel sends, I/O, other packages): copy what you need, unlock, then call.
• Timeouts on anything that waits for another party (context), so a deadlock becomes an error with a stack trace rather than a hang.

Detection: the runtime's all goroutines are asleep only fires when every goroutine is blocked — rare in servers. In practice: goroutine dumps (look for many goroutines in [sync.Mutex.Lock] for minutes), the mutex profile, and tests with a timeout (go test -timeout=60s prints all goroutine stacks when it fires).

Testing time-dependent code with synctest¶

Code with timers, tickers, timeouts, and background goroutines is hard to test: real sleeps make tests slow and flaky. testing/synctest (Go 1.25) runs a test inside a bubble with a fake clock: time only advances when every goroutine in the bubble is blocked, and then jumps straight to the next timer.

A TTL cache with a background janitor goroutine that deletes expired entries every minute:

func TestExpiry(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        c := New(time.Hour)
        stop := make(chan struct{})
        defer close(stop)
        c.StartJanitor(time.Minute, stop)
        c.Set("k", "v")

        time.Sleep(59 * time.Minute) // fake clock: returns instantly
        if _, ok := c.Get("k"); !ok {
            t.Fatal("expired too early")
        }
        time.Sleep(2 * time.Minute)
        synctest.Wait() // run every bubble goroutine until all are blocked again
        if c.Len() != 0 {
            t.Fatalf("janitor did not remove expired entry, len=%d", c.Len())
        }
    })
}

--- PASS: TestExpiry (0.00s)

61 minutes of simulated time in under a millisecond, deterministically. synctest.Wait() is the key primitive: "let everything that can run, run" — it replaces the time.Sleep(10 * time.Millisecond) "give the goroutine a chance" that makes ordinary tests flaky. The bubble also reports a deadlock if all its goroutines block with no timer pending, and synctest.Test fails if goroutines started in the bubble are still running when the test function returns — a built-in leak check.

A testing toolbox for concurrent code¶

Tool	Catches
go test -race	Data races in executed interleavings
go test -count=100 -race -run TestX	Rare interleavings — run flaky-looking tests many times
-cpu=1,2,8	Bugs that appear only with real parallelism (or only without it)
testing/synctest	Deterministic timers, timeouts, background loops
go.uber.org/goleak	Goroutines still running after a test
Parallel benchmarks / load tests	Logic races and deadlocks under contention
go test -timeout	Converts hangs into failures with full stack dumps
Model-based tests	Results checked against a sequential model (Testing and Benchmarking)

Common pitfall¶

"Fixing" a race by adding time.Sleep. A test is flaky because it checks a value before a background goroutine has updated it; someone adds time.Sleep(100 * time.Millisecond) before the assertion, and the test passes — locally. On a loaded CI machine the goroutine sometimes takes 150 ms, and the test is flaky again; and the sleep has hidden the real problem, which is that the code offers no way to know when the update happened. Make the synchronization explicit: the code under test returns a channel or accepts a callback when done, the test waits on that (with a timeout), or the test runs inside synctest and uses synctest.Wait(). A sleep in a test is a guess about scheduling, and scheduling guesses are what concurrency bugs are made of.

Review questions¶

1. What exactly does the race detector report, and why can it miss real races?
2. Give an example of a logic race with no data race. How do you fix it?
3. Why are Go mutexes' non-reentrancy and RWMutex's writer preference both sources of deadlocks?
4. How does synctest make a test that waits 61 minutes run instantly, and what does synctest.Wait() guarantee?
5. Why is time.Sleep in a test not a fix for a race?