Leases and Fencing¶

Locks that can expire¶

A distributed lock cannot be held forever: if the holder crashes, the lock must eventually be released, or the system stops. So distributed locks are leases — a lock with a time-to-live that the holder must renew. etcd leases, ZooKeeper ephemeral nodes, Consul sessions, Redis SET key value NX PX ttl, and Kubernetes leader election (coordination.k8s.io/v1 Lease objects) all work this way.

The expiry is what makes a lease unsafe on its own: the lock service decides the lease has expired based on its clock, while the holder believes it still holds the lease based on its view of time — and a process can be paused for longer than any TTL at any moment (Failure and Time).

The failure, reproduced¶

labs/lease has a lock service that grants exclusive leases with a TTL, and a storage service that the lease is supposed to protect.

type LockService struct {
    mu      sync.Mutex
    holder  string
    expires time.Time
    token   uint64
}

func (l *LockService) Acquire(client string, ttl time.Duration) (uint64, error) {
    l.mu.Lock()
    defer l.mu.Unlock()
    if l.holder != "" && l.holder != client && time.Now().Before(l.expires) {
        return 0, ErrHeld
    }
    l.holder, l.expires = client, time.Now().Add(ttl)
    l.token++ // strictly increasing: the fencing token
    return l.token, nil
}

The scenario, run with a fake clock under testing/synctest:

Result without fencing — both writes are accepted:

without fencing: ["B: write under valid lease" "A: write from expired lease"]

Two clients both believed they held the exclusive lock, and the storage applied writes from both, the stale one last. Nothing A could have done would prevent this: checking "do I still hold the lease?" right before writing does not help, because the pause can happen between the check and the write — or while the write is in flight in the network.

Pauses long enough to cause this are not exotic: stop-the-world GC in other runtimes (seconds on large JVM heaps), a VM live migration, a process stopped with SIGSTOP, a laptop lid closed, swapping, CPU throttling under a cgroup quota, or simply a network delay on the write itself.

Fencing tokens¶

The fix moves the check to the resource. Every lease grant carries a strictly increasing number. The storage remembers the highest token it has seen and rejects any request with a lower one:

func (s *Storage) Write(token uint64, v string) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.Fencing {
        if token < s.maxSeen {
            return ErrStaleToken
        }
        s.maxSeen = token
    }
    s.Data = append(s.Data, v)
    return nil
}

Same scenario, fencing on:

A rejected: storage: stale fencing token
with fencing: ["B: write under valid lease"]

As soon as B's write with token 2 reaches the storage, every request from an earlier lease holder is refused. Correctness no longer depends on clocks or pause durations, only on the token order, which the lock service controls.

Where the tokens come from in real systems:

The resource must participate. If the protected thing cannot check a token (an external API, a file on a plain filesystem), a lease alone cannot make access exclusive — design the operation to be idempotent or conflict-detecting instead.

Leader leases and lease-based reads¶

Raft leaders sometimes use a lease to serve reads without contacting a majority: after a successful heartbeat round at time t, no other leader can be elected before t + election timeout (followers won't start an election while hearing from the leader), so the leader can serve reads locally until shortly before then. This is faster than ReadIndex but depends on bounded clock drift across nodes — violate it (a paused leader, a clock running fast) and stale reads return. etcd and TiKV offer it as an option; the safe default is ReadIndex, measured in Replicated Key-Value Store.

Using leases correctly, as a checklist¶

1. Fence every write to the protected resource with the lease's token (or term/revision).
2. TTL well above the renewal interval (e.g. renew every TTL/3) and above expected pauses; but remember no TTL makes an unfenced lease safe.
3. Stop work before expiry: when renewal fails, stop issuing new operations at expiry − safety margin measured on the monotonic clock.
4. Treat losing the lease as an event: cancel in-flight work via context, don't just let it finish.
5. Use leases for efficiency, fencing for correctness. A lease that occasionally lets two holders run is fine if the resource rejects one of them; it's a disaster if it doesn't.

Common pitfall¶

Using a Redis lock to make a job "run on exactly one instance". A cron job acquires SET lock:billing <id> NX PX 30000, then charges customers. One instance hits a long GC pause (or a slow network call to the payment provider) past 30 seconds; another instance acquires the lock and starts the same billing run; the first resumes and continues charging. Customers are charged twice. The lock was never the problem — the payment calls were unfenced and not idempotent. The fix is on the resource side: an idempotency key per charge (the payment provider deduplicates), or a per-run record in the database updated with compare-and-set on a token, so a stale holder's writes fail.

Review questions¶

1. Why can't the lease holder protect itself by checking that its lease is still valid right before writing?
2. Walk through the scenario with fencing. At which moment does A's write become impossible to accept?
3. Which value plays the role of a fencing token in Raft, and where is it checked?
4. What does a leader-lease read assume that ReadIndex does not?
5. The protected resource is a third-party API that cannot check tokens. What can you do instead?