Raft on Real Hardware¶

Leaving the simulation¶

Everything on the previous Raft pages ran on a simulated network with storage that costs nothing. That is the right way to test correctness and the wrong way to measure performance: in the simulation, the leader's batches never grew beyond one entry, because nothing ever waits for a disk. This page runs the same code as three real processes, talking TCP, writing a real write-ahead log with real fsync, and measures what a replicated write actually costs.

Setup: Go 1.26.4, Apple M1 Pro, macOS, internal NVMe SSD; three raftkvd processes on localhost; Go's File.Sync on macOS is F_FULLFSYNC, about 4.4 ms per call (Files and Durability in Go).

The pieces¶

Component	What it does
labs/raftnet	Binary codec for RequestVote, AppendEntries, InstallSnapshot; a raft.Transport over TCP with one multiplexed connection per peer, redialed after errors, 500 ms per-RPC timeout. Built on the framed protocol from Designing a Binary Protocol; the decoder is fuzzed
labs/cmd/raftkvd	One server process: raftwal storage, a Raft node, the raftkv state machine on labs/kv, and a TCP listener serving Raft RPCs, client requests, and a stats endpoint
labs/cmd/kvbench	N concurrent closed-loop clients (each sends its next request when the previous one returns), 100-byte values over 10,000 keys, latency percentiles, plus the leader's batching stats and the WAL fsync count from every server

raftkvd -id 0 -peers 0=127.0.0.1:7100,1=127.0.0.1:7101,2=127.0.0.1:7102 -dir /data/kv0 &
raftkvd -id 1 -peers ... -dir /data/kv1 &
raftkvd -id 2 -peers ... -dir /data/kv2 &
kvbench -servers 127.0.0.1:7100,127.0.0.1:7101,127.0.0.1:7102 -clients 64 -conns 4 -duration 5s

The transport's codec is hand-written varint encoding rather than encoding/gob or JSON: AppendEntries carries every replicated byte, so its encoding cost is on the critical path of every write.

Results: durable writes¶

Every write is committed by a majority with its log entry fsynced on each node before acknowledgment — the real durability contract.

Clients	Writes/s	p50	p99	Leader's average batch	WAL fsyncs per write (all 3 nodes)
1	78	12.7 ms	23.6 ms	1.0	3.00
8	235	34.0 ms	46.0 ms	2.0	0.99
64	1,504	39.3 ms	90.2 ms	12.9	0.16
256	2,385	99.2 ms	227.4 ms	22.7	0.09

(64 and 256 clients with 4 connections per server; see the next section for why that matters.)

One client pays the full latency of the protocol: the leader fsyncs its log, sends the entry, the follower fsyncs, replies, the leader commits and applies. Two fsyncs in sequence at ~4.4 ms each plus processing gives the 12.7 ms median — the cost of durability and replication, before any load.

With more clients, batching appears — the effect the simulation could not show. While the leader is inside one fsync, new requests queue in the proposer's channel; the next StartBatch takes all of them, and AppendEntries carries all of them to the followers, which also persist them with one fsync. The leader's average batch grew to 12.9 entries at 64 clients and 22.7 at 256, and fsyncs per write fell from 3 to 0.09.

A simple model that explains the table¶

In every run the cluster completed roughly the same number of commit rounds: 526–581 batches in 5 seconds, about 110 rounds per second. That is what two sequential 4.4 ms fsyncs plus a little processing allow. So:

throughput ≈ commit rounds per second × average batch size
           ≈ 110 × 12.9 ≈ 1,420/s   (measured 1,504/s at 64 clients)
           ≈ 110 × 22.7 ≈ 2,500/s   (measured 2,385/s at 256 clients)

Throughput is bought with batch size, and batch size is bought with latency: each request waits for the round in progress and then its own round, which is why p50 grows from 12.7 ms to 99 ms. Little's Law ties them together — 256 requests in flight / 2,385 per second ≈ 107 ms in the system (Load Testing and Tail Latency).

The plateau that was the benchmark's fault¶

The first run at 256 clients used one connection per server and reached only 1,209 writes/s — less than 64 clients — with p50 of 223 ms and an average batch of 9.6. Adding connections fixed it:

256 clients	Writes/s	p50	Leader's average batch
1 connection per server	1,209	223 ms	9.6
4 connections per server	2,385	99 ms	22.7

The cause: the protocol server allows 64 in-flight requests per connection — deliberate backpressure, so one client cannot consume unbounded server memory. With one connection, at most 64 of the 256 clients' requests were ever inside the server; the rest waited in the client library, invisible to the cluster's batching, adding pure queueing delay. The cluster was not saturated; the load generator was. Before believing a throughput ceiling, check that the client, the connection limits, and the load generator are not the bottleneck.

Results: without fsync¶

The same cluster with -nosync (the WAL skips fsync; process crashes still lose nothing, power loss loses everything not yet written back):

Clients	Writes/s	p50	p99	Leader's average batch
1	3,622	0.26 ms	0.41 ms	1.0
8	14,567	0.43 ms	4.21 ms	1.7
64	28,399	1.27 ms	24.6 ms	8.6
256 (1 connection)	28,935	4.94 ms	45.1 ms	14.1

With one client, removing fsync makes writes 46x faster (78/s → 3,622/s): on this machine, durability is almost the entire cost of a replicated write. Without fsync the limit becomes CPU and the implementation itself — encoding, syscalls, goroutine handoffs, the single apply goroutine, and the leader holding its mutex while writing the WAL — at roughly 28,000–29,000 writes/s on one laptop running all three servers and the load generator.

What would change on servers¶

These numbers are specific to this machine, and two things differ sharply on typical Linux servers:

• fsync latency. Datacenter NVMe drives with power-loss protection can acknowledge a flush in tens of microseconds, because their write cache is protected by capacitors. With 0.05 ms instead of 4.4 ms per fsync, commit rounds per second rise by an order of magnitude or more, and the same code would be limited by CPU much sooner. Consumer SSDs and network block storage (cloud volumes) are closer to the milliseconds measured here. Measure the fsync latency of the disks you deploy on before estimating anything.
• Network latency. Here all three processes share one machine: a round trip is microseconds. Across availability zones, each commit round adds the inter-zone RTT (often 0.5–2 ms), and across regions, tens of milliseconds — at which point the network, not the disk, sets the commit rate, and batching matters even more.

What would make this implementation faster¶

Measured bottlenecks, in the order they would be attacked:

1. Write the leader's WAL outside the node mutex, in parallel with sending entries to followers. Today StartBatch holds the Raft mutex during the leader's fsync, so heartbeats and replies wait behind the disk. Raft only needs a majority of durable copies, and the leader may count itself once its own write completes — etcd and TiKV overlap the two.
2. Pipeline commit rounds: the leader can start fsyncing the next batch while the previous one is still being acknowledged.
3. Apply in batches: the state machine applies entries one at a time; one kv.Batch per Raft batch would cut its writes by the batch size.
4. Group commit in the state machine's own storage is already implicit (unsynced writes), and correct because the Raft log is the durability source of truth.

Common pitfall¶

Benchmarking a replicated store on a laptop and quoting the number. A team measures 25,000 writes/s on a developer machine with fsync disabled "because it's just a test", or with fsync on a filesystem that ignores it (some container overlay setups, tmpfs, macOS's raw fsync), and sizes production capacity on it. In production, every write waits for real flushes on real disks across real network links, and capacity is an order of magnitude lower — or, on servers with power-loss-protected NVMe, much higher than the laptop's durable number. A throughput figure for a durable system is meaningless without three facts next to it: whether every acknowledged write was flushed, the fsync latency of the device, and the network round-trip time between replicas.

Review questions¶

1. Why did leader-side batching appear on real disks but not in the simulation?
2. Using the commit-round model, estimate throughput if each fsync took 0.1 ms instead of 4.4 ms. What would then become the limit?
3. Why did 256 clients on one connection per server reach lower throughput than 64 clients, and how did the measurement show it was the benchmark's fault?
4. Why is p50 latency at 256 clients about 100 ms, and how does Little's Law confirm it?
5. Why does holding the Raft mutex during the leader's fsync limit throughput, and what do production implementations do instead?