Designing a Binary Protocol¶
Why build one¶
Redis, Kafka, PostgreSQL, Cassandra, NATS, and every database driver speak custom protocols over TCP. Designing one end to end — framing, encoding, request multiplexing, backpressure, versioning — exercises almost everything in Network Programming in Go. This page builds a small, complete, tested request/response protocol, and shows a real concurrency bug that its first version had.
Measurements: Go 1.26.4, Apple M1 Pro, loopback.
Framing: TCP is a byte stream¶
TCP delivers bytes, not messages. One Write of 100 bytes may arrive
as one Read of 100, two Reads of 50, or merged with the next
message. A protocol must define message boundaries. The options:
| Framing | Example | Trade-off |
|---|---|---|
| Delimiter | Redis RESP lines (\r\n), HTTP/1 headers |
Human-readable; payload must escape or length-prefix the delimiter; scanning is slower |
| Length prefix | Kafka, PostgreSQL, gRPC (inside HTTP/2 frames) | One read for the header, one for the body; trivially fast; needs a maximum size check |
| Fixed size | Some market-data feeds | Simplest; wastes space or can't carry variable data |
Length-prefixed binary framing is the default for performance-oriented systems. The frame for this protocol:
0 4 12 13
+-------+---------------+----+--------------------+
| len | request id | op | payload |
| u32 | u64 | u8 | len - 9 bytes |
+-------+---------------+----+--------------------+
big-endian (network byte order)
const (
headerLen = 4 + 8 + 1
MaxFrameSize = 1 << 20 // the length field is untrusted input
)
type Frame struct {
ID uint64
Op uint8
Payload []byte
}
func AppendFrame(dst []byte, f Frame) []byte {
dst = binary.BigEndian.AppendUint32(dst, uint32(8+1+len(f.Payload)))
dst = binary.BigEndian.AppendUint64(dst, f.ID)
dst = append(dst, f.Op)
return append(dst, f.Payload...)
}
// ReadFrame reads one frame. buf is reused for the payload when it is
// large enough; the returned payload aliases it until the next call.
func ReadFrame(r *bufio.Reader, buf []byte) (Frame, []byte, error) {
var hdr [headerLen]byte
if _, err := io.ReadFull(r, hdr[:]); err != nil {
return Frame{}, buf, err // io.EOF here = clean close between frames
}
n := binary.BigEndian.Uint32(hdr[0:4])
if n < 9 {
return Frame{}, buf, fmt.Errorf("proto: bad frame length %d", n)
}
if n-9 > MaxFrameSize {
return Frame{}, buf, ErrFrameTooLarge // check BEFORE allocating
}
plen := int(n - 9)
if cap(buf) < plen {
buf = make([]byte, plen)
}
buf = buf[:plen]
if _, err := io.ReadFull(r, buf); err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF // EOF inside a frame is truncation
}
return Frame{}, buf, err
}
return Frame{ID: binary.BigEndian.Uint64(hdr[4:12]), Op: hdr[12], Payload: buf}, buf, nil
}
Details that matter:
io.ReadFull, never a bareRead: aReadmay return fewer bytes than asked for.- Validate the length before allocating. Without
MaxFrameSize, four bytesff ff ff fffrom any client make the server allocate 4 GB — a one-packet denial of service. The test suite sends exactly that frame and expectsErrFrameTooLarge. - Distinguish clean EOF from truncation: EOF before a header is a normal close; EOF inside a frame is an error.
- Append-style encoding (
AppendFrame(dst, f)) and a reusable read buffer keep the hot path allocation-free (Reducing Allocations). - Byte order: big-endian is the network convention; little-endian
is what CPUs use natively and what most storage formats choose.
Either is fine if documented —
encoding/binarymakes both cheap. Use varints (binary.AppendUvarint) for small, usually-small integers; fixed width when you need random access or to patch the value later (as with the length).
Multiplexing: many requests, one connection¶
A protocol where the client sends a request and waits for its response before sending the next one uses one round trip per request. Measured on loopback with this protocol: 40 µs per call sequentially — the cost of two wakeups, four syscalls, and the scheduler on each side.
The request ID lets many requests share a connection concurrently (multiplexing, as in HTTP/2 and most database protocols), and lets the server answer out of order — a slow request does not block the fast ones behind it (no head-of-line blocking at the application layer). The client:
type Client struct {
conn net.Conn
wmu sync.Mutex // serializes writers
w *bufio.Writer
wbuf []byte
mu sync.Mutex // protects pending, nextID, err
nextID uint64
pending map[uint64]chan Frame
err error
}
func (c *Client) Call(ctx context.Context, op uint8, payload []byte) ([]byte, error) {
ch := make(chan Frame, 1) // buffered: the reader never blocks on a caller that gave up
c.mu.Lock()
if c.err != nil {
c.mu.Unlock()
return nil, ErrClosed
}
c.nextID++
id := c.nextID
c.pending[id] = ch
c.mu.Unlock()
c.wmu.Lock()
c.wbuf = AppendFrame(c.wbuf[:0], Frame{ID: id, Op: op, Payload: payload})
_, err := c.w.Write(c.wbuf)
if err == nil {
err = c.w.Flush()
}
c.wmu.Unlock()
if err != nil {
c.conn.Close() // the reader loop fails all pending calls
return nil, err
}
select {
case f, ok := <-ch:
if !ok {
return nil, ErrClosed
}
return f.Payload, nil
case <-ctx.Done():
c.mu.Lock()
delete(c.pending, id) // a late response will be discarded
c.mu.Unlock()
return nil, ctx.Err()
}
}
A single reader goroutine routes responses by ID:
func (c *Client) readLoop() {
r := bufio.NewReader(c.conn)
var buf []byte
for {
f, _, err := ReadFrame(r, buf[:0])
if err != nil {
c.fail(err) // close every pending channel, remember the error
return
}
f.Payload = append([]byte(nil), f.Payload...) // caller keeps it: copy out of the reused buffer
c.mu.Lock()
ch := c.pending[f.ID]
delete(c.pending, f.ID)
c.mu.Unlock()
if ch != nil {
ch <- f
}
}
}
With 64 goroutines calling concurrently over the same single
connection, throughput rose from 25,000 calls/s (sequential) to about
143,000 calls/s (7.0 µs per call): the same connection now carries
many requests per round trip, and the buffered writers on both sides
coalesce them into fewer syscalls. The test suite checks that 10% of
requests sleeping 50 ms on the server do not delay the others, and that
a call whose context expires returns context.DeadlineExceeded without
leaking its pending entry.
The server: concurrency, backpressure, and flushing¶
func serveConn(conn net.Conn, h Handler) {
defer conn.Close()
r := bufio.NewReaderSize(conn, 64<<10)
w := bufio.NewWriterSize(conn, 64<<10)
var wmu sync.Mutex
var waiting atomic.Int32 // responses computed but not yet written
sem := make(chan struct{}, 64) // at most 64 in-flight requests per connection
var wg sync.WaitGroup
defer wg.Wait()
for {
conn.SetReadDeadline(time.Now().Add(2 * time.Minute)) // drop idle connections
f, _, err := ReadFrame(r, nil)
if err != nil {
return
}
sem <- struct{}{} // backpressure: stop reading when 64 are in flight
wg.Add(1)
go func() {
defer func() { <-sem; wg.Done() }()
resp := h(f.Op, f.Payload)
waiting.Add(1)
wmu.Lock()
defer wmu.Unlock()
w.Write(AppendFrame(nil, Frame{ID: f.ID, Op: f.Op, Payload: resp}))
if waiting.Add(-1) == 0 { // last one in a burst flushes for everyone
w.Flush()
}
}()
}
}
- Backpressure comes from the semaphore: when 64 requests are in flight, the reader stops reading; the kernel's receive buffer fills; TCP flow control tells the client to stop sending. Memory per connection is bounded no matter how fast the client sends.
- Flush coalescing: a burst of responses is written with one syscall — whoever writes the last pending response flushes.
The bug in the first version¶
The first version decided whether to flush with if len(sem) == 1
("I am the only request in flight"). All unit tests passed. Under the
parallel benchmark the client hung, and the server eventually logged
i/o timeout from its idle read deadline. The interleaving:
- Requests A and B finish at about the same time. A takes
wmu, writes, seeslen(sem) == 2(B still holds its slot), and does not flush. - A unlocks
wmu; its deferred<-semhas not run yet. - B takes
wmu, writes, and seeslen(sem) == 2again (A has not released its slot yet) — does not flush either. - Both responses sit in the
bufio.Writer. The client waits for them; the server waits for new requests. Nothing moves.
len(sem) measured the wrong thing (requests in flight, released
after the write) and was read without synchronization relative to the
releases. The fix counts exactly the responses that are ready but not
yet written, incremented before taking the lock and decremented under
it: whoever brings the count to zero is provably the last writer in the
burst, so some writer always flushes. The race detector did not report
this — there was no data race, only a logic race — but a parallel
benchmark found it in seconds. Always run concurrent code under a
concurrent load test, not only unit tests.
Versioning and evolution¶
A protocol outlives its first client. Plan for change from the first frame:
- A handshake at connection start: magic bytes (to fail fast on a wrong port), protocol version, feature flags. Negotiate down to the highest common version.
- Leave room: an op code space with unknown ops answered by an error (not a disconnect), reserved flag bits that must be zero.
- Payload encoding with schema evolution: protobuf (gRPC and Protocol Buffers) or a hand-rolled TLV (tag-length-value) format, where readers skip unknown tags. Never rely on struct layout matching across versions.
- Error responses as frames, with a code and message, so a request can fail without killing the connection.
Testing a protocol¶
- Round-trip encode/decode tests, including empty payloads, maximum size, and every op.
- Fuzzing the decoder — it parses untrusted input:
func FuzzReadFrame(f *testing.F) {
f.Add(AppendFrame(nil, Frame{ID: 1, Op: 1, Payload: []byte("x")}))
f.Fuzz(func(t *testing.T, data []byte) {
r := bufio.NewReader(bytes.NewReader(data))
for {
if _, _, err := ReadFrame(r, nil); err != nil {
return // must return an error, never panic or allocate wildly
}
}
})
}
- Concurrency tests with many goroutines, slow handlers, cancellations,
and connection kills mid-request, all under
-race. - A parallel benchmark, which is what found the flush bug above.
Common pitfall¶
Assuming one Read returns one message. A first protocol
implementation does n, _ := conn.Read(buf); handle(buf[:n]). On a
developer laptop, client and server exchange small messages over
loopback and every Read happens to return exactly one message, so
all tests pass. In production, under load and across a real network,
messages arrive split across reads and merged with each other; the
server parses half a message as a whole one, misreads the next
message's length as payload, and from then on every frame is garbage.
TCP never preserved boundaries — the tests just never exercised it. Use
io.ReadFull with explicit framing, and test with a reader that
returns one byte at a time (testing/iotest.OneByteReader) to force
every split.
Review questions¶
- Why does a protocol over TCP need framing, and why is length-prefixing usually preferred over delimiters?
- What attack does
MaxFrameSizeprevent, and why must the check happen before allocation? - How does a request ID enable both multiplexing and out-of-order responses?
- Where does backpressure come from in the server, and how does it reach the client?
- Why did
len(sem) == 1fail as a flush condition, and why does thewaitingcounter work?