Runtime Scheduler (GMP)¶

The problem the scheduler solves¶

The OS schedules threads. A thread costs about 1–8 MB of reserved stack, a kernel object, and a few microseconds per context switch through the kernel (Process vs. Thread, OS Scheduling). A server with 100,000 connections cannot have 100,000 threads. Go's answer is to multiplex many cheap goroutines onto a few OS threads with a scheduler that runs in user space, inside the runtime. Knowing how it works explains goroutine costs, why blocking calls are (mostly) fine, what GOMAXPROCS really controls, and how to read scheduler latency in traces.

Measurements: Go 1.26.4, Apple M1 Pro (8 cores).

G, M, and P¶

The rule that ties them together: an M must hold a P to run Go code. So GOMAXPROCS bounds how many goroutines execute Go code simultaneously, not how many threads exist.

The scheduling loop¶

When the goroutine running on a P blocks, yields, or is preempted, the M calls schedule(), which looks for the next G in this order (findRunnable in runtime/proc.go):

1. Every 61st scheduling tick, check the global run queue first — otherwise a busy local queue could starve it.
2. runnext, then the local run queue.
3. The global run queue.
4. The netpoller (non-blocking): goroutines whose sockets became ready.
5. Steal half of another P's local queue (trying Ps in random order), including its timers.
6. If nothing: release the P, park the M (sleep on a futex).

runnext exists for communication patterns: when G1 sends on a channel and wakes G2, G2 goes into runnext and runs next on the same P, inheriting G1's remaining time slice and warm caches. That is why an unbuffered channel ping-pong between two goroutines costs about 245 ns per round trip (measured) rather than the cost of two OS thread wakeups.

Spinning threads. An M that finds no work spins briefly (checking queues, netpoller, stealing) before parking, and the runtime keeps at most GOMAXPROCS/2 spinning Ms. Spinning costs some CPU but avoids the latency of waking a parked thread for every new goroutine — the reason a mostly idle Go service shows a few percent of CPU in runtime.findRunnable and runtime.usleep in profiles.

Costs, measured¶

The last two rows are the most important fact on this page. A goroutine blocked in the Go runtime — on a channel, mutex, timer, or network socket — is parked: it holds no thread. A goroutine blocked in the kernel or in C holds its thread, and the runtime must create another thread to keep the P busy.

Blocking: syscalls, cgo, and the netpoller¶

Network I/O never blocks a thread. Sockets are put in non-blocking mode. When conn.Read would block (EAGAIN), the goroutine registers interest with the netpoller (epoll on Linux, kqueue on macOS/BSD), parks, and the M runs something else. When data arrives, the netpoller marks the goroutine runnable. Details in Network Programming in Go.

Blocking syscalls and cgo calls do. File I/O on regular files (read, write, fsync — page-cache I/O has no readiness notification), cgo calls, and some other syscalls block the thread:

For short syscalls (most read/write on cached files) the P is not handed off at all: sysmon retakes a P only after its goroutine has been in the syscall for at least one sysmon tick (≥ 20 µs), and when there is no other runnable work, only after 10 ms (retake in runtime/proc.go). For long syscalls the thread count grows. Consequences:

• Thousands of goroutines doing slow disk I/O, or calling a slow C library, create thousands of threads. The runtime aborts the process at 10,000 threads (runtime: program exceeds 10000-thread limit, adjustable with debug.SetMaxThreads). Bound such work with a semaphore or worker pool.
• DNS resolution through cgo (getaddrinfo) blocks threads; the pure-Go resolver (GODEBUG=netdns=go) uses the netpoller.

sysmon¶

sysmon is a runtime thread that runs without a P, waking every 20 µs to 10 ms (backing off when idle). It:

• retakes Ps from Ms stuck in syscalls (above);
• preempts goroutines running for more than 10 ms (forcePreemptNS);
• polls the network if no one has for 10 ms;
• triggers a forced GC if none ran for 2 minutes.

Preemption¶

Before Go 1.14, preemption was cooperative: a goroutine could only be descheduled at a function call (the stack-growth check in the prologue doubles as a preemption check). A tight loop without calls could hold its P forever and block the GC's stop-the-world phase.

Since Go 1.14 there is also asynchronous preemption: sysmon sends the thread a signal (SIGURG), the signal handler checks that the goroutine is at an async-safe point and, if so, injects a call to the scheduler. Preempted goroutines go to the global run queue.

Trace of 64 CPU-bound goroutines on 8 Ps (GODEBUG=schedtrace=500):

SCHED 510ms: gomaxprocs=8 idleprocs=0 threads=9 spinningthreads=0 needspinning=1 idlethreads=0 runqueue=41 [ 0 1 1 11 0 0 1 1 ] schedticks=[ 19 17 17 17 17 17 19 19 ]
SCHED 1018ms: gomaxprocs=8 idleprocs=0 threads=9 spinningthreads=0 needspinning=1 idlethreads=0 runqueue=47 [ 1 2 1 1 2 1 1 0 ] schedticks=[ 34 32 32 32 32 32 34 34 ]

Add ,scheddetail=1 to see every G and M. For anything beyond a quick look, use the execution tracer (Execution Tracer), which shows exactly when each goroutine was runnable versus running.

GOMAXPROCS in containers¶

GOMAXPROCS defaults to the number of usable CPUs. Before Go 1.25 that meant the host's CPU count, even in a container limited to 2 CPUs by a cgroup quota: a 64-core node gave a 2-CPU pod GOMAXPROCS=64. The 64 Ps ran 64 threads against a quota of 2 CPUs per period; the kernel's CFS bandwidth controller then throttled the whole container for the rest of each 100 ms period, causing latency spikes of tens of milliseconds that look like GC pauses but are not. The standard fix was go.uber.org/automaxprocs.

Since Go 1.25 the runtime reads the cgroup CPU limit on Linux at startup, sets GOMAXPROCS to it (rounded up, minimum 2), and re-checks periodically if the limit changes. Setting GOMAXPROCS explicitly (env var or runtime.GOMAXPROCS(n)) disables the automatic updates. Only the CPU limit is used, not the CPU request — a pod with only requests.cpu still gets the node's CPU count. Check what your service actually runs with: log runtime.GOMAXPROCS(0) at startup.

Goroutines are cheap, not free¶

• Memory: a parked goroutine costs its stack (≥ 2 KB; Go 1.19+ starts new goroutines at the average stack size observed so far) plus whatever it references. Measured: 5,000 idle TCP connections served by io.Copy goroutines used ~4 KB of stack each — and 33 KB of heap each, because io.Copy allocates a 32 KB buffer per call.
• Stack growth: when a goroutine needs more stack, the runtime allocates a 2x larger stack and copies the old one (adjusting pointers into it). Deep recursion in a hot goroutine shows up as runtime.morestack / runtime.copystack in profiles.
• Unbounded spawning turns overload into memory exhaustion: under a burst, go handle(req) for every request queues work in RAM instead of pushing back. Bound concurrency explicitly (semaphore, errgroup.SetLimit, worker pools).
• Leaks: a goroutine blocked forever is never collected — Goroutine Leaks.

runtime.LockOSThread¶

Pins the goroutine to its current M (and the M to it) until unlocked. Needed when the code relies on thread-local state outside Go: OpenGL/GUI main threads, some C libraries, Linux namespaces (setns) and per-thread credentials. If a goroutine exits while locked, its thread is destroyed rather than reused — which is deliberately used to discard a thread whose state was modified.

Common pitfall¶

Treating GOMAXPROCS as a concurrency knob. A team sees a service doing slow file I/O and raises GOMAXPROCS from 8 to 64 to "allow more concurrent requests". It changes nothing for I/O-bound work — goroutines blocked in syscalls already release their P — and makes CPU-bound phases worse: 64 Ps on 8 cores means more threads competing for the same cores, more context switches, and, under a cgroup quota, throttling. GOMAXPROCS should match the CPU the process can actually use. To change how much work runs concurrently, bound the work itself (worker pools, semaphores), and look at the tracer's "scheduler latency" to see whether goroutines are actually waiting for a P.

Review questions¶

1. Why can 10,000 goroutines sleep on 5 threads, while 200 goroutines blocked in cgo need about 200 threads?
2. In what order does findRunnable look for work, and why is the global queue checked first every 61 ticks?
3. What does runnext optimize, and which benchmark shows its effect?
4. How does asynchronous preemption work, and what could a tight loop do before Go 1.14?
5. A 2-CPU-limit pod on a 64-core node built with Go 1.22 has periodic 50 ms latency spikes. What is happening, and what changed in Go 1.25?