Thread Pool & Web Server Concurrency¶

What is it: A thread pool is a fixed set of pre-created threads that pull work from a shared queue instead of spawning a new thread per task. The pool size is set at startup; threads are reused across many tasks.

Why: Creating an OS thread costs ~1ms and ~1MB of stack. At 10,000 requests/second, spawning a thread per request = 10,000 thread creations/second = unsustainable overhead. A pool of N threads amortizes this cost — threads are created once and reused indefinitely.

How a thread pool works: At startup: create N worker threads. Each thread blocks on the work queue (queue.take() — blocks until work is available). When a request arrives: enqueue the task. One idle worker dequeues it, processes it, returns to waiting. If all N workers are busy and the queue is full: reject the request (or block the producer).

Thread per request (blocking model — traditional Java/Spring): Each incoming request is handled by one dedicated thread from a pool. The thread synchronously performs all work: parse request → query DB → call external API → serialize response. Simple to code — sequential logic, familiar stack traces. Problem: a thread blocked waiting for a slow DB query holds a thread from the pool. With 200 threads and a DB query taking 100ms, max throughput = 200 threads / 100ms = 2,000 requests/second. Beyond that, requests queue up. At 10,000 concurrent slow-DB requests: need 10,000 threads → ~10GB stack RAM, ~10,000 context switches/second between sleeping threads.

Single-threaded event loop (Node.js): One thread handles all I/O via libuv (epoll on Linux). When I/O would block, register a callback and continue processing other events. When I/O completes, run the callback. Advantage: one thread handles 100,000+ concurrent connections. No thread-pool exhaustion. No context switch overhead. Problem: CPU-bound work (JSON parsing, image processing, crypto) blocks the entire event loop, freezing all connections. Fix: offload CPU work to a worker thread pool (worker_threads).

Goroutine per connection (Go): Create one goroutine per connection. Goroutines are cheap: ~2KB stack, ~1µs to create. 10,000 connections = 10,000 goroutines = ~20MB stack RAM. The GMP scheduler maps goroutines to OS threads. Goroutines blocking on I/O (via netpoller) do not hold OS threads — the OS thread is released for other goroutines. Combines: the simplicity of blocking-style code (goroutine writes sequential logic) with the efficiency of async I/O (OS threads never idle). Go's web server handles 100K+ concurrent connections on a modest number of OS threads (~GOMAXPROCS = number of CPU cores).

Nginx: Event-driven, one worker process per CPU core, epoll for I/O multiplexing. No threads — handles 10K connections per worker on ~4 processes (total 40K connections). Extremely efficient for serving static files and proxying. Not suitable for CPU-heavy work per request.

How a Go server handles 10K requests: - Accept: net.Listener.Accept() returns a new net.Conn. The runtime spawns a goroutine: go handleConn(conn). - Read: goroutine calls conn.Read() → parked (netpoller registers with epoll). OS thread continues other goroutines. - DB query: goroutine blocks on DB connection (from pool). Pool has 20 connections. 10K concurrent requests, each taking 5ms DB query → throughput = 20 connections / 5ms = 4,000 DB queries/second = bottleneck. OS threads: ~8 (GOMAXPROCS). No thread exhaustion. - The DB connection pool is the limiting factor, not the goroutine count.

Trade-offs: Thread pool (Java): simple mental model, poor resource usage under high concurrency with slow I/O. Event loop (Node.js): very efficient for I/O-bound, fragile under CPU load. Goroutine-per-connection (Go): best of both worlds — sequential code, efficient scheduling. Requires understanding goroutine leaks and pool sizing.