NIO Servers, Async Responses, and Virtual Threads

Understand why Java server frameworks moved from one-thread-per-connection to NIO selectors and async response patterns, what problem each layer of that ceremony actually solves — and why virtual threads have since made most of it unnecessary to write by hand.

• Explaining why a server framework has both "selector" threads and "worker" threads instead of one undifferentiated pool.
• Sizing a REST server's worker thread pool correctly when most request time is spent blocked on a downstream call, not doing CPU work.
• Recognizing when reaching for an async/reactive response pattern solves a real problem versus adding real complexity that a simpler, blocking-style approach on a virtual thread would remove for free.

Blocking I/O doesn't scale past one thread per connection

Early Java I/O only had one mode: a thread that reads from a socket blocks until data arrives, with no way to check readiness without attempting the read. That forces a rigid one-to-one mapping between client connections and server threads. It's wasteful in a very specific, quantifiable way: 100 HTTP keep-alive clients with a 30-second think time and a 500ms request each have, on average, fewer than two requests in flight — yet a blocking server still needs 100 live threads just to hold those connections open.

NIO selectors decouple connections from threads

NIO's Selector lets one (or a few) threads watch many sockets at once and get notified only when a socket actually has data ready, instead of a thread blocking per-socket. A notified selector hands the ready client off to a worker thread pool sized for how many requests are actually concurrent, not how many connections happen to be open — the earlier 100-client example needs a worker pool of maybe 5-6 threads, not 100.

Sizing the worker pool means knowing what "blocked" means

If a request is purely CPU-bound, the worker pool ceiling is the core count, same as any other compute-bound workload. It gets more interesting once a request makes an outbound blocking call — say 900ms in a downstream database call plus 100ms of local processing, on a 2-core machine. That server can handle 20 requests/second of CPU work, but if a request arrives from each of 600 clients every 30 seconds, roughly 20 requests will be blocked on the database at any given moment — the worker pool needs at least 20 threads just to hold those blocked calls, even though only 2 threads' worth of CPU is ever actually busy. Make that downstream call non-blocking too, and the requirement collapses back to 2 threads, since nothing is parked waiting anymore.

Async responses: escaping the request-thread throttle by hand

When the downstream call can't be made non-blocking, frameworks like JAX-RS offer an escape hatch: defer the actual work to a second, independently-sized thread pool, freeing the request thread immediately:

java
ThreadPoolExecutor tpe = Executors.newFixedThreadPool(64);

@GET @Path("/asyncsleep")
public void sleepAsyncEndpoint(@QueryParam("delay") long delay, @Suspended final AsyncResponse ar) {
    tpe.execute(() -> {
        try { Thread.sleep(delay); } catch (InterruptedException ie) {}
        ar.resume("{\"sleepTime\": \"" + delay + "\"}");
    });
}

This works, but notice what it costs: a second explicitly-managed thread pool, sized by its own separate reasoning, purely to work around the first pool's throttle — none of this extra machinery exists because the business logic needs it.

• NIO's efficiency only holds if everything downstream is also non-blocking — a single blocking call anywhere in the chain reintroduces the "one parked thread per in-flight request" cost, just moved to whichever pool made that call.
• Async response patterns trade code complexity for thread economy — a second executor, careful bookkeeping about which pool does what, and a callback-shaped API purely to avoid the cost of a blocked platform thread, not because splitting the work across pools helps correctness or throughput on its own merits.
• Book vs today: virtual threads (Project Loom, JEP 444, finalized JDK 21) remove most of the reason for this entire dance. A virtual thread that blocks on Thread.sleep(), a socket read, or (since later virtual-thread-friendliness improvements) most blocking I/O releases the platform thread underneath it automatically — you write the sleep-endpoint example above as plain blocking code, on one virtual thread per request, with no selector tuning and no second executor, and get comparable scalability. Spring Boot 3.2+ can run Tomcat/servlet request handling on virtual threads with a single configuration flag, no code changes. One historical caveat worth knowing: virtual threads used to "pin" (block the underlying platform thread instead of releasing it) inside a synchronized block or certain native calls — that specific synchronized pinning case was fixed in JDK 24 (JEP 491). NIO itself isn't obsolete — it's still how the platform thread underneath a blocked virtual thread gets freed in the first place, and it still matters for genuinely huge numbers of long-idle connections (long-poll, SSE) — but hand-writing selector/async-response ceremony to size a typical REST server's thread pool is largely a solved problem today, not something to reach for by default.