Virtual threads: the limits of infinity

Java threads were expensive at scale. When your server needs to handle many requests concurrently, and each request needed a thread, the number of threads spawned could be performance-prohibitive. This led to all sorts of thread-conserving measures. The default server worker pool limit for Spring Boot was just 200, which meant you could only process 200 requests at a time. In desperation, people may switch to Reactive programming to eliminate the thread-per-request association, but the work is still scheduled off thread pools that do the real work. 

With Java 21, virtual threads are now an officially supported feature. This means threads are suddenly very cheap, blowing away those assumptions that led to thread pooling and Reactive programming. In fact, Java language architect Brian Goetz boldly predicted the death of Reactive programming (see link below). But oxymoronic as it may sounds, we should know the limits of infinite threads.

We have always been able to increase the number of server threads. Java servers spend much of their time waiting for I/O, so CPU contention is generally not a problem. The original 200 thread limit was never a hard rule, and your containers can likely accommodate much larger thread pools. In Spring Boot, a single line would bump up the limit:

server.tomcat.threads.max=9999

With Java 21, we could instead just turn on virtual thread support for essentially infinite server threads:

spring.threads.virtual.enabled=true

"Infinite"? Really? We're told the JVM handles a million virtual threads on commodity hardware like your laptop. So yeah, we can treat them as unlimited. And like real threads, virtual threads have their own stack trace and support ThreadLocals. But when I tried cranking up the thread limits, I quickly discovered that threads are not the main limit, even without virtual threads. 

Connections are expensive too

Once I tried hitting a previously limited server with thousands of concurrent requests, I started seeing JDBC connection timeout errors. The requests were timing out waiting for a DB connection from the connection pool. A database transaction needs a dedicated DB connection for the duration of the transaction. You can imagine other scarce resources. It turns out a Java server rarely stands alone and tends to need things like databases and message brokers to do anything useful. There are also OS process limits that your JVM might be subject to.

There are certainly desirable aspects of virtual threads. After all, as Goetz puts it:

• Writing straight sequential code should be plan A
• Doing something weird should be plan B

Reactive programming was what he meant by plan B, but until recently it was the only way to get desirable throughput at scale. The programming model has a reputation for being hard to comprehend and debug, and should not be the default approach for most applications. Also, long-lived threads in thread pools play poorly with ThreadLocals, but it's hard to stop random 3rd party libraries from stuffing ThreadLocals into your shared threads. Virtual threads by comparison give you straightforward stack traces and should never be pooled. 

Despite their promise, I think virtual threads' full potential might still require some still require plan B-ish measures or at least some care to avoid holding on to limited, blocking resources. For some applications, you can go full throttle on those threads. For others, well, not so much.

 

Reference

Brian Goetz: "I think Project Loom is going to kill Reactive Programming" :Brian Goetz's 2021 interview on YouTube discussed in Reddit. His statement about Project Loom killing Reactive is about 20min in. In interesting video on Java in its own right.