A few bumps on the road to HTTP/2

I had recently blogged about making concurrent REST calls at scale using HTTP. I had observed that, thanks to async I/O and/or virtual threads, it's feasible to make large numbers of requests in parallel. But while threads are cheap, there is typically another limiting factor: the HTTP connection pool. This pool serves as a semaphore: if the pool is empty you'll have to wait. I've seen default pool configurations that have even tighter limits on connections to the same host, which is problematic for internal service-to-service calls.

On the other hand, the HTTP/2 protocol offers an end run around the pool: requests are multiplexed onto a single connection, and you get responses back in any order. HTTP/2 seems to promise some serious performance benefits. We could send numerous requests without the overhead of setting up and tearing down a new connection every time, and the protocol is more compact and requests can processed in parallel. Unfortunately, I found some gotchas while experimenting with HTTP/2 requests which I'm going to note here for future reference.

Virtual threads: the limits of infinity

Java threads were expensive at scale. When your server needs to handle many requests concurrently, and each request needed a thread, the number of threads spawned could be performance-prohibitive. This led to all sorts of thread-conserving measures. The default server worker pool limit for Spring Boot was just 200, which meant you could only process 200 requests at a time. In desperation, people may switch to Reactive programming to eliminate the thread-per-request association, but the work is still scheduled off thread pools that do the real work. 

With Java 21, virtual threads are now an officially supported feature. This means threads are suddenly very cheap, blowing away those assumptions that led to thread pooling and Reactive programming. In fact, Java language architect Brian Goetz boldly predicted the death of Reactive programming (see link below). But oxymoronic as it may sounds, we should know the limits of infinite threads.

Making concurrent HTTP requests in Java

There may be occasions when you need to make multiple HTTP requests. The easy way is to simply make one request after another in sequence. If that's all you need, there is no need to read further. This will only complicate your life. Go away. Shoo.

The hard way to do this is to make multiple requests concurrently. Generally at scale (otherwise, why bother?). That's what I will play with in this post. Most of the time spent in a HTTP request is waiting for the response, so we want to get those requests all out at once, then wait for their responses. The typical pattern is to use async requests.

Test scenario:

• Make 1000 concurrent HTTP calls to a REST endpoint
• For each request, the server will wait 3 seconds before responding, simulating "work".

The results of my experimentation follows.