A few bumps on the road to HTTP/2

I had recently blogged about making concurrent REST calls at scale using HTTP. I had observed that, thanks to async I/O and/or virtual threads, it's feasible to make large numbers of requests in parallel. But while threads are cheap, there is typically another limiting factor: the HTTP connection pool. This pool serves as a semaphore: if the pool is empty you'll have to wait. I've seen default pool configurations that have even tighter limits on connections to the same host, which is problematic for internal service-to-service calls.

On the other hand, the HTTP/2 protocol offers an end run around the pool: requests are multiplexed onto a single connection, and you get responses back in any order. HTTP/2 seems to promise some serious performance benefits. We could send numerous requests without the overhead of setting up and tearing down a new connection every time, and the protocol is more compact and requests can processed in parallel. Unfortunately, I found some gotchas while experimenting with HTTP/2 requests which I'm going to note here for future reference.

Virtual threads: the limits of infinity

Java threads were expensive at scale. When your server needs to handle many requests concurrently, and each request needed a thread, the number of threads spawned could be performance-prohibitive. This led to all sorts of thread-conserving measures. The default server worker pool limit for Spring Boot was just 200, which meant you could only process 200 requests at a time. In desperation, people may switch to Reactive programming to eliminate the thread-per-request association, but the work is still scheduled off thread pools that do the real work. 

With Java 21, virtual threads are now an officially supported feature. This means threads are suddenly very cheap, blowing away those assumptions that led to thread pooling and Reactive programming. In fact, Java language architect Brian Goetz boldly predicted the death of Reactive programming (see link below). But oxymoronic as it may sounds, we should know the limits of infinite threads.

Making concurrent HTTP requests in Java

There may be occasions when you need to make multiple HTTP requests. The easy way is to simply make one request after another in sequence. If that's all you need, there is no need to read further. This will only complicate your life. Go away. Shoo.

The hard way to do this is to make multiple requests concurrently. Generally at scale (otherwise, why bother?). That's what I will play with in this post. Most of the time spent in a HTTP request is waiting for the response, so we want to get those requests all out at once, then wait for their responses. The typical pattern is to use async requests.

Test scenario:

• Make 1000 concurrent HTTP calls to a REST endpoint
• For each request, the server will wait 3 seconds before responding, simulating "work".

The results of my experimentation follows.

Is Scrum obsolete?

The blog title may be mischievous, but that thought did run through my mind when I last interviewed for a job. I had spoken to 4 companies at the time, and 3 out of those 4 places deployed to production at least daily. If Scrum is a series of time-boxed sprints or iterations what meaning does it have when you ship stuff daily? I'm not saying Scrum should die, but I do wonder whether it deserves to be the default choice at so many software development organizations these days. It certainly did not seem to be a good fit for those doing daily deploys. Should we assume it's a good fit for most places?

Annotations, for better or for worse

A recent provocatively titled blog post bashing Java annotations kicked up an amusing amount of controversy. What I find notable about the post and its responses is that there doesn’t seem to be much agreement on annotations’ place in Java. My own take on this is that many popular usages of annotations are questionable, but we’re stuck with them.

Git, reconsidered

I’ve expressed my skepticism before that DVCS (distributed version control systems) like Git offer a lot of benefits over traditional version control systems like Subversion. I’ve had the opportunity to use Git exclusively for quite a few months now, and have had the opportunity to appreciate its benefits. My opinion on this matter is somewhat tempered now. Here are my current thoughts on this.

Don't swallow Hibernate exceptions!

Every once in a while, I see a code pattern that makes me go "hmmm". Once instance of this is code that catches a Hibernate exception within a transaction and swallows it before doing some compensating action. Normally, the usual pattern is that if Hibernate throws an exception, you let it propagate past the transaction boundary, rolling back the transaction, before handling the error. You violate the established practice at your peril.

What peril? Well, Hibernate's reference manual says the following (section 13.2.3 on Exception Handling):

If the Session throws an exception, including any SQLException, immediately rollback the database transaction, call Session.close() and discard the Session instance. Certain methods of Session will not leave the session in a consistent state. No exception thrown by Hibernate can be treated as recoverable.

I have emphasized the relevant text in bold. If you're catching the exception to do some other action within the same transaction, there is a chance you're corrupting data. If you're lucky, at some point the transaction got marked rollback-only and Hibernate will not let you commit the mischief. But the Hibernate folks are saying that all bets are off if you swallow that exception.

If you think about it, this sort of pattern makes no sense in a transactional setting. When you demarcate an operation as being transactional, you are saying "this operation either succeeds entirely, or it fails entirely as if it never happened". Atomicity is a guarantee of a transaction. So if Hibernate throws an exception it means the operation has failed, and you're mucking with the meaning of a transaction if you attempt a compensating action. If you want to log an error, retry or otherwise record a failure, you are free to do it outside of that transaction.

Dynamically changing a log4j configuration

For most of my career, I've used log4j as my logging back end. Whether during development, testing or production, I've encountered situations where I need to change the log level of a running web app on Tomcat. Restarting the application is often not feasible: either the situation I'm trying to diagnose would be wiped out, or other users would be impacted.  Here's an easy tweak, if you're using Spring, to dynamically change a log4j configuration. With this addition to your web.xml, log4j will pick up any changes you make to your log4j.properties file within 5 seconds:

<listener> 
    <listener-class>org.springframework.web.util.Log4jConfigListener</listener-class> 
</listener>
<context-param> 
    <param-name>log4jConfigLocation</param-name> 
    <param-value>WEB-INF/classes/log4j.properties</param-value>
</context-param> 
<context-param> 
    <param-name>log4jRefreshInterval</param-name> 
    <param-value>5000</param-value> 
</context-param>

The log4jRefreshInterval parameter is the key. Without that, the log4j configuration won't reload. While this solution is documented separately elsewhere, I wanted to call attention to this tweak because it really deserves wider knowledge.
There are, of course, other possible approaches such as writing your own custom JMX MBean, a custom servlet or using JRebel. These have their own advantages and disadvantages, but I feel the Spring Log4jConfigListener approach is the easiest and most straighforward for a developer.

Is your language strongly, weakly, statically or dynamically typed?

Would you describe a programming language to be strongly typed, weakly typed, statically typed or dynamically typed? People often use the terms "strongly typed" and "statically typed" interchangeably, and likewise "weakly typed" and "dynamically typed" are often used to mean the same thing. I think this imprecision of terminology is unfortunate. That is, the definition of strong v.s. weak typing is so vague that I would claim it's possible to see them as completely orthogonal concepts.

Yammer/Chatter as an alternative to your daily scrum

The daily standup meeting, also known as the daily scrum, did not work very well for me. One day, we looked at each other and asked "does anyone actually want to do this?" Nobody did, so that was the end of the daily scrum. The intention is admirable: you want team members to get a status update and know what's going on with the rest of the team. But the daily scrum seems the wrong way to accomplish that. There must be better ways.

Don’t forget to index your Oracle foreign keys

This is another note-to-myself blog post. Like PostgreSQL, Oracle does not create indexes on foreign key columns. This can lead to both poor performance and deadlocks. This is because when the parent table’s column where the foreign key points to is updated Oracle needs to verify the child table’s FK constraints. In the absence of an index, this means a full table scan and a table lock. Normally foreign keys point to primary keys in the parent table, and you rarely need to change a primary key, so you’d think this should not happen often. But deleting rows from the parent table also has the potential of violating FK constraints. So operations that delete rows in the parent table can result in deadlocks in some cases.
Anyway, here’s the Oracle script to identify unindexed foreign keys:

JmsTemplate is not evil

A while back, I watched a presentation on JMS messaging where the presenter (Mark Richards) declared that Spring’s JmsTemplate is “evil,” and he had the benchmarks to prove it. Since Mark was kind enough to provide the source code (based on ActiveMQ), I decided to dig a little deeper to determine the cause. It turns out that the apparent poor performance of JmsTemplate was not due to anything intrinsic to this class, and you can indeed get message sending performance with it comparable to using the standard JMS API. It’s not evil. Really.

Read on if you want the details.