A few bumps on the road to HTTP/2

I had recently blogged about making concurrent REST calls at scale using HTTP. I had observed that, thanks to async I/O and/or virtual threads, it's feasible to make large numbers of requests in parallel. But while threads are cheap, there is typically another limiting factor: the HTTP connection pool. This pool serves as a semaphore: if the pool is empty you'll have to wait. I've seen default pool configurations that have even tighter limits on connections to the same host, which is problematic for internal service-to-service calls.

On the other hand, the HTTP/2 protocol offers an end run around the pool: requests are multiplexed onto a single connection, and you get responses back in any order. HTTP/2 seems to promise some serious performance benefits. We could send numerous requests without the overhead of setting up and tearing down a new connection every time, and the protocol is more compact and requests can processed in parallel. Unfortunately, I found some gotchas while experimenting with HTTP/2 requests which I'm going to note here for future reference.

Virtual threads: the limits of infinity

Java threads were expensive at scale. When your server needs to handle many requests concurrently, and each request needed a thread, the number of threads spawned could be performance-prohibitive. This led to all sorts of thread-conserving measures. The default server worker pool limit for Spring Boot was just 200, which meant you could only process 200 requests at a time. In desperation, people may switch to Reactive programming to eliminate the thread-per-request association, but the work is still scheduled off thread pools that do the real work. 

With Java 21, virtual threads are now an officially supported feature. This means threads are suddenly very cheap, blowing away those assumptions that led to thread pooling and Reactive programming. In fact, Java language architect Brian Goetz boldly predicted the death of Reactive programming (see link below). But oxymoronic as it may sounds, we should know the limits of infinite threads.

Making concurrent HTTP requests in Java

There may be occasions when you need to make multiple HTTP requests. The easy way is to simply make one request after another in sequence. If that's all you need, there is no need to read further. This will only complicate your life. Go away. Shoo.

The hard way to do this is to make multiple requests concurrently. Generally at scale (otherwise, why bother?). That's what I will play with in this post. Most of the time spent in a HTTP request is waiting for the response, so we want to get those requests all out at once, then wait for their responses. The typical pattern is to use async requests.

Test scenario:

• Make 1000 concurrent HTTP calls to a REST endpoint
• For each request, the server will wait 3 seconds before responding, simulating "work".

The results of my experimentation follows.

Is Scrum obsolete?

The blog title may be mischievous, but that thought did run through my mind when I last interviewed for a job. I had spoken to 4 companies at the time, and 3 out of those 4 places deployed to production at least daily. If Scrum is a series of time-boxed sprints or iterations what meaning does it have when you ship stuff daily? I'm not saying Scrum should die, but I do wonder whether it deserves to be the default choice at so many software development organizations these days. It certainly did not seem to be a good fit for those doing daily deploys. Should we assume it's a good fit for most places?

Annotations, for better or for worse

A recent provocatively titled blog post bashing Java annotations kicked up an amusing amount of controversy. What I find notable about the post and its responses is that there doesn’t seem to be much agreement on annotations’ place in Java. My own take on this is that many popular usages of annotations are questionable, but we’re stuck with them.

Git, reconsidered

I’ve expressed my skepticism before that DVCS (distributed version control systems) like Git offer a lot of benefits over traditional version control systems like Subversion. I’ve had the opportunity to use Git exclusively for quite a few months now, and have had the opportunity to appreciate its benefits. My opinion on this matter is somewhat tempered now. Here are my current thoughts on this.

Don't swallow Hibernate exceptions!

Every once in a while, I see a code pattern that makes me go "hmmm". Once instance of this is code that catches a Hibernate exception within a transaction and swallows it before doing some compensating action. Normally, the usual pattern is that if Hibernate throws an exception, you let it propagate past the transaction boundary, rolling back the transaction, before handling the error. You violate the established practice at your peril.

What peril? Well, Hibernate's reference manual says the following (section 13.2.3 on Exception Handling):

If the Session throws an exception, including any SQLException, immediately rollback the database transaction, call Session.close() and discard the Session instance. Certain methods of Session will not leave the session in a consistent state. No exception thrown by Hibernate can be treated as recoverable.

I have emphasized the relevant text in bold. If you're catching the exception to do some other action within the same transaction, there is a chance you're corrupting data. If you're lucky, at some point the transaction got marked rollback-only and Hibernate will not let you commit the mischief. But the Hibernate folks are saying that all bets are off if you swallow that exception.

If you think about it, this sort of pattern makes no sense in a transactional setting. When you demarcate an operation as being transactional, you are saying "this operation either succeeds entirely, or it fails entirely as if it never happened". Atomicity is a guarantee of a transaction. So if Hibernate throws an exception it means the operation has failed, and you're mucking with the meaning of a transaction if you attempt a compensating action. If you want to log an error, retry or otherwise record a failure, you are free to do it outside of that transaction.

Dynamically changing a log4j configuration

For most of my career, I've used log4j as my logging back end. Whether during development, testing or production, I've encountered situations where I need to change the log level of a running web app on Tomcat. Restarting the application is often not feasible: either the situation I'm trying to diagnose would be wiped out, or other users would be impacted.  Here's an easy tweak, if you're using Spring, to dynamically change a log4j configuration. With this addition to your web.xml, log4j will pick up any changes you make to your log4j.properties file within 5 seconds:

<listener> 
    <listener-class>org.springframework.web.util.Log4jConfigListener</listener-class> 
</listener>
<context-param> 
    <param-name>log4jConfigLocation</param-name> 
    <param-value>WEB-INF/classes/log4j.properties</param-value>
</context-param> 
<context-param> 
    <param-name>log4jRefreshInterval</param-name> 
    <param-value>5000</param-value> 
</context-param>

The log4jRefreshInterval parameter is the key. Without that, the log4j configuration won't reload. While this solution is documented separately elsewhere, I wanted to call attention to this tweak because it really deserves wider knowledge.
There are, of course, other possible approaches such as writing your own custom JMX MBean, a custom servlet or using JRebel. These have their own advantages and disadvantages, but I feel the Spring Log4jConfigListener approach is the easiest and most straighforward for a developer.

Is your language strongly, weakly, statically or dynamically typed?

Would you describe a programming language to be strongly typed, weakly typed, statically typed or dynamically typed? People often use the terms "strongly typed" and "statically typed" interchangeably, and likewise "weakly typed" and "dynamically typed" are often used to mean the same thing. I think this imprecision of terminology is unfortunate. That is, the definition of strong v.s. weak typing is so vague that I would claim it's possible to see them as completely orthogonal concepts.

Yammer/Chatter as an alternative to your daily scrum

The daily standup meeting, also known as the daily scrum, did not work very well for me. One day, we looked at each other and asked "does anyone actually want to do this?" Nobody did, so that was the end of the daily scrum. The intention is admirable: you want team members to get a status update and know what's going on with the rest of the team. But the daily scrum seems the wrong way to accomplish that. There must be better ways.

Don’t forget to index your Oracle foreign keys

This is another note-to-myself blog post. Like PostgreSQL, Oracle does not create indexes on foreign key columns. This can lead to both poor performance and deadlocks. This is because when the parent table’s column where the foreign key points to is updated Oracle needs to verify the child table’s FK constraints. In the absence of an index, this means a full table scan and a table lock. Normally foreign keys point to primary keys in the parent table, and you rarely need to change a primary key, so you’d think this should not happen often. But deleting rows from the parent table also has the potential of violating FK constraints. So operations that delete rows in the parent table can result in deadlocks in some cases.
Anyway, here’s the Oracle script to identify unindexed foreign keys:

JmsTemplate is not evil

A while back, I watched a presentation on JMS messaging where the presenter (Mark Richards) declared that Spring’s JmsTemplate is “evil,” and he had the benchmarks to prove it. Since Mark was kind enough to provide the source code (based on ActiveMQ), I decided to dig a little deeper to determine the cause. It turns out that the apparent poor performance of JmsTemplate was not due to anything intrinsic to this class, and you can indeed get message sending performance with it comparable to using the standard JMS API. It’s not evil. Really.

Read on if you want the details.

Multithreaded programming is easy!

Multithreaded programming is supposed to be perilous. Even for experienced practitioners who are familiar with the concepts of multithreading, the cognitive load of this work is supposed to be too taxing for the human mind to handle reliably. Yet your typical server-side enterprise Java application is inherently multithreaded. Whether deployed on a full Java EE application server or something lighter like Tomcat, your typical application runs in a shared process space with hundreds or thousands of threads. Somehow, we manage to release and keep in production real working applications. And truth be told, in all my years as a programmer very few bugs I encounter actually has anything to do with multithreading logic.

Plight of the introvert programmer

I have grumbled before about the trend away from private offices towards cubicles and open plan offices. At the same time, there are people who genuinely prefer these offices, which I considered inhumane. Then I read a book: "Quiet: the power of introverts in a world that can't stop talking", by Susan Cain. It helped clarify the differences between introverts and extroverts, the rise of the latter, and the implications all this has on workplace productivity.

Introvert, extrovert

Cain's definition of introversion and extroversion is pretty much what you would expect if you are familiar with Myers-Briggs. She emphasizes that "introvert" does not mean shy nor anti-social. Rather, an introvert tends to focus on the inner world of thought and feeling, enjoys solitude, dislike conflict and focus well. An extrovert would be the opposite, focusing on the exterior world and social interaction, are comfortable with conflict and enjoy multitasking. Another way to look at this is to say that introverts are more sensitive to outside stimulation, such as noise or people, and thrive at lower levels of such stimulation. Conversely, an extrovert needs a greater intensity of stimulation to feel "right". An introvert needs solitude to recharge after experiencing elevated levels of stimulation. An extrovert is the opposite: he recharges with social interaction.

The difference between introverts and extroverts -- keep in mind this is a continuum rather than a binary distinction -- in terms of optimal stimulation levels helps explain work environment preferences. An introvert works best in a relatively quiet, distraction-free environment, such as a private office. An extrovert is energized by the constant social interaction of pair programming and the bustling activity of an open plan office.

AppFog: the GMail of PaaS

AppFog sent out an email announcement a couple of days ago that made my jaw drop. AppFog is one of the commercial implementations of the open source CloudFoundry PaaS (Platform as a Service). I blogged briefly before about CloudFoundry: it's one of the new generation of PaaS that do not lock you in with vendor-specific APIs. You can provision a standard DB service, upload a war and watch your application go live.

So AppFog is a commercial offering of an open source PaaS over generic public infrastructure: sounds about as generic as it gets, right? The difference is in their business model: calling themselves the "GMail of PaaS", they intend to shake up the market in the same way. Remember GMail? It arrived as a free email service with virtually unlimited storage. Suddenly, the idea of a paid email service seemed ridiculous. AppFog's announcement is comparable. The part of the announcement that made my jaw drop is the price: free for an application with 2GB of RAM, 1GB of DB storage and a 50GB transfer limit. By contrast, Jelastic's prices at ServInt work out to $14.40 per 128MB per month. The next step up is a $100/month plan, but the free plan alone is so generous many users won't need anything more. AppFog could kill the VPS market altogether.

There are details I'm still not clear about. For example, the default infrastructure is Amazon's AWS: is the disk storage the EC2 ephemeral storage that disappears when an instance shuts down or is that EBS or S3? Of course, you also have other infrastructure options: HP's OpenStack, Microsoft's Azure (no Java) or Rackspace. Moreover, I'm still skeptical about how AppFog intends to make money. They made vague noises enterprise customers, but the fact is their free product will be very popular. And while AppFog will not charge you money, AppFog itself still has to pay Amazon for the infrastructure. Still, why look a gift horse in the mouth? One thing is for sure: the PaaS market will not be the same again.

Where are the QA engineers?

I heard again today from my employer that we still really, really need software QA engineers. This has been going on for quite a while now. I realize that good tech talent is often hard to come by, and newer skills like HTML5 and mobile are red hot. But SQA -- while obviously an important position -- has been around for a long time, yet we've had a hard time hiring them. Is there something in the Boston tech scene that is sucking up all the QA talent?

Revealed: my top secret Grails project

I have mentioned in previous postings that I had been working on a Grails project, but neglected to actually say what that project was. Jelastic not only hosts this project on their PAAS cloud, but Judah Johns of Jelastic was kind enough to interview yours truly on their blog:

The Jelastic Spotlight: My Hometown Video

Now that My Hometown Video (HTV) is out there, it seems unnecessarily bashful to avoid blogging about this project. HTV is the brainchild of my client, Rob Roddenberry, who wanted a way for communities to form around towns of interest. Perhaps you might be nostalgic about a town you grew up in, or would like to tour attractions (parks, beaches). That's where HTV comes in: to provide location-centric videos and discussions. A longer explanation and more screenshots are in the Jelastic spotlight link above, but the best way to check it out is to visit the HTV site.

This has been a fun project. While I have a day job building enterprise Java applications, this is one site that I could sort of call my own. It really belongs to Rob, of course, and is entirely his idea. But it is satisfying to be get to do "everything" at the technical level including UI, server, DB, provisioning, deployment and operations. Grails has been fun to work with, and I find it interesting that every single Jelastic Spotlight application up to HTV has been a Grails app. The presence of public APIs like Facebook's and easy-to-use PAAS offerings like Jelastic has made such applications so easy to turn into reality.

Finding table for Oracle constraint violation

When Oracle reports a constraint violation, you might get something unhelpfully cryptic like:

ORA-00001: unique constraint (FOO.SYS_C003567231) violated

The constraint is system-generated, based on UNIQUE being specified in the DDL, so the name is unhelpful. With no other information, finding the affected table can be difficult. You can query for the table with the following SQL:

select constraint_name, table_name
from user_constraints
where constraint_name='SYS_C003567231

This is a handy query to know, so I'm noting it here for my future reference. It's also described in a number of other blogs like this one.

Programming at the speed of thought II: JRebel

In my last blog post, I lamented the slow turnaround and overall heaviness of enterprise Java development. I also said that I would be checking out JRebel, a tool designed to let you make code changes and push them out to a running application without restarting the application. My verdict? I loved it, convinced my boss to buy it, and am an enthusiastic user. But it is not flawless, so adopters should keep expectations realistic.

Programming at the speed of thought

In his classic "No Silver Bullets" paper in 1986, Fred Brooks claimed that the "accidents" of software development had been substantially addressed. He used the word "accident" in the Aristotelian sense, meaning tasks not inherently essential to programming like waiting for a program to compile. We were getting close to the being limited only by the essential tasks, where the limit is the human mind. I wonder how he might amend his views had he known how enterprise Java would introduce one bad accident after another to the profession. Fortunately, I think things are turning around.

The contrast between accident and essence came into sharp relief to me recently because of two things that happened one day. The first was when a colleague mentioned to me that he had wasted half a day trying to get Maven to build our application. He wasn't changing the build system: all he wanted to do was check out the latest code, build it, and work on it in Eclipse. This took literally hours to get working. The second thing was that I had been working on a personal project based on Grails.