There are updates to this page that haven't been applied because you've entered text. Refresh this page to see updates.
Hide this message.

Tuning a back-end Java process: Where should the knobs go?

Eric WadsworthEric Wadsworth, technologist
Let's say we're developers building a back-end process that wakes up periodically and does stuff. We need to tell it how much work to do on each invocation, represented by a single number. We want a value that will result in the process taking about an hour, give-or-take, so we choose 40.

You can think of this value as a knob that can be adjusted. The question is, where do we put it? Here are some options:

OPTION A: Include it in the class itself:
static final int NUM_PARTITIONS_TO_PROCESS = 40;

But then we realize that since the environment can change, this value might need to be adjusted from time-to-time on the production system,at runtime. If it's buried in the code like this, how can the developers make needed changes?

OPTION B: Pull the value out into a properties file, and have the code load it in at runtime.

This seems pretty good. The developers have some control over the value now, and can make tuning adjustments in production. But all is not well: A quality piece of code now needs tests for multiple values of this parameter. Sure, there may not be a bug with a value of 40, but what about 10000? Or 0? Don't forget -1 either. Ugh.

Another downside is that this properties file is likely buried inside a JAR file somewhere. Getting the systems guys to dig down there and make an adjustment to the value in the file is painful, plus that value will be stomped on at the next deploy. Double-ugh. This scheme is getting worse and worse, as what we are trying to do is construct a knob that can be intelligently adjusted to tune a back-end process, but the knob is stuffed down in the guts of the engine, not on a handy dashboard.

OPTION C: Build a dashboard web-application, with the knobs on it. After all, not only do the developers need to make adjustments to these kinds of values from time-to-time, but these adjustments are not in a vacuum, they are in response to feedback from the system about, in this case, how long each invocation is actually taking. That feedback could be exposed on the dashboard, although then the dashboard has evolved into a metrics reporting system as well.

This is okay, but that dashboard web-application is big. Probably needs tables to hold timings of things. And it doesn't solve the test issue mentioned previously. What if someone turns that knob and breaks things? Again, tests need to be expanded to cover all possible values that the user can specify.

OPTION D: Take the humans out of the loop entirely. Use a database table to track how long it's taking to do the processes, and adjust the value accordingly. When the code needs to know how many partitions to process, it can get that value from the table.

This is pretty sophisticated, and a ton of work. A new database table, and accompanying code, is created for each value we want to track. If you've got the time to build code like this, that's great. The tests are going to be a bear, though, and you still have to test all possible values of each knob.

OPTION E: Build a separately deployed application that acts as a configuration service. It provides configuration values to other applications. It can be tested and deployed off-cycle to effect configuration changes.

One downside to this approach is that there may be a performance hit if the configuration service is handled over the network. Consuming applications would probably just call the service once to get the values, but what if a new configuration service is deployed? You'd need to either bounce all the other applications, or have them poll for changes. Then there is the issue of synchronizing various running applications that might have different versions of the same value. Yikes.

But haven't we almost come full circle at this point? Since we deploy every week or two anyway, why even mess with configuration? After all this, it turns out that option A is the best solution. The knob is built into the engine, but we'll just adjust it when we replace the engine, which happens all the time anyway.
Comments1+

Blog

This blog contains some tech stuff, some though...

Author