continuous integration


One of the most common arguments in favor of FeatureBranch is that it provides a mechanism for pending features that take longer than a single release cycle. Imagine you are releasing into production every two weeks, but need to build a feature that's going to take three months to complete. How do you use Continuous Integration to keep everyone working on the mainline without revealing a half-implemented feature on your releases? We run into this issue quite a lot and feature toggles are a handy tool to deal with it.

(I've seen a lot of names for this concept tossed around: feature bits, flags, flippers, switches and the like. There seems no generally accepted name yet.)

The basic idea is to have a configuration file that defines a bunch of toggles for various features you have pending. The running application then uses these toggles in order to decide whether or not to show the new feature.

Most of these decisions occur in the user-interface of the application. So if you are building a web application using jsp, you may use a set of jsp tags to surround any user-interface parts of a pending feature.

    <toggle name="petSurvey">
      <p>Take our new <a href = 'petSurvey'>pet survey</a></p>

The implementation of the toggle tag then just passes through the content if the toggle is set to on, and skips it otherwise. Other UI technologies will use different details, but the basic notion of wrapping the pending elements is the same.

Some features may be like introducing a new pricing algorithm, where there might be no user-interface elements. Here the test of the toggle would be in the application code, it could be as crude as a conditional test, or something more sophisticated like a strategy wired through dependency injection.

Toggle tests should only appear at the minimum amount of points to ensure the new feature is properly hidden. There could be many screens in the pet survey feature, but if there's only one link on the home page that gets you there, then that's the only element that needs to be protected with the toggle tag. Don't try to protect every code path in the new feature code with a toggle, focus on just the entry points that would lead users there and toggle those entry points. If you find that creating, maintaining, or removing the toggles takes significant time, then that's a sign that you have too many toggle tests. Remember that although simple conditionals are the easiest way to implement a toggle, you should use techniques like polymorphic substitution to minimize how many points the toggle is tested.

So far I've described feature toggles as something you use to hide partly build features, a kind of feature toggle I call release toggles. There is another form of feature toggle, called a business toggle which we can use to selectively turn on features in regular use. An example might be to only expose certain features to certain classes of user. These two styles of toggle are implemented in the same manner, but differ greatly in how they are used. Release toggles are primarily visible to the development organization and should be retired once the feature has bedded down in the application. Business toggles are visible to the business sponsors and are a permanent feature of the application. This means that the two kinds of toggles need to be clearly separated, usually appearing in separate configuration files. You often need to build some admin features to help control of business toggles, while release toggles can the managed through edits in the code base.

Most feature toggles I've heard about are set at run-time, but I've also seen cases where release toggles are set at build time. The small advantage of a build time toggle is that none of the new feature's code gets compiled into the released executable. Run-time toggles make it easier to set up your DeploymentPipelines and to run tests with various configurations of features. It also facilitates canary releasing, A/B testing, and makes it easier to roll-back should a new feature misbehave in production.

One danger with feature toggles is an accidental exposure, when someone forgets to wrap the UI feature in a toggle tag. This is awkward to test, since it's difficult to form a test that nothing that should be hidden is visible without calling out the individual elements - which are likely to be forgotten at the same time.

A common question we hear about feature toggles concerns testing - does using feature toggles mean a combinatorial explosion of tests? In general there's no need to test all combinations of features. For release toggles it's usually sufficient to run two combinations

This is pretty much the same as what you need to do with feature branches if you want to find any integration bugs.

For business toggles the combination problem is greater, since you do need to consider the various combinations that appear in practice and how they might interfere. You usually won't need to test every combination, but what subset of combinations you need depends on your knowledge of how the application performs in production.

It's very important to retire release toggles once the pending features have bedded down in production. This involves removing the definitions on the configuration file and all the code that uses them. Otherwise you will get a pile of toggles that nobody can remember how to use. In one memorable example I heard of, it required making a special recompilation of the linux kernel to handle enough command line switches.

Release toggles are the last thing you should do

Release toggles are a useful technique and lots of teams use them. However they should be your last choice when you're dealing with putting features into production.

Your first choice should be to break the feature down so you can safely introduce parts of the feature into the product. The advantages of doing this are the same ones as any strategy based on small, frequent releases. You reduce the risk of things going wrong and you get valuable feedback on how users actually use the feature that will improve the enhancements you make later.

If you really must hide a partly built feature, then the best way is to build all of it save the UI entry point and add that UI in a single release cycle. This way the non-ui code is fully integrated with everything else, but nothing is visible or used until you add the last bit at the end. The problem with this approach is that you can't do any testing that requires the UI, such as BroadStackTests through the UI or, more importantly, exploratory tests with the UI, until the final release cycle, just before the feature goes live. You can (and should) still use other tests, such as subcutaneous tests or perhaps a backdoor into the UI.

Only if you can't do small releases or UI last should you employ release toggles.

To probe further...

(Thanks to Charles Bradley, Kent Beck and Christian Gruber for tweets that reminded me of points I forgot to include.)

if you found this article useful, please share it. I appreciate the feedback and encouragement