ReportingDatabase

database · application architecture

tags:

Most EnterpriseApplications store persistent data with a database. This database supports operational updates of the application's state, and also various reports used for decision support and analysis. The operational needs and the reporting needs are, however, often quite different - with different requirements from a schema and different data access patterns. When this happens it's often a wise idea to separate the reporting needs into a reporting database, which takes a copy of the essential operational data but represents it in a different schema.

Such a reporting database is a completely different database to the operational database. It may be a completely different database product, using PolyglotPersistence. It should be designed around the reporting needs.

A reporting database has a number of advantages:

The downside to a reporting database is that its data has to be kept up to date. The easiest case is when you do something like use an overnight run to populate the reporting database. This often works quite well since many reporting needs work perfectly well with yesterday's data. If you need more timely data you can use a messaging system so that any changes to the operational database are forwarded to the reporting database. This is more complicated, but the data can be kept fresher. Often most reports can use slightly stale data and you can produce special case reports for things that really need to have this second's data [1].

A variation on this is to use views. This encapsulates the operational data and allows you to denormalize. It doesn't allow you to separate the operational load from the reporting load. More seriously you are limited to what views can derive and you can't take advantage of derivations that are written in an in-memory programming environment.

A reporting database fits well when you have a lot of domain logic in a domain model or other in-memory code. The domain logic can be used to process updates to the operational data, but also to calculate derived data which to enrich the reporting database.

I originally wrote this entry on April 2nd 2004. I took advantage of its ten-year anniversary to update the text.

Notes

1: These days the desire seems to be for near-real time analytics. I'm skeptical of the value of this. Often when analyzing data trends you don't need to react right away, and your thinking improves when you give it time for a proper mulling. Reacting too quickly leads to a form of information hysteresis, where you react badly to data that's changing too rapidly to get a proper picture of what's going on.


EnterpriseApplication

application integration · application architecture

tags:

In the early part of this century, I worked on my book Patterns of Enterprise Application Architecture. One of the problems I had when writing the book was how to title it, or rather what to call the kinds of software systems that I was writing about. I've always been conscious that my experience of software development has always been focused on one particular form of software - things like health care records, foreign exchange trading, payroll, and lease accounting. These are very different to embedded software inside printers, games, flight control software, or telephone switches. I needed a name to describe these kinds of systems and settled on the term "enterprise application".

As I so often have to say, there is no formal definition for this term. However there are some characteristics that enterprise applications have in common.

Enterprise applications usually have a lot of persistent data, usually managed by some kind of database management system. Usually this database is relational, but increasingly we're seeing NoSQL alternatives. This data will usually be longer lasting and more valuable than the applications that process it.

This data is accessed and manipulated concurrently. The numbers vary a lot, in-house applications may have a few tens of users, but customer-facing web applications can easily have tens of thousands. Despite high levels of concurrency, many enterprise application developers don't think much about critical regions, race conditions and other elements of classic concurrent programming. Instead they build their thinking on top of transactions managed by databases or specialized transaction management tools.

With so much data, enterprise applications have a lot of user interface screens to handle it. Usually the same data is manipulated in different ways in different contexts. Users vary from regular to occasional users, so the interfaces need to match different levels of familiarity. There is also a significant amount of offline (batch) processing that is easily forgotten.

Even if you are building a brand new enterprise application, you don't do so in isolation. Instead you'll need to integrate with other enterprise applications. These systems are built by a wide range of teams, some from vendors who sell to many customers, others built internally just for your organization. These applications will have been written over many decades in a host of different technologies, some of which you'll have to ask your mother about. There are many integration mechanisms to deal with - file exchange, shared databases, messaging middleware. Every so often there will be an attempt to rationalize all this communication technology, but they never entirely succeed leaving behind more complexity in their wake.

Even when different applications access the same data there is considerable conceptual dissonance between them, a customer may mean something quite different to the sales organization than it does to technical support. The same sounding entity has different fields in different contexts, or worse have fields with the same name yet different meanings.

And then there's so-called "business logic". When you are writing an operating system you strive to keep the whole thing logical and stive to discover and implement simplifications to keep the software straightforward and reliable. But business rules are given to you as they stand, and if you want to change them you need sixty-seven meetings and three vice-presidents retiring. They are usually a haphazard array of strange conditions that interact in surprising ways. Their insanity derives from a good reason, each one is a case where salesman could close a particular deal by offerring some special one-off condition. Do this a thousand times and you have the complex business "illogic" that lies in the heart of many enterprise applications.

Enterprise applications can be large or small. Often discussion focuses on large, complex applications, but there is also a challenge for smaller applications that need to be built quickly. Big systems make a lot of noise when they go wrong, yet the cumulative effect of small systems can have a surprising effect on an enterprises's health.

Coming up with names for things is always tricky. You need to use a minimum number of words, and want them to trigger the right connotations in the readers' minds, so that you don't have to constantly remind them what the definition means. On the whole I've been reasonably happy with my choice, but since I finished the book the word enterprise has taken on connotations which don't quite fit my usage.

One problem that's emerged since the book is that "enterprise" now usually means a large, well-established company. People think of G.E. or Siemens rather than Facebook, Etsy, or a company of a hundred people producing custom T-shirts. But according to my definition above, even small start-ups rely on software that I would call an enterprise application. So even though the Ruby on Rails community has ended up using enterprise as an insult, I would call Ruby on Rails a framework for building enterprise applications and BaseCamp a classic example of an enterprise application. (Just don't tell DHH I said so or he'll turn me into a hood ornament.)

These connotations around "enterprise" have made me muse about whether we need a different term. When I was writing P of EAA my working title was "Information Systems Architecture", but we felt that "information systems" had its own undesirable connotations of elder technologies. I guess I could go really retro and use "data processing", but on the whole "enterprise application" still seems a better term than anything else I could come up with.

This post is adapted from the definition of Enterprise Application in the introduction of P of EAA.


CircuitBreaker

delivery · application architecture

tags:

It's common for software systems to make remote calls to software running in different processes, probably on different machines across a network. One of the big differences between in-memory calls and remote calls is that remote calls can fail, or hang without a response until some timeout limit is reached. What's worse if you have many callers on a unresponsive supplier, then you can run out of critical resources leading to cascading failures across multiple systems. In his excellent book Release It, Michael Nygard popularized the Circuit Breaker pattern to prevent this kind of catastrophic cascade.

The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you'll also want some kind of monitor alert if the circuit breaker trips.

Here's a simple example of this behavior in Ruby, protecting against timeouts.

I set up the breaker with a block (Lambda) which is the protected call.

  cb = CircuitBreaker.new {|arg| @supplier.func arg}

The breaker stores the block, initializes various parameters (for thresholds, timeouts, and monitoring), and resets the breaker into its closed state.

class CircuitBreaker...
  attr_accessor :invocation_timeout, :failure_threshold, :monitor
  def initialize &block
    @circuit = block
    @invocation_timeout = 0.01
    @failure_threshold = 5
    @monitor = acquire_monitor
    reset
  end

Calling the circuit breaker will call the underlying block if the circuit is closed, but return an error if it's open

# client code
    aCircuitBreaker.call(5)


class CircuitBreaker...
  def call args
    case state
    when :closed
      begin
        do_call args
      rescue Timeout::Error
        record_failure
        raise $!
      end
    when :open then raise CircuitBreaker::Open
    else raise "Unreachable Code"
    end
  end
  def do_call args
    result = Timeout::timeout(@invocation_timeout) do
      @circuit.call args
    end
    reset
    return result
  end

Should we get a timeout, we increment the failure counter, successful calls reset it back to zero.

class CircuitBreaker...
  def record_failure
    @failure_count += 1
    @monitor.alert(:open_circuit) if :open == state
  end
  def reset
    @failure_count = 0
    @monitor.alert :reset_circuit
  end

I determine the state of the breaker comparing the failure count to the threshold

class CircuitBreaker...
  def state
     (@failure_count >= @failure_threshold) ? :open : :closed
  end

This simple circuit breaker avoids making the protected call when the circuit is open, but would need an external intervention to reset it when things are well again. This is a reasonable approach with electrical circuit breakers in buildings, but for software circuit breakers we can have the breaker itself detect if the underlying calls are working again. We can implement this self-resetting behavior by trying the protected call again after a suitable interval, and resetting the breaker should it succeed.

Creating this kind of breaker means adding a threshold for trying the reset and setting up a variable to hold the time of the last error.

class ResetCircuitBreaker...
  def initialize &block
    @circuit = block
    @invocation_timeout = 0.01
    @failure_threshold = 5
    @monitor = BreakerMonitor.new
    @reset_timeout = 0.1
    reset
  end
  def reset
    @failure_count = 0
    @last_failure_time = nil
    @monitor.alert :reset_circuit
  end

There is now a third state present - half open - meaning the circuit is ready to make a real call as trial to see if the problem is fixed.

class ResetCircuitBreaker...
  def state
    case
    when (@failure_count >= @failure_threshold) && 
        (Time.now - @last_failure_time) > @reset_timeout
      :half_open
    when (@failure_count >= @failure_threshold)
      :open
    else
      :closed
    end
  end

Asked to call in the half-open state results in a trial call, which will either reset the breaker if successful or restart the timeout if not.

class ResetCircuitBreaker...
  def call args
    case state
    when :closed, :half_open
      begin
        do_call args
      rescue Timeout::Error
        record_failure
        raise $!
      end
    when :open
      raise CircuitBreaker::Open
    else
      raise "Unreachable"
    end
  end
  def record_failure
    @failure_count += 1
    @monitor.alert(:open_circuit) if :open == state
    @last_failure_time = Time.now
  end

This example is a simple explanatory one, in practice circuit breakers provide a good bit more features and parameterization. Often they will protect against a range of errors that protected call could raise, such as network connection failures. Not all errors should trip the circuit, some should reflect normal failures and be dealt with as part of regular logic.

With lots of traffic, you can have problems with many calls just waiting for the initial timeout. Since remote calls are often slow, it's often a good idea to put each call on a different thread using a future or promise to handle the results when they come back. By drawing these threads from a thread pool, you can arrange for the circuit to break when the thread pool is exhausted.

The example shows a simple way to trip the breaker — a count that resets on a successful call. A more sophisticated approach might look at frequency of errors, tripping once you get, say, a 50% failure rate. You might also have different thresholds for different errors, such as a threshold of 10 for timeouts but 3 for connection failures.

The example I've shown is a circuit breaker for synchronous calls, but circuit breakers are also useful for asynchronous communications. A common technique here is to put all requests on a queue, which the supplier consumes at its speed - a useful technique to avoid overloading servers. In this case the circuit breaks when the queue fills up.

On their own, circuit breakers help reduce resources tied up in operations which are likely to fail. You avoid waiting on timeouts for the client, and a broken circuit avoids putting load on a struggling server. I talk here about remote calls, which are a common case for circuit breakers, but they can be used in any situation where you want to protect parts of a system from failures in other parts.

Circuit breakers are a valuable place for monitoring. Any change in breaker state should be logged and breakers should reveal details of their state for deeper monitoring. Breaker behavior is often a good source of warnings about deeper troubles in the environment. Operations staff should be able to trip or reset breakers.

Breakers on their own are valuable, but clients using them need to react to breaker failures. As with any remote invocation you need to consider what to do in case of failure. Does it fail the operation you're carrying out, or are there workarounds you can do? A credit card authorization could be put on a queue to deal with later, failure to get some data may be mitigated by showing some stale data that's good enough to display.

Further Reading

The netflix tech blog contains a lot of useful information on improving reliability of systems with lots of services. Their Dependency Command talks about using circuit breakers and a thread pool limit.

Netflix have open-sourced Hystrix, a sophisticated tool for dealing with latency and fault tolerance for distributed systems. It includes an implementation of the circuit breaker pattern with the thread pool limit

There are other open-source implementations of the circuit breaker pattern in Ruby, Java, Grails Plugin, C#, AspectJ, and Scala


AlienatingAtmosphere

diversity · internet culture

tags:

There are many factors that lead to the troubling DiversityImbalance that we find in the software community. Some of these, like the problems in teenage education that discourages girls from STEM subjects is a long term problem where our profession can't play a central role in fixing [1]. But one factor that comes down directly to us is the alienating atmosphere that hangs over the tech community.

One of the hardest things about talking about this alienating atmosphere is that it's something that's experienced very differently depending on who you are. For most middle-class white guys like me, it's something we've rarely experienced. Yet for many people in HistoricallyDiscriminatedAgainst populations it's a constant indication that "you're not welcome here".

One of the sharpest technologists I know is a woman who has been in the industry for a long time, much of it in technical leadership roles. There's nobody I'd rather have at my side in an important architecture meeting or reviewing a draft of mine that's veering into deep language theory. Yet she still says that she feels a reaction as she walks into a gathering of techies where nearly every face is different to hers, a little voice saying "this place is not for you". If such a reaction still strikes someone as accomplished as this, we shouldn't be surprised to see it elsewhere.

That basic reaction is set off just by the disparity in numbers, which is a long term problem to solve - but it can amplified by things within our short term influence. Sexually-oriented jokes tend to reinforce a men-only locker-room atmosphere which many women do find alienating. Many women have had real incidents of harassment, such jokes remind them of that. And although the chances of a serious incident are slim, they are not zero. [2]

If you have any doubt about how ugly this can get, just peruse the comments on a blog or news entry whenever this kind of thing comes up. The bile that falls on people who criticize alienating incidents is both vile and sadly predictable. Ironically the bilers often complain about censorship infringing their right to make sexual jokes, but not acknowledging that fear of the intermobs imposes its own censorship. I know several articulate women who will not post their opinions because they don't want to have to deal with this inevitable backlash.

There's an important asymmetry here. I've occasionally been on the receiving end of some nasty satire, but that's not such a big issue for me because I'm in a position of relative influence in the software world as well as part of a historically powerful group. For people from historically-discriminated-against (HDA) populations the dynamic changes. Such attacks reinforce the history of exclusion and oppression, and trigger fears of physical attack. Merely the fact that people think it's ok to make such attacks makes it clear that "your kind is not welcome here".

Trying to clear internet atmosphere of this feels like trying to deal with global warming - what can we do?

A first start is just to be aware that different people will react differently to the same things. One person might not find a remark offensive while it really hurts someone else. A phrase I've found useful in the past is I can't choose whether someone is offended by my actions. I can choose whether I care. If someone reacts in an hurt way to something I say, I don't get to say "you can't be offended by that". Their reaction is a result of their life experiences, which will be different to mine. Often HDA people gets lots of these small hurts, individually they don't mean much - but the sheer quantity has a big cumulative effect.

While I can't tell someone if they should be hurt or not, I can choose whether I care about the hurtful feeling. And it is a choice. I have offended some people when I've said we should treat gay people equally. In that case say I don't care very much. It's my choice that I'm not concerned about hurting the feelings of homophobes and I accept responsibility for that decision.

I do care, and think we as a profession should care, about this kind of thing with HistoricallyDiscriminatedAgainst groups. Correcting these long-term discriminations takes time, and can't be solved by just saying "we're all equal now". An alienating atmosphere against women, both in our profession and on the wider internet, reinforces an injustice that's been perpetrated for thousands of years. While I cannot be responsible for what my ancestors did, I can take responsibility to play what small part I can in cleaning up their mess.

I don't find it straightforward to learn about what causes alienation and how to avoid contributing to it. I try to pay attention to incidents when they come up and to listen to those who feel attacked. Not all women are alienated by an act, so I have to make a judgement about what proportion of people are going to react - is it just this one person, or is this individual the only one of many who felt brave enough to speak out?

I've found it valuable to find people who I can confidently talk to about these issues, and I certainly recommend searching out a few people who can act as guides. It's important to get a variety here, as different people react differently - it's easy to miss things by generalizing from opinions that are too few or not diverse enough. I feel that these steps have increased my awareness over the years, but it's far from being a territory I can be confident in.

Once we can begin to sense the alienation in the atmosphere the next step is to try to create environments with cleaner air. An example is how many conferences now have code of conduct policies designed to spell out that certain behaviors are not good manners. They can (and must) reinforce this by making it easy for people to make complaints and dealing with them swiftly and fairly. Yes, there is a risk that specious complaints will be made, and these are frustrating, but in practice the problems due to these are much smaller than the problems that never get reported. [3]

One question is what people in the majority groups should do when they detect some miasma in the air. If I hear a bunch of guys making sexual jokes in clear earshot, should I intervene? I don't find this an easy question to answer. I don't have high confidence in my ability to detect whether a group I care about is going to be offended by things that I personally don't find offensive, so I'd be reluctant to overreact myself.

But I agree with the notion that the standard you walk past is the standard you accept. Many women are sick of complaining about this kind of behavior because these complaints often only lead to more trouble. So I think it's important for all of us to battle against this miasma, to intervene when someone is fueling the bad air and to support someone's right to speak up and say they are feeling excluded.

So I believe I have the responsibility to act when such things happen but I say this with trepidation, I have that British thing about wanting to avoid fuss-and-bother, so my natural inclination is to let things go when I'm feeling offended. I know that even in the last few years I've missed opportunities to speak up when I should, but I know that I should not let that stop me from doing better in the future [4]. We must change this culture of silence.

In particular we must avoid supporting this alienating atmosphere. Sadly such support appears when people attack those who are trying to fight it. Often women who complain about alienating incidents are told to "grow a thick skin". I find this kind of support for NetNastiness deeply annoying.

I value people with good ideas and don't consider the thickness of their skin. Anyone driven away from expressing innovation or writing excellent code is a loss to all of us, however unoffended we think they should be.

Further Reading

Amanda Hess gives a sad but worthwhile summary of attacks against women online. These are wider than just within the tech community, but are typical of what women have to deal with here too.

Ashe Dryden writes about the risks and rewards of speaking up to oppose the alienating atmosphere in tech. The post includes responses to common misconceptions and suggestions of how to help.

Notes

1: There are a number of initiatives out there to encourage HDA teenagers to get involved in programming, such as Black Girls Code and Technovation Challenge.

2: Fear of physical attacks

Physical attacks are rare, but not negligible. Recent statistics in the US indicate that 2 per thousand women suffer rape or sexual abuse each year (4 per thousand for younger women). Another survey focusing on college women found 35 women per thousand per year suffered sexual victimization in college.

It's also important to note that the fear of an attack, even if irrational, is still a factor. As one reviewer commented "I once had a rather unpleasant experience with a male colleague, I spoke up and took a stand and pretty much held my own. But after that, I couldn't shake off the fear for days together, that he might turn around and physically harm me! This was probably the farthest thing from a possibility - but that thought was with me with an annoying amount of conviction. And every time I take a stand I feel a pinch of that."

Someone can be very sure intellectually that there isn't a threat of a physical attack, yet the debilitating emotion is still there. These reactions come from triggers that are deeply ingrained in our culture.

3: The best study into this that I could find indicated that false report rate was somewhere in the range of 2-8%

4: It's difficult talk about what to do because different cases are so varied. Even with a specific example, I frequently find myself at a loss. I've often thought about what action I should take if I found myself in an audience for a talk that uses sexual imagery to illustrate its concepts. I know I should do something, but I confess that I'm unsure what it should be - which gives me little confidence that I'd do the right thing.


Pink and Blue

When preparing the illustrations for the article, I puzzled a good bit about whether to use the pink and blue colors you see here. Many people, including me, object to the gender stereotyping of these colors and I don't like to reinforce it. But in the end I felt that using the gender associations of these colors helped the illustrations communicate better (as opposed to using more neutral colors). I also tried using light and dark brown in a similar way (as the alienating atmosphere is true of race too) but pink/blue scheme provides more striking color.

Acknowledgements

My thanks to Ashe Dryden, Adria Richards, Adrienne Hisbrook, Andrew Slocum, Chitra Sachdeva, Graham Brooks, Jim Gumbley, Linda Rising, Peter Gillard-Moss, Rebecca Parsons, and Trisha Gee for their feedback on this post.

SegregatedDOM

web development

tags:

Single-page web applications often turn into jQuery soup, where application logic, DOM manipulation, and server access are all mixed together. This mixing of concerns makes such applications harder to understand and test than they ought to be. Segregated DOM is a modularization tactic that separates all manipulation of the DOM into dedicated JavaScript objects.

For an example, consider the recent filter catalog page I developed to discuss my favorite eurogames. A key behavior of this page is your ability to click on the filter panel on the left side to filter which games display in the entry list panel. All this is done via JavaScript. I have JavaScript classes for the filter panel and the entry list, each of which encapsulates the DOM access for that part of the page. The logic of deciding which games to show is kept in a controller class which talks to the DOM manipulation classes using an interface that is independent of the DOM. The controller class doesn't make any use of jQuery.

During initialization the controller passes its update function to the Filter DOM to be called whenever a box in the filter changes state, the Filter DOM sets this function to the click events of the check boxes. So when anything changes in the filter display this update function in the controller asks the Filter DOM class for its active filters ➊. The Filter DOM class responds to this call by using jQuery to find the appropriate HTML elements ➋, filtering out those boxes that are checked, and responding to the controller with a simple data structure that indicates which boxes in each group are checked ➌.

The controller uses this information about the filter state to decide which games should now be visible, divides them into the left and right columns, and calls the Entry List DOM with the details of which element ids should be displayed in the list ➍. The Entry List DOM uses jQuery to shuffle the HTML items into the right divs to do the job ➎.

The principal benefit of doing this is to make it easier to reason about each class by keeping it focused on a single task. When I'm working on the Filter DOM class, I concentrate on how to do the DOM manipulations on the HTML elements. When I work on the controller I can ignore the details of the HTML structure, css class names, and the like.

In more abstract pattern terms, these DOM objects act as Gateways, a passage between two BoundedContexts (the application and the HTML DOM). Like any gateway the DOM object has an interface that speaks the vocabulary of its client (the application) and an implementation that translates this into the barbaric (HTML DOM) land beyond. [1]

A good modular design tends to correlate to testability. Each class can be tested in relative isolation. A particular advantage with a Segregated DOM is that I can test the controller without using a browser or mock browser such as PhantomJS. Since I don't need a DOM to test the controller, I can test it in node and just supply some simple test doubles for the DOM gateways. This makes it quicker and easier to modify the controller. By shifting as much logic out of the DOM gateway as possible, I can increase how much testing I can do without resorting to PhantomJS and the like (an application of the Humble Object pattern).

Further Reading

Pete Hodgson was a valuable source for ideas for this article. Coincidentally, his own article on using Segregated DOM was published the same day. It has more details and a short example.

Acknowledgements

Pete Hodgson gave me some useful ideas for improving this article.

Notes

1: You can use a similar separation for server access - this is the classic application of Gateway.


BoundedContext

team organization · requirements analysis · application integration · domain driven design

tags:

Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships.

DDD is about designing software based on models of the underlying domain. A model acts as a UbiquitousLanguage to help communication between software developers and domain experts. It also acts as the conceptual foundation for the design of the software itself - how it's broken down into objects or functions. To be effective, a model needs to unified - that is to be internally consistent so that there are no contradictions within it.

As you try to model a larger domain, it gets progressively harder to build a single unified model. Different groups of people will use subtly different vocabularies in different parts of a large organization. The precision of modeling rapidly runs into this, often to leading to a lot of confusion. Typically this confusion focuses on the central concepts of the domain. Early in my career I worked with a electricity utility - here the word "meter" meant subtly different things to different parts of the organization: was it the connection between the grid and a location, the grid and a customer, the physical meter itself (which could be replaced if faulty). These subtle polysemes could be smoothed over in conversation but not in the precise world of computers. Time and time again I see this confusion recur with polysemes like "Customer" and "Product".

In those younger days we were advised to build a unified model of the entire business, but DDD recognizes that we've learned that "total unification of the domain model for a large system will not be feasible or cost-effective" [1]. So instead DDD divides up a large system into Bounded Contexts, each of which can have a unified model - essentially a way of structuring MultipleCanonicalModels.

Bounded Contexts have both unrelated concepts (such as a support ticket only existing in a customer support context) but also share concepts (such as products and customers). Different contexts may have completely different models of common concepts with mechanisms to map between these polysemic concepts for integration. Several DDD patterns explore alternative relationships between contexts.

Various factors draw boundaries between contexts. Usually the dominant one is human culture, since models act as Ubiquitous Language, you need a different model when the language changes. You also find multiple contexts within the same domain context, such as the separation between in-memory and relational database models in a single application. This boundary is set by the different way we represent models.

DDD's strategic design goes on to describe a variety of ways that you have relationships between Bounded Contexts. It's usually worthwhile to depict these using a context map.

Further Reading

The canonical source for DDD is Eric Evans's book. It isn't the easiest read in the software literature, but it's one of those books that amply repays a substantial investment. Bounded Context opens part IV (Strategic Design).

Vaughn Vernon's Implementing Domain-Driven Design focuses on strategic design from the outset. Chapter 2 talks in detail about how a domain is divided into Bounded Contexts and Chapter 3 is the best source on drawing context maps.

I love software books that are both old and still-relevant. One of my favorite such books is William Kent's Data and Reality. I still remember his short description of the polyseme of Oil Wells.

Eric Evans describes how an explicit use of a bounded context can allow teams to graft new functionality in legacy systems using a bubble context. The example illustrates how related Bounded Contexts have similar yet distinct models and how you can map between them.

Notes

1: Eric Evans in Domain-Driven Design


BranchByAbstraction

version control · continuous integration

tags:

"Branch by Abstraction" is a technique [1] for making a large-scale change to a software system in gradual way that allows you to release the system regularly while the change is still in-progress.

We begin with a situation where various parts of the software system are dependent on a module, library, or framework that we wish to replace

We create an abstraction layer that captures the interaction between one section of the client code and the current supplier. We change that section of the client code to call the supplier entirely through this abstraction layer.

We gradually move all client code over to use the abstraction layer until all interaction with the supplier is done by the abstraction layer. As we do this we take the opportunity to improve the unit test coverage of the supplier through this abstraction layer.

We build a new supplier that implements the features required by one part of the client code using the same abstraction layer [2]. Once we are ready we switch that section of the client code to use the new supplier.

We gradually swap out the flawed supplier until all the client code uses the new supplier. Once the flawed supplier isn't needed, we can delete it. We may also choose to delete the abstraction layer once we no longer need it for migration.

My description above describes a common case, but there are plenty of variation that can occur. Sometimes you can't swap-out the supplier for only some clients, you have to do it all at once. Sometimes you can break down the features of the supplier into different sub-components and carry out this whole procedure one sub-component at a time.

Despite these variations, there is a common theme. Use an abstraction layer to allow multiple implementations to co-exist in the software system. Use the notion of one abstraction and multiple implementations to perform the migration from one implementation to the other. Ensure that the system builds and runs correctly at all times, so you can continue to use Continuous Delivery while you are doing the replacement. Look for as many ways as possible to make changes gradually.

Further Reading

Jez Humble describes how his team used Branch-by-Abstraction to replace both an object-relational mapping framework (ibatis to hibernate) and the web UI framework (velocity/JsTemplate to Ruby on Rails) for ThoughtWorks's Continuous Delivery tool "Go".

Paul Hammant provides more details of using Branch-by-Abstraction, particularly as an alternative to using version-control branching.

Steve Smith describes a variation which involves verifying that the two implementations return the same results to requests.

Acknowledgements

Thanks to Paul Hammant for his detailed suggestions as well as being an important primary source.

Notes

1: This technique has been around for a long time, although it never got a popular name. Paul Hammant introduced the term "Branch by Abstraction" while arguing in favor of Trunk Based Development (he credits Stacy Curl with originally coming up with it). This name seems to have caught on and is now the term I most commonly hear.

2: While we are building the new feature we can use FeatureToggles to run the new supplier in test environments and compare its behavior to the flawed supplier.