Agile bliki

Like any profession, software development has it's share of oft-forgotten activities that are usually ignored but have a habit of biting back at just the wrong moment. One of these is data migration.

Most new software projects involve data that's lived somewhere else and now needs to be moved into the new system once it's live. A system replacement might have to move all the old data, new functionality may lead to data being loaded from some other system.

It's common to not take this task very seriously. After all, it's just reading some data, munging it a bit, and loading into the new system. Furthermore the code only ever has to be run once, so there's no point making particularly fast or pretty. Once the migration is done the code can be safely chucked away.

And of course there's no need to worry about it till the end of the project since you only want to run the migration just before the new system goes live.

I have a high opinion of my readers, if only for their taste in software writing, so I'm sure I can see the wistful smiles. Data migration often looks easy from the safety of whiteboard abstractions, but is usually full of nasty details to trip you up.

• You may suspect that the existing data is somewhat messy, but everyone is usually taken aback at how dirty the data really is. As a result the whole exercise is often far more complicated than it ought to be.
• Because it's single use, throw-away code people don't tend to put much design effort into migration code since they assume it's below the DesignPayoffLine. That assumption is often wrong, especially with the previous bullet point.
• Doing an activity that balloons into something harder than you think is never fun, but when you leave it till close to the ship date you're offering trouble a big signing bonus.

There's a soundbite I like to use in an agile context: if it hurts do it more often. Its surface illogicality makes it memorable, and there's a real truth in there. Many difficult activities can be made much more straightforward by doing them more frequently. XPers are particularly well known for applying this principle to testing, integration, design, and planning - so it shouldn't surprise anyone to see it applied to data migration.

I first saw this done by my colleague Josh Mackenzie on a moderately sized project (dozen developers for one year) with two failed attempts in its recent past. He decided he would migrate data with every two-week iteration. Each iteration the team figured out what data they needed to add to support the new functionality that was being built and updated the data migration system to migrate that extra data from the live system.

As is often the case with these things it ended up being much less impossible than people feared and the resulting reduction of risk and stress made it a worthwhile choice. They appreciated the obvious benefits, which boiled down to a distinct lack of hasty panic close to going live.

The most interesting benefit, however, was the one they didn't expect. Incremental migration made a significant improvement in communication with the domain experts. Usually when you want to talk about use cases with domain experts, you make up some pretend scenario. By using incremental data migration the team got into the habit of using real examples, which were much easier for the domain experts to relate to. Furthermore when the development made builds available for the domain experts to look at, it included a copy of the live data. As a result the domain experts could investigate how the new system worked with tricky cases they had run into recently. Particularly juicy predicaments could easily be copied over into the test environment.

Even without the improved communication it's worth the effort to do incremental migration. If you do, be prepared to take advantage of the opportunity to use real data to talk to domain experts.

I'm thinking of using agile software development - but should I use Lean software development instead?

This question is one I've run into a few times recently. It's not a question I can answer quickly as the question is based on a false premise about the relationship between lean and agile. So first I need to go into some history to help explain that relationship.

"Lean" fundamentally refers an approach in the manufacturing world that was originally developed by Toyota in the 1950's. At this time Japanese industry was recovering from the ravages of the second world war. This approach, often called the Toyota Production System is mostly credited to Taiichi Ohno, although he was influenced by various western thinkers - particularly Deming. The Toyota Production System became well known in the rest of the world in the 1990's when westerners started writing books to explain why the Japanese were beating the US at so many industries. The western writers called this approach Lean Manufacturing. Although Japanese industry in general has run into stickier times since then, Toyota continues to outperform most western auto companies.

Agile software development is an umbrella term for several software development methods (including Extreme Programming and Scrum) that were developed in the 1990s. These methods share a common philosophy which was described as values and principles in the Manifesto for Agile Software Development. (My essay "The New Methodology" goes into this in more depth.)

There was a connection between lean manufacturing and agile software from the beginning in that many of the developers of the various agile methods were influenced by the ideas of lean manufacturing. This connection was made more explicit by Mary and Tom Poppendieck. Mary had worked in a manufacturing plant that had adopted lean manufacturing and her husband Tom is an experienced software developer. They have written a couple of books on the application of Lean ideas in the software world. When people talk about Lean Software they are usually referring to the ideas in these books, although others have been making similar links.

Lean manufacturing and agile software methods have a very similar philosophy. Both place a lot of stress on adaptive planning and a people focused approach. As a result lean's ideas fit in very well with the agile software story. Mary and Tom have both been very active in the agile community - indeed I'd credit Mary with an important role in forming the Agile Alliance. (Like me, she was a founding board member of the Agile Alliance, but she was far more effective in it than I was.)

I've already mentioned that lean manufacturing was an influence on agilists from the beginning, and in the last few years more ideas have appeared in the agile world with a clear lean manufacturing heritage. These range from concrete techniques like Value Stream Maps, to articulations of existing agile concepts such as the Last Responsible Moment for making design decisions. The idea of thinking of analysis and design documentation as inventory came from the Poppendiecks. Several agilists I know emphasize the importance of reducing cycle time - another strongly lean-influenced idea. My colleague Richard Durnall has a nice summary of how lean knowledge can blend in with agile thinking.

A particularly strong appeal to many people about lean ideas is that they provide a way of explaining agile software development to people - particularly senior people both within IT and senior customers. This explanation ranges from a crude appeal to emulate Toyota to detailed discussions of how lean manufacturing's benefits translate to the ideas in agile software development.

So as you can see, lean and agile are deeply intertwined in the software world. You can't really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don't do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing.

The Poppendiecks didn't introduce lean as a separate idea, nor did they introduce lean as a published process in the style of Scrum or XP. Rather they introduced lean as a set of thinking tools that could easily blend in with any agile approach. I think of lean as a strand of thinking within the agile community, like a pattern in a rich carpet.

There is a distinct lean software community, as in a mailing list calling itself lean and people who label themselves as lean thinkers. But this is no different to the fact that there are also strong XP, Scrum, and other communities. Most people in these communities consider themselves part of the broader agile movement and many people are active in more than one of these agile communities. The whole point of coining the word 'agile' comes from a recognition that we share a core set of values and principles and this common core means what we have in common is greater than our differences.

For nth, and I'm sure not last time, I'm sliding into a conversation about defining practices, labeling some of them as "best", and probably the C-word (certification). It's a familiar discussion, and although we've barely started it, I can predict much of where it will go. It's driven by a perfectly reasonable desire to identify who are the better software developers, and how existing developers can improve their abilities.

When people get into these conversations, they usually end up in trouble. Either the group gets into heated discussions and cracks up, or the group doesn't have heated discussions and produces something that others deride. The heart of why this happens, and why I don't see any single, widely-recognized certification program for software development coming soon is that there is no single, well-agreed way to develop software effectively.

Instead what we see is a situation where there are several schools of software development, each with its own definitions and statements of good practice. As a profession we need to recognize that multiple schools exist, and their approach to software development development is quite different. Different to the point that what one school considers to exemplary is considered by other schools to be incompetent. Furthermore, we don't know which schools are right (in part because we CannotMeasureProductivity) although each school thinks of itself as right, with varying degrees of tolerance for the others.

I'm using "school" here in the style of this definition:

4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) <the Aristotelian school>; also : the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also : the shared opinions or behavior of such a group <other schools of thought>

--Merriam-Webster

I came across this notion explicitly from the Context-Driven School of Software Testing (see James Bach and Brett Pettichord). I like their way of looking at this because it's a model that explains why intelligent software developers have such different approaches.

The Context-Driven folks have done some looking at different schools within the testing world, but I don't know of any good attempt to classify the schools within the broader world of software development. I feel a sense of belonging to a school, one that for me is rooted in the people I met through OOPSLA in the 90's. Object-orientation is a key practice of this school, as is agile methods. You could reasonably argue that this is the agile school, except I think that agile methods are a core component of this school's thinking but not the whole picture. The leaders of this school include people like Ward Cunningham, Ralph Johnson, Kent Beck, and Robert Martin. ThoughtWorks is, on the whole, an organization that follows this school (which is why I'm comfortable here).

But despite this sense of a somewhat coherent school, there's still many open questions. Is it best to think of the agile world as one school or many (are Scrum and XP different schools or part of the same)? What are the major schools out there? What exactly defines a school of thought?

I don't have much of an answer to these questions, but the key point to remember is that there are multiple schools of thought about how to develop software effectively. We may not think much of the other schools to our particular one, but we are foolish not to recognize that other schools exist.

Imagine a hiring situation. There's two candidates both with a few years of experience. In the blue corner we have someone with good broad design skills in the style of design that you favor (in my case that would be things like DRY, judicious use of patterns, TDD, communicative code etc, but the actual list isn't important - just that it's what you favor). However she knows nothing of the particular platform technology that you're using. In the red corner we have someone who has little knowledge (or interest) in those issues, but knows your platform really well - edge cases in the language, what libraries are available, fingers move naturally over the tools. Assume all else about them is equal (which it never is except for thought experiments like this) and that your team doesn't have any gaping holes that this candidate might fill. Which one would you prefer?

My answer is simple, I'd take the one in the with broad design skills. I've always held the view that a good programmer should be able to pick up a new platform relatively quickly. Learning basic design aesthetics is both harder and carries over better to new platforms. Good design practices that matter in Java are equally valuable in .NET. Not being familiar with the platform does slow you down (how do I get a literal class name in C# again?), but producing well designed code is what really makes a difference.

Broad design skills aren't completely portable. Java and .NET are mostly equivalent as languages - moving to Ruby, however, changes more. Moving to a significantly different beast, like functional languages, is a bigger shift. In any case, you can't just blindly replicate all design habits in a new environment. But if you're aware of the new world, an awful lot does carry over.

We've seen this principle prove itself at ThoughtWorks. In our early days with Java, we found the skills experienced developers had learned in Forte gave us excellent instincts for working with Java. We moved away early from the EJB-dominant thinking, and I think it was experience with other platforms that guided us. We saw it even more strongly with .NET. Time and again we saw that good developers with a Java background were rapidly more effective than those with a longer .NET or Microsoft background who lacked those skills. The difference was visible in weeks, not months (and sometimes days).

At the moment we see this shift most notably in Ruby. We've had quite the run of Ruby projects this year, and often we turn to people with long experience in curly-brace languages to fill the need. Again we've seen the value that broad design skills gives us.

It's not always a sure thing. I have seen cases where someone experienced in another platform just doesn't desire to get in and learn the new one. Desire to learn is a necessary component here - I'd take the single platform specialist if he wanted to learn broad design and the broad designer didn't want to learn the new platform. It's also essential to have someone on the team who knows the platform well.

I'd say most people at ThoughtWorks prefer design skills over platform knowledge. Many clients don't share that point of view - which can lead to some difficult pragmatic and ethical choices.

What happens if you have someone you want to bring onto the team with strong design skills and no platform background - yet the client insists on at least two years experience on the platform. In your professional judgment, the broad candidate is going to be more productive than anyone else available. You need to be honest with your client, but on the other hand he is paying you for your professional judgment. Does this change if the client has given you responsibility for delivery of the project?

For us these questions are more charged because there is a financial interest involved. If we add a ThoughtWorker to a team, then we bill for that person. If a client hires a platform specialist contractor, we don't get that income. For many people this is a crucial fact in the situation, yet I expect our project managers are wise enough to know that the risk of adding the wrong person is much more important than one billable income.

Consider another case where you're open with the client and the client demands a reduced rate for the broad design person due to her lack of platform knowledge as she'll be learning on the job. You're sure that she, despite that lack, will be more productive than the competing platform expert due to those design skills. Should you accept a reduced rate?

It's the nature of our, and most other, professions that you learn on the job. A platform specialist also has to learn broad design skills if he's going to produce maintainable code. Here it's important to remember that not just is it usually harder to learn design than platforms, it's also less certain. Given a motivated broad-designer, I can be pretty sure she'll pick up a platform in time. But there's no guarantee the other way around. Some people are good at learning details of a platform, but never figure out how to write clear code.

I've talked here about broad design skills - and I do believe this makes a difference on the technical axis. But there are other dimensions of broadness too. Most risk in software projects lies in the communication between businesspeople and programmers, so a candidate who can communicate well with users brings a great deal to a team.

A similar issue is knowledge of the problem domain. Often clients want people who already know their business, yet are surprised when someone rapidly gains enough understanding to be useful. I've long held that it's the ability to collaborate with others which is central here. By collaborating with a domain expert, or a platform expert, someone with broad skills can be become effective very quickly. Knowledge of other domains often introduces surprising insights into a project and similarities often crop up in sup-rising places. It's remarkable how often things like core accounting patterns crop up in places that don't look like accounting on the surface. In the end it's the ability to work with others, coupled with being a fast learner, that is the critical skill.

I'm not dismissing deep platform knowledge here. In an ideal world every team member would be excellent broad programmers with several years platform experience, good familiarity with the problem domain, and written similar systems at least twice before. But we all know how far our world is from that ideal. You need some platform knowledge on a team, and if it were a gap I would reach for the platform specialist to fill it. But that doesn't alter my default position to prefer broad design skills most of the time.

A key property of agile development is figuring out how to make a system go live with a small subset of features. We build software for the business value it offers, the quicker we go live, the faster we get at least some of that business value.

My colleague Dave Leigh-Fellows told me one of my favorite examples of this kind of thinking. It came when we has working for a brokerage firm. They had a new kind of product that they wanted to get into the market. The full software support for this was a web page that the customer filled in that generated the necessary transactions against the back-end system. But Dave came up with a way to get the product into the market faster than that.

• Version 1 was a static web page that described the product and provided a telephone number to call. Some temporary staff then spoke to the customer and entered the information into the back end system.
• Version 2 was a web form that captured the data the customer filled in. However this version didn't load that data into to the back end system. Instead the web form generated a fax. They hired some more temps to get the orders from the fax machine to the people that keyed the information into the back end system. Since the fax machines were a bit of a distance away, this is where the roller-skates came in.
• Version 3 hooked the web form into the back-end system directly.

The first two versions may not have been the most elegant solutions ever conceived, but they did get the product into the market much more quickly. I've not come across any other examples of iterative development that use roller-skates, but that may be more due to lack of imagination rather than lack of need.

I'm a big fan of Continuous Integration, it's an relatively simple practice that can make a huge difference to most development teams. However like most practices it has its flaws^H^H^H^H^H opportunities for improvement. Paul Duvall, author of the soon-to-be-standard book on the subject, pointed out one of these recently. If the commit build breaks, the whole team is affected and potentially slowed until it's fixed.

When we first started doing Continuous Integration at ThoughtWorks, this one of the of the things that worried me about the way we were doing it. It worried me because there was an important difference between between the ThoughtWorks 2000 style and the style we'd used at C3.

The ThoughtWorks 2000 style is pretty much the canonical style of CI used today. Once you are happy with your work you commit it to the repository, and then build it on the build machine (either manually for with a CI server like CruiseControl). The problem lies if your commit is bad, anyone who updates will get failing code until you fix it.

In the C3 way of doing it we didn't commit to the head of repository directly. C3 was a Smalltalk project and used Envy, a Smalltalk-oriented repository system. Envy had some different concepts to mainstream repositories. Since it's ages since I used it my memory on exactly how it worked has gone all fuzzy, but the basic idea was that when you were working on your feature you committed to editions. An edition was like a private branch, visible to everyone but not blessed. Only when you had a successful build on the build machine would you upgrade your edition into a release, which was the equivalent to the mainline. This way you never got broken code into the mainline of the project.

Envy made it easy to work this way, the version control systems we mostly use now make it more tricky. Ideally you want to create a working copy that updates from the true head (to keep you in sync) but commits to a different pending-head branch. Only a successful integration build can actually commit to the true project head. A continuous integration server would check out from the pending head and, if successful, commit to the true head.

How difficult is it to set something like this up yourself? I'm not sure, I haven't chatted with a team that's done it2. However a number of team oriented tools are providing this kind of capability. For example JetBrains's TeamCity does it under the name "delayed commit". Paul also mentions Borland's Gauntlet.

The other question is how much it matters. Despite my worries we didn't get enough pain from broken builds to want to install a pending head in 2000. If you get a lot of broken integration builds there are other ways to fix it. Often the main problem is that people aren't doing a private build before they commit. As usual the people-issue is often a more important issue to deal with before introducing more complicated technology.