Martin Fowler's Bliki

SchoolsOfSoftwareDevelopment

2008-04-12T09:53:00-04:00

For nth, and I'm sure not last time, I'm sliding into a conversation about defining practices, labeling some of them as "best", and probably the C-word (certification). It's a familiar discussion, and although we've barely started it, I can predict much of where it will go. It's driven by a perfectly reasonable desire to identify who are the better software developers, and how existing developers can improve their abilities.

When people get into these conversations, they usually end up in trouble. Either the group gets into heated discussions and cracks up, or the group doesn't have heated discussions and produces something that others deride. The heart of why this happens, and why I don't see any single, widely-recognized certification program for software development coming soon is that there is no single, well-agreed way to develop software effectively.

Instead what we see is a situation where there are several schools of software development, each with its own definitions and statements of good practice. As a profession we need to recognize that multiple schools exist, and their approach to software development development is quite different. Different to the point that what one school considers to exemplary is considered by other schools to be incompetent. Furthermore, we don't know which schools are right (in part because we CannotMeasureProductivity) although each school thinks of itself as right, with varying degrees of tolerance for the others.

I'm using "school" here in the style of this definition:

4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) <the Aristotelian school>; also : the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also : the shared opinions or behavior of such a group <other schools of thought>

--Merriam-Webster

I came across this notion explicitly from the Context-Driven School of Software Testing (see James Bach and Brett Pettichord). I like their way of looking at this because it's a model that explains why intelligent software developers have such different approaches.

The Context-Driven folks have done some looking at different schools within the testing world, but I don't know of any good attempt to classify the schools within the broader world of software development. I feel a sense of belonging to a school, one that for me is rooted in the people I met through OOPSLA in the 90's. Object-orientation is a key practice of this school, as is agile methods. You could reasonably argue that this is the agile school, except I think that agile methods are a core component of this school's thinking but not the whole picture. The leaders of this school include people like Ward Cunningham, Ralph Johnson, Kent Beck, and Robert Martin. ThoughtWorks is, on the whole, an organization that follows this school (which is why I'm comfortable here).

But despite this sense of a somewhat coherent school, there's still many open questions. Is it best to think of the agile world as one school or many (are Scrum and XP different schools or part of the same)? What are the major schools out there? What exactly defines a school of thought?

I don't have much of an answer to these questions, but the key point to remember is that there are multiple schools of thought about how to develop software effectively. We may not think much of the other schools to our particular one, but we are foolish not to recognize that other schools exist.

UpcomingTalks

2008-04-08T17:27:00-04:00

My next appearance at a conference will be at the JAOO conference in Australia - specifically Brisbane and Sydney. This is JAOO's first venture into Australia and I have high hopes for it. One thing I've learned from my recent visits to our offices in Oz is that Australia really lacks for decent conferences. JAOO does an excellent job, so it should do well.

I have three talks on my list, two of which I'll be doing with my colleague Erik Dörnenburg.

The first one to mention is our tutorial on Test Driven Development. We've done this a couple of times in the past. It's a very informal tutorial - Erik codes up an example from scratch using TDD which we use as a vehicle to explaining how the process works. We show the basic red-green-refactor cycle, and go into the use of state and interaction based verification. It's very much an introductory tutorial, aimed at people who haven't done TDD before.

Our second effort will be a keynote: on Simplicity in Design. Erik is both a proponent and exponent of simple architectural designs, as shown by his work for The Guardian. The problem with simplicity is that it's a complicated subject to talk about, but I think we shall be able to give some useful principles to think about.

I'm very much looking forward to the keynote, as I've wanted to do a keynote with Erik for a long time. I really enjoy giving joint keynotes, and have managed to now with Neal, Dan, and Jim. Erik and I just haven't found a chance to team up together before.

My final talk is a solo. I'll be talking about patterns in enterprise software. With this talk about the role I see patterns playing in software design and talking about some of the more important patterns that I've written up.

ThoughtWorks Anthology

2008-04-03T12:08:00-04:00

General News: I'm sure that putting it together was a frustrating exercise in cat herding, but the Prags have put out a book of essays by ThoughtWorkers. It's a great collection of writings, mostly by people who aren't well known outside TW, despite their great know-how. It includes one from me on DSLs in Ruby, which the prags currently have made available as a PDF download.

REST Podcast

2008-04-03T12:07:00-04:00

General News: I got involved in another ThoughtWorks podcast, this time on the subject of REST with Jim Webber, Chris Stevenson, and Sriram Narayan.

CheaperTalentHypothesis

2008-02-08T19:32:00-05:00

One of the commonly accepted beliefs in the software world is that talented programmers are more productive. Since we CannotMeasureProductivity this is a belief that cannot be proven, but it seems reasonable. After all just about every human endeavor shows some people better than others, often markedly so. It's also commonly observed by programmers themselves, although it always seems to be remarked on by those who consider themselves to be in the better talented category.

Naturally better programmers cost more, either as full-time hires or in contracting. But the interesting question is, despite this, are more expensive programmers actually cheaper?

On the face of it, this seems a silly question. How can a more expensive resource end up being cheaper? The trick, as it is so often, is to think about the broader picture of cost and value.

Although the technorati generally agree that talented programmers are more productive than the average, the impossibility of measurement means they cannot come up with an actual figure. So let's invent one for argument sake: 2. If you can find a factor-2 talented programmer for less than twice of the salary of an average programmer - then that programmer ends up being cheaper. To state this more generally: If the cost premium for a more productive developer is less than the higher productivity of that developer, then it's cheaper to hire the more expensive developer. The cheaper talent hypothesis is that the cost premium is indeed less, and thus it's cheaper to hire more productive developers even if they are more expensive.

In case anyone hasn't noticed this hypothesis is a key part of our philosophy at ThoughtWorks and is one of the main reasons why I ended up switching from an independent consultant to join. We believe we actually end up cheaper for our clients, even though our rates were higher. Of course, we do have difficulty persuading many clients that this is true - that lack of objective productivity measures strikes again. I still remember a meeting with one prospective client complaining about how our rates were higher than a company who had made a previous, failed, attempt at the system we were bidding on. We had to politely point out that paying less rates for a project that delivered no value was hardly a financially prudent strategy.

There are some notable consequences to the the cheaper talent hypothesis. Most notably is one that it actually follows a positive scaling effect - the bigger the team the bigger the benefits of cheaper talent. Let's assume we actually have put together a team of ten talented developers to run a project in some alternative universe where we have actually measures that they are twice as productive as the average - and thus do cost exactly twice as much to hire. In this case you might naturally assume that a rival team of average programmers would be a team of twenty.

The trouble is that that assumption assumes productivity scales linearly with team size, which again observation indicates isn't the case. Software development depends very much on communication between team members. The biggest issue on software teams is making sure everyone understands what everyone else is doing. As a result productivity scales a good bit less than linearly with team size. As usual we have no clear measure, but I'm inclined to guess at it being closer to the square root. If we use my evidence-free guess as the basis then to get double the productivity we need to quadruple the team size. So our average talent team needs to have forty people to match our ten talented people - at which point it costs twice as much.

Another factor that plays a role here is time-to-market. Let's assume two teams of four people, one talented and one average. To stack the deck of our argument against our talented team, discount the previous paragraphs, and assume the talented team is only twice as productive as the average team. If the talented team charges twice as much then can we assume that it doesn't matter financially which team we pick?

I'm afraid the talented team wins again. They'll complete the project in half of the time of the average team, which means that the customer will start yielding value from the delivered software earlier. This earlier value, compounded by the time value of money, represents a financial gain for picking the talented team, even thought their cost per output is the same.

Agile development further accelerates this effect. A talented team has a faster cycle time than an average team. This allows the full team to explore options faster: building, evaluating, optimizing. This accelerates producing better software, thus generating higher value. This compounds the time-to-market effect. (And it's natural to assume that a talented team is more likely to produce better software in any case.)

Faster cycle time leads to a better external product, but perhaps the greatest contribution a talented team can make is to produce software with greater internal quality. It strikes to me that the productivity difference between a talented programmer and an average programmer is probably less than the productivity difference between a good code-base and an average code-base. Since talented programmer tend to produce good code-bases, this implies that the productivity advantages compound over time due to internal quality too.

All this sounds, at least to me, like a highly compelling argument. It's also one that's widely accepted (at least by programmers who consider themselves talented). But it's far off being accepted by the software industry as a whole. We can tell this because the premium for talented developers (in terms of salary/contracting fees) is less than the productivity difference. Probably the major reason for this the inability to objectively measure productivity. A hirer cannot have objective proof that a more expensive programmer is actually more productive. Only the higher cost is objective. As a result a hirer has to match a subjective judgment of higher value against an objective higher cost. Many hirers, even if they believe the talented programmer is worthwhile personally, isn't prepared to justify the full higher cost to managers, HR, and purchasing.

This effect is compounded by the difficulty in making even a subjective assessment. At ThoughtWorks we rely on peer assessment - developers abilities are assessed by fellow team members. The result is hardly pinpoint precision, but it's the best anyone can do.

Which all points out that hiring and retaining talented programmers is hard work. Hiring and assessment is hard work. You have to deal with people with very individual desires, which are even more important to track as they are effectively underpaid. So a hirer is faced with certain extra work and higher costs versus only a judgment call for higher productivity.

So I understand the situation but don't accept it. I believe that if the software industry is to fulfill its potential it needs to recognize the cheaper talent hypothesis and close the gap between high productivity and higher compensation.

PreferDesignSkills

2008-01-17T15:39:00-05:00

Imagine a hiring situation. There's two candidates both with a few years of experience. In the blue corner we have someone with good broad design skills in the style of design that you favor (in my case that would be things like DRY, judicious use of patterns, TDD, communicative code etc, but the actual list isn't important - just that it's what you favor). However she knows nothing of the particular platform technology that you're using. In the red corner we have someone who has little knowledge (or interest) in those issues, but knows your platform really well - edge cases in the language, what libraries are available, fingers move naturally over the tools. Assume all else about them is equal (which it never is except for thought experiments like this) and that your team doesn't have any gaping holes that this candidate might fill. Which one would you prefer?

My answer is simple, I'd take the one in the with broad design skills. I've always held the view that a good programmer should be able to pick up a new platform relatively quickly. Learning basic design aesthetics is both harder and carries over better to new platforms. Good design practices that matter in Java are equally valuable in .NET. Not being familiar with the platform does slow you down (how do I get a literal class name in C# again?), but producing well designed code is what really makes a difference.

Broad design skills aren't completely portable. Java and .NET are mostly equivalent as languages - moving to Ruby, however, changes more. Moving to a significantly different beast, like functional languages, is a bigger shift. In any case, you can't just blindly replicate all design habits in a new environment. But if you're aware of the new world, an awful lot does carry over.

We've seen this principle prove itself at ThoughtWorks. In our early days with Java, we found the skills experienced developers had learned in Forte gave us excellent instincts for working with Java. We moved away early from the EJB-dominant thinking, and I think it was experience with other platforms that guided us. We saw it even more strongly with .NET. Time and again we saw that good developers with a Java background were rapidly more effective than those with a longer .NET or Microsoft background who lacked those skills. The difference was visible in weeks, not months (and sometimes days).

At the moment we see this shift most notably in Ruby. We've had quite the run of Ruby projects this year, and often we turn to people with long experience in curly-brace languages to fill the need. Again we've seen the value that broad design skills gives us.

It's not always a sure thing. I have seen cases where someone experienced in another platform just doesn't desire to get in and learn the new one. Desire to learn is a necessary component here - I'd take the single platform specialist if he wanted to learn broad design and the broad designer didn't want to learn the new platform. It's also essential to have someone on the team who knows the platform well.

I'd say most people at ThoughtWorks prefer design skills over platform knowledge. Many clients don't share that point of view - which can lead to some difficult pragmatic and ethical choices.

What happens if you have someone you want to bring onto the team with strong design skills and no platform background - yet the client insists on at least two years experience on the platform. In your professional judgment, the broad candidate is going to be more productive than anyone else available. You need to be honest with your client, but on the other hand he is paying you for your professional judgment. Does this change if the client has given you responsibility for delivery of the project?

For us these questions are more charged because there is a financial interest involved. If we add a ThoughtWorker to a team, then we bill for that person. If a client hires a platform specialist contractor, we don't get that income. For many people this is a crucial fact in the situation, yet I expect our project managers are wise enough to know that the risk of adding the wrong person is much more important than one billable income.

Consider another case where you're open with the client and the client demands a reduced rate for the broad design person due to her lack of platform knowledge as she'll be learning on the job. You're sure that she, despite that lack, will be more productive than the competing platform expert due to those design skills. Should you accept a reduced rate?

It's the nature of our, and most other, professions that you learn on the job. A platform specialist also has to learn broad design skills if he's going to produce maintainable code. Here it's important to remember that not just is it usually harder to learn design than platforms, it's also less certain. Given a motivated broad-designer, I can be pretty sure she'll pick up a platform in time. But there's no guarantee the other way around. Some people are good at learning details of a platform, but never figure out how to write clear code.

I've talked here about broad design skills - and I do believe this makes a difference on the technical axis. But there are other dimensions of broadness too. Most risk in software projects lies in the communication between businesspeople and programmers, so a candidate who can communicate well with users brings a great deal to a team.

A similar issue is knowledge of the problem domain. Often clients want people who already know their business, yet are surprised when someone rapidly gains enough understanding to be useful. I've long held that it's the ability to collaborate with others which is central here. By collaborating with a domain expert, or a platform expert, someone with broad skills can be become effective very quickly. Knowledge of other domains often introduces surprising insights into a project and similarities often crop up in sup-rising places. It's remarkable how often things like core accounting patterns crop up in places that don't look like accounting on the surface. In the end it's the ability to work with others, coupled with being a fast learner, that is the critical skill.

I'm not dismissing deep platform knowledge here. In an ideal world every team member would be excellent broad programmers with several years platform experience, good familiarity with the problem domain, and written similar systems at least twice before. But we all know how far our world is from that ideal. You need some platform knowledge on a team, and if it were a gap I would reach for the platform specialist to fill it. But that doesn't alter my default position to prefer broad design skills most of the time.

RepositoryBasedCode

2008-01-14T15:29:00-05:00

An alternative to SourceBasedCode is the idea that the core definition of a system should be held in a model and edited through projections.

To talk about this style of environment I find it handy to think in terms of multiple representations of the system:

editable representation: what you edit in order to change the system.
storage representation: the persistent record of the system definition.
executable representation: what is executed to make the system run - the executable.
abstract representation: used to manipulate and reason about system definition.
visualization representation: a non-editable view of the system definition.

A source based system combines the editable and storage representations in the source file. It executes the source by transforming the source into an executable representation either in one observable step (interpretation) or multiple steps via a compiler. In order to do this it usually transforms the source into an abstract representation as an intermediate step, but this abstract representation is transitory and only around during compilation. The source is seen as the core definition of the system.

With a repository based system the abstract representation is the is core definition of the system. A tool manipulates the abstract representation and projects multiple editable representations for the programmer to change the definition of the system. The tool persists the abstract representation in a storage representation, but this is entirely separated from any of the editable representations that it projects. The relationship to the executable representation is pretty much the same - the executable is produced through a series of transformations from the abstract representation.

An important difference between repository and source based environments is the split between persistent storage and editing. Repositories can choose any persistence mechanism that they choose, while source systems need to have some universal storage mechanism - which is why they are almost always text files.

The abstract representation may be edited through multiple projections, each projection can show a limited amount of the total information which isn't tied to the actual structure of the abstract representation. Repository systems thus usually show a wider range of editing environments - including graphical and tabular structures - rather than just a textual form.

Sophisticated source based IDEs also show multiple projections - for instance a side pane showing a list of methods for a class with graphical annotations to indicate their AccessModifiers. However these projections are usually very much secondary to a source editor, and often the projections can't be edited directly - you have to change the source and see the projection update.

Such PostIntelliJ IDEs do this by creating an abstract representation when they load the source files (which is why they can take a while to start up). They also use the abstract representation to do perform lots of other code-assistance features such as contextual code completion and refactoring.

A significant pragmatic problem with repository based systems is the fact that there is no generally accepted way format for the storage representation. The fact that programmer-readable text is the universal choice for source files means that a whole slew of tools can be built to process them: editors, source-code control, difference visualizers etc. Repositories have to do all this themselves, which is often why these things are often lacking. In particular many repository based environments suffer greatly because they don't have a decent configuration control system, which makes it much harder for multiple people to collaborate on the same system definition. This is a big contrast to source based environments that have a plethora of source code control systems to do this task.

Repository based systems are closely connected with Model-Driven Development (MDD), although I don't think the two are entirely synonyms. In an MDD context the abstract representation is usually referred to as the model. Certainly almost all MDD tools are repository based, but many all repository based tools, eg Microsoft Access, would not consider themselves to be MDD.

(I first explored this way of looking at environments in my essay on Language Workbenches. I've described it here because I think the notion of repository based environments is broader than just in Language Workbenches.)

TestCancer

2007-12-06T13:42:00-05:00

As my career has turned into full-time authorship, I often worry about distancing myself from the realities of day-to-day software development. I've seen other well-known figures lose contact with reality, and I fear the same fate. My greatest source of resistance to this is ThoughtWorks, which acts as a regular dose of reality to keep my feet on the ground.

ThoughtWorks also acts as a source of ideas from the field, and I enjoy writing about useful things that my colleagues have discovered and developed. Usually these are helpful ideas, that I hope that some of my readers will be able to use. My topic today isn't such a pleasant topic. It's a problem and one that we don't have an answer for.

The scenario runs like this. We carry out a project for a client and hand over a shiny new piece of software. As is our habit these days, we also hand over a bevy of automated tests for this software (typically there are as many lines of code of tests as there are of functional code). These tests are usually a mix of unit tests and broader ranging functional and acceptance tests. Either way the tests act as an active description of what the software does and a bug detector to quickly find problems as we evolve the software. We treasure these tests, they are a key to our success in building software systems.

Some months later the happy customer calls us back to do some further work on the software, adding new features and capabilities. We come in, keen to work on a code base that may have faults - but at least are our faults. Then we make an unpleasant discovery.

The tests no longer run.

Sometimes the tests are excluded from the build scripts, and haven't been run in months. Sometimes the "tests" are run, but a good proportion of them are commented out. Either way our precious tests are afflicted with a nasty cancer that is time-consuming and frustrating to eradicate.

We ask what happened and are told things like "we made a change and some tests broke, so we removed the tests". You can look at this as our failing - we haven't managed to fully teach the client teams about the value of the tests. We need to do more to pass on that failing tests need to be investigated, not simply ignored. But whatever anyone says, we've discovered that cancer of the tests is a common disease.

We don't think that the fact that Test Cancer appears is a reason against writing tests. Even if a particularly virulant strain wipes them all out the day after we leave, we still got value from them while we were building the system. And tests don't always get cancer. We recently spoke to a developer who had become a convert to TDD after maintaining a system we'd handed over a few years ago. The tests made our code much easier to work with than code that other firms had added later.

BookCode

2007-12-04T22:24:00-05:00

I don't not write much production code these days, but I still spend quite a few hours writing code. This code is a particular form of code, meant for explaining ideas in books. Book code isn't quite like real code, there are some different forces to consider when writing it.

One question is the choice of language. I need to write code in a language that as many of my readers can read and follow. I try to write about ideas that are platform independent, but I need code to be precise. So I need to pick some widely readable language that people can follow.

In my early days the two largest OO languages were Smalltalk and C++. Both had faults, Smalltalk was too weird for non-smalltalkers and C++ was too full of sharp edges to get right. Java was a godsend for me. Anyone with a C/C++ background could read it. Even most smalltalkers could hold their noses and understand what I was coding. That's why the refactoring book was in Java.

Later on .NET appeared. The nice thing here was the C# was mostly the same as Java, so I could use the two pretty interchangeably. I liked to use both to reinforce that the ideas I was writing about were useful in either case.

That situation is getting more difficult these days, particularly with writing about DomainSpecificLanguages. Java and C# are diverging, and some things I want to illustrate require features that neither of them have. I do much of my personal programming in Ruby, which is very well suited to DSLs, so I will use Ruby as my first choice for this situation. But other languages have contributions too. I need to balance illustrating various language capabilities for DSLs against letting the book become a hodgepodge of language tidbits.

Another issue with book code is to beware of using obscure features of the language, where obscure means for my general reader rather than even someone fluent in the language I'm using. A good example of this is that when I write examples using Ruby, I've often shied away from using CollectionClosureMethods, even though I use them heavily in my own Ruby code. This is because I consider that programmers from a curly-brace background will find it harder to understand those kinds of expressions. So I sacrifice fluent Ruby in order to reach those readers.

Again this is much harder for a DSL book. Internal DSLs tend to rely on abusing the native syntax in order to get readability. Much of this abuse involves quirky corners of the language. Again I have to balance showing readable DSL code against wallowing in quirk.

Podcast on DSLs

2007-12-04T00:07:00-05:00

General News: ThoughtWorks has started organizing some podcasts. The first one is on Domain Specific Languages and features Rebecca Parsons, Neal Ford, Jay Fields and myself.

GroovyOrJRuby

2007-11-28T14:29:00-05:00

Currently there's quite a debate raging over the relative merits of Groovy and JRuby as scripting languages running on the Java virtual machine. Curious minds want to know - which of these languages will win this upcoming language war? People want to know which language to pick for a project, or which language to commit to learn.

Perhaps the first thing to point out is that it's perhaps rather unfair to see this as a race between these particular two horses. Scripting has been available on the JVM for a long time. Jython, a Java implementation of Python, has been around for several years. There's plenty of other, more obscure languages, which I daren't mention for fear of offending all the ones I miss out.

JRuby has got a lot of attention due to the attention of the Ruby language generally - attention particularly ignited by the interest around Rails. We've seen a sharp spike of interest around Ruby and Rails work at ThoughtWorks, and JRuby adds an extra dimension since it allows people to deploy Rails applications using their existing Java infrastructure.

Groovy gets its attention because it, more than any other language, is designed to work seamlessly with the JVM, and got a lot of attention from an early JSR.

Personally I'd dropped Groovy from my radar a couple of years ago when its development seemed to bog down. With its 1.0 release and further interesting positive vibes from some of my colleagues I've started to pay attention again.

Lets begin by talking about similarities. Both JRuby and Groovy (and indeed Jython) are modern OO scripting languages. They combine the well-chosen terseness of scripting languages with good solid structures for building larger programs. As such they are suitable both for classical scripting and for writing larger programs. Both are comfortable with dynamic type checking, although Groovy does offer some static facilities too. Both support Closures which are an important feature for the greater expressiveness that people want from this kind of language.

The biggest difference between them is their broader platform philosophy. Groovy is designed to be a scripting language for Java. As much as possible its syntax tries to match the equivalent in Java. (Including such ugly things as the default fall-through in switch statements.) It also works with Java's class library directly, although it dynamically adds many methods to Java's classes, vital in order to make use of things like closures.

JRuby, however, is a Java implementation of the Ruby platform. Ruby can run directly on mainstream operating systems with a C runtime, and is starting to run on .NET's CLR. When you program in JRuby you primarily use Ruby's libraries which are implemented in Java, and may also use Java's libraries at your discretion. If you stick to Ruby's libraries, or at least wrap any foreign elements, you can run Ruby programs on the C, Java, or (in time) .NET runtimes. So you can use JRuby to both run Ruby programs on the JVM and as a language for scripting the JVM.

One of the big differences between JRuby and Jython is around the libraries. One of the tricky aspects of porting this kind of scripting language to the JVM is that these languages are usually closely intertwined with libraries implemented in C. Porting these libraries to Java involves rewriting the libraries in Java. Jython didn't do much of this, as a result many Python apps can't run in Jython. However the JRuby implementers decided from early on that their goal was to run Rails apps, as a result many libraries including all the Ruby standard libraries needed to be ported.

The fact that JRuby is a Ruby platform on the JVM means that in JRuby you have two kinds of objects - JRuby objects and Java objects. Although there are ways for the two to talk to each other and to convert there is a difference. There are times when you need to know whether you're dealing with a Java string or a JRuby string. With Groovy you don't have that boundary, there are just Java objects.

It's too early, or rather too difficult, to say if one language will win out. Both are pretty young, only just finding their feet on the JVM. On a more personal level, your choice has a lot to do with what you expect to do with it. If you are only interested in running on the JVM, then Groovy could well be the easier choice. You are working directly with Java's library and object model, and the syntax requires less getting used to. A strong reason to prefer Ruby is the fact that it lives in multiple implementations. Ruby is a tool you can use in a lot of other places. As a long time Rubyist, there's not much incentive for me personally to get heavily into Groovy, even though I actually like the language a lot from what I've seen of it.

Rails is an important factor. The Java world is hardly lacking in web frameworks, but Rails is widely liked by those who've used it. I've not got many reports yet about Grails (the Groovy knock-off) so can't give a firm opinion on that. But I can imagine that the ability to deploy web apps with Rails could be a major factor in making JRuby popular. Something else to look at is the growth of RSpec as a new spin on testing environments.

With any platform it's as important to consider the people involved in the community as much as any technical factors. Good people can overcome technical weaknesses quickly and a vibrant community is a potent source for big innovations. RubyPeople have formed a particularly strong community, which has already spawned things like Rails, Rake, and Rspec.

Will either matter to Java? After all Jython's been around for a long time without making a huge impact on the JVM. Tool support is frankly pathetic for any of these languages when you compare it to what you have for Java at the moment.

I think we're actually at an inflection point with Java. Until recently the Java cry was One VM and OneLanguage. (As opposed to the CLR which started with the cry of one VM and many languages - providing they're C# or VB.) This seems to be changing as people realize the limitations of Java and begin to seek out different capabilities. It's likely the future will see multiple languages closely integrated within the JVM.

There are plenty of people who dislike the hype around Rails and Ruby. But even if you dislike Ruby, the hype has led to a resurgence of interest in new languages. I doubt if the interest in Groovy would be anywhere near as great as it is if it wasn't for this hype, nor would Jython be awaking from its slumbers. The ruby/rails hype has also generated interest in other exotic JVM languages. The really nice thing here is that the JRuby people have been encouraging their dynamic rivals - recognizing that the point here is to encourage multi-lingual inter-operability and innovation.

DSL book Work In Progress

2007-11-15T16:41:00-05:00

General News: Over the last few months (actually many months), I've been working on a book on Domain Specific Languages. I'm now at the point where I think it's worth pushing out my work in progress. This page will keep you informed on where things are (and there's an atom feed too). Rather than drop it all out in one huge dollop, I shall release what I have so far in bits over the next few weeks. Once I'm caught up I'll release material as I'm writing it.

Modifiability: Or is there Design in Agility

2007-10-15T13:07:00-04:00

General News: The organizers of QCon London, earlier this year, asked me to do a conference session on modifiability of architecture. I thought that rather than listening to me, the audience might prefer listen to some of the ThoughtWorks architects whose ideas I usually repackage: Dave Farley, Ian Cartwright, Fred George, Erik Doernenberg, and Dan North. InfoQ has now put up a video of session.

AltNetConf

2007-10-09T18:38:00-04:00

Last weekend I attended the Alt.NET conference. It was the first named gathering of a group of people I've been watching on the blogosphere for quite a long time. A group of long-time users of Microsoft technologies who feel that their development philosophy has been getting out of sync with the perceived orthodoxy from Redmond. While some have considered moving away this group is keen to stay and try to influence the Microsoft world.

The term alt.net was coined by David Laribee in his blog. The conference was organized using OpenSpace, a style that I thought was particularly apropos to the nature of this community. I'm not claiming to speak for this community, or define it; these words are my interpretations from what I saw and heard.

Alternative

One question that came up was the name alternative. This made a few people uncomfortable as it suggested to them that this was a group in opposition to Microsoft. A different view of "alternative" is that it's something that embraces choice. Many communities believe that having many alternatives makes them stronger (the Unix community leaps to mind). In software development it's rare that one solution is the right one for all possible circumstances. Having alternatives does mean you have to think about which solution is the right one for your situation, but I prefer that to trying to use a hammer to turn a nut. And yes, this does mean that personal experience and preference leads to different choices. We may be programmers but we are still human, not embodied algorithms.

The alt.net mind-set is one that is very familiar to me. It has that mix of agile + object-orientation + patterns + TDD + DDD which is very much the school of software development that I favor. (Lacking a proper name for it, I'm inclined to call it the OOPSLA school of software development.) There is certainly a belief that there is a mainstream Microsoft orthodoxy at the moment, one that doesn't fit the OOSPLA school. And there's some frustration with that. But the point here is that it's not that the alt.net community thinks that the perceived mainstream Microsoft route should be erased, but that the Microsoft world is big enough for different approaches.

As well as discussion about whether "alternative" is a good name, there's also discussion about whether you need a name at all. As a compulsive maker of Neologisms you shouldn't be surprised to find out that I think a name is useful. There's clearly a common style of software development that's formed here and giving it a name makes it easier to talk about. Inevitability some people will be annoyed by coining terms, but I think the usefulness outweighs that objection.

Participative Community

An important feature of alt.net is that it's a participative community. Traditionally when user conferences get together, the vendor is in charge of the agenda. Most sessions are about the vendor showing the community how to use the tools it provides. Good vendors listen to their customer community, reacting by developing new products that respond to their community's wishes.

A participative community is different, they don't just want the vendor to listen and provide suitable products - they want to participate in the development of new products. It's just such a participative community that's taken the initiative in the Java world. JUnit, IBatis, Spring, Hibernate et al didn't come out of the vendors, but were developed by "customers". One of the things about the nature of the software industry is that many customers are every bit as capable of producing vital products as vendor companies, especially when combined with the community and ethos of open source.

The great question ahead for Microsoft is how to engage with a participative and opinionated community like this. Treating such a group as an opponent will result in the loss of valuable products, and more importantly the capable people connected with them. Engaging with a community like this brings great opportunity. I would argue that the participative community around enterprise Java has saved the enterprise Java platform. A big challenge for Microsoft in all this is that this means finding a way to accommodate with open source development. Recent signs, particularly around Iron Ruby, suggest that at least some bits of Microsoft are heading in the right direction.

More signs of the right direction was Scott Guthrie's demonstration of the ASP.NET MVC framework (also see Scott Hanselmans's video). This was very interesting to me, not so much because of the product itself, which didn't seem to have anything particularly innovative (and that's not a complaint), but the philosophical signs around it.

First off there's the fact that Scott Guthrie made the commitment to come to a small, conference like this to reveal this product. Next was the clear design goal around testability. Add a rich sauce of clear understanding and learning from other work in this space. Add a garnish of plugability that allows the framework to work with non-Microsoft tools and encourages extension. Many people present said they hadn't been so excited by a product announcement since .NET.

It's also a fine example of "alternative". The MVC framework isn't intended to replace webforms, programmers can choose whether to use webforms or MVC.

One other issue in a community like this is that it's a community that doesn't equate criticism with animosity. Many vendors suffer from the belief that anyone who criticizes them is their enemy. In truth often your friends are at their most valuable when they are critical. Like any large corporation, Microsoft can exhibit contradictory reactions. There are certainly parts of the organization that do think that friends should never criticize. Part of working with a participative community is to learn to value friendly criticism. Equally people in the community need to learn how to criticize without being nasty - a particularly rare quality in our profession.

Exclusive Community

There's been some debate (particularly in the blogosphere) about whether the alt.net community is an exclusive community (in a bad way). I find a useful way to think about it is a question that did come up a couple of times during the conference. Should people form separate alt.net user groups or should they try to influence and change the current user groups? My answer to this is "both". A focused alt.net user group sets an expectation about the material being discussed and the kind of values and principles that ground the group. People who are doing this style of development need to talk to others doing it so they can learn from them. I've been engaged in this style for over a decade but I still have tons to learn about it. The advantage of a focused gathering is that I get to concentrate on this type of content.

A specialized alt.net user group is only exclusive if it tries to exclude others. The alt.net conference was not exclusive because, at least until the conference filled up, anyone could turn up. As long as an alt.net user group is open to accepting anyone, it's a good thing.

At the same time it's important for alt.net people to engage the wider .NET community. I encourage this style of developing software because I genuinely think it's effective. Therefore I think it's important for me to try and communicate how and why to do it to as big an audience as I can. This way other people can be exposed to the ideas and get the opportunity to understand the techniques so they can choose whether they want to try them. I expect to see many alt.neters looking to talk about these techniques and their experiences with them at a wide range of Microsoft-oriented conferences.

I have high hopes for the alt.net community. I believe this kind of community is important to the viable future of the Microsoft ecosystem, and I want a healthy Microsoft world. My hope is that Microsoft participates in this community, so that many leading Microsofties can happily state they are alt.neters. I hope that alt.neters can strike the tricky balance between sustaining themselves and being open to people coming into the community. I hope I can play a role in making this happen. There was an excellent spirit at this first conference, one that provides lots of good fuel for the future.

RollerSkateImplementation

2007-09-09T15:56:00-04:00

A key property of agile development is figuring out how to make a system go live with a small subset of features. We build software for the business value it offers, the quicker we go live, the faster we get at least some of that business value.

My colleague Dave Leigh-Fellows told me one of my favorite examples of this kind of thinking. It came when we has working for a brokerage firm. They had a new kind of product that they wanted to get into the market. The full software support for this was a web page that the customer filled in that generated the necessary transactions against the back-end system. But Dave came up with a way to get the product into the market faster than that.

Version 1 was a static web page that described the product and provided a telephone number to call. Some temporary staff then spoke to the customer and entered the information into the back end system.
Version 2 was a web form that captured the data the customer filled in. However this version didn't load that data into to the back end system. Instead the web form generated a fax. They hired some more temps to get the orders from the fax machine to the people that keyed the information into the back end system. Since the fax machines were a bit of a distance away, this is where the roller-skates came in.
Version 3 hooked the web form into the back-end system directly.

The first two versions may not have been the most elegant solutions ever conceived, but they did get the product into the market much more quickly. I've not come across any other examples of iterative development that use roller-skates, but that may be more due to lack of imagination rather than lack of need.