Martin Fowler

photostream 25

2012-05-10T12:46:00-04:00

Winterthur, DE

Retreaded: StandardStoryPoints

2012-05-10T12:30:00-04:00

Retread of post orginally made on 06 Sep 2004

I've heard a couple of questions recently about coming up with a standard story point mechanism for multiple teams using extreme programming's planning approach. The hope is have several teams all using equivalent story points, so that three story points of effort on one team is the same as on another.

I think trying to come up with this at best of limited value, and at worst dangerous.

The estimating system of extreme programming is based on XpVelocity and YesterdaysWeather. Inherent in this is the idea that when you make estimates, the actual units you estimate aren't important - what's important is you estimate by rough comparative value and use YesterdaysWeather for calibration.

In this situation the story points act as an anchor for the feedback loop that Yesterday's Weather provides - nothing more. Baked into them are all sorts of assumptions about the nature of the team's task, the capability of the team, and whether the team are optimistic or pessimistic estimators. Once you try to come up with a standard across teams you are trying to normalize all of these factors. Trying to do this sounds very hard to me, and I'm not aware of anyone who has done this effectively. This doesn't mean its impossible, it just means it's hard.

The dangerous aspect comes from once you have a standard unit for measurement across teams, someone is inevitably going to use it to compare the performance of teams. Even if everyone swears till they are blue in the face that they won't use it for cross team measurement, there will always be the suspicion that this will happen eventually. This will cause teams to distort their measurements so that it seems that they get more story points done. My fear is that this will break the feedback loop of yesterday's weather and knock the planning process off kilter. I'm always suspicious about these things because while it would be incredibly valuable to have a way to measure productivity I think the nature of software is such that we CannotMeasureProductivity.

So to be worth trying, this has to yield some valuable benefits - but I don't see any. One reason that I've heard is to help people move onto teams and estimate more quickly. But you can't estimate on a new team until you get reasonably familiar with the problem and the current code base. As you do that you'll also get a feel for the relative sizes of story points on that team. We move people around between projects at ThoughtWorks and I've never heard anyone complain about difficulty of estimating when coming onto a new team due to differences in story points.

reposted on 10 May 2012

Bliki: OrmHate

2012-05-08T11:31:00-04:00

While I was at the QCon conference in London a couple of months ago, it seemed that every talk included some snarky remarks about Object/Relational mapping (ORM) tools. I guess I should read the conference emails sent to speakers more carefully, doubtless there was something in there telling us all to heap scorn upon ORMs at least once every 45 minutes. But as you can tell, I want to push back a bit against this ORM hate - because I think a lot of it is unwarranted.

The charges against them can be summarized in that they are complex, and provide only a leaky abstraction over a relational data store. Their complexity implies a grueling learning curve and often systems using an ORM perform badly - often due to naive interactions with the underlying database.

There is a lot of truth to these charges, but such charges miss a vital piece of context. The object/relational mapping problem is hard. Essentially what you are doing is synchronizing between two quite different representations of data, one in the relational database, and the other in-memory. Although this is usually referred to as object-relational mapping, there is really nothing to do with objects here. By rights it should be referred to as in-memory/relational mapping problem, because it's true of mapping RDBMSs to any in-memory data structure. In-memory data structures offer much more flexibility than relational models, so to program effectively most people want to use the more varied in-memory structures and thus are faced with mapping that back to relations for the database.

The mapping is further complicated because you can make changes on either side that have to be mapped to the other. More complication arrives since you can have multiple people accessing and modifying the database simultaneously. The ORM has to handle this concurrency because you can't just rely on transactions- in most cases, you can't hold transactions open while you fiddle with the data in-memory.

I think that if you're going to dump on something in the way many people do about ORMs, you have to state the alternative. What do you do instead of an ORM? The cheap shots I usually hear ignore this, because this is where it gets messy. Basically it boils down to two strategies, solve the problem differently (and better), or avoid the problem. Both of these have significant flaws.

A better solution

Listening to some critics, you'd think that the best thing for a modern software developer to do is roll their own ORM. The implication is that tools like Hibernate and Active Record have just become bloatware, so you should come up with your own lightweight alternative. Now I've spent many an hour griping at bloatware, but ORMs really don't fit the bill - and I say this with bitter memory. For much of the 90's I saw project after project deal with the object/relational mapping problem by writing their own framework - it was always much tougher than people imagined. Usually you'd get enough early success to commit deeply to the framework and only after a while did you realize you were in a quagmire - this is where I sympathize greatly with Ted Neward's famous quote that object-relational mapping is the Vietnam of Computer Science [1].

The widely available open source ORMs (such as iBatis, Hibernate, and Active Record) did a great deal to remove this problem [2]. Certainly they are not trivial tools to use, as I said the underlying problem is hard, but you don't have to deal with the full experience of writing that stuff (the horror, the horror). However much you may hate using an ORM, take my word for it - you're better off.

I've often felt that much of the frustration with ORMs is about inflated expectations. Many people treat the relational database "like a crazy aunt who's shut up in an attic and whom nobody wants to talk about"[3]. In this world-view they just want to deal with in-memory data-structures and let the ORM deal with the database. This way of thinking can work for small applications and loads, but it soon falls apart once the going gets tough. Essentially the ORM can handle about 80-90% of the mapping problems, but that last chunk always needs careful work by somebody who really understands how a relational database works.

This is where the criticism comes that ORM is a leaky abstraction. This is true, but isn't necessarily a reason to avoid them. Mapping to a relational database involves lots of repetitive, boiler-plate code. A framework that allows me to avoid 80% of that is worthwhile even if it is only 80%. The problem is in me for pretending it's 100% when it isn't. David Heinemeier Hansson, of Active Record fame, has always argued that if you are writing an application backed by a relational database you should damn well know how a relational database works. Active Record is designed with that in mind, it takes care of boring stuff, but provides manholes so you can get down with the SQL when you have to. That's a far better approach to thinking about the role an ORM should play.

There's a consequence to this more limited expectation of what an ORM should do. I often hear people complain that they are forced to compromise their object model to make it more relational in order to please the ORM. Actually I think this is an inevitable consequence of using a relational database - you either have to make your in-memory model more relational, or you complicate your mapping code. I think it's perfectly reasonable to have a more relational domain model in order to simplify your object-relational mapping. That doesn't mean you should always follow the relational model exactly, but it does mean that you take into account the mapping complexity as part of your domain model design.

So am I saying that you should always use an existing ORM rather than doing something yourself? Well I've learned to always avoid saying "always". One exception that comes to mind is when you're only reading from the database. ORMs are complex because they have to handle a bi-directional mapping. A uni-directional problem is much easier to work with, particularly if your needs aren't too complex and you are comfortable with SQL. This is one of the arguments for CQRS.

So most of the time the mapping is a complicated problem, and you're better off using an admittedly complicated tool than starting a land war in Asia. But then there is the second alternative I mentioned earlier - can you avoid the problem?

Avoiding the problem

To avoid the mapping problem you have two alternatives. Either you use the relational model in memory, or you don't use it in the database.

To use a relational model in memory basically means programming in terms of relations, right the way through your application. In many ways this is what the 90's CRUD tools gave you. They work very well for applications where you're just pushing data to the screen and back, or for applications where your logic is well expressed in terms of SQL queries. Some problems are well suited for this approach, so if you can do this, you should. But its flaw is that often you can't.

When it comes to not using relational databases on the disk, there rises a whole bunch of new champions and old memories. In the 90's many of us (yes including me) thought that object databases would solve the problem by eliminating relations on the disk. We all know how that worked out. But there is now the new crew of NoSQL databases - will these allow us to finesse the ORM quagmire and allow us to shock-and-awe our data storage?

As you might have gathered, I think NoSQL is technology to be taken very seriously. If you have an application problem that maps well to a NoSQL data model - such as aggregates or graphs - then you can avoid the nastiness of mapping completely. Indeed this is often a reason I've heard teams go with a NoSQL solution. This is, I think, a viable route to go - hence my interest in increasing our understanding of NoSQL systems. But even so it only works when the fit between the application model and the NoSQL data model is good. Not all problems are technically suitable for a NoSQL database. And of course there are many situations where you're stuck with a relational model anyway. Maybe it's a corporate standard that you can't jump over, maybe you can't persuade your colleagues to accept the risks of an immature technology. In this case you can't avoid the mapping problem.

So ORMs help us deal with a very real problem for most enterprise applications. It's true they are often misused, and sometimes the underlying problem can be avoided. They aren't pretty tools, but then the problem they tackle isn't exactly cuddly either. I think they deserve a little more respect and a lot more understanding.

1: I have to confess a deep sense of conflict with the Vietnam analogy. At one level it seems like a case of the pathetic overblowing of software development's problems to compare a tricky technology to war. Nasty the programming may be, but you're still in a relatively comfy chair, usually with air conditioning, and bug-hunting doesn't involve bullets coming at you. But on another level, the phrase certainly resonates with the feeling of being sucked into a quagmire.

2: There were also commercial ORMs, such as TOPLink and Kodo. But the approachability of open source tools meant they became dominant.

3: I like this phrase so much I feel compelled to subject it to re-use.

photostream 24

2012-05-05T10:04:00-04:00

Boston, MA

Retreaded: PublishedInterface

2012-05-03T10:18:00-04:00

Retread of post orginally made on 26 Dec 2003

Published Interface is a term I used (first in Refactoring) to refer to a class interface that's used outside the code base that it's defined in. As such it means more than public in Java and indeed even more than a non-internal public in C#. In my column for IEEE Software I argued that the distinction between published and public is actually more important than that between public and private.

The reason is that with a non-published interface you can change it and update the calling code since it is all within a single code base. Such things as renames can be done, and done easily with modern refactoring tools. But anything published so you can't reach the calling code needs more complicated treatment.

reposted on 03 May 2012

Bliki: TestPyramid

2012-05-01T11:27:00-04:00

The test pyramid is a concept developed by Mike Cohn, described in his book Succeeding with Agile. Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI.

For much of my career test automation meant tests that drove an application through its user-interface. Such tools would often provide the facility to record an interaction with the application and then allow you to play back that interaction, checking that the application returned the same results. Such an approach works well initially. It's easy to record tests, and the tests can be recorded by people with no knowledge of programming.

But this kind of approach quickly runs into trouble, becoming an ice-cream cone. Testing through the UI like this is slow, increasing build times. Often it requires installed licences for the test automation software, which means it can only be done on particular machines. Usually these cannot easily be run in a "headless" mode, monitored by scripts to put in a proper deployment pipeline.

Most importantly such tests are very brittle. An enhancement to the system can easily end up breaking lots of such tests, which then have to be re-recorded. You can reduce this problem by abandoning record-playback tools, but that makes the tests harder to write. [1] Even with good practices on writing them, end-to-end tests are more prone to non-determinism problems, which can undermine trust in them. In short, tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run. So the pyramid argues that you should do much more automated testing through unit tests than you should through traditional GUI based testing.

The pyramid also argues for an intermediate layer of tests that act through a service layer of an application, what I refer to as SubcutaneousTests. These can provide many of the advantages of end-to-end tests but avoid many of the complexities of dealing with UI frameworks. In web applications this would correspond to testing through an API layer while the top UI part of the pyramid would correspond to tests using something like Selenium or Sahi..

The test pyramid comes up a lot in Agile testing circles and while its core message is sound, there is much more to say about building a well-balanced test portfolio. In particular a common problem is that teams conflate the concepts of end-to-end tests, UI tests, and customer facing tests. These are all orthogonal characteristics. For example a rich javascript UI should have most of its UI behavior tested with javascript unit tests using something like Jasmine. A complex set of business rules could have tests captured in a customer-facing form, but run just on the relevant module much as unit tests are.

In particular I always argue that high-level tests are there as a second line of test defense. If you get a failure in a high level test, not just do you have a bug in your functional code, you also have a missing unit test. Thus whenever you fix a failing end-to-end test, you should be adding unit tests too.

1: Record-playback tools are almost always a bad idea for any kind of automation, since they resist changeability and obstruct useful abstractions. They are only worth having as a tool to generate fragments of scripts which you can then edit as a proper programming language, in the manner of Twist or Emacs.

photostream 23

2012-04-27T21:10:00-04:00

Middleton Place, Charleston, SC

An English Literature Podcast

2012-04-26T19:02:00-04:00

I’ve added another recommended podcast to my list, this one a is a treasure if you enjoy good fiction. Why I Really Like This Book is a quirky sail through the lesser known waters of English Literature. The pilot is Kate Macdonald, who teaches English Literature but was frustrated at the limited range of books she could cover. Each podcast spends ten minutes or so exploring a single suggested book. Her first series was a random alphabetical wander, now she’s taken to talking about sets of five books with a common theme. Recent sets include Five Appalling Fictional Women and Five Classic Detective Novels. If you’re looking for ideas for books to curl up with, this is an excellent source of suggestions.

Retreaded: CustomerAffinity

2012-04-26T09:55:00-04:00

Retread of post orginally made on 28 Jul 2006

When someone is looking at what makes up a top-class enterprise software developer, often the conversation may turn to knowledge of frameworks and languages, or perhaps the ability to understand complicated algorithms and data structures. For me, one of the most important traits in a programmer, or indeed in a development team, is something that I'll call Customer Affinity. This is the interest and closeness that the developers have in the business problem that the software is addressing, and in the people who live in that business world.

There are many aspects to customer affinity. The first one is just the interest in the business, its processes and rules. I've always been fascinated by domains I've worked in: health care, derivatives, payroll, leasing - are all examples of really interesting domain problems with a lot of complexity that has to be organized and understood. For me aspects of a project such as object-relational mapping, remote process communication - what I think of as the plumbing of enterprise systems, don't have that same interest.

It's important that a team has the plumbing working and under control, but I believe that the more energy goes into the business problem, the more effective at providing value a team will be. Which is a good reason to use frameworks that solve and simplify as much of the plumbing as possible.

Another aspect of customer affinity is the ability to collaborate with domain experts. I don't think that detailed knowledge of the domain is an important thing for programmers to have; useful yes, but not important. What's more critical than knowledge is the ability to collaborate with those that have the knowledge.

My high regard for customer affinity is one the main reasons why I'm such a fan of Extreme Programming and other agile methods. I found it very significant that when Kent Beck summarized XP to his agile peers at the snowbird workshop which coined 'agile', he chose to describe not the technical elements of XP, but his desire to change the nature of the customer/developer interaction.

I've often heard this relationship mis-characterized. In particular there is a belief in some quarters that the customer team just comes up with stories which they throw at the developers. This characterization is rather like the notion that requirements are just lying out there to be gathered. Either way that's not much of a collaboration. Instead developers need to work together with domain people to generate ideas with developers learning a lot about the business in the process.

One of Kent's suggested names for 'Agile' was conversational software development - the point being that it's a two way communication. This isn't something like a telecoms protocol that you can define, but the back and forth discussions about how software can enhance the business are where the real value lives. Much of this conversation is of half-baked ideas, some of which grow into valuable features - often ones that aren't things that the customer originally thought of.

One of the things that frustrates me is how organizations often try to squelch customer affinity. It's not something people admit to doing, but it's a common consequence of other decisions. Organizational barriers can contribute to squelching - I've come across places where you have to get your manager to arrange with another manager just so you can have a conversation with someone on the business side. That hardly encourages you to get inside the workings of the business.

I've often heard it said that enterprise software is boring, just shuffling data around, that people of talent will do "real" software that requires fancy algorithms, hardware hacks, or plenty of math. I feel that this usually happens due to a lack of customer affinity. The real intellectual challenge of business software is figuring out what the real contribution of software can be to a business. You need both good technical and business knowledge to find that. Working closely with business people to develop this knowledge, and PleasingTheCustomer as you do it, is what makes enterprise software development fun - and motivation is the key to good and productive work.

There are plenty of bright and capable people who want to learn about the businesses they write software for. Too often organizations make it hard for them to do so. Until that changes, our profession will continue to under-deliver on our potential.

reposted on 26 Apr 2012

Bliki: EditingPublishingSeparation

2012-04-24T10:41:00-04:00

In my conversations with ThoughtWorks project teams over the last year or so, a regular theme has been the growing impact of content management systems (CMS). They aren't usually seen as helpful, indeed there is a clear sign that they are becoming a worryingly invasive tool - used for more than their core purpose in such a manner that they hinder overall development.

Amongst the other irritations, a common failing is that they keep one copy of each article[1]. This single copy is edited as part of creating the content and published to readers (usually on some kind of state-change flag).

The idea of keeping a single copy of some piece of data is a common one. It's the underlying principle behind the relational concept of normalization, and enterprise architects often try to ensure that critical data has a single authoritative copy.

Yet for a CMS there's a clear downside - the data access patterns for editing and publication are very different. Editing involves small number of people frequently accessing the article, doing both reads and updates. Publishing involves many more people (we hope) accessing the article, but all doing reads. There are some edits done to fix problems in published articles, but these are far fewer than the reads and are from a well-controlled group of people.

With two such different access paths, a few CMSs keep separate copies of the articles, controlled by relatively independent modules. The editing module is geared around the frequent updates, it provides support for editing, tracking changes and monitoring the workflow of the editing process. When an article is published it's copied over the publishing module.

The publishing module treats the article as largely read-only, updated rarely and only by the editing module. Consequently the publishing module is designed around serving that article to a large number of readers. At the least this involves a different configuration of the data storage. The publishing module can be freely replicated across many nodes in a cluster, while it's usually better for the editing module to be centralized on a single node. There's also an argument for different data storage technology, allowing each module to use something appropriate to its access patterns.

The articles can also be stored in different formats. Often articles are edited in one form but published in another, such as editing in markdown and publishing in html. In this case the editing module should store the markdown form while the publishing module stores html. The publishing module can also do some page composition work on the stored copy. So if you have a static header, this can be added to the stored article's html when the article is published, saving the effort of recomposing it for every read.[2]

Separating these modules can also help with the editing workflow. Often people want to preview changes before publishing to the world, this is easy to do with the separation because you can publish to a private publishing module in a staging area. This can nicely finesse otherwise awkward logic to figure out what to publish from single storage.

User-generated content does add something of a wrinkle to this approach. A wiki, which is entirely user-generated, is going to have a larger and less-well controlled group of editors than a curated site. Similarly reader comments will come from a wider range of writers. But even with user-generated content you should get many more readers than writers, so it makes sense to separate handling updates from serving published pages.

Teams that use the rare systems that support editing-publishing separation find it works very well and most teams who use tools that don't have this style think it would improve things. If you're evaluating a CMS, or building one for your own needs, you should certainly consider editing-publishing separation as a key feature to look for.

1: I'm using "article" here to mean any item of content that the CMS manages.

2: This does mean that any change to the header would require all the published articles to be rebuilt. Usually this would not be an issue compared to composing the page for each read.

photostream 22

2012-04-17T15:35:00-04:00

London

Bliki: TestCoverage

2012-04-17T11:17:00-04:00

From time to time I hear people asking what value of test coverage (also called code coverage) they should aim for, or stating their coverage levels with pride. Such statements miss the point. Test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are.

Let's look at the second statement first. I've heard of places that may say things like "you can't go into production with less than 87% coverage". I've heard some people say that you should use things like TDD and must get 100% coverage. A wise man once said:

I expect a high level of coverage. Sometimes managers require one. There's a subtle difference.

-- Brian Marick

If you make a certain level of coverage a target, people will try to attain it. The trouble is that high coverage numbers are too easy to reach with low quality testing. At the most absurd level you have AssertionFreeTesting. But even without that you get lots of tests looking for things that rarely go wrong distracting you from testing the things that really matter.

Like most aspects of programming, testing requires thoughtfulness. TDD is a very useful, but certainly not sufficient, tool to help you get good tests. If you are testing thoughtfully and well, I would expect a coverage percentage in the upper 80s or 90s. I would be suspicious of anything like 100% - it would smell of someone writing tests to make the coverage numbers happy, but not thinking about what they are doing.

The reason, of course, why people focus on coverage numbers is because they want to know if they are testing enough. Certainly low coverage numbers, say below half, are a sign of trouble. But high numbers don't necessarily mean much, and lead to ignorance-promoting dashboards. Sufficiency of testing is much more complicated attribute than coverage can answer. I would say you are doing enough testing if the following is true:

You rarely get bugs that escape into production, and
You are rarely hesitant to change some code for fear it will cause production bugs.

Can you test too much? Sure you can. You are testing too much if you can remove tests while still having enough. But this is a difficult thing to sense. One sign you are testing too much is if your tests are slowing you down. If it seems like a simple change to code causes excessively long changes to tests, that's a sign that there's a problem with the tests. This may not be so much that you are testing too many things, but that you have duplication in your tests.

Some people think that you have too many tests if they take too long to run. I'm less convinced by this argument. You can always move slow tests to a later stage in your deployment pipeline, or even pull them out of the pipeline and run them periodically. Doing these things will slow down the feedback from those tests, but that's part of the trade-off of build times versus test confidence.

So what is the value of coverage analysis again? Well it helps you find which bits of your code aren't being tested. [1] It's worth running coverage tools every so often and looking at these bits of untested code. Do they worry you that they aren't being tested?

If a part of your test suite is weak in a way that coverage can detect, it's likely also weak in a way coverage can't detect.

-- Brian Marick

Progress on NoSQL Distilled

2012-04-17T10:42:00-04:00

Time for a progress report on NoSQL Distilled. We’ve done a round of technical review and are now pretty happy with the text, happy enough to move it into production.

Production for a book these days is both more complicated and more responsive than it used to be. The next step on the book text is to do copy-editing and indexing. It sounds like we are going to get the same people I worked with on my DSL book, which will be excellent news since they did a good job and were happy to work directly on the XML source in my repository. This simplifies and speeds up the whole process. They will also do the book design and convert the book in camera-ready for printing the physical books.

I suspect, however, that this book will see more electronic copies than physical ones. In parallel with the copy-editing we’ll be preparing the electronic copies. We intend to do a rough-cut version of the book as soon as we can. This should be available through InformIT, Amazon, and Safari. I’ll make further announcements here as these become visible.

Rebecca Parsons and I speak on The Evolving Panorama of Data

2012-04-16T10:57:00-04:00

Our keynote at QCon London 2012 looks at the role data is playing in our lives (and that it’s doing more than just getting bigger).

How the world of data is changing: its growing, becoming more distributed and connected.
The industry’s response: the rise of NoSQL, the shift to service integration, the appearance of event sourcing, the impact of clouds and new analytics with a greater role for visualization.
How data is being used now, with a particular emphasis from Rebecca on data in the developing world.
What does all this mean to our personal responsibilities as software professionals.

Jez Humble and I speak on Continuous Delivery at YOW

2012-02-14T15:46:00-05:00

Video of the one-hour overview presentation that Jez and I gave at YOW at the end of last year. We cover the rationale for Continuous Delivery, the deployment pipeline, continuous integration, devops, and deployment strategies. The highlight is Jez’s personification of a release candidate as a hero in a greek myth.

photostream 21

2012-02-12T18:32:00-05:00

Èze, France

Bliki: PresentationSmells

2012-02-09T08:16:00-05:00

I've given lots of presentations, and since I go to a lot of conferences I see a lot too. This means I see a lot of problems, where people are doing things that reduce the efficacy of their talks. I've not tried to come up with a comprehensive list, so the ones I'm raising here are just a few things off the top of my head. Like most smells, these aren't always wrong, but should always make you think.

Sentences on slides. I could say bullet points here, as that's the most common way to misuse slides by turning them into Slideuments. But I think the real rogue is the sentence. My approach with slides is to treat them as a VisualChannel, following this it's important that the audience is listening to the speaker for the flow of words. Reading sentences breaks that up.

Floodmarks. Lots of slides have repetitive elements on the slide master that show up on every slide, these include corporate logos, legal boilerplate (such as copyrights), occasionally even tag lines. The audience either tunes out the floodmark (which makes them useless) or is distracted by them (which is worse). I recommend you only put logos and the like on the first and last slide, that's quite enough to get them noticed, but not enough to be distracting. You only need a copyright statement on one slide - and even then I'd argue you don't really need it.[1] Are your slides really so amazing that you'll lose a fortune if someone copies them?

Repetitive Pointer Movement. It's wise to always carry a laser pointer when giving a talk - indeed I suggest a combined remote-clicker laser device. But if you find yourself pointing to the same sequence of things every time you give the talk, then you should put that pointer sequence into the slides themselves. This may be as simple as an arrow or a highlight on the existing slides, or it may be rethinking the slides themselves to help bring out the point better.

Backtracking Slides. If you find yourself going back to a previous slide, that indicates the slide sequence isn't right. Instead you should copy the slide so that you keep going forwards through the deck. Once you've copied it, you can then think about whether it should be a true copy, or whether it should look different when used again in order to highlight how you are using it in the further sequence.

Slide Titles It seems to be mandatory that every slide has a title - but ask yourself if the slide's title is really needed. I've found I've often improved the visual channel by consciously removing slide titles.

Having said all this, the most important thing is your content. If the content isn't solid, or you don't really care about it, then even outstanding slidecraft isn't going to save you. Conversely an audience will forgive all sorts of crummy presentation techniques if you have interesting things to say.

1: In many places there is implied copyright, even if you don't have a statement on a document. Even so, it's wise to keep one on the document somewhere to make it easier should you need to enforce copyright. I have to say, however, that I haven't heard of anyone suing anyone over copyright infringement of a slide deck.

ThoughtWorks recruiting event in New York

2012-02-07T15:17:00-05:00

This weekend, ThoughtWorks is hold a recruiting event in New York City. The idea is to compress the whole recruiting cycle down to a single weekend. The coding problem will be on Saturday, and those that pass that hurdle will be invited back for interviews on Sunday. The idea is that we’ll get job offers to any successful candidates by Sunday evening. I’ll be there for the first couple of hours on Saturday.

Infodeck on NoSQL and Polyglot Persistence

2012-02-07T10:26:00-05:00

Pramod and I have put together a short infodeck (pdf) outlining the reasons why NoSQL databases are an important new dish on the software development menu. It’s directed at non-technical managers and customers to provide an outline of why the data world has seen this new entrant and some direction as to the kinds of projects which should give them a closer look.

photostream 20

2012-02-04T16:54:00-05:00

Stowe, VT

Rebecca and I will be Keynoting at QCon London

2012-01-26T10:24:00-05:00

QCon invited Rebecca and I to do a keynote at QCon London (March 7). Over the last year or so we’ve been seeing more and more of a change in how organizations think about data, so that seems a natural choice to talk about. Our clients are seeing larger volumes of data, that they need to analyze more quickly and more cleverly. We’re seeing different technologies for data storage that have loosened the vice-like grip of relational databases. And we find our profession needs to start thinking about what our responsibilities are in managing all of this information.

Rebecca and I talk about DSLs on Software Engineering Radio

2012-01-26T09:36:00-05:00

I’ve long been a fan of the podcast series Software Engineering Radio. The team there have built an excellent series of podcasts on various aspects of software development and I often listen to them while taking my afternoon walk. So I’m glad to get a spot on there myself. In this episode I’m on the program with Rebecca Parsons, my colleague and co-author of my DSL book to talk about DSLs. We talk about what DSLs are, the differences between internal and external DSLs, and when you should (and shouldn’t use DSLs). The episode is hosted by Markus Völter, who, of course, is a considerable expert on DSLs too - so it’s really a three way conversation on the topic.

Bliki: CharityCodeJam

2012-01-25T09:39:00-05:00

Over the last couple of years several of my colleagues have been organizing code jam[1] events where developers get together to write software for charitable causes. A good example is a regular code-jam in New York that works on RapidFTR. Chris George, a ThoughtWorker based in New York, helped organize a one-off event in New York in August 2010. The group didn't get as much done on the day as they had hoped, but in a bar afterwards decided to try to get together more regularly. Since then they've been meeting every week. It's a small group, still mostly ThoughtWorkers and friends, with a core of 3-4 people rising to a dozen when we've had a big project in town.[2]. (Chris is happy to have more people join the group, so if you are interested drop him an email.)

Many people have found these events to an enjoyable way to use our skills for purposes that we find rather more fulfilling than many day jobs, and a way both to learn new skills and learn from a different group of people. So I thought I should share our thoughts on how to set one up.

The first thing is to find a suitable effort to contribute to. We look to contribute to projects producing open-source software for NGOs - the open source model fits well for such organizations. The two that we've built up most of a relationship are RapidFTR and OpenMRS. RapidFTR is a system designed to help reunite families after a natural disaster or other calamity. It allows people to quickly input information about either a lost child, or a child found without parents - then provide search facilities to match them up. OpenMRS is an open source medical records system, designed to support various forms of health care delivery work. It's used by many health care groups all over the world (and not just in the developing world).

Like New York, most of our code-jams begin as one-off events, a single evening or all day event. These days we advertise heavily, and hope to get a good sample of local developers to come along and take part. One-off code jams like this don't usually produce much useful software, since they are too short to really accomplish much. But they still have value. Firstly they generate awareness, exposing the local development community both to the specific project and the notion of working on open-source efforts for NGOs. More directly they can be the seed for an regular code jam, so it's useful to put together activities that will encourage getting back together later.

A regular code jam gets together on a schedule, with core of people who come most weeks. Such a group can make some significant contributions to a code base. People come because they get to work on some different technologies, with a different group of co-developers, to an audience that (unlike most open-source projects) isn't just other developers.

To make meaningful progress, you need someone to prepare for each code jam by breaking down work-items into something small enough that people will be able to finish them during the time at the jam. Whatever people may say and hope, they'll rarely work on the project outside code jam hours, and the schedule is too infrequent to want half-done things hanging over. Small tasks allow teams to make perceptible progress each jam - which helps keep motivation high. We like to put these tasks online before each event so people can prepare if they want to, or just get a feel for what we're working on. We also set up a mailing list to keep up regular communication on the jam and support anyone who does contribute outside of the jam.

Our regular code-jams succeed best when the group has a couple of champions who take the lead in organizing the event. It's best to have more than one champion, to cope with the work load and provide some resilience if they are absent for a while.

We try to ensure the development environment is set up to allow people to come in quickly and become productive. Much of this is the same kind of thing that we do on projects anyway to enable continuous integration - make sure that installation and build are automated so people can quickly install the code base and get it working. It's important to mention this in the advertisements for the event - people are often put off from coming due to a concern that they'll never get started due to these issues. Even so, make sure that each event has at least one person who is familiar with the code base and build environment, she can then help others find their way around. Often someone will give a short overview of what the system does and how it works for new people at the start of the jam.

We usually provide food to each event - that's an easy thing for us to do as a corporate contribution. As any XPer knows, sharing food when working is an important part of gelling a team.

So, if the idea of a code-jam appeals to you, why not give it a try? Find a suitable project to contribute to, a group of a few people to act as a core, and spend a few sessions to get things going. (There are developer guides for both OpenMRS and RapidFTR to help you get started if those projects appeal to you.) If you get going on a stable basis - post in a blog somewhere so we see what code-jams are avaialable and learn more about how to get them going.

1: "Code-jam" is a problematic name for these events. As far as I can determine, the term "code-jam" was originally used for competitive events where programmers would try to best their peers in some programming challenge. The events I describe here are the utter opposite of this, on many levels, but have attracted the same name.

2: When one of the team went down to our Porto Alegre office, he got a group contributing there too.

photostream 19

2012-01-22T19:49:00-05:00

Boston, MA

Bliki: AggregateOrientedDatabase

2012-01-19T09:11:00-05:00

One of the first topics to spring to mind as we worked on NosqlDistilled was that NoSQL databases use different data models than the relational model. Most sources I've looked at mention at least four groups of data model: key-value, document, column-family, and graph. Looking at this list, there's a big similarity between the first three - all have a fundamental unit of storage which is a rich structure of closely related data: for key-value stores it's the value, for document stores it's the document, and for column-family stores it's the column family. In DDD terms, this group of data is an aggregate.

The rise of NoSQL databases has been driven primarily by the desire to store data effectively on large clusters - such as the setups used by Google and Amazon. Relational databases were not designed with clusters in mind, which is why people have cast around for an alternative. Storing aggregates as fundamental units makes a lot of sense for running on a cluster. Aggregates make natural units for distribution strategies such as sharding, since you have a large clump of data that you expect to be accessed together.

An aggregate also makes a lot of sense to an application programmer. If you're capturing a screenful of information and storing it in a relational database, you have to decompose that information into rows before storing it away.

An aggregate makes for a much simpler mapping - which is why many early adopters of NoSQL databases report that it's an easier programming model.

This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster.

There is a significant downside - the whole approach works really well when data access is aligned with the aggregates, but what if you want to look at the data in a different way? Order entry naturally stores orders as aggregates, but analyzing product sales cuts across the aggregate structure. The advantage of not using an aggregate structure in the database is that it allows you to slice and dice your data different ways for different audiences.

This is why aggregate-oriented stores talk so much about map-reduce - which is a programming pattern that's well suited to running on clusters. Map-reduce jobs can reorganize the data into different groups for different readers - what many people refer to as materialized views. But it's more work to do this than using the relational model.

This is part of the argument for PolyglotPersistence - use aggregate-oriented databases when you are manipulating clear aggregates (especially if you are running on a cluster) and use relational databases (or a graph database) when you want to manipulate that data in different ways.

An open letter to Pearson opposing their support of SOPA and PIPA

2012-01-12T14:04:00-05:00

Jez Humble and I have written an open letter to Pearson opposing their support of the SOPA and PIPA bills currently under consideration from the US congress.

Bliki: DiversityImbalance

2012-01-11T10:57:00-05:00

Although it's easy to become accustomed to it, it's pretty obvious the software development world has some serious issues in diversity. By this I mean that we have some notable differences in proportions of people compared to the general population. One of the most obvious differences is the low proportion of women, which is true all over the world (albeit noticeably less so in China). In the US, where I spend a good chunk of my time, the lack of African-Americans is also obvious. There's a lot been written on why such imbalances might exist, and what might be done about it. [1] But here I want to concentrate on a more fundamental question - does it matter?

One point of view I hear fairly regularly is that these diversity imbalances are natural - because women don't have the aptitude or inclination for programming. This point of view upsets a lot of people but I think it's important to treat it seriously. I think of it as a hypothesis, which I'll call the natural balance hypothesis. It needs to be treated seriously because there's plenty of people who feel it explains the current situation - but I argue that it has two serious flaws, which mean that I must vigorously reject it.

The first flaw is a simple one of evidence. There are (roughly) 50% women in the world, so we should expect the ratio for women in computing to be 50% - unless there's real evidence that some other ratio is natural.[2] So far there's no such evidence. Sure, it's obvious that there are biological differences between men and women, and there is evidence that there are differences in brain function between the sexes. But there is no evidence that indicates that the skills that make people better programmers are more common in men.

The only evidence that seems to occur to people who promote the natural balance hypothesis is the fact that there are less female programmers.[3] Personally I find it troubling when software professionals, who ought to be good at logical thinking, can reach so easily for such circular logic.

It is the second flaw in the natural imbalance hypothesis that brings the heat into the discussion. Men have spent centuries using this kind of argument to deny women equal rights in all sorts of fields. Over the last century we've seen tons of evidence that this isn't true elsewhere, so why should it be true in software? As far as I'm concerned this shoddy history should make us doubly wary of the any suggestion that a diversity imbalance is natural. Unless someone comes up with decent evidence that there is a relevant biological difference, we must operate on the assumption that women are equally well suited to programming.

You'll notice above that I said "inclination or aptitude". I've noticed a lot recently that advocates of the natural-balance hypothesis say less frequently that women have less aptitude than men for programming, instead they say that women don't want to do programming. But making statements with inclination is little better than with aptitude - there's still no evidence and it has just the same shoddy history.

So, accepting that there is no good reason for a diversity imbalance, does such an imbalance matter? That is, given we have a unnatural imbalance, is it a problem that's sufficiently serious to spend energy on fixing it? I think there are many reasons why it matters to tackle our imbalances. First and foremost, there's a moral argument. I'm a strong meritocrat, who believes that we should strive for a society where everyone has an equal opportunity to fulfill their potential. A diversity imbalance suggest that there are many women, who would have good careers as programmers, who are not getting the opportunity to do so. I agree with Eric Ries's view that diversity imbalances suggest we are not as meritocratic as we like to think we are.

This waste hurts our profession too. We need more and better software developers to produce valuable software that improves our lives. By not bringing enough women into the profession, we are handicapping ourselves. This will only become more serious as the demand for talent increases in the future. How can we say we are hiring the best people when we ignore significant chunks of our population. Critics of efforts to fix the diversity imbalance often fret that we risk failing to hire a well-qualified male, when we habitually fail to hire well-qualified females.

Lack of diversity is itself a problem. Different people think differently, and consequently come up with different ways to solve problems. If you have a bunch of people with the same background, they miss lots of ideas - leading to inefficiencies and lack of innovation. A diverse group is usually more effective.

This lack of diversity also contributes to our marginalization as a profession. We are already in the situation where the opinions of programmers aren't taken as seriously as they should be by people outside our profession. We see this regularly in our discussions with business people who dismiss us as mere nerds. A diversity imbalance makes us look even more like some marginalizable outsider.

Left to themselves, these kinds of imbalances tend to get worse. People have a natural, often sub-conscious, tendency to be around people like themselves. Consequently as a group becomes a smaller minority, they get excluded more. A warning sign is when people are turned away because they "won't fit in".

There is a great deal of good potential in the software profession. We have a strong tendency towards meritocracy, a natural first position with exploiting the power of computers to enhance our lives, a lack of historical baggage in how we organize ourselves and our work. As a result I think we could provide a model to influence other social groups and lead the way in demonstrating how humans can collaborate. Diversity imbalances are a cancer to that position - how can we claim to be forward thinking when our diversity looks it comes from where the rest of the world was a couple of generations ago?

photostream 18

2012-01-10T18:25:00-05:00

Lindau, Germany

Bliki: NosqlDefinition

2012-01-09T09:30:00-05:00

As soon as we started work on NosqlDistilled we were faced with a tricky conundrum - what are we writing about? What exactly is a NoSQL database? There's no strong definition of the concept out there, no trademarks, no standard group, not even a manifesto.

The term originally surfaced at an informal meetup on June 11 2009 in San Francisco organized by Johan Oskarsson. At the session there were presentations from Voldemort, Cassandra, Dynomite, HBase, Hypertable, CouchDB, and MongoDB. The term caught on rapidly and few people would argue that only the databases mentioned at that meeting should be called NoSQL.

Indeed there's often a twist in the name itself: many advocates of NoSQL say that it does not mean a "no" to SQL, rather it means Not Only SQL. On this point I think it's useful to separate an individual database from the kind of ecosystem that NoSQL advocates see as the future. When we say "x is a NoSQL database" I think it's silly to interpret NoSQL as "Not Only" because that would render the term meaningless. (You could then reasonably argue that SQL Server (say) is a NoSQL database.) So I think it's best to say a "NoSQL database" is a "no-sql" database. You should separately interpret the NoSQL ecosystem as a "not only" - although I prefer the term PolyglotPersistence for this usage. [1]

Even with this matter out of the way, it's still not easy to define a NoSQL database. Does any database that doesn't use SQL qualify? How about older database technologies such as IMS or MUMPS? How about a relational system that didn't have SQL (such as the early Ingres)? What happens if someone manages to bolt a SQL interface onto one of the original septet?

So for our book we took a view that NoSQL refers to a particular rush of recent databases. Some characteristics are common amongst these databases, but none are definitional.

Not using the relational model (nor the SQL language)
Open source
Designed to run on large clusters
Based on the needs of 21st century web properties
No schema, allowing fields to be added to any record without controls

While I'm used to the blurry lines of definitions in the software industry, I confess my heart sinks at yet another one. But the important thing is that these databases provide a important addition to the way we'll be building application in next couple of decades. A lack of a clear definition will be no more than a gnat bite on their future successes.

1: If we take the "not-only" interpretation, then we should write "NOSQL" rather than "NoSQL". I almost always see it written as "NoSQL".

Bliki: Slideument

2011-12-19T09:41:00-05:00

A slideument is a cross between a slide deck and a document. The idea is that you can use a single slide deck both for slides during your presentation and as a handout for people to read afterwards. The trouble is that those two needs lead to very different requirements on your slides, so you can't satisfy them both. The result is that slideuments usually fail at both.

The main reason they fail is due to the amount of words and detail you need in the deck. If you want a stand-alone document you need enough context to make sense without the speaker being there. This requires a lot of words. But if the speaker is there, and speaking, then the audience ends up both reading and listening - and usually not concentrating properly on either. A good slide deck is a VisualChannel, it provides an accompaniment to the spoken words, reinforcing but not duplicating what is being said.

If people want a stand-alone document, then you can provide them with one - but it should be a different document, one that's designed for reading and not speaking.

The most sensible argument I've heard in favor of slideuments is that audience expects them, even if they are useless and never read later. I've a lot of sympathy with this argument, as it goes back to my first major presentation - a tutorial at OOPSLA in 1992. Since this was a big deal for me I decided to work hard to provide an extra-special contribution. So instead of the usual copies of slides, I wrote a special document to support the talk - about 40 or so pages - and provided that as a handout. That document was a much better coverage of the topic then the slides would be. But on the day I got lots of complaints becuase I didn't provide a copy of the slides. I am forever grateful to Rebecca Wirfs-Brock, who was the tutorial chair that year and backed me up.

People do expect slides, even when they don't really make any sense, and often it's easiest just to run off a pdf. I think that ideally you should try for more. My approach in later tutorials was to give out both the document and the slide copies. These days I give people my TalkNotes URI that points to relevant articles I've written on the topic of my talk.

Slideuments shouldn't be confused with Infodecks

photostream 17

2011-11-22T17:35:00-05:00

Wachusett Meadow, Princeton, MA

Retreaded: SpecificationByExample

2011-11-17T08:38:00-05:00

Retread of post orginally made on 18 Mar 2004

I was attending a workshop at XP/Agile Universe in 2002 when the phrase 'Specification By Example' struck me as a way to describe one of roles of testing in XP.

(These days it's terribly unfashionable to talk about Test Driven Development (TDD) having anything to do with testing, but like Jon I do think that having a comprehensive automated test suite is more valuable than the term 'side-effect' implies. Rather like if someone offered me a million dollars to hike up a mountain. I may say that the main purpose of the hike is the enjoyment of nature, but the side-effect to my wallet is hardly trivial....)

Specification By Example isn't the way most of us have been brought up to think of specifications. Specifications are supposed to be general, to cover all cases. Examples only highlight a few points, you have to infer the generalizations yourself. This does mean that Specification By Example can't be the only requirements technique you use, but it doesn't mean that it can't take a leading role.

So far the dominant idea with rigorous specifications, that is those that can be clearly judged to be passed or failed, is to use pre and post conditions. These techniques dominate in formal methods, and also underpin Design by Contract. These techniques have their place, but they aren't ideal. The pre-post conditions can be very easy to write in some situations, but others can be very tricky. I've tried to use them in a number enterprise application settings, and I've found that in many situations it's as hard to write the pre and post conditions as it is to write the solution. One of the great things about specification by example is that examples are usually much easier to come up with, particularly for the non-nerds who we write the software for. (Timothy Budd pointed out that while you can show a lot of stack behavior with pre and post conditions, it's very tricky to write pre and post conditions that show the LIFO property.)

An important property of TDD tests is that they involve a double-check. In fact this highlights an amusing little lie of the XP community. They make a very strong point of saying things Once and Only Once, but in fact they always say things twice: once in the code and once in the tests. Kent has pointed out that this double-check principle is a vital principle, and it's certainly one that humans use all the time. The value of the double check is very much tied into using different methods for each side of the double check. Combining Specification By Example with production code means that you have both things said quite differently, which increases their ability to find errors.

The formal specification community have constantly had trouble verifying that a design satisfies a specification, particularly in doing this without error prone humans. For Specification By Example, this is easy. Another case of Specification By Example being less valuable in theory but more valuable in practice.

One Design by Contract fan pointed out that if you write a specification in terms of tests, then the supplier could satisfy the specification by just returning hard-coded responses to the specific test inputs. My flippant answer to this is that if you are concerned about this then the issue of tests versus Design by Contract is the least of your worries. But there is a serious point here. Tests are always going to be incomplete, so they always have to be backed up with other mechanisms. Being the twisted mind that I am, I actually see this as a plus. Since it's clear that Specification By Example isn't enough, it's clear that you need to do more to ensure that everything is properly communicated. One of the most dangerous things about a traditional requirements specification is when people think that once they've written it they are done communicating.

Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting. The examples trigger abstractions in the design team while keeping the abstractions grounded. You do need more - things like regular conversation, techniques like Domain Driven Design, indeed even doses of Design by Contract. While I've never had the opportunity to use Design by Contract fully (i.e. with Eiffel) I regularly think about interfaces in contractual terms. Specification By Example is a powerful tool, perhaps my most used tool, but never my only tool.

(For more thinking on Specification By Example, if not by that name, take a look at Brian Marick's writings. Somewhere on his site there probably is one super page that sums up his view on it. Of course finding it is less valuable than reading all the stuff there while you're trying)

Some Later Comments

A couple of years after I first wrote this, there was a post by Cedric Beust that was critical of agile methods that caused a minor blog spat. There were rebuttals by Jeff Langr and Bob Martin, and I sent this post through the feed again. Jeff Langr later added a nice example using using tests as specification by example for Java's Math.pow function.

reposted on 17 Nov 2011

photostream 16

2011-11-11T09:22:00-05:00

Boston, MA

New series book - Service Design Patterns

2011-11-10T10:38:00-05:00

Addison-Wesley has published a new book in my Signature Series. It’s by Robert Daigneau and it’s called Service Design Patterns. It’s a topic that’s already had too many books on it, but I added this one to the series because I think Robert has done a particularly good job of collecting together the best advice on the topic and organizing it into a useful handbook. This is the book that I think ought to become the standard book on the topic.

Retreaded: ContextualValidation

2011-11-03T11:00:00-04:00

Retread of post orginally made on 07 Dec 2005

In my writing endeavors, I've long intended to write a chunk of material on validation. It's an area that leads to a lot of confusion and it would be good to get some solid description of some of the techniques that work well. However life is full of things to write about, rather more than time allows.

Some recent readings made me think about saying a few preliminary things on the topic. One common thing I see people do is to develop validation routines for objects. These routines come in various ways, they may be in the object or external, they may return a boolean or throw an exception to indicate failure. But one thing that I think constantly trips people up is when they think object validity on a context independent way such as an isValid method implies.

I think it's much more useful to think of validation as something that's bound to a context - typically an action that you want to do. Is this order valid to be filled, is this customer valid to check in to the hotel. So rather than have methods like isValid have methods like isValidForCheckIn.

One of the consequences of this is that saving an object to a database is itself an action. Thinking about it that way raises some important questions. Often when people talk about a context-free validity, they mean it in terms of saving to a database. But the various validity checks that make this up should be interrogated with the question "should failing this test prevent saving?"

In About Face Alan Cooper advocated that we shouldn't let our ideas of valid states prevent a user from entering (and saving) incomplete information. I was reminded by this a few days ago when reading a draft of a book that Jimmy Nilsson is working on. He stated a principle that you should always be able to save an object, even if it has errors in it. While I'm not convinced that this should be an absolute rule, I do think people tend to prevent saving more than they ought. Thinking about the context for validation may help prevent that.

reposted on 03 Nov 2011

ThoughtWorks Agile East

2011-10-13T16:20:00-04:00

ThoughtWorks is organizing an Agile East event: a one day conference that we’re holding in New York (November 2) and Atlanta (November 7). I’ll be speaking, doing my usual suite of talks. More importantly there’s an impressive line up of my colleagues - the people whose ideas I clip and package up into those talks. So this is a great opportunity to listen to:

Graham and Andy talk from their experiences in the technical leadership of agile projects
project management lessons from Joe Z
hot devops and continuous delivery juice from Rolf and Jez
Ross’s advice on IT budgeting
Linda’s take on integrating UX with an agile project
Matt on using offshore teams for agile (he played a big role in building our offshore teams)
a trio of talks on the gnarly topic of organizational change from Tiffany, Greg and Manu
Chad explaining how to avoid the pitfalls of automated tests

and a special guest in New York who will make all our talks seem pedestrian.

photostream 15

2011-09-17T15:55:00-04:00

Miyajima, Japan

Interviewed about Agile in 2001 by Jim Highsmith

2011-09-16T08:20:00-04:00

When I went to Snowbird in 2001 for the meeting that led to the Manifesto, Jim interviewed me for the book he was writing on Agile Software Development Ecosystems. It provides a snapshot on my thinking on extreme programming and this thing that a few days later we called agile software development.

Retreaded: SoftwareDevelopmentAttitude

2011-09-08T10:04:00-04:00

Retread of post orginally made on 08 Mar 2004

Many debates in software development are underpinned by whether the speaker has a DirectingAttitude or an EnablingAttitude. These different attitudes affect choices over languages, designs, tools, processes, and lots more.

Here's some examples of this dichotomy:

A debate a while ago triggered by Joel Spolsky's post on exceptions. He didn't like exceptions because they could be misused badly, leading to confusing code (directing). Bill Caputo pointed out that exceptions, when used well, make life much easier (enabling).
Some of the static/dynamic typing debate brings up these points. Some arguments in favor of static typing talk about how they prevent people from making certain kinds of mistake (directing) while dynamic typers point out how static typing restricts some useful idioms (enabling).
Agile processes are PeopleOriented (enabling), while plan-driven methods seek to ensure that even a poor team can do an acceptable job (directing).

These aren't hard-wired attitudes. Often people are directing in some cases and enabling in others. But I think there is a deep strain running through here, often a personality issue, that runs underneath much discussion on how we do software. (I'm very much in the enabling category, as if you can't tell.)

You might think that all restrictions on what a developer does imply a directing attitude, but it's not that simple. As an example, consider memory management. You can think of this as a directing feature: programmers can't be trusted to manage memory correctly so take away their ability to allocate memory. But I look at memory management as an enabling technology - it takes away something I don't want to worry about, so I can concentrate better on those things I do care about. Steve tied this thought nicely onto the difference between problems and difficulties.

reposted on 08 Sep 2011

photostream 14

2011-09-07T19:01:00-04:00

Gourdon, France

Speaking in Frankfurt and Berlin in October

2011-09-06T09:04:00-04:00

I’ll be taking part in a couple of ThoughtWorks events in Germany during October. They are both evening events (Oct 17 Frankfurt and Oct 18 Berlin). I shall do my usual suite of talks on some mix of topics. I’ll be joined by two of my colleagues: Wolf Schlegel, who will be talking about continuous delivery and Erik Dörnenburg who will be talking about using lean techniques with enterprise architecture.

photostream 13

2011-08-31T22:29:00-04:00

Highgate, London

Update of "It's Not Just Standing Up"

2011-08-29T09:21:00-04:00

Short daily “stand-up” meetings are a regular feature of agile projects. Done well they allow frequent communication across the team with a high signal to noise ratio. Done badly, they can be pretty grim. For several years, this article by Jason Yip has become the definitive source of information on how to make them better. Jason has just finished a comprehensive revision of the article, so if you’re looking to get the best from your stand-ups, you should take a good look at the tips in here.

Speaking at goto Aarhus

2011-08-29T09:12:00-04:00

As in so many previous years, I’ll be speaking at the the goto conference (formerly known as JAOO) in early October. This is an excellent conference with a good range of material across both technology and process. This year I’ll be participating in a pair of tutorials.

On Oct 9 I’ll be with Jez Humble doing our full-day tutorial on Continuous Delivery, the first time we’ve brought this tutorial to Aarhus. This will cover much of the ground in Jez and Dave’s award winning book including the business case for Continuous Delivery, deployment pipelines, testing strategies, version control organization, and recipes for going live. We’ve had a lot of good feedback on this tutorial so far, so I’m glad to finally bring this to my favorite conference.

On Oct 13 I’ll join Rebecca Parsons to talk about Evolutionary Architecture. Rebecca’s long been our chief technical firefighter on difficult projects, which often have complicated dependencies to enterprise architecture. I’ve long been urging her to talk more about her experiences and techniques for handling these issues, so it will be good to work with her on this tutorial.

In between I’ll be hanging around the conference catching up on all the good pickings that will be available.

(If you want to register, using the discount code “fowl1250” will get you a worthwhile discount.)

Neal Ford's video workshop on agile engineering

2011-08-26T10:00:00-04:00

My colleague Neal Ford is a frequent writer and speaker who shares many of my interests and opinions (and is an excellent source of ideas for me to clip). In 2009-10 he wrote a series of fifteen articles for IBM developerWorks which dug deeper into the question of evolutionary architecture and design. His topics included Test Driven Development, leveraging reusable code, refactoring towards design and Domain-Specific Languages.

Now he’s followed on from this and recorded a video workshop on Agile Engineering Practices for O’Reilly that’s available on safari books online. These cover much of the topics that make evolutionary design possible, including Test-Driven Design, automation, and testing strategies.

photostream 12

2011-08-25T15:35:00-04:00

Mesa Arch, Canyonlands N.P., Utah

ThoughtWorks Microsite on Continuous Delivery

2011-08-24T10:02:00-04:00

ThoughtWorks has launched a microsite devoted to Continuous Delivery. At the moment you’ll find some short videos from the ThoughtWorks Live event we did in May, plus links to recent writing by Jez Humble (oops, now I have to call him the award-winning Jez Humble). We also post details of our upcoming events in the field of Continuous Delivery and Devops.

Beaten Again

2011-08-24T09:49:00-04:00

In the software world, we have the Jolt awards. These were originally run by Software Development magazine and are now run by Dr Dobbs magazine. My books have won a couple of ‘productivity awards’ - which are essentially the runner-up awards - but I’ve never snagged to top award (now called the ‘excellence award’). This year my DSL book was up for an award and won a ‘finalist award’ (essentially this means I was runner-up to the runner-ups).

So I failed again, but this year I would have been very annoyed if I had won. I’ve argued all year that Jez and Dave’s book on Continuous Delivery was the most important book released this year, and so I’m glad that Dr Dobbs agreed with me and gave them the excellence award. Congratulations to Jez and Dave in joining the long list of people who have kept me away from this award.

Although I keep missing the top prize, my signature series has a very good record with these awards. The 10 books in my series so far have picked up 1 finalist, 4 productivity and 2 excellence awards. Not a bad hit rate.

photostream 11

2011-08-03T18:14:00-04:00

Villa Rufolo, Ravello, Italy

ThoughtWorks Technology Radar

2011-08-02T10:58:00-04:00

We have now released the latest version of the ThoughtWorks Technology Radar. This is where a bunch of my colleagues get together and discuss the technologies that they think are worth keeping an eye on. It’s not intended to be some comprehensively researched list, as it’s just the aggregation of the opinions of the small group of people that create it. But since that group is an experienced group of ThoughtWorks consultants, I think it’s a useful pointer to things to pay attention to in enterprise software.

photostream 10

2011-07-12T19:56:00-04:00

Canterbury Cathedral, Canterbury, England

The LMAX Architecture

2011-07-12T09:43:00-04:00

When I was at QCon London last year, there was much buzz around a talk about a new retail trading system. The thing that got people’s attention was its approach to achieving its high performance needs - reaching 6 million TPS. It does this by running all its business logic on a single JVM thread, getting high speed without all the complications of concurrent programming. This article describes the architecture they use to pull this off.

photostream 9

2011-07-06T18:45:00-04:00

Gloucester, MA

Mike Mason and I talk about Feature Branching

2011-07-05T10:05:00-04:00

In this video (12 minutes) Mike Mason and I talk about the perils of Feature Branching and its alternatives.

photostream 8

2011-06-23T19:41:00-04:00

Boscastle, Cornwall

Sam Newman talks about Continuous Delivery

2011-06-21T09:37:00-04:00

Sam Newman is one of our Principal Consultants based in London who has been heavily involved in the development of our approaches to Continuous Delivery. Recently he gave a series of QTB (Quarterly Technology Briefing) talks and one of these was videoed and put on the ThoughtWorks site. It’s an excellent overview of why it’s important to learn how to do Continuous Delivery on the modern web, together with an overview of the practices needed to achieve it.

photostream 7

2011-05-28T09:54:00-04:00

Jordan Pond, Acadia N.P., ME

Reference pages for DSL patterns

2011-05-13T09:35:00-04:00

I’ve been very tardy in doing this, but I’ve finally put up a set of reference pages for the patterns in the DSL Book. There’s not much on each page, just the intent, chapter reference, and sometimes some summary text. I’ve primarily put these up to give them a stable URI for reference purposes. There’s a chance I’ll put supplemental material up there too, but I’ve had that intention before without anything happening.

photostream 6

2011-05-12T20:41:00-04:00

St Malo, Brittany, France

photostream 5

2011-05-07T18:07:00-04:00

Castle Hill, Ipswich, MA

Musings on ebooks

2011-05-05T09:18:00-04:00

It’s only just over a year since I got my first ebook reader. Now I only buy paper books when I really have to. I wrote my last book thinking of it primarily as a paper book, but that will be the last time, in the future electronic forms will be at the front of my mind. It’s clear ebooks will change the book industry in a dramatic way, but it’s not clear exactly how. Here are my current thoughts on what’s happening.

Tutorial on Continuous Delivery at goto; Aarhus

2011-05-03T16:17:00-04:00

I’ll be joining my colleague Jez Humble to give another of our tutorials on Continuous Delivery at goto; Aarhus. This is the rebranded incarnation of JAOO - which has been my favorite conference for a long time now. This will be our first time doing the full day continuous delivery tutorial in Aarhus.

Reflections on 10 years since the agile manifesto at SD Times

2011-05-03T13:05:00-04:00

SD Times has published an interview with me on my thoughts on ten years since the Agile Manifesto. I answer questions about what brought me to the meeting, whether I think the manifesto should be changed, and what I see in the next five years of the agile movement.

photostream 4

2011-05-02T14:17:00-04:00

Kumarakom, Kerala, India

Signature Series 40% off until May 15

2011-04-28T14:49:00-04:00

It’s been going on for a bit, but I’ve been tardy in letting people know. Addison-Wesley are currently having a promotion on signature series books. You get 40% off if you buy through informIT and buy at least two books in any of the series (ie mine, Kent’s or Mike’s).

Adding photos to my feed

2011-04-23T16:10:00-04:00

You may have noticed some photos cropping up in my news feed. Like many geeks I indulge in photography. So far this has been a harmless personal diversion, but I’ve reached the point where I’m feeling a desire to share some of my photos with the Internet - just in case we have a shortage. So you’ll see photos appear in my photostream from time to time. I intend to post no more than one or two a week.

photostream 3

2011-04-23T15:12:00-04:00

Mont Saint-Michel, Normandy, France

photostream 2

2011-04-23T15:11:00-04:00

Charlestown, MA

photostream 1

2011-04-23T15:10:00-04:00

401 Trail, Crested Butte, CO

Going Beyond Curly Braces

2011-04-19T16:54:00-04:00

On my long flight down to Australia recently I dug into a copy of Seven Languages in Seven Weeks. I spent less than seven hours on it, but found it a worthwhile read. He takes a lightening tour through seven languages. It’s not enough to go into much depth on any them, but it is enough to get a rough feel what’s interesting about them. I’d particularly recommend it if you are just starting to explore beyond the curly-brace languages that dominate so much of current programming.

One thing I particularly appreciate about pragprog books is that you get access to electronic versions in multiple formats. This book worked very well on the Kindle, despite the small screen.

Article on Non-Determinism in Tests

2011-04-14T09:19:00-04:00

In conversations with project teams, I’ve heard a lot about problems with flaky, unreliable tests. These non-deterministic tests are a serious problem with testing efforts. So I’ve written an article that talks about how to deal with these non-determinisms, discussing how to cope with some common causes: lack of isolation, asynchronous behavior, remote services, time, and resource leaks.

Q&A on Story Points

2011-03-29T08:57:00-04:00

My colleague Anand Vishwanath has put together a nice Q&A on the role Story Points play on a project.

InfoQ interviews Jez Humble and me about Continuous Delivery

2011-02-18T15:39:00-05:00

After our tutorial at QCon San Francisco last year, the InfoQ people sat us down for an interview on Continuous Delivery.

Update on malware issue with ThoughtWorks web sites

2011-02-18T11:19:00-05:00

I’ve posted an article detailing our malware attack. The short form is that we believe our sites are clear, but we currently have to operate them with reduced functionality. The malware only attacks a small proportion of visitors (which is part of what makes it difficult to deal with). However if you have visited our sites since Christmas with a windows machine, you should read more.

Speaking at ThoughtWorks Australia QTB

2011-02-17T14:01:00-05:00

Next week I’m heading off to spend some time in Australia (thankfully away from all this snow-shovelling). While I’m there I’ll be participating in a series of Quarterly Technology Briefings. These will take the form of a panel Q&A on Continuous Delivery. On the panel I shall be over-shadowed by Neal Ford and we will both be over-shadowed by Evan Bottcher.

Investigating ThoughtWorks in Vancouver

2011-02-08T15:53:00-05:00

Sid Pinney often takes on the job of investigating new locations for ThoughtWorks offices. He’s been one of the first to help start our offices in England, China, and Brazil. Last week I got an email saying he’s investigating possibilities in Vancouver. He’s looking for people to talk to: so if you’re a prospective client, possible employee, or are just friendly and looking to have Sid buy you a free food - get in touch with him by email or twitter.

Malware alert on thoughtworks.com

2011-02-03T12:14:00-05:00

We seem to have a shy but annoying piece of malware hanging out on thoughtworks.com. As far as we can tell, it manifests itself as a hidden iframe that redirects you to a site that hosts malware. We’re a bit vague on this, as it only appears rarely so only a couple of people have seen it.

Our biggest sign of it is via google. Google reported our site on Jan 7th as having problems, we looked into it, didn’t find anything, and Google’s complaint went away very quickly. The flag went up again on Jan 31st and this time they emailed our webmaster. We looked again for a problem, without any success, so we asked them to re-review our site. Since then various Google diagnostics have raising and lowering the suspicious flag at a dizzying rate. As I write this the site is considered safe, but given the volatility of the opinion, we aren’t sure about how things will go. (If you’re concerned you can check Google’s safebrowsing link.)

Clearly this is a bit of malware that’s set to only show itself occasionally, which makes it harder to deal with. The tricky bit is to find how it’s getting into the site. We’ve checked all our content and not found anything suspicious, but it could be third party javascript libraries, could be our Drupal instance, could even be a problem with our apache installation - but we don’t know for sure.

We’re investigating this further (including following the recommendations of stopbadware) but haven’t yet found the root cause, although we do have some strong suspects. If you see anything suspicious on our site or would like to pass on any suggestions, please contact Andy Yates.

As far as I know, this problem does not affect martinfowler.com - which is a static site built with my custom scripts, so is less likely to attract such a problem. But do let me know if you see anything.

Video of Wikileaks panel

2011-01-21T11:18:00-05:00

I enjoyed the Churchill Club panel discussion on wikileaks on Wednesday. It’s thoughtful and well worth watching the video.

I particularly appreciated Daniel Ellsberg’s picture of the climate of government secrecy and Roy’s call to follow the example of librarians resisting secret gag orders. (There was a bit were it threatens to sink deep into the nature of libertarianism, but fortunately that didn’t last long.) While my thoughts on wikileaks are complicated (and I’ve not had the energy to write about them) I think the incident raises some very serious issues around the contending forces of privacy and openness. This panel provides a useful briefing on some of these perspectives.

Cindy is doing the Avon Walk for Breast Cancer

2011-01-20T10:43:00-05:00

Last year Cindy, my wife, did Boston’s Avon walk to raise money for breast cancer research. She decided to it again this year. Last year a good proportion of the money she raised came from my readers, and we’re hoping we can get another round of support from you this year. If you’d like to donate, please go to her walk page.

Speaking event in Hamburg

2011-01-18T09:28:00-05:00

I’m going to Germany next week and we’ve added a third speaking event to my trip. As well as speaking at OOP in Munich and in Cologne, ThoughtWorks has organized a speaking event in Hamburg on Jan 31st. My colleagues Erik Döernenburg and Wolf Schlegel will give a talk on Continuous Deployment and I’ll follow with my usual SuiteOfTalks.

WikiLeaks: Why it Matters. Why it Doesn’t?

2011-01-17T11:56:00-05:00

ThoughtWorks and The Churchill Club are presenting a panel discussion on Wikileaks which will explore the consequences of this saga on the freedom of the internet. The panel features:

Daniel Ellsberg (of Pentagon Papers fame)
Clay Shirky (thoughtful writer about the internet)
Roy Singham (ThoughtWorks’s founder)
Peter Thiel (co-founder of PayPal)
Jonathan Zittrain (law professor at Harvard).

The panel is moderated by Paul Jay from The Real News Network.

It’s being held at 5.30 PST on Wednesday January 19th at the Santa Clara Marriott. You can also watch the event live on Fora.tv and The Real News Network. Streaming starts at 7pm PST.

Martin Fowler

photostream 25

Retreaded: StandardStoryPoints

Bliki: OrmHate

A better solution

Avoiding the problem

photostream 24

Retreaded: PublishedInterface

Bliki: TestPyramid

photostream 23

An English Literature Podcast

Retreaded: CustomerAffinity

Bliki: EditingPublishingSeparation

photostream 22

Bliki: TestCoverage

Further Reading

Progress on NoSQL Distilled

Rebecca Parsons and I speak on The Evolving Panorama of Data

Jez Humble and I speak on Continuous Delivery at YOW

photostream 21

Bliki: PresentationSmells

ThoughtWorks recruiting event in New York

Infodeck on NoSQL and Polyglot Persistence

photostream 20

Rebecca and I will be Keynoting at QCon London

Rebecca and I talk about DSLs on Software Engineering Radio

Bliki: CharityCodeJam

photostream 19

Bliki: AggregateOrientedDatabase

An open letter to Pearson opposing their support of SOPA and PIPA

Bliki: DiversityImbalance

Further Reading

photostream 18

Bliki: NosqlDefinition

Bliki: Slideument

Further Reading

photostream 17

Retreaded: SpecificationByExample

Some Later Comments

photostream 16

New series book - Service Design Patterns

Retreaded: ContextualValidation

ThoughtWorks Agile East

photostream 15

Interviewed about Agile in 2001 by Jim Highsmith

Retreaded: SoftwareDevelopmentAttitude

photostream 14

Speaking in Frankfurt and Berlin in October

photostream 13

Update of "It's Not Just Standing Up"

Speaking at goto Aarhus

Neal Ford's video workshop on agile engineering

photostream 12

ThoughtWorks Microsite on Continuous Delivery

Beaten Again

photostream 11

ThoughtWorks Technology Radar

photostream 10

The LMAX Architecture

photostream 9

Mike Mason and I talk about Feature Branching

photostream 8

Sam Newman talks about Continuous Delivery

photostream 7

Reference pages for DSL patterns

photostream 6

photostream 5

Musings on ebooks

Tutorial on Continuous Delivery at goto; Aarhus

Reflections on 10 years since the agile manifesto at SD Times

photostream 4

Signature Series 40% off until May 15

Adding photos to my feed

photostream 3

photostream 2

photostream 1

Going Beyond Curly Braces

Article on Non-Determinism in Tests

Q&A on Story Points

InfoQ interviews Jez Humble and me about Continuous Delivery