Martin Fowler

May 2013 edition of ThoughtWorks Technology Radar

2013-05-22T16:35:00-04:00

The ThoughtWorks Technology Advisory Board (TAB) has released the latest edition of our technology radar. This is where we highlight some of the technologies that are currently attracting our attention and that we feel are worth you taking a look at. In this edition our themes include my long term interest in breaking down boundaries between people and groups, lightweight option for analytics, infrastructure as code, and applying the practices that have worked well for us in development to places that are missing them.

photostream 46

2013-05-21T09:34:00-04:00

Longwood Gardens, PA

Expansion to DIP in the Wild

2013-05-21T08:50:00-04:00

Brett’s article on the Dependency Inversion Principle (DIP) and how it works in practice has been a popular read on this site (over 24,000 views since it was published at the beginning of the month). Brett has now expanded the article with two more examples from his delivery experience: questioning requirements and handling time.

How do you estimate on an Agile project?

2013-05-03T13:36:00-04:00

If you’re interested in techniques for estimation, you should take a look at this pdf ebook. It contains half-a-dozen essays on estimating in agile projects, drawn from our experiences with a wide range of clients. We explain approaches based on story points and on story counting, which should give you a good overview for you to explore an approach that will work for you.

DIP in the Wild

2013-05-01T12:13:00-04:00

The Dependency Inversion Principle (DIP) was coined by (Uncle) Bob Martin several years ago. It’s a useful principle to help organize the dependencies in your system. Here Brett Schuchert shows how this principle works in practice to simplify designs with a couple of real world examples from his recent ThoughtWorks projects.

photostream 45

2013-04-24T19:18:00-04:00

Segovia, Spain

Bliki: StoryTest

2013-04-24T10:34:00-04:00

Story tests are BusinessFacingTests used to describe and verify the software delivered as part of a UserStory. When a story is elaborated the team creates several story tests that act as acceptance criteria for the story. The story tests can be combined into a regression suite for the software and provide traceability from the requirements (user stories) to tests and (through execution) to the behavior of the system. Story tests are usually BroadStackTests.

User stories are popular because they offer a simple workflow, each story adds new tests to the story test suite. However story tests often lead to problems. Regularly adding story tests leads to a large body of tests, often with significant duplication between them. When behavior needs to change in later iterations of the project, duplication in tests can take a painful amount of time to update. Furthermore broad-stack story tests take a long time to execute, which is why having a lot of them violates the TestPyramid. As a result many people recommend using just a few UserJourneyTests together with business-facing ComponentTests.

Bliki: UserJourneyTest

2013-04-24T10:34:00-04:00

User-journey tests are a form of BusinessFacingTest, designed to simulate a typical user's "journey" through the system. Such a test will typically cover a user's entire interaction the system in order to achieve some goal. They act as one path in a use case.

They are usually BroadStackTests and as such, are usually slow to execute and prone to being brittle. Consequently suites of user journey tests usually aren't built to be comprehensive tests of a system's behavior. Usually you will have only a few user journey tests to exercise the integration of the system as a whole - probably only one path for each use case (usually the happy path). Verification of all the variations in behavior is left to tests done in different styles, usually with more focused coverage.

In contrast to StoryTests, user journey tests are not tied to user stories. When you play a story you look at the existing user journey tests and modify them to support any change in behavior implied by the user story, only rarely does a user story lead to an entirely new user journey test.

Bliki: BusinessFacingTest

2013-04-24T10:32:00-04:00

A business-facing test is a test that's intended to be used as an aid to communicating with the non-programming members of a development team such as customers, users, business analysts and the like. When automated, they describe the system in domain-oriented terms, ignoring the component architecture of the system itself. Business-facing tests are often used as acceptance criteria, having such tests pass indicates the system provides the functionality that the customer expects.

Automated business-facing tests are often represented in some form of DomainSpecificLanguage, since this helps communication with non-programmers and also helps give programmers a mechanism that helps them step back from the details of the code. Tools like Cucumber and Twist help design such DSLs and provide mechanisms to bind them to the system under test.

Business-facing tests are commonly implemented as BroadStackTests since their user-oriented expression suggests treating the system under test as a black box. However there are significant advantages to implementing business-facing tests as ComponentTests since this often results in easier maintenance and faster execution.

I'm a big fan of automated testing, but it's important to recognize that manual tests play a significant role in business-facing testing. Techniques such as exploratory testing and usability testing are inherently manual activities and are essential parts of a well-balanced test portfolio.

StoryTests and UserJourneyTests are two common forms of business-facing test. The term business-facing test comes from Brian Marick's test quadrant.

The Architecture of Gap Inc's SCMS-PO

2013-04-23T11:24:00-04:00

I’m hoping to spend more time writing about expositional architectures, systems that show interesting characteristics that will act as seeds for ideas for other software systems. To make a start I’ve written an infodeck describing the SCMS-PO system of Gap Inc. It’s a system for managing purchase orders with a rich javascript client talking to a java server system using a resource API. Interesting features include using knockout.js form of the Presentation Model pattern, a javascript validator running on both client and server, encapsulating data access with repositories, using MongoDB as an application database, and the testing portfolio.

photostream 44

2013-04-22T10:54:00-04:00

Himeji, Japan

Bliki: UserStory

2013-04-22T10:42:00-04:00

User Stories are chunks of desired behavior of a software system. They are widely used in agile software approaches to divide up a large amount of functionality into smaller pieces for planning purposes. You also hear the same concept referred to as a feature, but the term "story" or "user story" has become prevalent in agile circles these days.

Kent Beck first introduced the term as part of Extreme Programming to encourage a more informal and conversational style of requirements elicitation than long written specifications. The essence of a story can be written on a single note card (Kent and I prefer 3" by 5"). Stories are deliberately not fleshed out in detail until they are ready to be developed, you only need enough understanding to allow prioritization with other stories.

Bill Wake came up with the INVEST mnemonic to describe the characteristics of good stories:

Independent: the stories can be delivered in any order
Negotiable: the details of what's in the story are co-created by the programmers and customer during development.
Valuable: the functionality is seen as valuable by the customers or users of the software.
Estimable: the programmers can come up with a reasonable estimate for building the story
Small: stories should be built in a small amount of time, usually a matter of person-days. Certainly you should be able to build several stories within one iteration.
Testable: you should be able to write tests to verify the software for this story works correctly.

A common way to formulate stories is the "As a … I want … So that …" form. The "As a" clause refers to who wants the story, "I want" describes what the functionality is, "so that" describes why they want this functionality. The "so that" part provides important context to understand to help get from what the customer think they want to providing what they actually need.

Mike Cohn wrote what is now the standard book on writing user stories. To understand the roots of user stories in XP consider the white book, or the tasteful green book. In an earlier bliki entry I discuss why UseCasesAndStories are different.

Bliki: ComponentTest

2013-04-22T10:41:00-04:00

A component test is a test that limits the scope of the exercised software to a portion of the system under test. It is in contrast to a BroadStackTest that's intended to exercise as much of the system as is reasonable.

The difference between broad-stack and component tests is one of degree rather than an absolute difference. Component tests can be as large or small as you define your components. The essence of the difference is that component tests deliberately neglect parts of the system outside the scope of the test. This is usually done by manipulating the system through internal code interfaces, using tools like xunit testing tools, and by using TestDoubles to isolate the code under test from other components.

Component tests are usually easier to write and maintain than broad-stack tests. They are also faster to run, since they only hit part of the code base. In theory a system with excellent component test coverage should be free of bugs, but in practice bugs like to lurk in the interactions between components. Therefore it's good to use the TestPyramid and combine a large quantity of component tests with a smaller amount of broad-stack tests.

Bliki: BroadStackTest

2013-04-22T10:40:00-04:00

A broad-stack test is a test that exercises most of the parts of a large application. It's often referred to as an end-to-end test or full-stack test. It lies in contrast to a ComponentTest, which only exercises a well-defined part of a system.

The difference between a broad-stack test and a component test is a continuum rather than a clear line. One area where bits can be missing from the fullness of the stack is how the test manipulates the application. Broad-stack tests often manipulate the application through a UI, such as testing web applications with tools like Selenium and Sahi. However a SubcutaneousTest can also be a broad-stack test if it continues to exercise most of the rest of the software. To further limit the scope, a test that exercises an application through a service interface can also be considered to be broad-stack test of the the server.

The other area where these tests don't cover the full breadth of the stack lies in connection to remote systems. Many people, including myself, think that tests that call remote systems are unnecessarily slow and brittle. It is usually better to use TestDoubles for these remote systems and check the doubles with IntegrationContractTests

Broad-stack tests have the advantage of exercising the application with all its parts connected together and thus can find bugs in the interaction between components in the way that component tests cannot. However broad-stack tests also tend to harder to maintain and slower to run than component tests. As a result the TestPyramid suggests using fewer broad-stack tests.

Bliki: JavascriptPromise

2013-04-22T10:39:00-04:00

In Javascript, promises are objects which represent the pending result of an asynchronous operation. You can use these to schedule further activity after the asynchronous operation has completed by supplying a callback.

    aPromise = someAsyncOperation();
    aPromise.done(function() {
      // runs if all went well
    });
    aPromise.fail(function() {
      // runs if something went wrong
    });
    aPromise.always(function() {
      // runs either way
    });

As well as providing a clear interface to schedule activity with asynchronous tasks, they also compose.

      composedPromise = $.when(anAsyncFunction(), anotherAsyncFunction());

In this form (using jQuery promises) the composed promise will run its done handlers when all the passed promises succeed and its fail handlers if any of them fail.

There are various forms of promises in javascript, annoyingly they have subtly different APIs and vocabularies. Probably the most used is jQuery's Deferred Object.

You also hear these concepts described as futures and deferreds. These concepts appear in many languages, not just javascript, often with concurrency in mind as much as asynchrony.

For more information I suggest getting a copy of Trevor Burnham's Async JavaScript. If you want a web article, I found Burnham has a short but useful article summarizing them.

A short statement on the Adria Richards / Pycon incident

2013-04-15T09:52:00-04:00

A few weeks ago I watched in depressed fascination at the internet explosion about Adria Richards and her reaction to offensive comments made in the audience of the Pycon conference. (If you would like more information on this incident I would suggest starting with the Guardian article. ) Although I did contemplate writing something about it, I felt that I didn’t have anything new or constructive to say. I don’t really have anything terribly enlightening to say now, but I do feel the need to state my opinions.

For me, the supreme issue here is the appalling internet attacks on Ms Richards, which included a barrage of viciousness that included rape threats, death threats, publishing her private information, and a DDOS attack on her employer. In the discussion over these events, many people have spent energy discussing such things as whether the jokes that triggered all this were truly offensive, and whether Ms Richards’s reaction was appropriate. Even if we were to accept that her action, a tweet including a photograph of the jokers, was an overreaction which violated reasonable etiquette - any wrong that she had done is insignificant compared to the wrong heaped upon her by those attacks.

I have long despaired of the net nastiness that hangs over our profession and how its alienating atmosphere contributes to our diversity imbalance. This incident sadly makes clear the misogynistic streak in the tech world and further deters women from speaking out against it. I have no solutions to offer, but will continue to listen to historically disadvantaged groups and hope to be able to support those trying to correct these problems. I know that many of my colleagues at ThoughtWorks are also eager to see this disease eradicated and for our profession to actually approach being the meritocracy it likes to think it is.

I would also like to state a loud “+1” to Alexis Ohanian for eloquently expressing much of my feelings.

photostream 43

2013-03-25T10:59:00-04:00

Martindale, Cumbria, England

Ruby Rogues Podcast on P of EAA

2013-03-25T08:32:00-04:00

The Ruby Rogues are a popular podcast where a regular panel discusses topics in the Ruby programming community. They have a regular book club and recently selected P of EAA as their featured book. Consequently they asked me to appear as a guest on their show to discuss the book and the patterns that it describes, in particular the interesting relationship between these patterns and the Rails framework.

Retreaded: TypeInstanceHomonym

2013-03-19T09:25:00-04:00

Retread of post orginally made on 11 Jan 2007

"'War and Peace' is a wonderful book.

"Let me see... pity this book has such a tattered cover"

Two sentences, each uses the word 'book'. We glance over combinations like this every day without remarking that the word 'book' means something completely different in each of those sentences.

In the first sentence 'book' refers to a literary work, one that's been with us for over a hundred years. In the second sentence 'book' now refers to a physical object that's probably been around for a lot less. I can easily destroy the latter in my fire, but the former isn't touched by my flames and will doubtless last longer than I will. I can put the former on my hard disk with many siblings that collectively weigh less than the latter.

This particular ambiguity is startlingly common in natural language. Think about words like 'car', 'oak', and 'programmer'. Trying to straighten this out can easily lead you into the darker corners of semantics in philosophy. For the moment, I'll just explain how I think about this.

The word 'book' is a term that can refer to multiple concepts. The two sentences above illustrate two concepts that 'book' refers to: the literary work and the physical copy. Since most of my readers are programmers I'll use a programming analogy and say that there are two distinct classes both of which are called 'book'. We can disambiguate them by referring to them as the classes LiteraryWork and PhysicalCopy. When in the first sentence I say that 'War and Peace' is a book, I'm saying that 'War and Peace' is an instance of the class LiteraryWork. The second sentence refers to the properties of an instance of PhysicalCopy.

These two concepts are not independent. Unlike a homonym like right and right (the opposites of left and wrong), these two concepts have a close relationship [1]. A book [PhysicalCopy] carries on it a rendering of the book [LiteraryWork]. So I might say that my War and Peace has a tattered cover. Similarly my Subaru Legacy [instance of physical car] has a relationship with Subaru Legacy [instance of car model]. Indeed we might say that the type of my car is Subaru Legacy [car model]. This kind of relationship is very common and is familiar to people who do any kind of object or data modeling. You often see similar structures involving things like Product and Product Type. I wouldn't use the terms Book and Book Type, but it's fundamentally the same relationship.

Spend time modeling, or programming, in almost any domain and you'll reach a point where you have to take some common term in that domain and realize it denotes two separate concepts which mean separate classes (or tables) with different names in your code. When you get homonyms in your domain language, it's often because the terms are used differently in different contexts - what Eric Evans calls Bounded Contexts. As Jim Odell liked to point out: "Mary had a little lamb" means quite different things to a veterinarian than to a restaurateur. The trouble with the type-instance homonym is that it occurs even within very tight bounded contexts.

My purpose here isn't to talk much further about modeling these kind of situations, that's quite a topic on its own, but to reflect on the ambiguity that this common type of homonym causes. The really interesting thing about this ambiguity is how little trouble it causes. Humans seem to have no trouble at all switching between the multiple concepts and only rarely getting tripped up in the process. What's worse, if you actively try to use different terms in everyday conversation, it's nearly impossible to keep it up. I certainly tried, both in conversation and writing and quickly realized it was a lost cause. Not just did I not have the discipline but it made me sound like someone with some obscure mental condition. (The type-instance case isn't the only one with this property, as the phrase "go through the door we painted last week" suggests.)

There's a particularly fascinating case of this in my own specialty of object-oriented design. We often talk about things like "add a method to an object". Yet, of course, we don't add methods to objects, we add them to classes. At some point in your career you've been the pedant who's pointed that out in a meeting. You've also effortlessly glossed over that point in countless conversations since.

So I came to the conclusion that we absolutely have to be aware of this homonym. We need to represent the concepts separately and with different names in our software. But we shouldn't expect to make a distinction between them in our conversation unless we're in a context where there is a real danger of confusion, because most of the time our brains resolve the ambiguity automatically.

So what does this mean in the context of a UbiquitousLanguage? Firstly we have to do the modeling and realize we have different concepts (aware of the homonym). We then come up with names for the different concepts that we'll use in our Ubiquitous Language and thus in the software representation. When coming up with these names, I like to avoid the bare homonym: so I wouldn't use "book" in the library context, preferring something like 'Literary Work' and 'Physical Copy'. These disambiguating terms need to be comprehensible to the domain experts, as with anything else in the Ubiquitous Language. Therefore the domain experts need to be aware of the homonym and help come up with the new terms. We then use the new terms whenever we need the precision we gain. However we should be wary of trying to completely exorcise the homonym. It effectively becomes a synonym for each of the new terms. So if a domain person says "a book has authors" we might say "you mean a literary work there, right?" Remember that it will feel unnatural to everyone to not use the homonym term, so apply the precision with a light touch. Fortunately awareness is 90% of the battle here.

A further consequence of this homonym is that you should always have your antenna scanning for it - particularly in the early days of working in a new domain. Usually it occurs in terms that are absolutely central to that domain - as with books and libraries. I've liked using informal class diagrams to model a ubiquitous language and I've found it does a good job of rooting out these kinds of homonyms.

1: Jim White emailed me to let me know that in Linguistics, the term "polysemy" is sometimes used to indicate to indicate a word that has different, but related, meanings.

reposted on 19 Mar 2013

Video of Introduction to NoSQL Talk

2013-03-04T14:18:00-05:00

Last fall goto Aarhus continued its ongoing interest in new approaches to data with a track gathering together various experiences with NoSQL databases. I was asked to kick off this track with an introductory talk for an hour giving an outline to the important issues in thinking about NoSQL. I talk about the origins of NoSQL, forms of NoSQL data models, the way many NoSQL databases consider the problem of consistency, and the importance of Polyglot Persistence. I was pretty happy with how this turned out and think it’s a good way to spend an hour if you want to start a journey in understanding NoSQL.

There is also a video of a panel of NoSQL vendors that I participated on. This wasn’t a great discussion - lots of talking past each other, but maybe worth a look for people who are already into the NoSQL world.

photostream 42

2013-03-02T09:58:00-05:00

Vieste, Puglia, Italy

Bliki: PurposeOfEstimation

2013-02-27T09:56:00-05:00

My first encounter with agile software development was working with Kent Beck at the dawn of Extreme Programming. One of the things that impressed me about that project was the way we went about planning. This included an approach to estimating which was both lightweight yet more effective than what I'd seen before. Over a decade has now passed, and now there is an argument amongst experienced agilsts about whether estimation is worth doing at all, or indeed is actively harmful [1]. I think that to answer this question we have to look to what purpose the estimates will be used for.

A common scenario runs like this:

Developers are asked for (or given) estimates for upcoming work.

People are optimists, so these estimates tend to be too low, even without pressure to make them low (and there's usually at least some implicit pressure)

These tasks and estimates are turned into release plans tracked with burn-down charts
Time and effort goes into monitoring progress against these plans. Everyone is upset when actuals end up being more than estimates. In effort to increase pace with the estimates, developers are told to sacrifice quality, which only makes things worse.

In this narrative, effort put into estimates is, at best, waste - since "an estimate is a guess in a clean shirt" [2]. Usually estimates end up being actively harmful as they encourage FeatureDevotion, a nasty condition where people start valuing ticking off features more than tracking the real outcome of the project.[3]

Estimates also set expectations, and since estimates are usually too low, they set unrealistic expectations. [4] Any increase in time or reduction in features is then seen as a loss. Due to loss-aversion, these losses have a magnified effect. [5]

Faced with situations like this, it's easy to see how people turn their angry glares towards estimation. This leads to an increasing notion that anyone indulging in estimating is Not a True Agilist. Critics of agile say this means that agile development is about developers going off and doing vague stuff with promises that it'll be done when its done and you'll like it.

I don't share this view of estimation as an inherently evil activity. If I'm asked if estimation is a Bad Thing my answer is the standard consultants' answer of "it depends". Whenever someone answers "it depends" the follow-up question is "upon what". To answer that we have to ask why we are doing estimation - as I like to say "if it's worth doing well, it's worth asking why on earth you're doing it at all".

For me, estimation is valuable when it helps you make a significant decision.

My first example of an estimation-informed decision is allocation of resources. Organizations have a mostly fixed amount of money and people, and usually there are too many worthwhile things to do. So people are faced with decisions: do we do A or B? Faced with such a decision it's useful to know how much effort (and cost) each will involve. To make sensible decisions about what to do, you need to have a feel for both the cost and the benefits.

Another example is to help with coordination. The blue team wants to release a new feature to their web site, but cannot do so until the green team builds a new service to give them crucial data. If the green team estimates they will be done in two months and the blue team estimates that it will take them a month to build the feature, then the blue team knows it's not worthwhile to start today. They can spend at least a month working on some other feature that can be released earlier.

So whenever you're thinking of asking for an estimate, you should always clarify what decision that estimate is informing. If you can't find one, or the decision isn't very significant, then that's a signal that an estimate is wasteful. When you do find a decision then knowing it focuses the estimate because the decision provides context. It should also clarify the desired precision and accuracy.

Understanding the decision may also lead you to alternative actions that may not involve an estimate. Maybe task A is so much more important than B that you don't need an estimate to put all your available energies into doing it first. Perhaps there is a way for blue team members to work with the green team to get the service built more quickly.

Similarly, tracking against a plan should also be driven by how it informs decision making. My usual comment here is that a plan acts as a baseline to help assess changes - if we want to add a new feature, how do we fit it into the FivePoundBag? Estimates can help us understand these trade-offs and thus decide how to respond to change. On a larger scale re-estimating a whole release can help us understand if the project as a whole is still the best use of our energy. A few years ago we had a year-long project that was cancelled after a re-estimate a couple of months in. We saw that as a success because the re-estimate suggested the project would take much longer than we had initially expected - early cancellation allowed the client to move resources to a better target.

But remember with tracking against plans that estimates have a limited shelf life. I once remember a gnarly project manager say that plans and estmates were like a lettuce, good for a couple of days, rather wilty after a week, and unrecognizable after a couple of months.

Many teams find that estimation provides a useful forcing function to get team members to talk to each other. Estimation meetings can help get better understanding of various ways to implement upcoming stories, future architectural directions, and design problems in the code base. In this case any output estimation numbers may be unimportant. There are many ways such conversations can happen, but estimation discussions can be introduced if these kinds of conversations aren't happening. [6] Conversely if you're thinking of stopping estimation, you need to ensure that any useful conversation during estimation still continues elsewhere.

Go to any conference with agile leanings and you'll hear talks of teams that work effectively without estimation. Often this works because they, and their customers, understand that making estimates isn't going to affect significant decisions. An example is a small team working closely with business. If the broader business is happy with allocating some people to that business unit, then work can be carried out in priority order; often this is helped by the team breaking down work into small enough units. [7] A team's level in the agile fluency model plays a big role here. As teams progress they first struggle with estimation, then can get quite good at it, and then reach a point where they often don't need it. [8]

Estimation is neither good or bad. If you can work effectively without estimation, then go ahead and do without it. If you think you need some estimates, then make sure you understand their role in decision making. If they are going to affect significant decisions then go ahead and make the best estimates you can. Above all be wary of anyone who tells you they are always needed, or never needed. Any arguments about use of estimation always defer to the agile principle that you should decide what are the right techniques for your particular context.

This bliki post was republished by ThoughtWorks studios in a pdf ebook on how to do estimation on agile projects. This ebook contains half-a-dozen useful essays on doing estimation with both story points and story counting techniques.

1: A recent read is Estimation is Evil an excellent discussion by Ron Jeffries of the problems that estimates can cause.

2: I got this analogy from Ron Jeffries, although I don't have a written reference for it.

3: This situation is an excellent example of an inappropriate use of metrics.

4: Estimates and expectations

I particularly liked a comment on this by my colleague Angela Ferguson "the way that estimates set expectations is up to us - it is poor project management (whether by project managers or other team members) that results in a client who thinks estimates are fixed, or that raw estimates = actual effort/duration"

"I, in fact, try to practice delivery bad news on a weekly basis with my key client, even when things are travelling as expected ... 'so we're looking quite well on track now, but if we had discovered something that took longer than expected, or a requirement had blown up to be larger than expected, or we found something new and very important, what do you think the best course of action would be?' And then you explore the options - cut stories, add time, add capacity, etc. This means that when the expected unexpected thing happens (because we know it will happen), the conversation doesn't seem new and scary to the client."

5: Very roughly people feel twice as much pain for a loss as pleasure for a gain.

6: If you do this an approach like ThrownEstimates can help the discussion move at a good pace.

7: Of course breaking work down into small units requires some implicit estimation, but that's really a different animal to the more common explicit estimation activity.

8: James Shore has a recent blog post that details his observations about how fluency influences estimation practice. I think a similar analysis of practices at various stages of fluency could be very useful.

Acknowledgements

I repeat myself again in thanking varous ThoughtWorkers on internal lists for their comments. In particuarly I'd like to call out Angela Ferguson, Dave Pattinson, and Pat Kua. I should also thank James Shore for responding so rapidly and well for my question about the links to the fluency model.

Bliki: NoDBA

2013-02-25T10:47:00-05:00

In many organizations, it's expected that any persistent data will be stored in relational databases that are managed by a central database management group. There are various reasons for such central control, usually centered around using IntegrationDatabases. Central data groups worry about keeping out malformed data, queries that can slow down important shared resources, and consistent data models across the enterprise.

Worthy these aims may be, but one consequence of them is considerable ceremony about storing data. I often hear complaints about change orders that take weeks to add a column to a database. For modern application developers, used to short-cycle evolutionary design, such ceremony is too slow, not to mention too annoying.

So application development groups tell me of using NoSQL databases to do an end-run around the DBAs. It helps that they are using a "mere datastore" here, not a "proper database". That way the DBAs can be kept out of the loop, often not told or happy to not care. [1]

There's both good and bad in all this. On the good side, this helps break what is often a troubling bottleneck for many organizations' application development. The sad divide between application developers and database professionals causes many problems, and the poor take up of many modern development techniques in the DBA community stifles much development. Shared databases are a poor integration mechanism, and NoDBA development leads to a useful push towards web services as the basis for integration. [2]

On the negative side, using a technology for social reasons can often lead to a poor use of the technology. It's not hard to imagine people using a NoSQL database to avoid the DBAs when a relational ApplicationDatabase would be a better choice. Data is often a vital asset, and bypassing DBA groups may also mean bypassing operations groups that know how to keep valuable data backed up and secure.

The real answer, of course, is engagement with data professionals along the lines of the devops movement. It's the barriers between groups that cause much of the ceremony, few data groups are populated by devils. We've seen success from inviting DBAs to stakeholder meetings where they get to engage with the broad aims of an application. We've also had NoSQL projects where we've reached out to the DBAs, helped them learn about this new technology, and got valuable help in supporting data needs. So although the NoDBA strategy is sometimes appropriate, engagement is often better.

1: It seems particularly common to use MongoDB in this context citing MongoDB's ease of installation, configuration, and use.

2: A large part of the slow response to change requests is that many incumbent databases are in essence complex legacy applications at the centre of a large web of dependencies. These databases are brittle and untested (Detestable), so it's no surprise that the DBAs who look after them are wary of making changes. Such systems would make even the smartest developer gun-shy of making changes.

Acknowledgements

As usual, I found the discussion of the draft of this article on our internal list very interesting. I particularly raided David Pattinson's contribution.

photostream 41

2013-02-23T14:35:00-05:00

William Ricketts Sanctuary, Victoria, Australia

Talks on schemalessness, consistency in NoSQL, and economics of software design

2013-02-20T09:53:00-05:00

A couple of weeks ago I dodged the historic snowstorm in New England by heading over to San Francisco to talk at a ThoughtWorks event. As is my habit these days I did a suite of talks, covering how and when to use schemaless data structures, why consistency in NoSQL databases is more than just ACID versus BASE, and the economic justification for well-designed software.

An Appropriate Use of Metrics

2013-02-19T09:41:00-05:00

Like many of us, my colleague Patrick Kua has seen a lot of use of metrics. Management love their metrics. The thinking goes something like this, “We need a number to measure how we’re doing. Numbers focus people and help us measure success.” Whilst well intentioned, management by numbers unintuitively leads to problematic behavior and ultimately detracts from broader project and organizational goals. Metrics inherently aren’t a bad thing; just often, inappropriately used. This essay demonstrates many of the issues caused by management’s traditional use of metrics and offers an alternative to address these dysfunctions.

photostream 40

2013-02-19T09:28:00-05:00

Block Island, RI

ThoughtWorks gets rid of sales commissions

2013-02-18T10:05:00-05:00

Sales commissions are almost universally used in the software business, as in all business sectors. In this short infodeck I explore why commissions are popular, the problems with them, and the thinking in ThoughtWorks that led us to getting rid of them in 2013.

Bliki: TransparentCompilation

2013-02-12T09:57:00-05:00

Increasingly web developers are using languages like CoffeeScript and SCSS that compile to other textual source languages that execute in the browser. Such source-to-source compilers (also called transpilers [1]) are not new, Cfront was widely used in the early days of C++ to generate target C code. But for me there is a difference that picks out CoffeeScript and SCSS as transparent compilers

With most compilers, you don't care very much about what is generated downstream. As long as it follows the semantics of the source language it's effectively a big lump of bits. But if you're going to generate JavaScript for the browser, this ignorance is hard to live with. Debugging environments are getting pretty nifty these days, but they are all in terms of the HTML/CSS/JavaScript triad. So it's important that you understand how your input language translates to its executable target.

This constraint has a big effect upon the source language. You need to ensure that the output corresponds very clearly to the source. When I write this CoffeeScript

$(window).on 'touchTap', (event) ->
  window.touchPanel.tap(event)

I can easily recognize the resulting JavaScript in my browser debugger

  $(window).on('touchTap', function(event) {
    return window.touchPanel.tap(event);
  });

This remains true even for more complicated bits of CoffeeScript's transformation, such as turning

  runSetupBuild: (slide, positionClass) ->
    switch positionClass
      when 'current', 'next'
        @buildsFor(slide)?.setupBuild?.forwards()
        # ...

into

    Infodeck.prototype.runSetupBuild = function(slide, positionClass) {
      var _ref, _ref1, _ref2, _ref3;
      switch (positionClass) {
        case 'current':
        case 'next':
          return (_ref = this.buildsFor(slide)) != null ? 
               (_ref1 = _ref.setupBuild) != null ? _ref1.forwards() : void 0 
               : void 0;
        /* ... */

There's a lot going on in that transformation - but still the correspondence is pretty clear. If I need to debug in that code, I can easily see how it matches up to the source CoffeeScript. This is the essence of what makes the compilation process transparent - the intention that you will work in the output language [2].

In contrast, there are source-to-source compilers that don't expect you to work in the output language, or see visibility of the output language as an unfortunate temporary mechanism. These can still be useful, and you see them in the javascript world with languages like Dart, GWT, and ClojureScript. It's this difference in intention which is what separates the transparent style of transpilation from the more common approach. [3]

The fact that you have to work at keeping the compilation transparent puts limits on what you can do in the source language. You don't have the degree of freedom in your language constructs that you get with a more unconstrained form of compilation. You have to follow the basic semantics of the target language, and keep to much the same program structure. These constraints haven't been widely talked about as a feature of language design.

CoffeeScript illustrates that despite these constraints you can get a considerable difference in syntax between source and target languages. CoffeeScript feels much more like Python in syntax than the C-like JavaScript. Such syntactic variation isn't always the case, indeed there is an important subset of transparent compilation languages that strive to be a superset of the target language. SCSS and TypeScript fit into this category - any CSS expression is valid in SCSS. Using a superset language makes the correspondence even clearer and I feel works well for CSS where CSS's syntax works well but the language misses some handy features.

Some say there's little point in using transparent compilation - if you have to understand the target code for debugging, what's the value in using a different source? For me, the value lies in a couple of directions. First off it's a way of getting useful language features that are missing in the target language. SCSS gives me handy capabilities such as variables (so I can say $light-purple instead of #f8c8fe and change it in only one place should I want to tweak it).

More drastic syntax changes, such as CoffeeScript, require stronger justification. One of my colleagues put it very well after finishing a project. He's an experienced JavaScript programmer and the project wrote well-disciplined JavaScript from the beginning. As a result he was very happy with the quality of the JavaScript codebase. However he still concluded that they would have been better off working in CoffeeScript, because it's easier to understand what is going on when you're reading the CoffeeScript, even when you're debugging in the generated JavaScript code. The transformation may not look like such a big deal for small fragments, such as those I show above. But it makes a big difference once you're up to hundreds of lines of code, let alone beyond.[4]

1: Hunting around usage, it seem to me that the the term "transpiler" is used as a synonym for source-to-source compiler. So transpilers may or may not be transparent. I've also seen the term "source-to-source translation" used equivalently to "source-to-source compilation".

2: Even with opaque compilation there are cases when people study the outputs. Occasionally there are odd behaviors or bugs that do require you to dig into the compiler outputs. Some programmers like to understand what the compiler is doing, although that's got less common as compilers and virtual machines get more sophisticated. But such activity is an exception.

3: It's interesting to see whether the development of sourcemaps will shift languages like coffeescript away from transparency.

photostream 39

2013-02-01T09:54:00-05:00

Ross Island, Andaman Islands, India

Infodeck on Big Data

2013-01-29T10:36:00-05:00

“Big Data” has leapt rapidly into one of the most hyped terms in our industry, yet the hype should not blind people to the fact that this is a genuinely important shift about the role of data in the world. The amount, speed, and value of data sources is rapidly increasing. Data management has to change in five broad areas: extraction of data from a wider range of sources, changes to the logistics of data management with new database and integration approaches, the use of agile principles in running analytics projects, an emphasis on techniques for data interpretation to separate signal from noise, and the importance of well-designed visualization to make that signal more comprehensible. Summing up this means we don’t need big analytics projects, instead we want the new data thinking to permeate our regular work. (go to the infodeck)

Speaking in San Francisco on Feb 7th

2013-01-22T16:53:00-05:00

I’ll be giving a talk in San Francisco on February 7th. I don’t know exactly what my topics will be, but it will be a Suite of Talks. Also on the program is my colleague Ken Collier, who will be talking about whether Big Data is any more than a load of old hype.

photostream 38

2013-01-10T11:00:00-05:00

Ironbridge, England

Schemaless Data Structures

2013-01-07T11:00:00-05:00

In recent years, there’s been an increasing amount of talk about the advantages of schemaless data. Being schemaless is one of the main reasons for interest in NoSQL databases. This infodeck explores the subtleties involved in schemalessness, both with respect to databases and in-memory data structures. These subtleties are present both in the meaning of schemaless and in the advantages and disadvantages of using a schemaless approach.

Activity Update

2012-12-18T14:13:00-05:00

My site hasn’t seen any updates for over a month, so I thought I’d make a quick note of what I’ve been up to. On the whole I’m not someone who tries to make a regular schedule of site updates - I write as I have the time, energy, and inspiration to do so. Regular articles were important in the days of magazines, but in the online world with notifications via RSS and twitter, I don’t see any reason to tie myself to a schedule.

Usually when I go quiet for a few weeks, it’s due to travel. When I’m on the grind of speaking I’m too busy, tired, and grumpy to do any writing. But that’s not the reason for my recent quiet period, as I’ve been happily at home. Indeed I’ve been working steadily at a couple more infodecks, and this involves both working on the content of the infodecks and on the toolchain to build them. I’ve added more features to the toolchain in addition to what I had for the multi-mobile deck, which should provide a more dynamic experience. I’ve been enjoying the mix of writing and building tools to support that writing - and I expect to be publishing these infodecks after the Christmas/New Year slowdown.

photostream 37

2012-11-16T09:31:00-05:00

Vancouver, BC

40% off Signature Series Promotion

2012-11-15T09:22:00-05:00

Addison-Wesley is doing a promotion on books in their signature series. This includes all the books in my signature series, but also the other signers: Kent Beck and Mike Cohn. The deal is that if you buy two or more signature series books from InformIT you get 40% off. The offer is good for physical and electronic books and is on until the end of 2012. To get the discount, use the discount code AWSS2012 during check-out. All my books pages have links to the relevant InformIT page - they do a good arrangement for multi-format ebooks.

photostream 36

2012-11-04T10:38:00-05:00

Aurora, NY, Finger Lakes

New ThoughtWorks Technology Radar

2012-10-24T10:14:00-04:00

The ThoughtWorks TAB has released the latest edition of our technology radar. As usual this provides our current view on what technologies are interesting, generated by a mix of our technological curiosity and project experience. I’ve also added a FAQ on my site to answer common questions about radar, such as who on earth the ThoughtWorks TAB is.

photostream 35

2012-10-19T15:01:00-04:00

Chinon, France

Calatrava

2012-10-15T10:07:00-04:00

A couple of months ago I wrote an infodeck about developing mobile applications for multiple devices. A core point of the deck is that for the best user-experience you need several native applications, but this is expensive. To reduce cost a mobile web application works on all suitable devices, but a web-app limits your user-interface. The result is an unavoidable choice between user-experience and cost.

While we can’t make that trade-off go away, we can turn a binary choice into a more graduated scale by developing hybrid applications that mix web and native technologies. At ThoughtWorks we’ve been pursuing this path on various client projects, with some success. We’ve now developed enough understanding on how to approach a hybrid application to make it worth sharing some code, so we recently released Calatrava, which is a framework to enable this web-native interoperation.

Calatrava is suitable for applications with significant client-side domain logic and when the application a channel for your product rather than the product itself. It is not suitable if most of your code is UI code (such as a game) or if a web app provides a sufficiently good user-experience. Calatrava is probably most valuable when you want to pursue an incremental release strategy, such as cover-your-bases.

For more information look at the calatrava home page on github, and the launch announcement by maintainer Giles Alexander. There’s also an overview on a single infodeck slide.

Appearing on Hanselminutes with Jez

2012-10-11T08:25:00-04:00

Last week I was at the goto conference in Aarhus. Also there was Scott Hanselman, and wherever Scott goes, he takes his portable recording studio and his titanium arm-twister. So before long I found myself in a quiet room with him and Jez Humble to talk about Continuous Delivery.

Key Points from NoSQL Distilled

2012-09-12T10:22:00-04:00

One the aspects of our design for NoSQL Distilled is that most chapters finish with a section of key points - short bullets that summarize the content of the chapter. I’ve now turned these into a web page that thus provides a summary of the key points of the book - which can act is a quick refresher for those that have the book and an indication of the content of the book for those who are contemplating buying it.

Modern Mocking Tools and Black Magic

2012-09-10T09:15:00-04:00

Modern mocking tools, such as JMockIt, have become extremely capable of interposing themselves into difficult places. As such they are able to deal with increasingly awkward legacy situations, allowing you to introduce seams to help get a code base under test. But in this article my colleague Brett Schuchert explores a dark side of this capability, one that casts serious concerns about using them in many situations.

photostream 34

2012-09-08T14:47:00-04:00

Malhamdale, Yorkshire, England

Proof-of-concept of using Google's BigQuery

2012-09-04T07:02:32-04:00

My colleague Ian Cartwright has contributed a short write-up of a week-long proof-of-concept project we did with AutoTrader to assess the viability of using Google’s BigQuery technology. The software produced 7-10 second response times for a dataset of 750 million rows. The article describes what they did and summarizes the lessons they learned from the experience.

photostream 33

2012-08-25T09:31:00-04:00

Mount Champlain, Acadia N.P., ME

NoSQL Distilled ebook from InformIT

2012-08-24T10:10:00-04:00

InformIT now has ebooks available, so this means that NoSQL Distilled is now available in all ebook formats. As all the formats get complicated, I’ve written a brief note on some options for buying paper and electronic books.

Some people have asked about when the paper books will reach other countries, including Amazon’s non-US stores. I’m afraid I have no information on that.

Speaking in Aarhus and Berlin

2012-08-22T15:47:00-04:00

Next month I’ll be spending a chunk of time in Europe. I have two talks lined up. Firstly I’ll be turning up in Aarhus at my favorite conference where I’ll be kicking off the NoSQL track on October 3rd with an introduction to NoSQL - roughly based on material from NoSQL Distilled.

Then, on October 4, I’ll be speaking in Berlin at a ThoughtWorks event with Jez Humble. For that I’ll do one of my usual Suites of Talks. There will be some NoSQL in the suite, but probably mixed with some other material too.

Best-Selling SQL book

2012-08-22T14:20:00-04:00

I love the irony.

NoSQL Guide Page

2012-08-22T10:54:00-04:00

I’ve added a guide page on NoSQL to martinfowler.com. I’ll use this to collect together links to material related to NoSQL, Polyglot Persistence, and other changes in the panorama of data.

NoSQL Distilled Available on Kindle

2012-08-22T10:33:00-04:00

Amazon now has NoSQL Distilled available on Kindle, so those of you who prefer that format can now get an instant fix of our deathless prose. If you prefer an epub ebook, then InformIT will have that soon, but they are still showing pre-order. I don’t know why the ebook versions have been so laggardly - but I know enough to blame computers.

photostream 32

2012-08-18T10:33:00-04:00

Jackson Lake Lodge, Grand Tetons N.P., WY

NoSQL Distilled Now Available

2012-08-17T13:35:00-04:00

It’s now August 17th, and NoSQL Distilled is now officially available. My copy arrived a day or two ago, and it’s good to see it in paper. Amazon has it available for order for physical copy. Currently they aren’t showing it as available for Kindle, but that should change in the next few days. InformIT is still showing a pre-order status, but again that should change soon.

It also seems I made a mistake earlier with the page count, which actually is only 152 pages. So we only busted our budget by two pages after all.

(I’ve updated my books pages to add a new page for this book.)

Mountain Lion and Nokogiri

2012-08-14T20:12:00-04:00

Today was the day I attempted the upgrade to Mountain Lion. Upgrading all the way from Snow Leopard (long story). The tedious first step is making sure I have backups, usually multiple backups, in addition to the ones I keep by default. Computers hate me, so I’ve learned to be paranoid.

My first serious hitch was with Aperture. I’d bought a download ages ago, when the route was to download a trial version and buy a product key. I still have the key, but there’s no place to download the trial version from. Fortunately there was a very helpful man on Apple’s telephone support who solved that one. +1 to him for being so helpful, but -2 for Apple for making it hard in the first place.

My more serious trouble was with Nokogiri, ruby’s XML processing library. Since almost all my website is XML transformed to HTML via Nokogiri and Ruby, this is rather crucial to me. But I had no end of trouble finding some way to get Nokogiri to install and work. Looking out on the web there are several pages of advice, with different combinations of animals to sacrifice in various orders.

In the end I got it working, but not in such a way that I have a confident trail for others to follow. I installed XCode command line tools, later the full XCode itself (don’t know if that was a factor). I initially used homebrew for supporting libraries but without success. I tried using rvm, but it failed (and I backed it out). In the end I installed libxml2 and libxslt using Macports and got it all working on the installed ruby 1.8.7.

I can only hope that this is just early days of Mountain Lion, but the situation is really too messy. It’s very troubling that the default install of ruby is still the elderly 1.8.7. I do need to share my scripts from time to time with others who are not active rubyists. It would be nice if there was a simpler installation process for them than my worry-ridden afternoon. I find myself pining for the days of the one-click windows install.

photostream 31

2012-08-11T11:19:00-04:00

Yellowstone N.P., WY

Your Path through Agile Fluency

2012-08-09T08:58:00-04:00

Diana Larsen and James Shore were actively involved with agile methods before they were fashionable. They have spent many years helping organizations explore agile development and I’ve always found their opinions worth listening to. When I get requests to review articles I rarely have time to read them, but a recent request by this pair piqued my interest.

When I read their draft on the stages of agile fluency, I immediately offerred to publish it on my site. The article looks at how teams pass through four stages of fluency with agile methods. Each stage brings benefits, but demands investment. Not all teams should go through all the stages. The model they’ve come up with is nicely simple and resonates with tales told by my colleagues. I think it provides a good framework for thinking about the style of agile development you are at and where you want to be.

photostream 30

2012-07-29T10:17:00-04:00

Yellowstone N.P., WY

Retreaded: RigorousAgile

2012-07-26T15:24:00-04:00

Retread of post orginally made on 29 May 2005

I often run into a complaint that agile methods don't have a rigorous definition. The complainer may talk about how this means that you can't tell if a particular team is using an agile method or not. They may also say that this makes it hard to teach people how to do agile methods - what's the curriculum?

To some degree I do feel the pain of this complaint - but I accept there is no cure. This lack of rigorousness is part of the defining nature of agile methods, part of its core philosophy.

One of the fundamental problems of thought processes in general - and of software development in particular - is the very varied nature of the settings. Different kinds of systems have different kinds of pressures and forces, which makes it very difficult to come up with a rigorous statement of what to do that's sufficient to cover them. This effect is compounded by the fact that software development is such a people-oriented activity, and people are naturally inconsistent and highly variable. The conclusion that agilists made from this is that's its not effective to try and bind software development to a rigorous process, because that's ignoring the essential nature of the primary (human) components that will execute that process.

(It's probably because our profession is to work with computers is what leads us to want to program human systems the same way that we program computers - despite the fact that they are so different.)

The upshot of all this is that agile methods fundamentally expect teams to decide what process to follow and furthermore expect teams to actively and regularly change their process. Any attempt to define a rigorous process that can be tested for conformance runs contrary to this philosophy.

I can't deny that this is frustrating. How can you do a survey on whether agile methods are more effective that alternatives, or whether Extreme Programming is more effective than Scrum, when you can't get a clear definition of what Scrum is in the first place? If a client wants a system built using Extreme Programming how can they tell if it's really being done?. There is a sense of "I know it when I see it", but it requires an experienced practitioner to have that sense and even then there's lots of room for such practitioners to disagree.

I don't have an answer to this conundrum. Indeed I don't think there is an answer. It's a an unfortunate consequence of the activity itself, just as getting wet is an unavoidable consequence of swimming.

reposted on 26 Jul 2012

NoSQL Distilled pre-order on InformIT with 35% discount

2012-07-26T10:26:00-04:00

I’ve been enjoying a brief vacation in Oregon (enjoying the excellent Oregon Shakespeare Festival and a nice mix of hikes). So I’m late in mentioning that NoSQL Distilled is now available for pre-order on InformIT. Its ship date is August 17.

If you pre-order you can get 35% off by entering discount code FOWLER6626 during checkout (this offer expires on August 17). InformIT is also running an offer where you get a 30 day trial to Safari Books Online if you place an order on InformIT.

The book has now completed production and has gone off to the printers. As I understand it the final ebook will appear when the printed book appears on August 17. Until then Safari subscribers can still read the rough cut, which includes the copy-edited text, but lacks the final layout and index.

The final book has 172 pages, which is sadly longer than we’d hoped. We hope you’ll forgive us for that.

photostream 29

2012-07-14T12:39:00-04:00

Outside Imperial Palace, Tokyo, Japan

Retreaded: SemanticDiffusion

2012-07-11T18:48:00-04:00

Retread of post orginally made on 14 Dec 2006

I have the habit of creating Neologisms to describe the things I see in software development. It's a common habit amongst writers in this field, for software development still lacks much useful jargon. One of the problems with building a jargon is that terms are vulnerable to losing their meaning, in a process of semantic diffusion - to use yet another potential addition to our jargon.

Semantic diffusion occurs when you have a word that is coined a person or group, often with a pretty good definition, but then gets spread through the wider community in a way that weakens that definition. This weakening risks losing the definition entirely - and with it any usefulness to the term.

I'm partly prompted to write this because I see this semantic diffusion happening very publicly to two terms at the moment: 'agile' and 'Web2.0'. Both were coined in the last few years and both come with lengthy definitions. (Agile has the agile manifesto, together with numerous books and articles by the authors of the manifesto. Web 2.0 has an excellent definitional article by Tim O'Reilly.) Yet both of these face a lot of corruption in the field. I've run into people who think agile methods mean you shouldn't do any planning and those who think Web 2.0 is solely about using AJAX. (A more personal case is RefactoringMalapropism, but I'll leave that aside for the moment.)

Semantic diffusion is essentially a succession of Chinese whispers where a different group of people to the originators of a term start talking about it without being careful about following the original definition. These people are listened to by a further group which then goes on to add their own distortions. After a few of these hand-offs it's easy to lose a lot of the key meaning of the term unless you make the point of going back to the originators. It's ironic that it's popular terms that tend to suffer from this the most. That's inevitable, of course, since unpopular terms have less people to create the Chinese whisper chains.

Semantic diffusion tends to coincide with hype phase of an idea. As an idea becomes attractive, lots of people are... well... attracted. Many of these followers talk about the term and teach about it. If these people aren't careful to go back to the source, the Chinese whispers begin. Popular ideas also spread primarily though communication media that are more likely to lead to misunderstanding - such as writing. Many followers aren't fortunate enough to work directly with the originators and thus learn directly from them.

A related indicator to popularity is desirability. A word that sounds good may be more likely to suffer from semantic diffusion. 'Agile' sounds like something you'd certainly want to be, the antonyms of agile aren't at all appealing. Who would want to still be merely 1.0 of the web? Kent Beck noticed this effect and thus deliberately picked Extreme Programming as a name because it less inherently desirable: 'extreme' is often used as a pejorative. Using a less desirable term may reduce semantic diffusion, but I don't think it avoids the problem completely. After all we saw semantic diffusion occur to 'object-oriented' which is a neutral term.

Semantic diffusion is more likely to occur with things that are broad concepts rather than hard technologies. Ruby on Rails is getting a lot of hype at the moment, but since it's a concrete tool it's hard for its meaning to weaken. Extreme Programming's list of concrete practices may also reduce this problem compared to agile's broader list of values and principles.

Semantic diffusion is a painful process to watch, particularly for those who find the ideas useful. At the moment I see signs of despair for both of these terms, some people in the agile world are talking of ditching the word agile.

I'm more sanguine about all this, largely because I've seen it before. There was a time when almost any software tool or method was described as object-oriented. But now, years later, the essential meaning of OO is reasonably well understood. I saw the same thing happen with patterns. So terms do recover their semantic integrity and the current diffusion doesn't inevitably mean the terms will lose their meaning. I'm also less worried about it because I expected it, again having gone through this process before. I think it's an inevitable consequence of ideas becoming popular and despite its problems, I prefer the hype to ignorance. A final comforting thought is that once the equally inevitable backlash comes we get a refocusing on the original meaning. As I'm writing this I'm comforted by the fact that I can't think of a term that lost its meaning entirely, although I'm sure it's happened. (SOA doesn't count, as I don't think there ever was a commonly agreed meaning for that to start with.)

Whatever the eventual outcome we still have to cope with semantic diffusion as it happens. One option is abandonment, to drop the diffuse term and pick a new one. I don't like this option because a new term only adds to the confusion and even if you're successful you just end up repeating the process for the new term. So my preference is to keep re-articulating the current terminology, pointing to those who understand the true meaning.

A further complication is that these terms also shift. I'm sure if you asked the original seventeen authors of the agile manifesto, they would come up with a different document today. (Let alone the fact that that group wouldn't be the best group to speak for the movement today.) Again it's a responsibility for those who are identified as originators to point this out, both by talking about the way the ideas evolve and by pointing to new people who are playing an active role in that evolution. (I'm pleased that the original seventeen "let the ship sail" to go its own way.) Let's not forget there's a tricky balance between holding to a clear definition and dogmatism.

All of this is hard work, but a good term is worth fighting for - particularly since the only bullets you need are words.

reposted on 11 Jul 2012

photostream 28

2012-06-21T20:03:00-04:00

Portland Sea Dogs, Portland, ME

Infodeck on handling multiple mobile devices

2012-06-19T10:43:00-04:00

A common question for people in mobile development is to how to deal with the wide range of mobile devices available. Do we have to write native applications for every single device? Or is there a way to reuse code for similar applications. I’ve written an infodeck to explore this problem and suggest how to think about dealing with it.

Facebook page for my tweets

2012-06-19T09:55:00-04:00

I’ve been using Twitter for a few years to post about what I’m up to, new activity on martinfowler.com, and other interesting things I find on the internets. Facebook has a facility to feed twitter activity to a Facebook page, which is handy for people who prefer tracking my posts on Facebook rather than Twitter. So I’ve set up a Facebook page to channel my tweets. If you “like” that page, Facebook will add my tweets to your Facebook news feed.

I don’t intend to do anything more with this page than be a channel for my tweets, so this won’t be anything new for people who use Twitter, but I’m giving it a try to see if at least some people prefer to use Facebook.

SE Radio podcast with Pramod on Agile Database Development

2012-06-06T21:05:00-04:00

SE Radio has long been my favorite podcast on software development, so I’m always happy to appear on it. I’m extra-pleased this time, because it gives me the chance to highlight some valuable software development techniques that I feel are particularly under-used.

When we first started using agile methods at the beginning of this century, one of our biggest questions was how to handle database design in an agile way. Fortunately we had Pramod Sadalage around, a DBA who was determined to find a way to incorporate databases with the agile working style. He developed a set of techniques, including regular database refactoring and scripted migrations that are version-controlled with application code. These techniques have now becomes habitual for ThoughtWorks but have not been as widely adopted in the industry as I believe they should. This podcast provides an hour or so that outlines how this all works, which I’m hoping will encourage greater usage of them.

photostream 27

2012-06-02T11:23:00-04:00

Salem, MA

NoSQL Distilled Rough Cut Now Visible

2012-05-31T09:45:00-04:00

The rough cut of NoSQL Distilled is now online. If you have a subscription that supports rough cuts you will see the whole book, if not you’ll see a preview of the book with the table of contents and the first paragraph or two from each section. Remember this is a rough cut and has not been copy edited.

I was a bit hesitant to announce this, after my false sighting on Tuesday, but I’ve had a few tweets now confirming that it’s online.

NoSQL Distilled rough cut available soon on Safari

2012-05-29T10:40:00-04:00

If you’re a subscriber to Safari Books Online, you will soon see a copy of a rough cut of NoSQL Distilled. This morning I said it was available, but I’ve since learned that my Safari account has special privileges, so I was seeing it but others cannot. We expect it to be available in the next couple of days for everyone who has an account that includes rough cuts.

It’s a rough cut, so it’s still early in the production process, with few copy-edits done. But it gives you an early chance to take a look at the text and to pass on comments.

Safari Books Online is an online library that gives access to lots of software books, including those of Pearson and O’Reilly. They have various levels of subscription, only some of which include rough cuts. For example there are two levels of individual subscription: bookshelf and library. Only the (more expensive) library individual subscription gives access to rough cuts.

I’m sorry for any confusion I’ve caused by my premature announcement.

Retreaded: ServiceCustodian

2012-05-24T09:01:00-04:00

Retread of post orginally made on 14 Nov 2008

Let's imagine a pretty world of SOA-happiness where the computing needs of an enterprise are split into many small applications that provide services to each other to allow effective collaboration. One fine morning a consumer service needs some information from a supplier service. The twist is that although the supplier service has the necessary data and processing logic to get this information, it doesn't yet expose that information through a service interface. The supplier has a potential service, but it isn't actually there yet.

In an ideal world the developers of the consumer service just asks the supplier service to develop the potential service and all is dandy. But life is not ideal - the sticking point here is that the developers of the supplier service have other things to do, usually things that are more important to their customer and management than helping out the consumer service team.

Recently I was chatting with my colleague Erik Dörnenburg and he told me about an approach he saw a client use to deal with this problem. They took a leaf out of the open source play-book and made all their services into internal open source systems. This allows consumer service developers write the service themselves.

I'm sure many readers are rolling their eyes at the visions of chaos this would cause, but just as open source projects don't allow just anyone to edit anything; this client uses open-source-style control mechanisms. In particular each service has a couple of custodians - people whose responsibility it is to keep the service in a healthy state. In the normal course of events the consumer developer wouldn't actually commit changes to the supplier source tree directly, instead they send a patch to the custodian. Just like an open-source maintainer, the custodian receives the patch and reviews it to see if it's good enough to commit. If not there's a dialog with the consumer developer.

As Erik knows well from his own open source work, reviewing a patch is much less effort than making a change yourself. So although the custodian approach doesn't entirely eliminate the problem of consumer developers needing to wait on supplier developers, it does a lot to reduce the difficulty. And again following the open-source model, a consumer developer can be made a committer once the custodians are comfortable. This still means that commits can get reviewed by the custodians, but avoids the custodians becoming a bottleneck.

Related to this was their approach to a service registry. We've seen a lot of fancy products being sold to provide service registry capabilities so that people can lookup services and see how to use them. This client discarded them and used a HumaneRegistry instead.

reposted on 24 May 2012

photostream 26

2012-05-22T18:16:00-04:00

Big Sur, CA

Speaking in China

2012-05-22T13:25:00-04:00

Here’s the information on my talks in China (the page is in Chinese). I’ll be giving talks in Chengdu 成都 on the 10th and 11th, Wuhan 武汉 on the 12th and Xi’an 西安 on the 13th.

Speaking in Singapore

2012-05-21T16:12:00-04:00

Next month (June 7) I’ll be speaking at a ThoughtWorks Quarterly briefing in Singapore. I’ll be doing a suite of talks, one of which I expect will be NoSQL-oriented. Also on the bill is Vivek Prahlad, a long-term colleague at ThoughtWorks, who I’m looking forward to catching up with. This will be the my first time in Singapore (other than the airport).

I’m then going on to talks in China: Chengdu, Wuhan, and Xi’an.

Cover for NoSQL Distilled

2012-05-21T16:10:00-04:00

Pramod and I now have a cover for our book NoSQL Distilled. Obsessive readers of my web site may recognize the photo.

Selecting a Mobile Implementation Strategy

2012-05-21T09:21:00-04:00

Over the last couple of years, we’ve seen mobile development become an increasing part of our work at ThoughtWorks. One question that clients regularly have is how to make the decision about which devices to support and what proportion of effort should go to each device. In this article Giles Alexander outlines two opening gambits - laser and cover-your-bases - and how to choose between and build on these approaches.

photostream 25

2012-05-10T12:46:00-04:00

Winterthur, DE

Retreaded: StandardStoryPoints

2012-05-10T12:30:00-04:00

Retread of post orginally made on 06 Sep 2004

I've heard a couple of questions recently about coming up with a standard story point mechanism for multiple teams using extreme programming's planning approach. The hope is have several teams all using equivalent story points, so that three story points of effort on one team is the same as on another.

I think trying to come up with this at best of limited value, and at worst dangerous.

The estimating system of extreme programming is based on XpVelocity and YesterdaysWeather. Inherent in this is the idea that when you make estimates, the actual units you estimate aren't important - what's important is you estimate by rough comparative value and use YesterdaysWeather for calibration.

In this situation the story points act as an anchor for the feedback loop that Yesterday's Weather provides - nothing more. Baked into them are all sorts of assumptions about the nature of the team's task, the capability of the team, and whether the team are optimistic or pessimistic estimators. Once you try to come up with a standard across teams you are trying to normalize all of these factors. Trying to do this sounds very hard to me, and I'm not aware of anyone who has done this effectively. This doesn't mean its impossible, it just means it's hard.

The dangerous aspect comes from once you have a standard unit for measurement across teams, someone is inevitably going to use it to compare the performance of teams. Even if everyone swears till they are blue in the face that they won't use it for cross team measurement, there will always be the suspicion that this will happen eventually. This will cause teams to distort their measurements so that it seems that they get more story points done. My fear is that this will break the feedback loop of yesterday's weather and knock the planning process off kilter. I'm always suspicious about these things because while it would be incredibly valuable to have a way to measure productivity I think the nature of software is such that we CannotMeasureProductivity.

So to be worth trying, this has to yield some valuable benefits - but I don't see any. One reason that I've heard is to help people move onto teams and estimate more quickly. But you can't estimate on a new team until you get reasonably familiar with the problem and the current code base. As you do that you'll also get a feel for the relative sizes of story points on that team. We move people around between projects at ThoughtWorks and I've never heard anyone complain about difficulty of estimating when coming onto a new team due to differences in story points.

reposted on 10 May 2012

photostream 24

2012-05-05T10:04:00-04:00

Boston, MA

Retreaded: PublishedInterface

2012-05-03T10:18:00-04:00

Retread of post orginally made on 26 Dec 2003

Published Interface is a term I used (first in Refactoring) to refer to a class interface that's used outside the code base that it's defined in. As such it means more than public in Java and indeed even more than a non-internal public in C#. In my column for IEEE Software I argued that the distinction between published and public is actually more important than that between public and private.

The reason is that with a non-published interface you can change it and update the calling code since it is all within a single code base. Such things as renames can be done, and done easily with modern refactoring tools. But anything published so you can't reach the calling code needs more complicated treatment.

reposted on 03 May 2012

photostream 23

2012-04-27T21:10:00-04:00

Middleton Place, Charleston, SC

An English Literature Podcast

2012-04-26T19:02:00-04:00

I’ve added another recommended podcast to my list, this one a is a treasure if you enjoy good fiction. Why I Really Like This Book is a quirky sail through the lesser known waters of English Literature. The pilot is Kate Macdonald, who teaches English Literature but was frustrated at the limited range of books she could cover. Each podcast spends ten minutes or so exploring a single suggested book. Her first series was a random alphabetical wander, now she’s taken to talking about sets of five books with a common theme. Recent sets include Five Appalling Fictional Women and Five Classic Detective Novels. If you’re looking for ideas for books to curl up with, this is an excellent source of suggestions.

Retreaded: CustomerAffinity

2012-04-26T09:55:00-04:00

Retread of post orginally made on 28 Jul 2006

When someone is looking at what makes up a top-class enterprise software developer, often the conversation may turn to knowledge of frameworks and languages, or perhaps the ability to understand complicated algorithms and data structures. For me, one of the most important traits in a programmer, or indeed in a development team, is something that I'll call Customer Affinity. This is the interest and closeness that the developers have in the business problem that the software is addressing, and in the people who live in that business world.

There are many aspects to customer affinity. The first one is just the interest in the business, its processes and rules. I've always been fascinated by domains I've worked in: health care, derivatives, payroll, leasing - are all examples of really interesting domain problems with a lot of complexity that has to be organized and understood. For me aspects of a project such as object-relational mapping, remote process communication - what I think of as the plumbing of enterprise systems, don't have that same interest.

It's important that a team has the plumbing working and under control, but I believe that the more energy goes into the business problem, the more effective at providing value a team will be. Which is a good reason to use frameworks that solve and simplify as much of the plumbing as possible.

Another aspect of customer affinity is the ability to collaborate with domain experts. I don't think that detailed knowledge of the domain is an important thing for programmers to have; useful yes, but not important. What's more critical than knowledge is the ability to collaborate with those that have the knowledge.

My high regard for customer affinity is one the main reasons why I'm such a fan of Extreme Programming and other agile methods. I found it very significant that when Kent Beck summarized XP to his agile peers at the snowbird workshop which coined 'agile', he chose to describe not the technical elements of XP, but his desire to change the nature of the customer/developer interaction.

I've often heard this relationship mis-characterized. In particular there is a belief in some quarters that the customer team just comes up with stories which they throw at the developers. This characterization is rather like the notion that requirements are just lying out there to be gathered. Either way that's not much of a collaboration. Instead developers need to work together with domain people to generate ideas with developers learning a lot about the business in the process.

One of Kent's suggested names for 'Agile' was conversational software development - the point being that it's a two way communication. This isn't something like a telecoms protocol that you can define, but the back and forth discussions about how software can enhance the business are where the real value lives. Much of this conversation is of half-baked ideas, some of which grow into valuable features - often ones that aren't things that the customer originally thought of.

One of the things that frustrates me is how organizations often try to squelch customer affinity. It's not something people admit to doing, but it's a common consequence of other decisions. Organizational barriers can contribute to squelching - I've come across places where you have to get your manager to arrange with another manager just so you can have a conversation with someone on the business side. That hardly encourages you to get inside the workings of the business.

I've often heard it said that enterprise software is boring, just shuffling data around, that people of talent will do "real" software that requires fancy algorithms, hardware hacks, or plenty of math. I feel that this usually happens due to a lack of customer affinity. The real intellectual challenge of business software is figuring out what the real contribution of software can be to a business. You need both good technical and business knowledge to find that. Working closely with business people to develop this knowledge, and PleasingTheCustomer as you do it, is what makes enterprise software development fun - and motivation is the key to good and productive work.

There are plenty of bright and capable people who want to learn about the businesses they write software for. Too often organizations make it hard for them to do so. Until that changes, our profession will continue to under-deliver on our potential.

reposted on 26 Apr 2012

photostream 22

2012-04-17T15:35:00-04:00

London

Progress on NoSQL Distilled

2012-04-17T10:42:00-04:00

Time for a progress report on NoSQL Distilled. We’ve done a round of technical review and are now pretty happy with the text, happy enough to move it into production.

Production for a book these days is both more complicated and more responsive than it used to be. The next step on the book text is to do copy-editing and indexing. It sounds like we are going to get the same people I worked with on my DSL book, which will be excellent news since they did a good job and were happy to work directly on the XML source in my repository. This simplifies and speeds up the whole process. They will also do the book design and convert the book in camera-ready for printing the physical books.

I suspect, however, that this book will see more electronic copies than physical ones. In parallel with the copy-editing we’ll be preparing the electronic copies. We intend to do a rough-cut version of the book as soon as we can. This should be available through InformIT, Amazon, and Safari. I’ll make further announcements here as these become visible.

Rebecca Parsons and I speak on The Evolving Panorama of Data

2012-04-16T10:57:00-04:00

Our keynote at QCon London 2012 looks at the role data is playing in our lives (and that it’s doing more than just getting bigger).

How the world of data is changing: its growing, becoming more distributed and connected.
The industry’s response: the rise of NoSQL, the shift to service integration, the appearance of event sourcing, the impact of clouds and new analytics with a greater role for visualization.
How data is being used now, with a particular emphasis from Rebecca on data in the developing world.
What does all this mean to our personal responsibilities as software professionals.

Jez Humble and I speak on Continuous Delivery at YOW

2012-02-14T15:46:00-05:00

Video of the one-hour overview presentation that Jez and I gave at YOW at the end of last year. We cover the rationale for Continuous Delivery, the deployment pipeline, continuous integration, devops, and deployment strategies. The highlight is Jez’s personification of a release candidate as a hero in a greek myth.

photostream 21

2012-02-12T18:32:00-05:00

Èze, France

ThoughtWorks recruiting event in New York

2012-02-07T15:17:00-05:00

This weekend, ThoughtWorks is hold a recruiting event in New York City. The idea is to compress the whole recruiting cycle down to a single weekend. The coding problem will be on Saturday, and those that pass that hurdle will be invited back for interviews on Sunday. The idea is that we’ll get job offers to any successful candidates by Sunday evening. I’ll be there for the first couple of hours on Saturday.

Infodeck on NoSQL and Polyglot Persistence

2012-02-07T10:26:00-05:00

Pramod and I have put together a short infodeck (pdf) outlining the reasons why NoSQL databases are an important new dish on the software development menu. It’s directed at non-technical managers and customers to provide an outline of why the data world has seen this new entrant and some direction as to the kinds of projects which should give them a closer look.

photostream 20

2012-02-04T16:54:00-05:00

Stowe, VT

Rebecca and I will be Keynoting at QCon London

2012-01-26T10:24:00-05:00

QCon invited Rebecca and I to do a keynote at QCon London (March 7). Over the last year or so we’ve been seeing more and more of a change in how organizations think about data, so that seems a natural choice to talk about. Our clients are seeing larger volumes of data, that they need to analyze more quickly and more cleverly. We’re seeing different technologies for data storage that have loosened the vice-like grip of relational databases. And we find our profession needs to start thinking about what our responsibilities are in managing all of this information.

Rebecca and I talk about DSLs on Software Engineering Radio

2012-01-26T09:36:00-05:00

I’ve long been a fan of the podcast series Software Engineering Radio. The team there have built an excellent series of podcasts on various aspects of software development and I often listen to them while taking my afternoon walk. So I’m glad to get a spot on there myself. In this episode I’m on the program with Rebecca Parsons, my colleague and co-author of my DSL book to talk about DSLs. We talk about what DSLs are, the differences between internal and external DSLs, and when you should (and shouldn’t use DSLs). The episode is hosted by Markus Völter, who, of course, is a considerable expert on DSLs too - so it’s really a three way conversation on the topic.

photostream 19

2012-01-22T19:49:00-05:00

Boston, MA

An open letter to Pearson opposing their support of SOPA and PIPA

2012-01-12T14:04:00-05:00

Jez Humble and I have written an open letter to Pearson opposing their support of the SOPA and PIPA bills currently under consideration from the US congress.

photostream 18

2012-01-10T18:25:00-05:00

Lindau, Germany

photostream 17

2011-11-22T17:35:00-05:00

Wachusett Meadow, Princeton, MA

Retreaded: SpecificationByExample

2011-11-17T08:38:00-05:00

Retread of post orginally made on 18 Mar 2004

I was attending a workshop at XP/Agile Universe in 2002 when the phrase 'Specification By Example' struck me as a way to describe one of roles of testing in XP.

(These days it's terribly unfashionable to talk about Test Driven Development (TDD) having anything to do with testing, but like Jon I do think that having a comprehensive automated test suite is more valuable than the term 'side-effect' implies. Rather like if someone offered me a million dollars to hike up a mountain. I may say that the main purpose of the hike is the enjoyment of nature, but the side-effect to my wallet is hardly trivial....)

Specification By Example isn't the way most of us have been brought up to think of specifications. Specifications are supposed to be general, to cover all cases. Examples only highlight a few points, you have to infer the generalizations yourself. This does mean that Specification By Example can't be the only requirements technique you use, but it doesn't mean that it can't take a leading role.

So far the dominant idea with rigorous specifications, that is those that can be clearly judged to be passed or failed, is to use pre and post conditions. These techniques dominate in formal methods, and also underpin Design by Contract. These techniques have their place, but they aren't ideal. The pre-post conditions can be very easy to write in some situations, but others can be very tricky. I've tried to use them in a number enterprise application settings, and I've found that in many situations it's as hard to write the pre and post conditions as it is to write the solution. One of the great things about specification by example is that examples are usually much easier to come up with, particularly for the non-nerds who we write the software for. (Timothy Budd pointed out that while you can show a lot of stack behavior with pre and post conditions, it's very tricky to write pre and post conditions that show the LIFO property.)

An important property of TDD tests is that they involve a double-check. In fact this highlights an amusing little lie of the XP community. They make a very strong point of saying things Once and Only Once, but in fact they always say things twice: once in the code and once in the tests. Kent has pointed out that this double-check principle is a vital principle, and it's certainly one that humans use all the time. The value of the double check is very much tied into using different methods for each side of the double check. Combining Specification By Example with production code means that you have both things said quite differently, which increases their ability to find errors.

The formal specification community have constantly had trouble verifying that a design satisfies a specification, particularly in doing this without error prone humans. For Specification By Example, this is easy. Another case of Specification By Example being less valuable in theory but more valuable in practice.

One Design by Contract fan pointed out that if you write a specification in terms of tests, then the supplier could satisfy the specification by just returning hard-coded responses to the specific test inputs. My flippant answer to this is that if you are concerned about this then the issue of tests versus Design by Contract is the least of your worries. But there is a serious point here. Tests are always going to be incomplete, so they always have to be backed up with other mechanisms. Being the twisted mind that I am, I actually see this as a plus. Since it's clear that Specification By Example isn't enough, it's clear that you need to do more to ensure that everything is properly communicated. One of the most dangerous things about a traditional requirements specification is when people think that once they've written it they are done communicating.

Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting. The examples trigger abstractions in the design team while keeping the abstractions grounded. You do need more - things like regular conversation, techniques like Domain Driven Design, indeed even doses of Design by Contract. While I've never had the opportunity to use Design by Contract fully (i.e. with Eiffel) I regularly think about interfaces in contractual terms. Specification By Example is a powerful tool, perhaps my most used tool, but never my only tool.

(For more thinking on Specification By Example, if not by that name, take a look at Brian Marick's writings. Somewhere on his site there probably is one super page that sums up his view on it. Of course finding it is less valuable than reading all the stuff there while you're trying)

Some Later Comments

A couple of years after I first wrote this, there was a post by Cedric Beust that was critical of agile methods that caused a minor blog spat. There were rebuttals by Jeff Langr and Bob Martin, and I sent this post through the feed again. Jeff Langr later added a nice example using using tests as specification by example for Java's Math.pow function.

reposted on 17 Nov 2011

photostream 16

2011-11-11T09:22:00-05:00

Boston, MA

New series book - Service Design Patterns

2011-11-10T10:38:00-05:00

Addison-Wesley has published a new book in my Signature Series. It’s by Robert Daigneau and it’s called Service Design Patterns. It’s a topic that’s already had too many books on it, but I added this one to the series because I think Robert has done a particularly good job of collecting together the best advice on the topic and organizing it into a useful handbook. This is the book that I think ought to become the standard book on the topic.

Retreaded: ContextualValidation

2011-11-03T11:00:00-04:00

Retread of post orginally made on 07 Dec 2005

In my writing endeavors, I've long intended to write a chunk of material on validation. It's an area that leads to a lot of confusion and it would be good to get some solid description of some of the techniques that work well. However life is full of things to write about, rather more than time allows.

Some recent readings made me think about saying a few preliminary things on the topic. One common thing I see people do is to develop validation routines for objects. These routines come in various ways, they may be in the object or external, they may return a boolean or throw an exception to indicate failure. But one thing that I think constantly trips people up is when they think object validity on a context independent way such as an isValid method implies.

I think it's much more useful to think of validation as something that's bound to a context - typically an action that you want to do. Is this order valid to be filled, is this customer valid to check in to the hotel. So rather than have methods like isValid have methods like isValidForCheckIn.

One of the consequences of this is that saving an object to a database is itself an action. Thinking about it that way raises some important questions. Often when people talk about a context-free validity, they mean it in terms of saving to a database. But the various validity checks that make this up should be interrogated with the question "should failing this test prevent saving?"

In About Face Alan Cooper advocated that we shouldn't let our ideas of valid states prevent a user from entering (and saving) incomplete information. I was reminded by this a few days ago when reading a draft of a book that Jimmy Nilsson is working on. He stated a principle that you should always be able to save an object, even if it has errors in it. While I'm not convinced that this should be an absolute rule, I do think people tend to prevent saving more than they ought. Thinking about the context for validation may help prevent that.

reposted on 03 Nov 2011

ThoughtWorks Agile East

2011-10-13T16:20:00-04:00

ThoughtWorks is organizing an Agile East event: a one day conference that we’re holding in New York (November 2) and Atlanta (November 7). I’ll be speaking, doing my usual suite of talks. More importantly there’s an impressive line up of my colleagues - the people whose ideas I clip and package up into those talks. So this is a great opportunity to listen to:

Graham and Andy talk from their experiences in the technical leadership of agile projects
project management lessons from Joe Z
hot devops and continuous delivery juice from Rolf and Jez
Ross’s advice on IT budgeting
Linda’s take on integrating UX with an agile project
Matt on using offshore teams for agile (he played a big role in building our offshore teams)
a trio of talks on the gnarly topic of organizational change from Tiffany, Greg and Manu
Chad explaining how to avoid the pitfalls of automated tests

and a special guest in New York who will make all our talks seem pedestrian.

photostream 15

2011-09-17T15:55:00-04:00

Miyajima, Japan

Interviewed about Agile in 2001 by Jim Highsmith

2011-09-16T08:20:00-04:00

When I went to Snowbird in 2001 for the meeting that led to the Manifesto, Jim interviewed me for the book he was writing on Agile Software Development Ecosystems. It provides a snapshot on my thinking on extreme programming and this thing that a few days later we called agile software development.

Retreaded: SoftwareDevelopmentAttitude

2011-09-08T10:04:00-04:00

Retread of post orginally made on 08 Mar 2004

Many debates in software development are underpinned by whether the speaker has a DirectingAttitude or an EnablingAttitude. These different attitudes affect choices over languages, designs, tools, processes, and lots more.

Here's some examples of this dichotomy:

A debate a while ago triggered by Joel Spolsky's post on exceptions. He didn't like exceptions because they could be misused badly, leading to confusing code (directing). Bill Caputo pointed out that exceptions, when used well, make life much easier (enabling).
Some of the static/dynamic typing debate brings up these points. Some arguments in favor of static typing talk about how they prevent people from making certain kinds of mistake (directing) while dynamic typers point out how static typing restricts some useful idioms (enabling).
Agile processes are PeopleOriented (enabling), while plan-driven methods seek to ensure that even a poor team can do an acceptable job (directing).

These aren't hard-wired attitudes. Often people are directing in some cases and enabling in others. But I think there is a deep strain running through here, often a personality issue, that runs underneath much discussion on how we do software. (I'm very much in the enabling category, as if you can't tell.)

You might think that all restrictions on what a developer does imply a directing attitude, but it's not that simple. As an example, consider memory management. You can think of this as a directing feature: programmers can't be trusted to manage memory correctly so take away their ability to allocate memory. But I look at memory management as an enabling technology - it takes away something I don't want to worry about, so I can concentrate better on those things I do care about. Steve tied this thought nicely onto the difference between problems and difficulties.

reposted on 08 Sep 2011

photostream 14

2011-09-07T19:01:00-04:00

Gourdon, France

Speaking in Frankfurt and Berlin in October

2011-09-06T09:04:00-04:00

I’ll be taking part in a couple of ThoughtWorks events in Germany during October. They are both evening events (Oct 17 Frankfurt and Oct 18 Berlin). I shall do my usual suite of talks on some mix of topics. I’ll be joined by two of my colleagues: Wolf Schlegel, who will be talking about continuous delivery and Erik Dörnenburg who will be talking about using lean techniques with enterprise architecture.

photostream 13

2011-08-31T22:29:00-04:00

Highgate, London

Update of "It's Not Just Standing Up"

2011-08-29T09:21:00-04:00

Short daily “stand-up” meetings are a regular feature of agile projects. Done well they allow frequent communication across the team with a high signal to noise ratio. Done badly, they can be pretty grim. For several years, this article by Jason Yip has become the definitive source of information on how to make them better. Jason has just finished a comprehensive revision of the article, so if you’re looking to get the best from your stand-ups, you should take a good look at the tips in here.

Speaking at goto Aarhus

2011-08-29T09:12:00-04:00

As in so many previous years, I’ll be speaking at the the goto conference (formerly known as JAOO) in early October. This is an excellent conference with a good range of material across both technology and process. This year I’ll be participating in a pair of tutorials.

On Oct 9 I’ll be with Jez Humble doing our full-day tutorial on Continuous Delivery, the first time we’ve brought this tutorial to Aarhus. This will cover much of the ground in Jez and Dave’s award winning book including the business case for Continuous Delivery, deployment pipelines, testing strategies, version control organization, and recipes for going live. We’ve had a lot of good feedback on this tutorial so far, so I’m glad to finally bring this to my favorite conference.

On Oct 13 I’ll join Rebecca Parsons to talk about Evolutionary Architecture. Rebecca’s long been our chief technical firefighter on difficult projects, which often have complicated dependencies to enterprise architecture. I’ve long been urging her to talk more about her experiences and techniques for handling these issues, so it will be good to work with her on this tutorial.

In between I’ll be hanging around the conference catching up on all the good pickings that will be available.

(If you want to register, using the discount code “fowl1250” will get you a worthwhile discount.)

Neal Ford's video workshop on agile engineering

2011-08-26T10:00:00-04:00

My colleague Neal Ford is a frequent writer and speaker who shares many of my interests and opinions (and is an excellent source of ideas for me to clip). In 2009-10 he wrote a series of fifteen articles for IBM developerWorks which dug deeper into the question of evolutionary architecture and design. His topics included Test Driven Development, leveraging reusable code, refactoring towards design and Domain-Specific Languages.

Now he’s followed on from this and recorded a video workshop on Agile Engineering Practices for O’Reilly that’s available on safari books online. These cover much of the topics that make evolutionary design possible, including Test-Driven Design, automation, and testing strategies.

photostream 12

2011-08-25T15:35:00-04:00

Mesa Arch, Canyonlands N.P., Utah

ThoughtWorks Microsite on Continuous Delivery

2011-08-24T10:02:00-04:00

ThoughtWorks has launched a microsite devoted to Continuous Delivery. At the moment you’ll find some short videos from the ThoughtWorks Live event we did in May, plus links to recent writing by Jez Humble (oops, now I have to call him the award-winning Jez Humble). We also post details of our upcoming events in the field of Continuous Delivery and Devops.

Beaten Again

2011-08-24T09:49:00-04:00

In the software world, we have the Jolt awards. These were originally run by Software Development magazine and are now run by Dr Dobbs magazine. My books have won a couple of ‘productivity awards’ - which are essentially the runner-up awards - but I’ve never snagged to top award (now called the ‘excellence award’). This year my DSL book was up for an award and won a ‘finalist award’ (essentially this means I was runner-up to the runner-ups).

So I failed again, but this year I would have been very annoyed if I had won. I’ve argued all year that Jez and Dave’s book on Continuous Delivery was the most important book released this year, and so I’m glad that Dr Dobbs agreed with me and gave them the excellence award. Congratulations to Jez and Dave in joining the long list of people who have kept me away from this award.

Although I keep missing the top prize, my signature series has a very good record with these awards. The 10 books in my series so far have picked up 1 finalist, 4 productivity and 2 excellence awards. Not a bad hit rate.

photostream 11

2011-08-03T18:14:00-04:00

Villa Rufolo, Ravello, Italy

ThoughtWorks Technology Radar

2011-08-02T10:58:00-04:00

We have now released the latest version of the ThoughtWorks Technology Radar. This is where a bunch of my colleagues get together and discuss the technologies that they think are worth keeping an eye on. It’s not intended to be some comprehensively researched list, as it’s just the aggregation of the opinions of the small group of people that create it. But since that group is an experienced group of ThoughtWorks consultants, I think it’s a useful pointer to things to pay attention to in enterprise software.

photostream 10

2011-07-12T19:56:00-04:00

Canterbury Cathedral, Canterbury, England

The LMAX Architecture

2011-07-12T09:43:00-04:00

When I was at QCon London last year, there was much buzz around a talk about a new retail trading system. The thing that got people’s attention was its approach to achieving its high performance needs - reaching 6 million TPS. It does this by running all its business logic on a single JVM thread, getting high speed without all the complications of concurrent programming. This article describes the architecture they use to pull this off.

photostream 9

2011-07-06T18:45:00-04:00

Gloucester, MA

Mike Mason and I talk about Feature Branching

2011-07-05T10:05:00-04:00

In this video (12 minutes) Mike Mason and I talk about the perils of Feature Branching and its alternatives.

photostream 8

2011-06-23T19:41:00-04:00

Boscastle, Cornwall

Sam Newman talks about Continuous Delivery

2011-06-21T09:37:00-04:00

Sam Newman is one of our Principal Consultants based in London who has been heavily involved in the development of our approaches to Continuous Delivery. Recently he gave a series of QTB (Quarterly Technology Briefing) talks and one of these was videoed and put on the ThoughtWorks site. It’s an excellent overview of why it’s important to learn how to do Continuous Delivery on the modern web, together with an overview of the practices needed to achieve it.

photostream 7

2011-05-28T09:54:00-04:00

Jordan Pond, Acadia N.P., ME

Reference pages for DSL patterns

2011-05-13T09:35:00-04:00

I’ve been very tardy in doing this, but I’ve finally put up a set of reference pages for the patterns in the DSL Book. There’s not much on each page, just the intent, chapter reference, and sometimes some summary text. I’ve primarily put these up to give them a stable URI for reference purposes. There’s a chance I’ll put supplemental material up there too, but I’ve had that intention before without anything happening.

photostream 6

2011-05-12T20:41:00-04:00

St Malo, Brittany, France

photostream 5

2011-05-07T18:07:00-04:00

Castle Hill, Ipswich, MA

Musings on ebooks

2011-05-05T09:18:00-04:00

It’s only just over a year since I got my first ebook reader. Now I only buy paper books when I really have to. I wrote my last book thinking of it primarily as a paper book, but that will be the last time, in the future electronic forms will be at the front of my mind. It’s clear ebooks will change the book industry in a dramatic way, but it’s not clear exactly how. Here are my current thoughts on what’s happening.

Tutorial on Continuous Delivery at goto; Aarhus

2011-05-03T16:17:00-04:00

I’ll be joining my colleague Jez Humble to give another of our tutorials on Continuous Delivery at goto; Aarhus. This is the rebranded incarnation of JAOO - which has been my favorite conference for a long time now. This will be our first time doing the full day continuous delivery tutorial in Aarhus.

Reflections on 10 years since the agile manifesto at SD Times

2011-05-03T13:05:00-04:00

SD Times has published an interview with me on my thoughts on ten years since the Agile Manifesto. I answer questions about what brought me to the meeting, whether I think the manifesto should be changed, and what I see in the next five years of the agile movement.

photostream 4

2011-05-02T14:17:00-04:00

Kumarakom, Kerala, India

Signature Series 40% off until May 15

2011-04-28T14:49:00-04:00

It’s been going on for a bit, but I’ve been tardy in letting people know. Addison-Wesley are currently having a promotion on signature series books. You get 40% off if you buy through informIT and buy at least two books in any of the series (ie mine, Kent’s or Mike’s).

Adding photos to my feed

2011-04-23T16:10:00-04:00

You may have noticed some photos cropping up in my news feed. Like many geeks I indulge in photography. So far this has been a harmless personal diversion, but I’ve reached the point where I’m feeling a desire to share some of my photos with the Internet - just in case we have a shortage. So you’ll see photos appear in my photostream from time to time. I intend to post no more than one or two a week.

photostream 3

2011-04-23T15:12:00-04:00

Mont Saint-Michel, Normandy, France

photostream 2

2011-04-23T15:11:00-04:00

Charlestown, MA

photostream 1

2011-04-23T15:10:00-04:00

401 Trail, Crested Butte, CO

Going Beyond Curly Braces

2011-04-19T16:54:00-04:00

On my long flight down to Australia recently I dug into a copy of Seven Languages in Seven Weeks. I spent less than seven hours on it, but found it a worthwhile read. He takes a lightening tour through seven languages. It’s not enough to go into much depth on any them, but it is enough to get a rough feel what’s interesting about them. I’d particularly recommend it if you are just starting to explore beyond the curly-brace languages that dominate so much of current programming.

One thing I particularly appreciate about pragprog books is that you get access to electronic versions in multiple formats. This book worked very well on the Kindle, despite the small screen.

Article on Non-Determinism in Tests

2011-04-14T09:19:00-04:00

In conversations with project teams, I’ve heard a lot about problems with flaky, unreliable tests. These non-deterministic tests are a serious problem with testing efforts. So I’ve written an article that talks about how to deal with these non-determinisms, discussing how to cope with some common causes: lack of isolation, asynchronous behavior, remote services, time, and resource leaks.

Q&A on Story Points

2011-03-29T08:57:00-04:00

My colleague Anand Vishwanath has put together a nice Q&A on the role Story Points play on a project.

InfoQ interviews Jez Humble and me about Continuous Delivery

2011-02-18T15:39:00-05:00

After our tutorial at QCon San Francisco last year, the InfoQ people sat us down for an interview on Continuous Delivery.

Update on malware issue with ThoughtWorks web sites

2011-02-18T11:19:00-05:00

I’ve posted an article detailing our malware attack. The short form is that we believe our sites are clear, but we currently have to operate them with reduced functionality. The malware only attacks a small proportion of visitors (which is part of what makes it difficult to deal with). However if you have visited our sites since Christmas with a windows machine, you should read more.

Speaking at ThoughtWorks Australia QTB

2011-02-17T14:01:00-05:00

Next week I’m heading off to spend some time in Australia (thankfully away from all this snow-shovelling). While I’m there I’ll be participating in a series of Quarterly Technology Briefings. These will take the form of a panel Q&A on Continuous Delivery. On the panel I shall be over-shadowed by Neal Ford and we will both be over-shadowed by Evan Bottcher.

Investigating ThoughtWorks in Vancouver

2011-02-08T15:53:00-05:00

Sid Pinney often takes on the job of investigating new locations for ThoughtWorks offices. He’s been one of the first to help start our offices in England, China, and Brazil. Last week I got an email saying he’s investigating possibilities in Vancouver. He’s looking for people to talk to: so if you’re a prospective client, possible employee, or are just friendly and looking to have Sid buy you a free food - get in touch with him by email or twitter.

Malware alert on thoughtworks.com

2011-02-03T12:14:00-05:00

We seem to have a shy but annoying piece of malware hanging out on thoughtworks.com. As far as we can tell, it manifests itself as a hidden iframe that redirects you to a site that hosts malware. We’re a bit vague on this, as it only appears rarely so only a couple of people have seen it.

Our biggest sign of it is via google. Google reported our site on Jan 7th as having problems, we looked into it, didn’t find anything, and Google’s complaint went away very quickly. The flag went up again on Jan 31st and this time they emailed our webmaster. We looked again for a problem, without any success, so we asked them to re-review our site. Since then various Google diagnostics have raising and lowering the suspicious flag at a dizzying rate. As I write this the site is considered safe, but given the volatility of the opinion, we aren’t sure about how things will go. (If you’re concerned you can check Google’s safebrowsing link.)

Clearly this is a bit of malware that’s set to only show itself occasionally, which makes it harder to deal with. The tricky bit is to find how it’s getting into the site. We’ve checked all our content and not found anything suspicious, but it could be third party javascript libraries, could be our Drupal instance, could even be a problem with our apache installation - but we don’t know for sure.

We’re investigating this further (including following the recommendations of stopbadware) but haven’t yet found the root cause, although we do have some strong suspects. If you see anything suspicious on our site or would like to pass on any suggestions, please contact Andy Yates.

As far as I know, this problem does not affect martinfowler.com - which is a static site built with my custom scripts, so is less likely to attract such a problem. But do let me know if you see anything.

Video of Wikileaks panel

2011-01-21T11:18:00-05:00

I enjoyed the Churchill Club panel discussion on wikileaks on Wednesday. It’s thoughtful and well worth watching the video.

I particularly appreciated Daniel Ellsberg’s picture of the climate of government secrecy and Roy’s call to follow the example of librarians resisting secret gag orders. (There was a bit were it threatens to sink deep into the nature of libertarianism, but fortunately that didn’t last long.) While my thoughts on wikileaks are complicated (and I’ve not had the energy to write about them) I think the incident raises some very serious issues around the contending forces of privacy and openness. This panel provides a useful briefing on some of these perspectives.

Cindy is doing the Avon Walk for Breast Cancer

2011-01-20T10:43:00-05:00

Last year Cindy, my wife, did Boston’s Avon walk to raise money for breast cancer research. She decided to it again this year. Last year a good proportion of the money she raised came from my readers, and we’re hoping we can get another round of support from you this year. If you’d like to donate, please go to her walk page.

Speaking event in Hamburg

2011-01-18T09:28:00-05:00

I’m going to Germany next week and we’ve added a third speaking event to my trip. As well as speaking at OOP in Munich and in Cologne, ThoughtWorks has organized a speaking event in Hamburg on Jan 31st. My colleagues Erik Döernenburg and Wolf Schlegel will give a talk on Continuous Deployment and I’ll follow with my usual SuiteOfTalks.

WikiLeaks: Why it Matters. Why it Doesn’t?

2011-01-17T11:56:00-05:00

ThoughtWorks and The Churchill Club are presenting a panel discussion on Wikileaks which will explore the consequences of this saga on the freedom of the internet. The panel features:

Daniel Ellsberg (of Pentagon Papers fame)
Clay Shirky (thoughtful writer about the internet)
Roy Singham (ThoughtWorks’s founder)
Peter Thiel (co-founder of PayPal)
Jonathan Zittrain (law professor at Harvard).

The panel is moderated by Paul Jay from The Real News Network.

It’s being held at 5.30 PST on Wednesday January 19th at the Santa Clara Marriott. You can also watch the event live on Fora.tv and The Real News Network. Streaming starts at 7pm PST.