Martin Fowler

Rebecca and I will be Keynoting at QCon London

2012-01-26T10:24:00-05:00

QCon invited Rebecca and I to do a keynote at QCon London (March 7). Over the last year or so we’ve been seeing more and more of a change in how organizations think about data, so that seems a natural choice to talk about. Our clients are seeing larger volumes of data, that they need to analyze more quickly and more cleverly. We’re seeing different technologies for data storage that have loosened the vice-like grip of relational databases. And we find our profession needs to start thinking about what our responsibilities are in managing all of this information.

Rebecca and I talk about DSLs on Software Engineering Radio

2012-01-26T09:36:00-05:00

I’ve long been a fan of the podcast series Software Engineering Radio. The team there have built an excellent series of podcasts on various aspects of software development and I often listen to them while taking my afternoon walk. So I’m glad to get a spot on there myself. In this episode I’m on the program with Rebecca Parsons, my colleague and co-author of my DSL book to talk about DSLs. We talk about what DSLs are, the differences between internal and external DSLs, and when you should (and shouldn’t use DSLs). The episode is hosted by Markus Völter, who, of course, is a considerable expert on DSLs too - so it’s really a three way conversation on the topic.

Bliki: CharityCodeJam

2012-01-25T09:39:00-05:00

Over the last couple of years several of my colleagues have been organizing code jam[1] events where developers get together to write software for charitable causes. A good example is a regular code-jam in New York that works on RapidFTR. Chris George, a ThoughtWorker based in New York, helped organize a one-off event in New York in August 2010. The group didn't get as much done on the day as they had hoped, but in a bar afterwards decided to try to get together more regularly. Since then they've been meeting every week. It's a small group, still mostly ThoughtWorkers and friends, with a core of 3-4 people rising to a dozen when we've had a big project in town.[2]. (Chris is happy to have more people join the group, so if you are interested drop him an email.)

Many people have found these events to an enjoyable way to use our skills for purposes that we find rather more fulfilling than many day jobs, and a way both to learn new skills and learn from a different group of people. So I thought I should share our thoughts on how to set one up.

The first thing is to find a suitable effort to contribute to. We look to contribute to projects producing open-source software for NGOs - the open source model fits well for such organizations. The two that we've built up most of a relationship are RapidFTR and OpenMRS. RapidFTR is a system designed to help reunite families after a natural disaster or other calamity. It allows people to quickly input information about either a lost child, or a child found without parents - then provide search facilities to match them up. OpenMRS is an open source medical records system, designed to support various forms of health care delivery work. It's used by many health care groups all over the world (and not just in the developing world).

Like New York, most of our code-jams begin as one-off events, a single evening or all day event. These days we advertise heavily, and hope to get a good sample of local developers to come along and take part. One-off code jams like this don't usually produce much useful software, since they are too short to really accomplish much. But they still have value. Firstly they generate awareness, exposing the local development community both to the specific project and the notion of working on open-source efforts for NGOs. More directly they can be the seed for an regular code jam, so it's useful to put together activities that will encourage getting back together later.

A regular code jam gets together on a schedule, with core of people who come most weeks. Such a group can make some significant contributions to a code base. People come because they get to work on some different technologies, with a different group of co-developers, to an audience that (unlike most open-source projects) isn't just other developers.

To make meaningful progress, you need someone to prepare for each code jam by breaking down work-items into something small enough that people will be able to finish them during the time at the jam. Whatever people may say and hope, they'll rarely work on the project outside code jam hours, and the schedule is too infrequent to want half-done things hanging over. Small tasks allow teams to make perceptible progress each jam - which helps keep motivation high. We like to put these tasks online before each event so people can prepare if they want to, or just get a feel for what we're working on. We also set up a mailing list to keep up regular communication on the jam and support anyone who does contribute outside of the jam.

Our regular code-jams succeed best when the group has a couple of champions who take the lead in organizing the event. It's best to have more than one champion, to cope with the work load and provide some resilience if they are absent for a while.

We try to ensure the development environment is set up to allow people to come in quickly and become productive. Much of this is the same kind of thing that we do on projects anyway to enable continuous integration - make sure that installation and build are automated so people can quickly install the code base and get it working. It's important to mention this in the advertisements for the event - people are often put off from coming due to a concern that they'll never get started due to these issues. Even so, make sure that each event has at least one person who is familiar with the code base and build environment, she can then help others find their way around. Often someone will give a short overview of what the system does and how it works for new people at the start of the jam.

We usually provide food to each event - that's an easy thing for us to do as a corporate contribution. As any XPer knows, sharing food when working is an important part of gelling a team.

So, if the idea of a code-jam appeals to you, why not give it a try? Find a suitable project to contribute to, a group of a few people to act as a core, and spend a few sessions to get things going. (There are developer guides for both OpenMRS and RapidFTR to help you get started if those projects appeal to you.) If you get going on a stable basis - post in a blog somewhere so we see what code-jams are avaialable and learn more about how to get them going.

1: "Code-jam" is a problematic name for these events. As far as I can determine, the term "code-jam" was originally used for competitive events where programmers would try to best their peers in some programming challenge. The events I describe here are the utter opposite of this, on many levels, but have attracted the same name.

2: When one of the team went down to our Porto Alegre office, he got a group contributing there too.

photostream 19

2012-01-22T19:49:00-05:00

Boston, MA

Bliki: AggregateOrientedDatabase

2012-01-19T09:11:00-05:00

One of the first topics to spring to mind as we worked on NosqlDistilled was that NoSQL databases use different data models than the relational model. Most sources I've looked at mention at least four groups of data model: key-value, document, column-family, and graph. Looking at this list, there's a big similarity between the first three - all have a fundamental unit of storage which is a rich structure of closely related data: for key-value stores it's the value, for document stores it's the document, and for column-family stores it's the column family. In DDD terms, this group of data is an aggregate.

The rise of NoSQL databases has been driven primarily by the desire to store data effectively on large clusters - such as the setups used by Google and Amazon. Relational databases were not designed with clusters in mind, which is why people have cast around for an alternative. Storing aggregates as fundamental units makes a lot of sense for running on a cluster. Aggregates make natural units for distribution strategies such as sharding, since you have a large clump of data that you expect to be accessed together.

An aggregate also makes a lot of sense to an application programmer. If you're capturing a screenful of information and storing it in a relational database, you have to decompose that information into rows before storing it away.

An aggregate makes for a much simpler mapping - which is why many early adopters of NoSQL databases report that it's an easier programming model.

This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster.

There is a significant downside - the whole approach works really well when data access is aligned with the aggregates, but what if you want to look at the data in a different way? Order entry naturally stores orders as aggregates, but analyzing product sales cuts across the aggregate structure. The advantage of not using an aggregate structure in the database is that it allows you to slice and dice your data different ways for different audiences.

This is why aggregate-oriented stores talk so much about map-reduce - which is a programming pattern that's well suited to running on clusters. Map-reduce jobs can reorganize the data into different groups for different readers - what many people refer to as materialized views. But it's more work to do this than using the relational model.

This is part of the argument for PolyglotPersistence - use aggregate-oriented databases when you are manipulating clear aggregates (especially if you are running on a cluster) and use relational databases (or a graph database) when you want to manipulate that data in different ways.

An open letter to Pearson opposing their support of SOPA and PIPA

2012-01-12T14:04:00-05:00

Jez Humble and I have written an open letter to Pearson opposing their support of the SOPA and PIPA bills currently under consideration from the US congress.

Bliki: DiversityImbalance

2012-01-11T10:57:00-05:00

Although it's easy to become accustomed to it, it's pretty obvious the software development world has some serious issues in diversity. By this I mean that we have some notable differences in proportions of people compared to the general population. One of the most obvious differences is the low proportion of women, which is true all over the world (albeit noticeably less so in China). In the US, where I spend a good chunk of my time, the lack of African-Americans is also obvious. There's a lot been written on why such imbalances might exist, and what might be done about it. [1] But here I want to concentrate on a more fundamental question - does it matter?

One point of view I hear fairly regularly is that these diversity imbalances are natural - because women don't have the aptitude or inclination for programming. This point of view upsets a lot of people but I think it's important to treat it seriously. I think of it as a hypothesis, which I'll call the natural balance hypothesis. It needs to be treated seriously because there's plenty of people who feel it explains the current situation - but I argue that it has two serious flaws, which mean that I must vigorously reject it.

The first flaw is a simple one of evidence. There are (roughly) 50% women in the world, so we should expect the ratio for women in computing to be 50% - unless there's real evidence that some other ratio is natural.[2] So far there's no such evidence. Sure, it's obvious that there are biological differences between men and women, and there is evidence that there are differences in brain function between the sexes. But there is no evidence that indicates that the skills that make people better programmers are more common in men.

The only evidence that seems to occur to people who promote the natural balance hypothesis is the fact that there are less female programmers.[3] Personally I find it troubling when software professionals, who ought to be good at logical thinking, can reach so easily for such circular logic.

It is the second flaw in the natural imbalance hypothesis that brings the heat into the discussion. Men have spent centuries using this kind of argument to deny women equal rights in all sorts of fields. Over the last century we've seen tons of evidence that this isn't true elsewhere, so why should it be true in software? As far as I'm concerned this shoddy history should make us doubly wary of the any suggestion that a diversity imbalance is natural. Unless someone comes up with decent evidence that there is a relevant biological difference, we must operate on the assumption that women are equally well suited to programming.

You'll notice above that I said "inclination or aptitude". I've noticed a lot recently that advocates of the natural-balance hypothesis say less frequently that women have less aptitude than men for programming, instead they say that women don't want to do programming. But making statements with inclination is little better than with aptitude - there's still no evidence and it has just the same shoddy history.

So, accepting that there is no good reason for a diversity imbalance, does such an imbalance matter? That is, given we have a unnatural imbalance, is it a problem that's sufficiently serious to spend energy on fixing it? I think there are many reasons why it matters to tackle our imbalances. First and foremost, there's a moral argument. I'm a strong meritocrat, who believes that we should strive for a society where everyone has an equal opportunity to fulfill their potential. A diversity imbalance suggest that there are many women, who would have good careers as programmers, who are not getting the opportunity to do so. I agree with Eric Ries's view that diversity imbalances suggest we are not as meritocratic as we like to think we are.

This waste hurts our profession too. We need more and better software developers to produce valuable software that improves our lives. By not bringing enough women into the profession, we are handicapping ourselves. This will only become more serious as the demand for talent increases in the future. How can we say we are hiring the best people when we ignore significant chunks of our population. Critics of efforts to fix the diversity imbalance often fret that we risk failing to hire a well-qualified male, when we habitually fail to hire well-qualified females.

Lack of diversity is itself a problem. Different people think differently, and consequently come up with different ways to solve problems. If you have a bunch of people with the same background, they miss lots of ideas - leading to inefficiencies and lack of innovation. A diverse group is usually more effective.

This lack of diversity also contributes to our marginalization as a profession. We are already in the situation where the opinions of programmers aren't taken as seriously as they should be by people outside our profession. We see this regularly in our discussions with business people who dismiss us as mere nerds. A diversity imbalance makes us look even more like some marginalizable outsider.

Left to themselves, these kinds of imbalances tend to get worse. People have a natural, often sub-conscious, tendency to be around people like themselves. Consequently as a group becomes a smaller minority, they get excluded more. A warning sign is when people are turned away because they "won't fit in".

There is a great deal of good potential in the software profession. We have a strong tendency towards meritocracy, a natural first position with exploiting the power of computers to enhance our lives, a lack of historical baggage in how we organize ourselves and our work. As a result I think we could provide a model to influence other social groups and lead the way in demonstrating how humans can collaborate. Diversity imbalances are a cancer to that position - how can we claim to be forward thinking when our diversity looks it comes from where the rest of the world was a couple of generations ago?

photostream 18

2012-01-10T18:25:00-05:00

Lindau, Germany

Bliki: NosqlDefinition

2012-01-09T09:30:00-05:00

As soon as we started work on NosqlDistilled we were faced with a tricky conundrum - what are we writing about? What exactly is a NoSQL database? There's no strong definition of the concept out there, no trademarks, no standard group, not even a manifesto.

The term originally surfaced at an informal meetup on June 11 2009 in San Francisco organized by Johan Oskarsson. At the session there were presentations from Voldemort, Cassandra, Dynomite, HBase, Hypertable, CouchDB, and MongoDB. The term caught on rapidly and few people would argue that only the databases mentioned at that meeting should be called NoSQL.

Indeed there's often a twist in the name itself: many advocates of NoSQL say that it does not mean a "no" to SQL, rather it means Not Only SQL. On this point I think it's useful to separate an individual database from the kind of ecosystem that NoSQL advocates see as the future. When we say "x is a NoSQL database" I think it's silly to interpret NoSQL as "Not Only" because that would render the term meaningless. (You could then reasonably argue that SQL Server (say) is a NoSQL database.) So I think it's best to say a "NoSQL database" is a "no-sql" database. You should separately interpret the NoSQL ecosystem as a "not only" - although I prefer the term PolyglotPersistence for this usage. [1]

Even with this matter out of the way, it's still not easy to define a NoSQL database. Does any database that doesn't use SQL qualify? How about older database technologies such as IMS or MUMPS? How about a relational system that didn't have SQL (such as the early Ingres)? What happens if someone manages to bolt a SQL interface onto one of the original septet?

So for our book we took a view that NoSQL refers to a particular rush of recent databases. Some characteristics are common amongst these databases, but none are definitional.

Not using the relational model (nor the SQL language)
Open source
Designed to run on large clusters
Based on the needs of 21st century web properties
No schema, allowing fields to be added to any record without controls

While I'm used to the blurry lines of definitions in the software industry, I confess my heart sinks at yet another one. But the important thing is that these databases provide a important addition to the way we'll be building application in next couple of decades. A lack of a clear definition will be no more than a gnat bite on their future successes.

1: If we take the "not-only" interpretation, then we should write "NOSQL" rather than "NoSQL". I almost always see it written as "NoSQL".

Bliki: NosqlDistilled

2012-01-04T10:01:00-05:00

Over the last few months[1] I've been helping my colleague Pramod Sadalage work on a book on NoSQL technologies to be titled NoSQL Distilled. (You may know of Pramod's work on refactoring databases and evolutionary database design.) In the last year we've been doing a few projects that have used NoSQL technology, and we think it's going to play an important role in the next few years of software development.

Pramod Sadalage

We're writing this book as a brief introduction to help people understand the issues involved in using this technology. As you might expect by the "distilled" in the title, it's modeled on the style of UML Distilled - a short overview aimed at giving you enough information to get started, providing core concepts and orientation for further study should you get deeper into it. Our target is a book of 100-150 pages, and we intend to be pretty firm on the upper limit.

It's a ill-defined and volatile field, which makes it tricky to decide what to include. This problem is exacerbated by the fact that we want a book that will be of lasting value - so we have to try and pick core concepts that span the various NoSQL systems out there and will continue to be important as they develop over the next years.

The current contents include such topics as aggregate-oriented data models, consistency in distributed data, and database evolution. We also have short example chapters that look at databases such as MongoDB, Neo4J, and Riak.

As I write this we are getting close to a first draft for technical review. We hope to reach a final draft by spring this year at which point there will be an electronic rough-cut available. The physical book, and production ebooks, should appear early this summer. I'll post here as we make progress, and I have a few related bliki posts brewing in the brain too.

1: I haven't talked about this during our work over the last few months because I don't like announcing a book project before I'm pretty sure it will ship. Now it's looking good, so it's time to reflect on the fact that it's a go.

Bliki: Slideument

2011-12-19T09:41:00-05:00

A slideument is a cross between a slide deck and a document. The idea is that you can use a single slide deck both for slides during your presentation and as a handout for people to read afterwards. The trouble is that those two needs lead to very different requirements on your slides, so you can't satisfy them both. The result is that slideuments usually fail at both.

The main reason they fail is due to the amount of words and detail you need in the deck. If you want a stand-alone document you need enough context to make sense without the speaker being there. This requires a lot of words. But if the speaker is there, and speaking, then the audience ends up both reading and listening - and usually not concentrating properly on either. A good slide deck is a VisualChannel, it provides an accompaniment to the spoken words, reinforcing but not duplicating what is being said.

If people want a stand-alone document, then you can provide them with one - but it should be a different document, one that's designed for reading and not speaking.

The most sensible argument I've heard in favor of slideuments is that audience expects them, even if they are useless and never read later. I've a lot of sympathy with this argument, as it goes back to my first major presentation - a tutorial at OOPSLA in 1992. Since this was a big deal for me I decided to work hard to provide an extra-special contribution. So instead of the usual copies of slides, I wrote a special document to support the talk - about 40 or so pages - and provided that as a handout. That document was a much better coverage of the topic then the slides would be. But on the day I got lots of complaints becuase I didn't provide a copy of the slides. I am forever grateful to Rebecca Wirfs-Brock, who was the tutorial chair that year and backed me up.

People do expect slides, even when they don't really make any sense, and often it's easiest just to run off a pdf. I think that ideally you should try for more. My approach in later tutorials was to give out both the document and the slide copies. These days I give people my TalkNotes URI that points to relevant articles I've written on the topic of my talk.

Slideuments shouldn't be confused with Infodecks

Bliki: Infodeck

2011-12-19T09:40:00-05:00

When I rant on at my colleagues about the evils of Slideuments, I do hear a useful push-back. Many people now like to communicate through slide decks that aren't meant to be used for presentations, just to be read. People like me can snark about managers these days being unable to read anything that doesn't look like a bullet point, but there are advantage to these infodecks.

You can use spatial layout to help explanation
They discourage long prose that people don't read
It's easy to include diagrams as primary elements in the communication

All this can lead to a document that's more approachable and easier to communicate information than a traditional prose text would be.

If a deck is intended for reading and not for projection, then I don't count it as a slideument. Slideuments are bad because they try to do two different things, but an infodeck is there only for reading, so can be done well. Often, of course, they aren't but that's a problem in the execution rather than the fundamental concept.

Infodecks are an interesting form to me, if only because it seems nobody takes them seriously. Most users look at them as something they just knock up with PowerPoint, and writers with pretentions to seriousness (such as myself) consider infodecks to be beneath us. I'm begining to think we should take them more seriously. We should think and talk more about how to make them effective as communciation vehicles.

One reason that these are likely to grow is that increasingly we are communciating on computer devices and not using paper. Modern computers have bright multi-color screens, and aren't worried about conserving paper. So a colorful, diagram-heavy approach that uses lots of virtual pages is an effective form of document - especially as tablets become more prevalent.

So what makes a good infodeck? Here's a few of my preliminary thoughts.

Although they are usually created with a presentation tool, such as PowerPoint or Keynote, they don't have to be. A page-layout style tool is often a better choice. An example of this is the Mac's Pages application. It's page-layout mode works very well for infodecks.
Landscape orientation is probably best, if only because they are often read on laptops.
Although bullets are the most common forms of using text, stretch out and write short paragraphs. That will convey more information without being too much text. There's no need to limit yourself to bullets if the deck isn't going to be projected.
Use diagrams as much as possible and make them the central element of the page with text around them. If you can't think of diagram try laying out the blocks of text with positioning and connecting lines to suggest any interrelationships between them.
Use PDF to send info decks around to others. PDF is the best document format for format-intensive documents like infodecks. (ePub is better for prose documents.)

Bliki: ThunderboltDisplay

2011-11-23T10:06:00-05:00

A few months ago I got issued a new company laptop - a Macbook Pro with the Thunderbolt port. As I got it started idly thinking about getting a Thunderbolt display. I've heard good things about Apple displays, despite their expense, and the idea of a display that would act as a docking station was appealing.

Sensing this laptop reached across the network to my Ubuntu desktop, seduced it, and persuaded it to kill one of its monitors. I took that as a sign, so once I got back from my travels a few weeks ago, I hit the apple store and came home with a heavy box.

I'll start with the bad. The display often doesn't wake from sleep. Each night I close my laptop lid, putting the mac to sleep. When I start in the morning I lift the lid, and roughly three quarters of the time the mac wakes up but the display doesn't. To get the display to start I have to unplug and replug the thunderbolt port. This is annoying, particularly if there's a disk drive plugged into the display (as it gets unceremoniously dismounted). I've had no success in tracking down why this happens, and it's erratic nature means it's hard to troubleshoot.

Another irritant is the glossy screen, which means I get some reflections from the light on my desk and window behind me. I've mostly got used to it, but it is annoying at times.

Having whined about that, I do need to confess that I like this display more than I thought. It's large size and high resolution means that I can fit lots on the screen at once. In my normal writing mode I can easily fit two emacs frames and a browser side by side. It's also really good when working on photographs in Aperture.

The suprising thing is that it's shifted my work habits. I used to do my general writing on my Ubuntu desktop, but since getting the Thunderbolt I've gravitated to doing my writing on the mac because the display is just so nice to work on. There's a serious danger that my Ubuntu box is going to become unused. (Which, I guess, is one way to cope with Unity.)

I've also found it makes video calls better. The built-in camera and speakers work very well for that. It's nice to have a decent-sized video feed and a document we're looking at right on the screen in front of me. (Although that's another reason to do something about the window behind me.)

I've always believed its worth spending money on a BigScreen - and this one, despite the sleep problem, is a winner for me.

Retreaded: SpecificationByExample

2011-11-17T08:38:00-05:00

Retread of post orginally made on 18 Mar 2004

I was attending a workshop at XP/Agile Universe in 2002 when the phrase 'Specification By Example' struck me as a way to describe one of roles of testing in XP.

(These days it's terribly unfashionable to talk about Test Driven Development (TDD) having anything to do with testing, but like Jon I do think that having a comprehensive automated test suite is more valuable than the term 'side-effect' implies. Rather like if someone offered me a million dollars to hike up a mountain. I may say that the main purpose of the hike is the enjoyment of nature, but the side-effect to my wallet is hardly trivial....)

Specification By Example isn't the way most of us have been brought up to think of specifications. Specifications are supposed to be general, to cover all cases. Examples only highlight a few points, you have to infer the generalizations yourself. This does mean that Specification By Example can't be the only requirements technique you use, but it doesn't mean that it can't take a leading role.

So far the dominant idea with rigorous specifications, that is those that can be clearly judged to be passed or failed, is to use pre and post conditions. These techniques dominate in formal methods, and also underpin Design by Contract. These techniques have their place, but they aren't ideal. The pre-post conditions can be very easy to write in some situations, but others can be very tricky. I've tried to use them in a number enterprise application settings, and I've found that in many situations it's as hard to write the pre and post conditions as it is to write the solution. One of the great things about specification by example is that examples are usually much easier to come up with, particularly for the non-nerds who we write the software for. (Timothy Budd pointed out that while you can show a lot of stack behavior with pre and post conditions, it's very tricky to write pre and post conditions that show the LIFO property.)

An important property of TDD tests is that they involve a double-check. In fact this highlights an amusing little lie of the XP community. They make a very strong point of saying things Once and Only Once, but in fact they always say things twice: once in the code and once in the tests. Kent has pointed out that this double-check principle is a vital principle, and it's certainly one that humans use all the time. The value of the double check is very much tied into using different methods for each side of the double check. Combining Specification By Example with production code means that you have both things said quite differently, which increases their ability to find errors.

The formal specification community have constantly had trouble verifying that a design satisfies a specification, particularly in doing this without error prone humans. For Specification By Example, this is easy. Another case of Specification By Example being less valuable in theory but more valuable in practice.

One Design by Contract fan pointed out that if you write a specification in terms of tests, then the supplier could satisfy the specification by just returning hard-coded responses to the specific test inputs. My flippant answer to this is that if you are concerned about this then the issue of tests versus Design by Contract is the least of your worries. But there is a serious point here. Tests are always going to be incomplete, so they always have to be backed up with other mechanisms. Being the twisted mind that I am, I actually see this as a plus. Since it's clear that Specification By Example isn't enough, it's clear that you need to do more to ensure that everything is properly communicated. One of the most dangerous things about a traditional requirements specification is when people think that once they've written it they are done communicating.

Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting. The examples trigger abstractions in the design team while keeping the abstractions grounded. You do need more - things like regular conversation, techniques like Domain Driven Design, indeed even doses of Design by Contract. While I've never had the opportunity to use Design by Contract fully (i.e. with Eiffel) I regularly think about interfaces in contractual terms. Specification By Example is a powerful tool, perhaps my most used tool, but never my only tool.

(For more thinking on Specification By Example, if not by that name, take a look at Brian Marick's writings. Somewhere on his site there probably is one super page that sums up his view on it. Of course finding it is less valuable than reading all the stuff there while you're trying)

Some Later Comments

A couple of years after I first wrote this, there was a post by Cedric Beust that was critical of agile methods that caused a minor blog spat. There were rebuttals by Jeff Langr and Bob Martin, and I sent this post through the feed again. Jeff Langr later added a nice example using using tests as specification by example for Java's Math.pow function.

reposted on 17 Nov 2011

Bliki: PolyglotPersistence

2011-11-16T10:03:00-05:00

In 2006, my colleague Neal Ford coined the term Polyglot Programming, to express the idea that applications should be written in a mix of languages to take advantage of the fact that different languages are suitable for tackling different problems. Complex applications combine different types of problems, so picking the right language for the job may be more productive than trying to fit all aspects into a single language.

Over the last few years there's been an explosion of interest in new languages, particularly functional languages, and I'm often tempted to spend some time delving into Clojure, Scala, Erlang, or the like. But my time is limited and I'm giving a higher priority to another, more significant shift, that of the DatabaseThaw. The first drips have been coming through from clients and other contacts and the prospects are enticing. I'm confident to say that if you starting a new strategic enterprise application you should no longer be assuming that your persistence should be relational. The relational option might be the right one - but you should seriously look at other alternatives.

One of the interesting consequences of this is that we are gearing up for a shift to polyglot persistence [1] - where any decent sized enterprise will have a variety of different data storage technologies for different kinds of data. There will still be large amounts of it managed in relational stores, but increasingly we'll be first asking how we want to manipulate the data and only then figuring out what technology is the best bet for it.

This polyglot affect will be apparent even within a single application[2]. A complex enterprise application uses different kinds of data, and already usually integrates information from different sources. Increasingly we'll see such applications manage their own data using different technologies depending on how the data is used. This trend will be complementary to the trend of breaking up application code into separate components integrating through web services. A component boundary is a good way to wrap a particular storage technology chosen for the way its data in manipulated.

This will come at a cost in complexity. Each data storage mechanism introduces a new interface to be learned. Furthermore data storage is usually a performance bottleneck, so you have to understand a lot about how the technology works to get decent speed. Using the right persistence technology will make this easier, but the challenge won't go away.

Many of these NoSQL option involve running on large clusters. This introduces not just a different data model, but a whole range of new questions about consistency and availability. The transactional single point of truth will no longer hold sway (although its role as such has often been illusory).

So polyglot persistence will come at a cost - but it will come because the benefits are worth it. When relational databases are used inappropriately, they exert a significant drag on application development. I was recently talking to a team whose application was essentially composing and serving web pages. They only looked up page elements by ID, they had no need for transactions, and no need to share their database. A problem like this is much better suited to a key-value store than the corporate relational hammer they had to use. A good public example of using the right NoSQL choice for the job is The Guardian - who have felt a definite productivity gain from using MongoDB over their previous relational option.

Another benefit comes in running over a cluster. Scaling to lots of traffic gets harder and harder to do with vertical scaling - a fact we've known for a long time. Many NoSQL databases are designed to operate over clusters and can tackle larger volumes of traffic and data than is realistic with single server. As enterprises look to use data more, this kind of scaling will become increasingly important. The Danish medication system described at gotoAarhus2011 was a good example of this.

All of this leads to a big change, but it won't be rapid one - companies are naturally conservative when it comes to their data storage.

The more immediate question is which types of projects should consider an alternative persistence model? My thinking is that firstly you should only consider projects that are at the strategic end of the UtilityVsStrategicDichotomy. That's because utility projects don't have enough benefit to be worth a new technology.

Given a strategic project, you then have two drivers that raise alternatives: either reducing development drag or dealing with intensive data needs. Even here I suspect many projects, probably a majority, are better off sticking with the relational orthodoxy. But the minority that shouldn't is a significant one.

One factor that is perhaps less important is whether the project is new, or already established. The Guardian's shift to MongoDB has been happening over the last year or so on a code base developed several years ago. Polyglot persistence is something you can introduce on an existing code base.

What all of this means is that if you're working in the enterprise application world, now is the time to start familiarizing yourself with alternative data storage options. This won't be a fast revolution, but I do believe the next decade will see the database thaw progress rapidly.

1: As far as I can tell, Scott Leberknight was the first person to start using the term "polyglot persistence".

2: Don't take the example in the diagram too seriously. I'm not making any recommendations about which database technology to use for what kind of service. But I do think that people should consider these kinds of technologies as part of application architecture.

Bliki: PrematureRampUp

2011-11-10T10:31:00-05:00

One of the good things about software is that people seem to want it, and want it quickly. It's usual for organizations to ask teams to speed up production of software and from time to time the organization seeks to help in the way that really shows its commitment - by spending money to add more people to the team.

Now there's many an argument that's been had about the true benefit of adding people to a software team. It's clear to me that you don't get a linear benefit, doubling a team won't double its productivity because the communication and coordination costs kick in and blunt the increase. My utterly unscientific rule of thumb is that any increase in productivity is likely to be proportional to the square root of the increase in people, so doubling a team will get you roughly a 50% increase in productivity. But in practice this will vary a great deal depending on individual factors. There are some people I know who are likely to double the productivity of even a sizable team just on their own, and we have run into people who will reduce a team's productivity.

But the issue I want to highlight here is that of the ramp-up. You have a small team that's working well, but you want more software and you are prepared to spend the money to get it. You're happy to pay quadruple, even sextuple to double your rate of progress. An important, yet not well understood factor is the rate at which you can safely add people to a team.

More than once, I've come across projects who added too many people too quickly. This manifests itself in a breakdown of the cohesion of the code base itself. Duplication runs rampant, several friends of mine know about the project that had three object-relational mapping frameworks within a single application. This breakdown occurs because the new people don't understand how the code base currently works, so they do something at odds with it, like adding a competing framework. If there's too many new people around, the team leadership can't keep track of it all and the code base sprouts problems. These problems then reinforce each other because nobody can find a consistent way to do things, the Broken Windows syndrome kicks in and you get a positive feedback loop. (And positive feedback loops are usually a bad thing.)

On top of this an overly rapid ramp up leads to a break down of the human communication mechanisms. It takes time for people to get used to working with each other and a rapid ramp up can stop a large team from forming the relationships it needs to succeed.

So how much ramp up can you safely do? It's difficult to give any concrete advice here, because any experienced project manager I ask always rightly points out that there are many variables that have to be taken into account. I pressed Joe Zenevitch, one of my most trusted PM sources, and he indicated that he would never want to more than double a team at once. Even doubling would be a risky call, with the risk increasing if the exisiting team was already a dozen or more people, or had a signficant amount of juniors.

If you do a significant increase in size you shouldn't add yet more people until the new folks have settled into the team. It will take a few weeks for this to happen. Developers need to get to know the code-base and the domain, BAs need to be familiar with the domain experts, everyone needs to get to know each other. At ThoughtWorks we expect people to come up to speed quickly, after all we hire bright, high motivated people who are quick learners. But even so it still take a week or two. For most teams you should allow a good bit longer.

When you add people to a team, you don't get an immediate increase in capability. It takes time for people to become productive on a new project. Worse still existing staff have to spend time helping them get up to speed, so your velocity may well drop at first. Joe Z's observation for ThoughtWorks teams is that there will be no net effect for the first couple of weeks as the new people and hit for existing people cancel out. [1] Most ThoughtWorkers like to point out that pair programming is a big enabler to on-boarding people more rapidly. Pat Kua also has some good advice on how to bring new people onto a team effectively.

Another thing to watch is to not ramp-up too early in the project. One of the firmest bits of advice I've heard from people who do large projects (anything over 50 people) is that the project should start small, maybe with just a dozen or so developers. They should figure out the key design elements and interactions of the system by building early parts of it. Only once that design has settled down should you think of increasing the team size to its full size. As part of that settling down put time into removing any design elements that you don't think should be copied. People will naturally copy stuff that's already there, so you should ensure that what's there is all going to make a good platform for further development. This is a time to err on the side of excessive attention to code-cleanliness.

Finally, when thinking of ramping up, think very carefully about whether it's worth it. I've rarely come across a large team where there isn't a feeling that the team could be significantly cut without reducing its productivity. As I once said "scaling agile is the last thing you want to do".

1: This couple-of-weeks rule is for ThoughtWorkers, so we would expect it would take longer for people who don't match our hiring standards.

Retreaded: ContextualValidation

2011-11-03T11:00:00-04:00

Retread of post orginally made on 07 Dec 2005

In my writing endeavors, I've long intended to write a chunk of material on validation. It's an area that leads to a lot of confusion and it would be good to get some solid description of some of the techniques that work well. However life is full of things to write about, rather more than time allows.

Some recent readings made me think about saying a few preliminary things on the topic. One common thing I see people do is to develop validation routines for objects. These routines come in various ways, they may be in the object or external, they may return a boolean or throw an exception to indicate failure. But one thing that I think constantly trips people up is when they think object validity on a context independent way such as an isValid method implies.

I think it's much more useful to think of validation as something that's bound to a context - typically an action that you want to do. Is this order valid to be filled, is this customer valid to check in to the hotel. So rather than have methods like isValid have methods like isValidForCheckIn.

One of the consequences of this is that saving an object to a database is itself an action. Thinking about it that way raises some important questions. Often when people talk about a context-free validity, they mean it in terms of saving to a database. But the various validity checks that make this up should be interrogated with the question "should failing this test prevent saving?"

In About Face Alan Cooper advocated that we shouldn't let our ideas of valid states prevent a user from entering (and saving) incomplete information. I was reminded by this a few days ago when reading a draft of a book that Jimmy Nilsson is working on. He stated a principle that you should always be able to save an object, even if it has errors in it. While I'm not convinced that this should be an absolute rule, I do think people tend to prevent saving more than they ought. Thinking about the context for validation may help prevent that.

reposted on 03 Nov 2011

Retreaded: SoftwareDevelopmentAttitude

2011-09-08T10:04:00-04:00

Retread of post orginally made on 08 Mar 2004

Many debates in software development are underpinned by whether the speaker has a DirectingAttitude or an EnablingAttitude. These different attitudes affect choices over languages, designs, tools, processes, and lots more.

Here's some examples of this dichotomy:

A debate a while ago triggered by Joel Spolsky's post on exceptions. He didn't like exceptions because they could be misused badly, leading to confusing code (directing). Bill Caputo pointed out that exceptions, when used well, make life much easier (enabling).
Some of the static/dynamic typing debate brings up these points. Some arguments in favor of static typing talk about how they prevent people from making certain kinds of mistake (directing) while dynamic typers point out how static typing restricts some useful idioms (enabling).
Agile processes are PeopleOriented (enabling), while plan-driven methods seek to ensure that even a poor team can do an acceptable job (directing).

These aren't hard-wired attitudes. Often people are directing in some cases and enabling in others. But I think there is a deep strain running through here, often a personality issue, that runs underneath much discussion on how we do software. (I'm very much in the enabling category, as if you can't tell.)

You might think that all restrictions on what a developer does imply a directing attitude, but it's not that simple. As an example, consider memory management. You can think of this as a directing feature: programmers can't be trusted to manage memory correctly so take away their ability to allocate memory. But I look at memory management as an enabling technology - it takes away something I don't want to worry about, so I can concentrate better on those things I do care about. Steve tied this thought nicely onto the difference between problems and difficulties.

reposted on 08 Sep 2011