Dsl bliki

I was asked to put together a discussion of DSLs for non-technical types. Maybe I've been reading too much Stephen O'Grady, but I felt an irresistible urge to do it in a Q and A manner. So here it comes.

What is a Domain Specific Language?

So you wouldn't write a whole project in a DSL?

Are they a new idea?

So why are they getting a lot of noise now?

Probably because of Ruby and Rails. Ruby as a language has many features that make it easy to develop DSLs and the people who got involved in the Ruby community have been familiar with this approach from elsewhere, so they took advantage of these features. In particular Rails uses several DSLs which have played a big role in making it so easy to use. This in turn encouraged more people to take up these ideas.

Another reason is that many Java and C# systems need to have some of their behavior defined in a more dynamic way. This led to complex XML files that are difficult to comprehend, which in turn led to people exploring DSLs again.

So DSLs can be used with languages other than Ruby?

What's the distinction between internal and external DSLs?

Why are people interested in DSLs?

I see DSLs as having two main benefits. The most common benefit is that they make certain kinds of code easier to comprehend, which makes it much easier to modify, thus improving programmer productivity. This is worthwhile all on its own and is relatively easy to achieve.

The most interesting benefit, however, is that a well designed DSL can be understandable by business people, allowing them to directly comprehend the code that implements their business rules.

So is this the hook - business people write the rules themselves?

Do you need special (ie expensive) tools?

In general, no. Internal DSLs just use the regular facilities of the programming language that you are using anyway. External DSLs do require you to use some special tools - but these are open source and are very mature. The biggest problem with these tools is that most developers aren't familiar with them and believe they are harder to use than they really are (a problem exacerbated by poor documentation).

There are exceptions on the horizon, however. These are a class of tools that I call a LanguageWorkbench. These tools allow you to define DSLs more easily, and also provide sophisticated editors for them. Tools like this make it more feasible to make business-writable DSLs.

So is this a repeat of the dream of developing software without programming (or programmers)?

When should I consider making a DSL?

Isn't this going to lead to a cacophony of languages that people will find hard to learn?

But won't people create lots of bad DSLs?

I've hit a significant, if purely internal, milestone in the DSL book recently. I also find that people regularly ask me what the status is of the book is. So it seems like a good moment to post a note about where I am with the book and where I see things going.

The most common question I get is "when will the book be out?" The way things look at the moment, I'd estimate it appearing some time in 2010. That's an estimate, with all the usual hedges that apply to such things.

To help see how the current state meshes in with that estimate, I'll describe the general progress of how books work, at least for me. The first phase is writing the First Review Draft. This means writing all the material in the book, to the level that I can submit it to peer review. For my larger books (eg Refactoring, P of EAA) this takes about 12-18 months. This book is taking longer, I've already been at it for nearly two years, and I suspect there's another year to go. Once I have the First Review Draft, it goes out to review. This takes time for people to read it, get their comments to me, and for me to modify the text based on the review. I usually do two rounds of review - and it takes around 6 months for that to happen. Once I'm done with formal review the book is a Final Draft. At this point it goes off for production - which includes copy-edit, indexing, layout, printing, etc. That takes another 6 months. So I estimate that the book will appear about a year after I get a First Review Draft.

Not all authors work this way, some write a few chapters and sent them for review while they write a few more chapters. I use the approach I do because it was how I did my first book and it seemed to work very well.

To describe the current state of the book, I need to talk a bit about its structure. The final structure of the book will be a DuplexBook, but while I'm working on it I tend to think of it in terms of a set of subject areas, where each subject area will end up as a set of narratives and topics within the duplex structure.

My recent milestone was completing my first coherent pass at the internal DSL subject area. By this I mean I now have what I think is a coherent draft of the that subject area. It will still need more work before I get to the First Review Draft, but reaching a coherent draft is a big point for me because it means I have reached a point where it's looking in good shape. Here's my current list of subject areas and their status:

• Introduction: coherent (this is the first three narratives: An Introductory Example, Using Domain Specific Languages, and Implementing DSLs).
• External DSLs: incoherent
• Internal DSLs: coherent
• Code Generation: coherent, but with some important bits missing
• Alternative Computational Models: incoherent
• Error Diagnostics: open.
• Language Workbenches: open.

I've been working since March on getting the internal DSL section into a coherent state. I suspect it will take me a similar amount of time to get external DSLs into coherence, and a similar time again to get alternative computational models into coherence. This is why I think I'm at least a year off First Review Draft. These are also the next two subject areas that I'm going to work on, in that order.

As well as those two subject areas to make coherent, I also have some holes to fill and a couple of elephants to deal with. The holes are topics within subject areas that aren't really complete yet, although the overall structure is reasonable. Code generation is a good example of that. I'm pretty happy with what I have, and it is coherent. But I need to address at least one topic (the Generation Gap pattern) before I can really say that subject area is done. These holes don't take a huge amount of time to fill and I tend to work on them as the mood takes me - often I like to take a break in the middle of a major topic for something like this.

The two elephants are the open topics of error diagnostics and language workbenches. These are elephants because I don't know how I'm going to tackle them, or indeed if I'm going to put much effort into tackling them. It may be that I'll do little more than a skimpy overview chapter, or I may spend six months each on them. I'm deliberately putting off those decisions while I work on the next two subject areas.

Of course this road-map could all change. One of the reasons I don't like doing incremental review (eg sending off the internal DSL section for a formal review now) is that working on one section can cause me to completely rethink another. This has already happened. I did a substantial chunk of work on the internal DSL section late in 2007 and then worked for a while on what is now the Computational Network and Dependency Network patterns at the end of 2007. Working on those patterns made me realize my internal DSL thinking was all wrong and I thus needed to rewrite that whole section. This kind of thing is common with a topic area like this where there is no pre-existing structure. (This is different to a book on an existing technology, eg UML Distilled, where the thing I'm describing already has an understood structure before I start.)

Having said that, if you're interested in what I'm doing I think I'm at the point where you can take a good look at the introductory and internal DSL sections and it should make sense. I'd certainly be happy for feedback on those sections. Just beware that there are cross links between those and other sections that may still get worked on.

I'll update this page when I hit a major milestone or major re-plan. If you want more granular updates I have a more detailed feed. If you want to send me comments on work so far, please do so. I have a mailing list for the book, I'd prefer all comments to go to that mailing list so that there's an opportunity for discussion. So if you do send me comments, please let me know if you're happy with them going on the list or not and whether you wish to join the list.

What is the connection between ModelDrivenSoftwareDevelopment (MDSD) and DomainSpecificLanguages (DSLs)?

It's pretty common to see the term "DSL" crop up in the context of MDSD. Indeed some MDSD people seem to think that DSLs only exist within the MDSD world. I've been writing a lot on DSLs recently for my book, but so far I haven't really touched on the MDSD angle much Instead I've concentrated on DSLs role in more conventional programming. DSLs exist in both the textual language and MDSD worlds and play pretty much the same role for both.

In an MDSD context DSLs are again a language targeted at a specific kind of problem as opposed to general purpose languages such as the UML. As a result they can have the same kind of relationship: build a system in the general purpose modeling language and use DSLs for various specific aspects. Since MDSD hasn't caught on that much, however, you also see a different approach where modeling DSLs are used in the context of a traditional language environment. Here you might use several modeling DSLs that generate Java code to be combined in a Java project. In this case there's no general purpose MDSD model around - you use MDSD for each DSL relatively independently.

In order to use model-oriented DSLs you need a different, RepositoryBasedCode, approach to tooling. This introduces quite a few pragmatic issues as the general support environment for such tools is less established. In order to define your own DSLs you need more specialized tooling - something I call a Language Workbench.

DSLs seem to have a proportionately higher emphasis in the MDSD world than they do in the mainstream programming world. Cynics think this is a result of the MDSD community desperately searching for a way to remain relevant, fans of MDSD regard it as a sign of MDSD's superior sophistication. I think this is mainly due to the fact that the MDSD community is smaller and has far less in the form of established practice.

A particularly visible sub-community of MDSD is centered around ModelDrivenArchitecture (MDA). I'm not much of a fan of MDA in particular, but am particularly skeptical of MDA DSLs.

There is much that model-oriented DSLs share with textual DSLs. I put a lot of emphasis with textual DSLs in basing work around a Semantic Model. MDSD, as its name indicates, is very much about driving a system from that kind of a model. A difference is that most MDSD people assume that you'll want to generate code from that model rather than executing the model directly.

As I write this, I'm not sure how much I'm going to cover language workbenches in my book. Certainly I'll at least discuss the overall concept behind them, but the coverage may not be that deep. This will be partly due to the large amount of material I seem to be generating on textual DSLs and partly due to the fact that language workbenches are much newer and thus more volatile and less mature.

A common phrase that's bandied about when talking about DomainSpecificLanguages (or indeed any computer language) is that of noisy syntax. People may say that Ruby is less noisy than Java, or that external DSLs are less noisy than internal DSLs. By Syntactic Noise, what people mean is extraneous characters that aren't part of what we really need to say, but are there to satisfy the language definition. Noise characters are bad because they obscure the meaning of our program, forcing us to puzzle out what it's doing.

Like many concepts, syntactic noise is both loose and subjective, which makes it hard to talk about. A while ago Gilhad Braha tried to illustrate his perception of syntactic noise during a talk at JAOO. Here I'm going to have a go at a similar approach and apply it to several formulations of a DSL that I'm using in my current introduction in my DSL book. (I'm using a subset of the example state machine, to keep the text a reasonable size.)

In his talk he illustrated noise by coloring what he considered to be noise characters. A problem with this, of course, is this requires us to define what we mean by noise characters. I'm going to side-step that and make a different distinction. I'll distinguish between what I'll call domain text and punctuation. The DSL scripts I'm looking at define a state machine, and thus talk about states, events, and commands. Anything that describes information about my particular state machine - such as the names of states - I'll define as domain text. Anything else is punctuation and I'll highlight the latter in red.

I'll start with the custom syntax of an external DSL.

events
  doorClosed  D1CL
  drawOpened  D2OP
  lightOn     L1ON
end
  
commands
  unlockDoor D1UL
  lockPanel   PNLK
end
   
state idle
  actions {unlockDoor lockPanel}
  doorClosed => active
end
   
state active
  drawOpened => waitingForLight
  lightOn    => waitingForDraw
end

A custom syntax tends to minimize noise, so as a result you see relatively small amount of punctuation here. This text also makes clear that we need some punctuation. Both events and commands are defined by giving their name and their code - you need the punctuation in order to tell them apart. So punctuation isn't the same as noise, I would say that the wrong kind of punctuation is noise, or too much punctuation is noise. In particular I don't think it's a good idea to try to reduce punctuation to the absolute minimum, too little punctuation also makes a DSL harder to comprehend.

Let's now look at an internal DSL for the same domain information in Ruby.

event :doorClosed, "D1CL"  
event :drawOpened,  "D2OP"  
event :lightOn, "L1ON"  

command  :lockPanel,   "PNLK" 
command  :unlockDoor,  "D1UL" 

state :idle do 
  actions :unlockDoor, :lockPanel
  transitions :doorClosed => :active
end 

state :active do 
  transitions :drawOpened => :waitingForLight, 
              :lightOn => :waitingForDraw
end 

Now we see a lot more punctuation. Certainly I could have made some choices in my DSL to reduce punctuation, but I think most people would still agree that a ruby DSL has more punctuation than a custom one. The noise here, at least for me, is the little things: the ":" to mark a symbol, the "," to separate arguments, the '"' to quote strings.

One of the main themes in my DSL thinking is that a DSL is a way to populate a framework. In this case the framework is one that describes state machines. As well as populating a framework with a DSL you can also do it with a regular push-button API. Let's color the punctuation on that.

Event doorClosed = new Event("doorClosed", "D1CL"); 
Event drawOpened = new Event("drawOpened", "D2OP"); 
Event lightOn = new Event("lightOn", "L1ON"); 
 
Command lockPanelCmd = new Command("lockPanel", "PNLK"); 
Command unlockDoorCmd = new Command("unlockDoor", "D1UL"); 

State idle = new State("idle"); 
State activeState = new State("active"); 
 
StateMachine machine = new StateMachine(idle); 

idle.addTransition(doorClosed, activeState);
idle.addCommand(unlockDoorCmd);
idle.addCommand(lockPanelCmd);

activeState.addTransition(drawOpened, waitingForLightState);
activeState.addTransition(lightOn, waitingForDrawState);

Here's a lot more punctuation. All sorts of quotes and brackets as well as method keywords and local variable declarations. The latter present an interesting classification question. I've counted the declaring of a local variable as punctuation (as it duplicates the name) but it's later use as domain text.

Java can also be written in a fluent way, so here's the fluent version from the book.

  Events doorClosed, drawOpened, lightOn; 
  Commands lockPanel, unlockDoor; 
  States idle, active; 

  protected void defineStateMachine() { 
    doorClosed. code("D1CL"); 
    drawOpened. code("D2OP"); 
    lightOn.    code("L1ON"); 

    lockPanel.  code("PNLK"); 
    unlockDoor. code("D1UL"); 
 
    idle 
        .actions(unlockDoor, lockPanel) 
        .transition(doorClosed).to(active) 
        ; 
 
    active 
        .transition(drawOpened).to(waitingForLight) 
        .transition(lightOn).   to(waitingForDraw) 
        ; 
 } 
 

Whenever two or three are gathered together to talk about syntactic noise, XML is bound to come up.

<stateMachine start = "idle"> 
    <event name="doorClosed" code="D1CL"/>  
    <event name="drawOpened" code="D2OP"/> 
    <event name="lightOn" code="L1ON"/> 

    <command name="lockPanel" code="PNLK"/> 
    <command name="unlockDoor" code="D1UL"/> 

  <state name="idle"> 
    <transition event="doorClosed" target="active"/> 
    <action command="unlockDoor"/> 
    <action command="lockPanel"/> 
  </state> 

  <state name="active"> 
    <transition event="drawOpened" target="waitingForLight"/> 
    <transition event="lightOn" target="waitingForDraw"/> 
  </state>
</stateMachine> 

I don't think we can read too much into this particular example, but it does provide some food for thought. Although I don't think we can make a rigorous separation between useful punctuation and noise, the distinction between domain text and punctuation can help us focus on the punctuation and consider what punctuation serves us best. And I might add that having more characters of punctuation than you do of domain text in a DSL is a smell.

(Mikael Jansson has put out a lisp version of this example. Mihailo Lalevic did one in JavaScript.)

I talk quite a bit with people about DomainSpecificLanguages these days and a common reaction I get to external DSLs is that it's hard to write a parser. Indeed one of the justifications for using XML as the carrier syntax for an external DSL is that "you get the parser for free". This doesn't jive with my experience - I think parsers are much easier to write than most people think, and they aren't really any harder than parsing XML.

I even have evidence. Well it's actually only one case, but I'll quote it anyway as it supports my argument. When I wrote the introductory example for my book I developed multiple external DSLs to populate a simple state machine. One of these was using XML (using it as a gateway drug) another was a custom syntax which I parsed with the help of Antlr. Writing the code to fully parse these formats took about the same amount of time.

Although you get an XML parser for free (I used Elliotte Rusty Harold's excellent XOM framework) the output of an XML parser is effectively a parse tree in the form of an XML DOM. In order to do anything useful with that you have to process it further. My practice with DSLs to is base them around a clear Semantic Model, so the true output of parsing in this case is a populated state machine model. In order to do this I have to write code that walks its way through the XML DOM. This isn't especially difficult, particularly since I can use XPath expressions to pick out the bits of the DOM I'm interested in. Indeed I'm not walking the DOM tree at all - for each thing I'm interested in I have a method that issues an XPath query, iterates through the resulting nodes and populates the state machine model.

So the XML processing is easy, but it isn't non existent - around a hundred lines of code. It took me a couple of hours. I hadn't used XOM in a while, so there was some familiarization required, but it's a very easy library to learn and use.

The Antlr processing was remarkably similar. Antlr has a notation that allows you to put some simple rules in the grammar file to populate an AST. The code to process the AST and populate the semantic model was very similar to the XML code - query for the right nodes in the tree and then process them. Including the grammar file the resulting code is around 250 lines, but took me about the same amount of time to write. I was familiar with most of Antlr before this, having used it a few times, but I hadn't actually used the tree construction stuff before. (If you're interested you can find a description of this example in my book's work in progress.)

Although my explorations of parser generators have got me used to the fact that they are much easier to write than many people think, I was surprised when I realized it was actually no slower than the XML case. In a more carefully controlled example, I would still expect it to take longer because I did the Antlr example second and as any programmer knows, things always go much faster with a second implementation. Even so, the difference is much less than what many people seem to expect - when the word "parser" seems to mean "too complicated".

I can't deny there is certainly a learning curve to get used to parser generators. You have to get used to grammar files and how they interact with code samples. There's different strategies you can use (what I currently refer to as Tree Construction, Embedded Translation and Embedded Interpretation). You also have to think about the syntax of your custom syntax, which involves more decisions than wondering whether to make something an attribute or an element in XML. But that curve isn't really that high. Modern tools make it much easier. Antlr is my current default choice, it comes with a very nice IDE which helps in exploring grammar expressions and seeing how they get parsed into an AST. But once you've got used to how one parser generator works, it's not hard to pick up others.

So why is there an unreasonable fear of writing parsers for DSLs? I think it boils down to two main reasons.

• You didn't do the compiler class at university and therefore think parsers are scary.
• You did do the compiler class at university and are therefore convinced that parsers are scary.

The first is easy to understand, people are naturally nervous of things they don't know about. The second reason is the one that's interesting. What this boils down to is how people come across parsing in universities. Parsing is usually only taught in a compiler class, where the context is to parse a full general purpose language. Parsing a general purpose language is much harder than parsing a Domain Specific Language, if nothing else because the grammar will be much bigger and often contain nasty wrinkles which you can avoid with a DSL.

This problem is compounded by encouraging code that tangles up parsing with output processing and code generation. For me the key to keeping things straight is to use a Semantic Model, so that your parser does no more than populate that model. Most of the time I can then do what I need by just executing that semantic model like any other OO framework. Most of the time I don't need to do code generation, and when I do I base it off the semantic model so it's independent of the parser. I think that if you've got code generation statements inside your grammars, things are way too coupled together.

For people to work effectively with external DSLs they have to be taught about it quite differently to how you'd teach parsing a general purpose language. The small size of both the language and the scripts in the language changes many of the concerns that people typically have with parsing. Avoiding code generation unless you really need it can remove a big hunk of the complexity. Using a clear semantic model can separate out the steps into much more tractable chunks.

The problem, of course, is that there isn't much written that follows these guidelines. (Which is one of the triggers for me to be spending so much time on it.) You're hard put to find any documentation out there on parser generator tools. When you do get some really nice documentation (like Terence Parr's Antlr book) it's still usually written with a general purpose language mindset. Don't get me wrong, I find the Antlr book very helpful (it's a big reason why Antlr is my default choice of parser generator) but I believe that there's an assumption there of parsing general purpose languages rather than domain specific languages that makes it harder to approach than it could be.

The nice thing, however, with all this is that you can still mount that learning curve. If you haven't tried working with a parser generator I'd certainly suggest giving it a try. Try writing a simple DSL of your own. Don't worry about code generation when you start, just create a domain model as you normally would and get the DSL to populate it. Start with something really silly (like I did with HelloAntlr) and gradually work it up from there. Poke around some open source projects that use a DSL and see what they do.

What we're trying to do is introduce the tools that are often used in compilers but are much more general than that to an audience that associates the tools only with compilers, because that's how they've always been taught.

--Rebecca Parsons

Update: the procedure and complaints here are no longer valid. Open ArchitectureWare has released a new version with Eclipse 3.3 that looks like it will install much more easily than what I just went through. There's also know a packaged distribution that includes eclipse and all the OAW stuff.

There are few things more frustrating than spending hours trying to install a piece of software and then having to delete everything and start again. Today at 9.30 I began installing openArchitectureWare, I finally had it installed (I think) at 15.30. So I thought I'd write this to help someone else do it more quickly.

OpenArchitectureWare is a set of tools, based on Eclipse, to support Model Driven Development. I'm interested in exploring some of its tools that are oriented towards DomainSpecificLanguages. (Xtext - which helps you develop textual languages - is something that's specifically been pointed out to me as worth looking at.) I don't know how worthwhile these tools are yet, after all it took me most of the day just to install the dratted thing, but we'll see.

One of my problems with the installation was that I'm not an Eclipse user - my usual Java IDE is IntelliJ. To install openArchitectureWare you need to know how to deal with the plugin system in Eclipse - and I'd never done anything with Eclipse before so that was new to me.

The first step was the easiest one - install Eclipse. I installed it on my Ubuntu machine, so all I had to do was wajig install eclipse (wajig is a unified command-line for various debian packaging and sysadmin tools). Then all hell broke loose. Rather than go through my miserable morning, I'll explain what I would do now.

The trouble with OpenArchitectureWare is that it has dependencies, other eclipse plugins that need to be installed before it can work. As anyone with experience in these things knows, sorting out dependencies can be a right pain without a good tool. apt-get for Debian and gem for ruby are examples of a good tool that resolves dependencies. When I installed eclipse, apt-get knew it had to pull down a whole host of dependencies and installed them for me. The situation in Eclipse is not so good.

To install openArchitectureWare you need a bunch of plugins: EMF, UML2, ATL, and GMF. I couldn't see from the web pages exactly how to get these things, or if they had their own dependencies.

There are several ways of installing plugins in Eclipse, although I had to hunt a bit for instructions. The easiest way is a menu option in Eclipse itself. In the menus pick [Help -> Software Updates -> Find and Install] (no I don't know why it's on the help menu). With a bit of button pushing you can get it to download a list of packages - the relevant source is the Callisto Discovery Site. Once you have that list downloaded look in the Models and Model Development section and select Eclipse Modeling Framework (EMF) and Graphical Modeling Framework (GMF). You'll get an error message saying that these have an unresolved dependency. Take note of the button on the right that says 'select required'. Hit it and it will find the dependency to GEF and its dependency on Batik. If you don't see that button and hit it you'll have a frustrating time trying to find them (believe me, I know).

That gets two of openArchitectureWare's dependencies. The others, and openArchitectureWare itself need to be done the harder way. Digging around the eclipse site I found the relevant web pages for UML2 and ATL. These need to be downloaded as zip files as does openArchitectureWare itself.

When you unzip the UML2 and openArchitectureWare folders they unzip into a folder called eclipse that contains subfolders for plugins and features. You can take the contents of these folders and put them into corresponding folders on your load environment (in my case /usr/local/lib/eclipse). As that didn't work for me when I tried it first, I found another way.

The way to tell if stuff has installed properly is to go to [Help -> Software Updates -> Manage Configuration]. When you open that you have the option of "Add an Extension Location". An extension location is (almost) any directory that contains an eclipse folder with subfolder for plugins and features. I say almost because the eclipse folder also needs a file called .eclipseextension. This is just an empty file so you can create it with touch .eclipseextension. What I did is created folders in /usr/local/lib for openArchitectureWare and uml2-eclipse, moved the unzipped eclipse folders in there, did touch .eclipseextension inside each of them and then added them using "Add an Extension Location". ATL just produces a plugin directory so I copied the contents of it into the plugin directory for openArchitectureWare.

It's important that you do this after you use the Find and Install tool because if you do it first, the Find and Install tool will tell you have an unresolved dependency and refuse to do anything until you fix it. When I was all installed it tells me "UML2 End-User Features (2.1.1.v200707181556) requires plug-in "org.eclipse.emf.ecore.xmi (2.3.0)". I don't know how to fix this and I have a bunch of emf.ecore jars present in EMF. However the rest of eclipse seems to work so far, so I'm carrying on regardless.