The Architecture of Gap Inc's SCMS

The following is dump of the text in the deck to help search engines perform indexing

---

The Architecture of Gap Inc.'s SCMS PO System

SCMS PO is an application that helps Gap Inc. manage purchase orders. The architecture of the application is well liked by its development team and thus makes a good expositional architecture for a system with a rich javascript front end working with a back-end serving json. Interesting design features include using knockout.js form of the Presentation Model pattern, a javascript validator running on both client and server, encapsulating data access with repositories, using MongoDB as an application database, and the testing portfolio.

Martin Fowler

2013-04-23 11:02 EDT

Hints for using this deck

My thanks to Ryan Murray, Steve Hill, Andrew Kiellor, Tobias Clemson, and Daniel Siwiec for providing information for this infodeck. Thanks to Gap Inc. for supporting us in writing about this project.

---

SCMS PO is an internal enterprise application

• Its purpose is to help Gap Inc. manage purchase orders from suppliers. (SCMS stands for “supply chain management system”.)
• Purchasing staff can put purchase orders (PO) on-screen and make changes depending on the state of the PO.
• Business logic includes validation of changes and figuring out consequences such as delivery routing and allocation of demands across suppliers.
• SCMS PO supports a separate service that takes demand orders from upstream forecasting systems and turns them into purchase orders. (An example might be a demand for 100 red T-shirts which SCMS PO might split 60:40 into POs against different suppliers according to its allocation rules.) This demand order service is only an API.

• There are a few dozen users, scattered around the globe.
• Development began in September 2011 and the application has been in production since December 2011
• Much of the talk about web applications centers on dealing with public-facing systems that must handle large volumes of users, but most corporate systems have much lower and constrained loads.
• On a typical day, 20-30 POs will be updated.
• On peak days about 200-300 POs will get updates
• The architecture is designed to handle loads one order of magnitude greater than these.
• The production code is around 50 KLOC evenly split between Java and Javascript. There is about 75 KLOC of tests, which are a mixture of cucumber, java, javascript, and ruby.

I have altered some names used by modules within SCMS PO to be consistent with the usual vocabulary I use on this website.

---

SCMS PO Architecture

The client is a single-page javascript application that communicates with the java server using JSON documents.

The javascript uses the Presentation Model pattern (aka MVVM) to provide a clear base for the HTML presentation. SCMS PO uses knockout.js to do data binding between the presentation model and the HTML DOM.

A javascript Repository handles all communication with the server. Its clients can ask for an order with an ID and get back the appropriate object.

Data validation uses the same javascript code on both client and server. The validation code is organized as a simple Production Rule System.

The server exposes a Resource API, providing json to clients, defined by Spring's http bindings. On receiving an http message it initiates service objects to coordinate business logic and data logistics.

There are a variety of data sources for the application, including a MongoDB data store for application data, external systems, integration databases, and a couple of CSV files. All access to these sources are through gateway objects with interfaces in domain object terms.

---

Knockout has been an effective way to use the Presentation Model pattern to organize the UI logic.Presentation Model

(aka MVVM)

Represent the state and behavior of the presentation independently of the GUI controls used in the interface.

more ...

• Knockout uses the vocabulary of MVVM (Model-View-ViewModel) to describe its use of the presentation model pattern.

The presentation model (aka view model) arranges information from the underlying model to provide a simple interface for the view. As such it hides the structure of the underlying model from the view.

All information, including display information for the view, is held in the presentation model in such a way as to make a simple mapping for the view elements

The UI controls (view) use a simple synchronization mechanism between their state and the presentation model's state. Knockout provides a data-binding mechanism between the the HTML DOM and the presentation model to do this.

The javascript code can be tested by tests that use the presentation model. These tests can be run outside of the browser and are thus easier and faster to run. (SCMS PO's test bed runs about 900 jasmine specs in around 70 seconds.) Such tests won't catch errors in the data binding, but these have not been a problem in practice.

Domain logic is kept in separate javascript modules. This ensures that presentational issues, and code using jquery and the DOM, is well separated from domain logic which makes both easier to work with.

---

A js repository manages communication with the server

• All javascript communication with the server is done through a repository.
• The repository receives calls to javascript domain objects and fires off the appropriate ajax calls to get the data from the server.
• The repository returns a promise that contains a purchase order object when it's resolved.

---

Validation is done by a general purpose javascript validation engine

• The validator takes a ruleset with a candidate object and any other required context and executes the rules against the candidate. The validator logs all errors in a Notification which it returns to the caller.
• This mechanism can be used for validating any kind of object, and is used in SCMS PO for both purchase orders and demand orders.
• Since the validator (and the rules) are written in javascript, identical rules can be run in the browser and on the server, avoiding duplication of rule definitions.

---

Rules are javascript functions that can be organized into rulesets

• The rules for the validator act as a simple Production Rule System. The validator runs all the rules in no particular order, any rule failure results in adding an error to the Notification
• Rules and Rulesets are composites, so rulesets can contain other rulesets.
• Rule composition is only conjunctive (ands). The team built more complex ways to combine expressions but didn't find them useful.

• By making rules and rulesets be objects, we can organize them freely into groups
• Grouping related rules together helps understand the rules and the relationships
• Different groups of rules can be run in different stages of the application workflow - providing Contextual Validation.

---

The SCMS PO team has differing views about implementing the rules in javascriptFor

• You can run complex validation code on the client that is the same code as used on the server

Running these validations on the client supports a very fast and responsive behavior

Against

• Is running validation on the client worthwhile?

It's not clear that faster responsiveness is worth enough to the users, and it might have been faster to call to the server and run rules in java.

• The interaction between Rhino and Java is awkward
• You cannot use any java domain logic for the rules
• It is awkward to incorporate interaction with remote systems for rule checking
• Load paths and asset handling are complicated to ensure everything works on both sides.

• Whatever the pros and cons of writing in javascript, the Production Rule System design proved valuable.

---

The web service uses a Resource API

• Data from the server is accessed as a Resource API: a set of resources manipulated through http verbs.
• This corresponds to RMM level 2, ie no use of hypermedia controls
• This kind of interface is handy to explore and debug with simple tools - even non-developers can easily get and understand json payloads.
• The URI structure emphasizes common sub-URI patterns, eg /search, /events.
• Resources correspond to natural aggregates in the domain, such as purchase order and demand plan.
• Using a json service with html rendering on the client helps enforce Separated Presentation.

• Using web services allows clients to use a uniform interface that encapsulates the actual data source: whether it be a database, application api, or a simple file.

---

Resource-oriented service acts as an excellent test point

• The service interface provides a good platform for testing the server-side behavior.
• These tests (called Service Tests) are broad-stack, business-facing tests written in Cucumber.
• A ruby service client provides a convenient interface for making the service calls
• Cucumber step definitions map from the steps to calls on the client library
• Although the cukes (cucumber test specifications) aren't read much by non-programmers, the team felt that the shift in syntax and style helps developers step back from the code and think more about the business problem.
• Cukes provide good documentation for service behavior, especially handy for communicating with QA..
• The tools provided by IntelliJ Idea were particularly helpful for manipulating the cukes and supporting ruby code.

• SCMS test bed runs around 900 cukes in about 7 minutes. This uses parallelization, runs through the full server-side stack, and drops the database between each scenario.

---

Access to Data Sources is done through GatewaysGateway

An object that encapsulates access to an external system or resource       (more…)

• The interface of the gateway is designed to fit the needs of the host application.
• Its vocabulary is the context of purchase orders
• Its operations reflect the intent of the interaction from SCMS PO's point of view.
• The gateway's value is that it encapsulates the complexity of talking to an external system, allowing code that uses the gateway to written in terms of its own intentions.
• The gateway makes a natural substitution point for testing, avoiding the slower speed and non-determinisms that external systems bring to tests.

(AES stands for An External System)

---

MongoDB has worked well for application persistence

While NoSQL databases are often touted for their performance characteristics, in this case the driving force was ease of development and deployment

• Mongo's aggregate-oriented data model is a good match for the application's data needs. Purchase Orders and Demand Plans are natural aggregates. Coupled with morphia it is trivially easy to manipulate aggregates in the store.
• All access to data is done via a repository, this both eases access and provides a substitution point should the project need to change its persistance technology in the future. This flexibility was an important factor in accepting the use of MongoDB in an enterprise environment.
• MongoDB's document model provides good affordances for indexing and querying, compared to pure key-value stores.

• In production the database set up with replicas which act as hot backups. Writes are set to be accepted when at least one replica has written to disk.
• When running tests, durability doesn't matter, so the fastest settings are used - this greatly helps test performance.

• As with many organizations, people are used to relational databases used as integration databases. This introduces considerable ceremony for updating data structure definitions. Using a NoSQL database allows the application team to manage the database directly.
• Operational management is done by the development team and has been straightforward.

---

SCMS PO has modest concurrency needs, thus uses simple mechanisms

• As is common in enterprise applications, SCMS PO uses Optimistic Offline Locks with version stamps on the purchase order. This is effective as it's very rare for conflicts to occur.
• Updates to the datastore on the server are naturally atomic if only one aggregate is modified.
• There are some operations that require multiple aggregates to be updated atomically. SCMS PO handles this by using a single global write lock - which is an appropriate solution at the moment given the small update volumes for this application.

• In time, the application will need a more sophisticated locking approach, but a simple global lock has worked well for the first year. With locking it's often a good strategy to go for the coarsest granularity that will work initially, and work in a more fine-grained approach later

---

The test portfolio has many styles of test

• Unit Tests using JUnit for java and Jasmine for javascript tests. Classical style using Test Doubles only for external services.
• Integration Tests check interaction with external services. These are smoke tests: they are not comprehensive they are there to check connectivity and basic operation.
• Service Tests are business-facing tests written in Cucumber. These are broad-stack tests, most of which test the server through the web service API but about ⅙ test the browser UI using Capybara and Webdriver.

• Integrity Tests check that locking is working correctly.
• User Journey Tests tested usage of the system in a user journey style, written in cucumber.
• Contract Tests check that external services operate the way SCMS PO thinks they should operate. Ideally these should be shared with the teams developing those external services.
• Monitoring Tests perform semantic monitoring against the production system to detect errors in real time. The team is only doing this in a very basic form at the moment but intends to increase usage to so they can detect problems more rapidly.

In terms of lines of test code (an admittedly crude measure), 50% of tests are unit tests, 30% are service tests, and 10% are user journey tests