The CONQUER Architecture for Distributed Systems, Part 1 of Probably Too Many but Hopefully Just Enough

Some time between 2010 and 2013 I began thinking differently about integration, both among components of distributed applications and among distributed applications per se. I know the rough time span because I published an integration reference architecture here in 2010 that did not feature the new ideas, and one in 2013 that did – well, early drafts of it did, but they were relegated to an appendix, then edited out completely, as too radical at the time. I pulled it all back to the front of the stove in 2017 when I first internally published our Command Notify Query (CNQ, ‘sync’) reference architecture.

It’s no longer quite as radical – part of the reconceptualization is becoming mainstream – for example Ben Stopford of Confluent calls out commands, events and queries as the three mechanisms through which services interact in his May 2017 post ‘Build Services on a Backbone of Events‘, which rolled into his book ‘Designing Event-Driven Systems‘ (which I was fanboying on recently in my post on the renaissance of complex event processing).

So it feels like high time to trot it out to a larger audience, rebranded as ‘CONQUER’. I fear, though, that this intended series of posts is at risk of becoming my Theory of Everything – an ego trap many of us old software types seem to fall prey to. Bertrand Russell, talking about Locke’s (I think) inconsistency, said – paraphrasing – the trouble with a completely coherent, consistent exposition is that it was either right or wrong, figure the odds, but in inconsistency there might be nuggets of truth. Getting to the rationale for CONQUER is going to take a lot of interrelated – and consistent – scaffolding. So damn the torpedoes.

(In Civil War days, when the quote originated, attributed to Admiral Farragut at the Battle of Mobile bay, ‘torpedoes’ were actually naval mines – making their eternal damnation even more appropriate.)

The original insight came from trying to create an abstract model of integration to serve as a structuring device for comparing integration architecture strategies.

In order to meaningfully discuss integration architecture at a strategic level we need a simple working model of integration that is independent of historical distinctions.

The primary underlying coarse-grained historical distinction in integration seems to be control flow versus data flow. EAI is primarily about control flow. ETL is internal data flow. EDI is external data flow.

‘Control’ is about how the so-called ‘flow of control’ is passed from endpoint to endpoint. Flow of control is the ‘bouncing ball’ in software – it represents the sequential execution of business logic over time. It is the moving front of computation and logic in executing software that carries an initiating action through a particular route to its end, sometimes splitting along the way.

Flow of control is launched in motion and then it flows within a system, and sometimes from that system to another, and so on, doing stuff until it ends, usually by presenting information to some other user.

Data flow is moving data from one endpoint to another, frequently in bulk.

The distinction between control flow and data flow is fed to programmers with our mothers’ milk. It is so embedded into both inter- and intra-application interactions that it is designed into our tools.

Even newer integration mechanisms such as messaging perpetuate the distinction. Hohpe and Wolff’s integration patterns distinguish command (control) messages from event messages from document (data) messages.

But if you press gently on the difference black and white quickly grey.

Control communications can and usually do carry data. Data movement frequently implies control flow. The explanatory framework breaks down.

So what is a more useful way of categorizing types of integration? Trying to answering that question led me to CONQUER .

Where I ended up – and where we will end up after the promised N more posts on CONQUER – is here:

Distributed systems are information tools enabling volitional entities in relationships of permission and obligation to achieve their ends via purposeful dialogs, communicated among software components proxying their wills, consisting of imperative, declarative, and interrogative sentences: commands, notifications, and queries.

‘CONQUER’ is an acronym shoe-horned together from from COmmand Notification and QUERy (and one, unlike ‘CNQ’, that doesn’t need a pronunciation guide : ). Integration simply means making it so. Interoperability is its execution among distributed volitional entities.

The coming CONQUER posts will unpack our definition.