CONQUER Part 3: Flames from Agile Burning in the Back Office Light the Way

This is the third part of a series describing the Command, Notify, Query (CONQUER) distributed systems architecture: distributed systems are information tools enabling volitional entities in relationships of permission and obligation to achieve their ends via purposeful dialogs, communicated among software components proxying their wills, consisting of imperative, declarative, and interrogative sentences: commands, notifications, and queries. First Part. Previous Part. Next Part.

O.K., so maybe the headline is a little click-bait-ey. But whatever we are calling Agile in the back office has lost the spirit of the Manifesto, despite Dean Leffingwell’s valiant attempts to resuscitate it with SAFe, and Craig Larman and Bas Vodde’s with LeSS. And the reason has mostly to do with our not recognizing the necessary role of CONQUER’s volitional entities in our distributed architectures. Analyzing the failure of agile in the back office was one of the key factors that lead me to CONQUER.

I’ve just implied something about CONQUER – let me make it explicit. I believe its underlying model is just how distributed software works. And that the more clearly our architectures are done in light of that model, the more elegant they will be. When we get into our discussion of CONQUER’s sentential communications model we will see how existing distributed software solutions tend to obfuscate the model.

Today let’s continue on the path of understanding the role of volitional entities, by looking at how and why Agile hit the shoals when it moved to the back office.

Agile grew out of best practices for app development in (mostly) Silicon Valley in the 1990’s. Kent Beck, Alstair Cockburn, Martin Fowler, Ward Cunningham and the other signatories of the 2001 Agile Manifesto had seen, and participated in, the backlash against the high ceremony waterfall processes inherited from big iron development, and the emergence of new methodologies such as XP and Scrum. They got together at Snowbird and derived, inferred, a common set of tenets – brief and to the point:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

They also published the twelve principals behind the Manifesto.

Now, almost twenty years later, the redoubtable ‘Uncle Bob’ Martin, who was there in Snowbird, has been compelled by the state of Agile today to write and publish ‘Clean Agile: Back to Basics‘, an entreaty to revisit the principles and practices on which Agile was founded. As Kent Beck says, “Bob’s frustration colors every sentence of Clean Agile, but it’s a justified frustration. What is in the world of Agile development is nothing compared to what could be.”

Claire Drumond of Atlassian has written a wonderful piece about the relevance of Agile in today’s world. In it she characterizes the emergence of ‘The Agile Industrial Complex’, driven by the growth of agile practice vendors and tools, which many believe has gutted the practice, devolving it into the aberrant  “’faux agile’…and its evil twin ‘dark agile’”.

Call it “faux,” “dark,” or “cargo cult,” these agile subversions often lead to situations that fly in the face of the Manifesto’s intentions—micromanagement, burnout-rate pacing, lack of delivery, and adherence to process over principles register as the most egregious…

Claire Drumond, ‘Is the Agile Manifesto Still a Thing‘.

In a thoughtful post from 2016, Antony Marcano, a deeply experienced Agile guide and coach, attributes the slide to the creeping use of user stories as a feature and action capturing device versus a user-intention capturing device.

Drummond and Marcano are both right, but are describing the disease’s symptoms, not its etiology.

When we moved Agile into the back office it rotted from the inside out. Most back office users are system users. But the nature of system users and system user stories has been ill-defined in agile theory and practice. As a result, we consistently write ill-formed, fundamentally broken system user stories. Everything that directly follows from them inherits their stank, and is cursed to be ill-designed and ill-executed. And worse – well, also bad – our bad practices with system user stories have infected our approach to all user stories. As a result, agile has devolved into a pale shadow of its Manifesto, and become a just-not-very-damn-good task management approach.

Let’s look at how we got here, and what we can learn about the nature of software from the failure of Agile.

Our journey for today begins with users and user stories.

Whether we are doing Kanban or Scrum, whether we feel SAFe or LeSS, we are all writing user stories. We don’t write them on cards much anymore – we are all capturing them in Gherkin features, in JIRA, in Rally.

So we obviously know what a user is, and a user story.

Or do we? Try to define ‘user’ and in turn ‘user story’ explicitly, and you may end up, as I did back in the day, at “well, I know ’em when I see ’em”.

If you turn to the Web for help, you will find there is very little original work on agile methodology. The overwhelming majority of what is there is simply parroted from elsewhere, usually without attribution. What passes for expertise seems very shallow.

So my research approach was to start with work from key signatories of the Agile Manifesto and move forward in time.

Here is what I was able to glean.

Kent Beck (inventor of the term ‘user story’) has a number of wiki pages on c2.com where he and fellow Manifesto-ers Alistair Cockburn, Ron Jeffries and other posters discuss various aspects of user stories. There is no single clear description of what a user story user is – users are generally referred to simply as the authors of user stories. In fact, there is a lot of discussion but little clarity on what user stories themselves are or are not.

Jeffries is the pithiest: “[a] UserStory is a story, told by the user, specifying how the system is supposed to work, written on a card, and of a complexity permitting estimation of how long it will take to implement.”

Jeff Sutherland – co-inventor of Scrum, and another original signatory of the Agile Manifesto, has published a presentation deck called ‘User Stories Done Right.’ But in it he largely borrows the actual user story stuff from one Mike Cohn.

And it is Cohn, Mr. Mountain Goat Software (cool logo and great website), who seems to have emerged as the leading acknowledged expert on user stories. He was a founding member of both the Agile Alliance and the Scrum Alliance. His 2004 book ‘User Stories Applied for Agile Software Development,’ published in Addison Wesley’s Kent Beck ‘Signature Series’, has become arguably the standard reference on Agile user stories. Despite it being 15 years old, it is still the third hit you get searching Amazon for “agile user stories.” (‘System user’ does not appear in the index of either of the first two hits.) According to the back cover blurb from ‘Applied’ “you’ll learn what makes a great user story, and what makes a bad one…”. Cohn has since published several other popular books on agile practice, publishes video courses, runs seminars, speaks at conferences.

We have Cohn to thank for popularizing what has become the most widely used template for user stories (which has survived into Gherkin features), usually called the ‘3Rs’ or ‘Connextra’ template (it captures Role, Requirement, and Reason, and was conceived by Rachel Davies at Connextra ~2001) :

“As a ______
I want to _____
So that _____”

Cohn, who is not only an author but also an instructor and a popular speaker on the conference circuit, is definitely one of the parent-most nodes in the agile IP parrot tree.

What does Cohn have to say in ‘Applied’ about user story users? Let’s dive in together and see. We are looking not only for explicit descriptions of user story users, but also for descriptions of user stories from which we might infer things about their users.

“What is a user story? A user story describes functionality that will be useful to either a user or a purchaser of a system or software.”

But then rather than trying to describe the difficult-to-describe, Cohn jumps right away to the ‘you’ll know it when you see it’ extensional device of some quick user story examples from the fictional used-for-illustration BigMoneyJobs website project:

“A user can search for jobs.
A company can post new job opening.
A user can limit who can see her resume.”

And he provides two examples of ‘bad’ user stories:

“The software will be written in C++.
The program will connect to the database through a connection pool.”

The users in two of the ‘good’ examples are people who will use the system, simply called ‘user’.

So is a user always a person who will use the system? No, the user in the other good example is ‘company’. Since the company has no fingers, we might assume that the actual user is a company employee.’

So we might infer a basic definition as a starting point: a user is a human who will interact with the software.

Cohn says the first example is not a good user story because BigMoneyJobs users would not care which programming language was used. I can’t see how to squeeze a reasonable user at all into the passive ‘the software will be written in C++,’ no matter how acrobatic my grammar skills.

Cohn then tries to clarify. “If this were an application programming interface, then the user of that system (herself a programmer) could very well have written that ‘the software will be written in C++’.” That doesn’t quite track for me at first read – I get sideways thinking about interfaces vs. implementations. But maybe he means API (a squishy acronym with many meanings) as in ‘a linkable library’.

In Chapter 3, Cohn suggests that it is a fallacy to write stories from the perspective of a single uber-user, that that approach can lead to undiscovered stories, and suggests user roles as the solution.

Using his example project, BigMoneyJobs, he asks “when we talk about user stories, who is the user we’re talking about?”

Finally!

But rather than providing intensional rigor, he simply identifies a few possible users of the system, both job seekers and job posters, by name and brief description – sketchy (in the older sense) Alan Cooperian personae.

He suggest that “[w]hile each user comes to the software with a different background and different goals it is still possible to aggregate individual users and think of them in terms of user roles” – “collections of defining attributes that characterize a population of users and their intended interactions with the system.”

So we still seem to be working in terms of the fundamental definition – humans (now in particular roles) who will interact with the software/system.

While our system will satisfy its set of high level goals – in his example the primary user goals are Find a Job and Find an Employee – they can be satisfied by a series of sub-goals which may be more or less important to particular segments of the highest level user demographic. And those segments may have other secondary goals in common, after Cooper, “some users favor convenience, others favor a rich experience, and so on.”

Cohn does call out system users in a boxed sidebar:

“As much as you can, stick with user roles that define people, as opposed to other systems. If you think it will help, then identify an occasional non-human user role.”

The klaxon should be blowing not-so-gently in your ear at this point – most of the stories we write these days for the services that make up distributed systems are for non-human users.

‘Occasional’ makes it sound as if there is a choice you are making willy-nilly among alternatives. Surely there is some set of necessary user roles for a given system. There must be a way of determining in a given instance who the appropriate users are.

Cohn does draw a perimeter of a kind:

“[W]e don’t need user roles for every conceivable user of the system, but we need roles for the ones who can make or break the success of our project. Since other systems are rarely purchasers of our system, they can rarely make or break the success of our system. Naturally, there will be exceptions to this, and if you feel that adding a non-human user role helps you think about your system, then add it.”

Write stories for the people writing the checks, and add a system user if it helps you think.

Cohn suggests capturing these user roles on user role cards. He then goes on to suggest that some teams may find value in fleshing out (Alan) Cooperian personae – detailed vignettes describing individual users based on role characteristics to help bring them to life, and Djajadiningratian ‘extreme characters’ (a technique where you use fringe user characters, almost caricatures, such as hookers and Popes, to effectively explore the skinny ends of your user demographic space’s bell-shaped curve). But in neither approach are the users usefully compared to Cohn’s base user story users.

We are provided one last chance to find out what bad users might be…which might let us infer what good users are. Cohn catalogs ‘story smells’ in Chapter 14. Unfortunately none of them captures the peculiar pong of a ‘bad user.’

He does advocate that we capture non-functional requirements as user stories about the system, which have no actual user defined: “[t]he system must support peak usage of up to 50 concurrent users”

Let’s try to synthesize Cohn’s definition of user:

“An agile user story user is a human or company who is paying for a System or a human or company working in a given role who will interact with the System, and who can make or break the System’s success, or is, occasionally, a System user if it will help the developers think or capture a non-functional requirement.”

Not the useful level of conceptual rigor we are looking for.

There might be an objection raised from the touchy-feely user story school that says a user story is just a promise for a conversation to be held later. So that not much rigor is required, and I should untwist my panties.

My first response is that they should go build a doghouse to hang their crystals in and stay away from my skyscraper project.

Biting that back let me just ask this. If you don’t get the user right, are you conversing with the right party? And if you don’t get the user story right, are you having the the right kind of conversation with that right party?

In his chapter on User Role Modeling Cohn acknowledges a debt not to any of the Agile Manifesto-ers but instead to the usage-driven design theory of Larry Constantine and Lucy Lockwood and the goal-directed design theory of Alan Cooper.

Let’s chase Cohn’s sources and see if we can find more clarity on users there.

Larry Constantine’s most recent concise description of users is in his 2005 article “Users, Roles and Personas.”

C&L’s original articulation of what are now called ‘task cases’ came about because use cases were seen to have “too many assumptions and premature decisions about the user interface being designed.” He says that task cases have had widespread adoption, and actually cites the Cohn book we have been discussing as an example. Cohn’s user stories == C&L’s task cases.

Forgive the extended quote, but after reading dozens of agile parrots’ prose I find Constantine downright lyrical:

“A task case represents a single, discrete intention carried out by a user in some role.

Task cases are also called essential use cases because they are stripped down to the barest essentials of the task. Each task case is expressed as a dialog in which user intentions and system responsibilities are abstract, simplified, and stripped of all assumptions about technology or implementation. This form of description is intended to get closer to the essence of what the task is really about from the perspective of the user in a role and to avoid making unintended or premature assumptions about the form of the user interface to be designed.

The need to anchor the task model in an appropriate understanding of users quickly became apparent to those working with essential use cases. User roles emerged as an elaboration of the software engineering construct of Actors (Jacobson et al., 1992), which, like use cases, required adaptation to better suit the needs of user interface designers.”

With a little software archeology we have unearthed the source of Agile Rot. There was a lack of clarity in the original exposition of agile methodologies around that nature of users and user stories. Early practitioners such as Cohn who popularized the user story approach based their work in part on the ‘essential use cases’ or ‘task cases’ in the usage-directed design tenets of Larry Constantine et al.

But the rigor present in Constantine’s full method was lost in their adaptations. While there is clear value in Cohn and other popularizers’ approaches for greenfield user application projects, such as Cohn’s ‘BigMoneyJobs’ website example, there is insufficient guidance and rigor around the definitions of users and user stories to help agile methodology adopters apply them effectively in the back office.

The key is here: “a task case represents a single, discrete intention carried out by a user.”

But a system per se – a ‘system user’ – has no intent of its own. Intent is the state of mind with which an act is done. That is volition. Volition is the power of choosing or determining. We call that power will.

Software is a tool which enables some volitional entity, some human or group of humans, to exercise their will to realize their intent.

These are CONQUER’s ‘volitional entities.’

In a distributed system, their will is communicated via a chain of software components that have been created to proxy their wills.

And that is the fundamental nature of a ‘system user’: a system user is an ‘actor’ or ‘agent’ serving as proxy for the will of the ‘upstream’ volitional entity they are channeling. The system user’s intent is only intelligible in the context of the volitional entity’s intent.

If agent-based architectures had emerged as the dominant idiom we would have inevitably discovered this sooner.

We didn’t. And we don’t write system user stories featuring upstream volition. But we must.

Instead, today we write most system user stories – and most user stories – about actions, not intentions. That breaks Agile’s back by crippling its fundamental mechanism for capturing requirements. And it denies us an elegant conceptual picture of what software is and does.

That was a key realization that led me to CONQUER.

Stay tuned.