We are plumbing our event-driven architecture here at Cambia, and, as we were socializing our approach internally, a question came up about how the stateful-streaming-based complex event processor we’re rolling out will fit into the overall flow of data.
As I struggled to answer it clearly, I realized that the layers of event-related technology that have accreted over the years, representing historical contexts in the evolution of software architecture, may be clouding our shared understanding.
So I thought I would take a break today from Informational Identity castle building and begin to peel back, sand off, strip off, a few of those layers.
Let’s talk events – a little history, a little theory, a little practice.
Hardware interrupts – signalling the processor to stop what it is executing and execute some other machine code – first appeared on the first mobile computer, circa 1954. It weighed 18 tons and was mounted on a semi-trailer.
‘Signals’ – async notifications that Something You Need to Know Has Happened, like ‘this computer is shutting down, time to save your bacon’ – have been around in Unix forever (forever began at Bell Labs in the 1970’s). Signals are software process interrupts. The signal mechanism, with its small footprint, has a kind of Spartan elegance. And like the anti-depressant buproprion being used for its side-effect of reducing nicotine cravings – ok, maybe not like that – signals started to be used for distributed interprocess communication.
With the broad emergence of Graphical User Interfaces (the original Mac was famously released in 1984, but it was probably Windows 3 in 1990 that counts as ‘broad’), user event-driven interface programming became popular, where the app sits in a loop waiting for a mouse or keyboard event and dispatches it to an event handler. (As memorialized in Morph’s Outpost on the Digital Frontier, an early 90’s multimedia technical rag…
…I hacked a DOS-based multimedia tool back in the day to create a mouse event queue so that mouse events would not be lost – the default behavior of DOS was to overwrite them – by writing new opcodes directly into the executable to change its mouse interrupt behavior. Had to recalculate the checksum in the executable header so it would still load. I’ve been <to the tune of ‘Desperado’>out riding events-es for so long now</>. Ok, reeling it in….)
The asynchronous programming model of the ‘event loop’ emerged as one of the primary internal application architectures. And it remains so. Node.js, bootstrapped by Ryan Dahl in 2009/10, revolutionized scalable server-side JavaScript programming by introducing a (mostly) single-threaded event loop that dispatches work efficiently. (Its widespread adoption was also driven by its sweet package manager).
Meanwhile, with the advent of object-oriented programming, around the same time as Windows 3, we found ‘event’ now classified as ‘a noteworthy change in state [of an object]’ (that from a 1994 James J. Odell article in the Journal of OOP).
There is an even deeper tie between events and object-oriented programming. Still true today, but even more true in the 1960’s given the slow, tiny processors of the day, one way of coming up with practical solutions to some computationally intractable problems, especially where the heuristics may not be clear, is to model them – to create and execute simulations. You build a software model of your Retail Empire store, with customers queuing up for check out, and play with it to figure out how to minimize both the customers’ waiting times and the number of checkers and their work shifts given your projected customer volumes at different times of day.
There are two broad classes of such simulations: continuous simulation, and discrete event simulation. Continuous simulation has been used since the earliest computers to simulate rocket trajectories and whatnot – lots of differential equations at play. Discrete event sims model a system – such as the store, customers, and checkers – as a series of events – the model jumps from one state to the next driven by events. (There are always some bits in there to randomize behaviors within limits, e.g. which line a customer chooses all else being equal.) The sim doesn’t have to evolve in real-time – it moves from state to state as quickly as the computer can make it so. So you can execute a lot of iterations of the test and do some aggregate analysis of the results.
SIMULA, based on ALGOL, was a computer language created in the early 60’s for Discrete Event Simulation (DES). It was the first object-oriented program, featuring classes, objects, and inheritance. Object-oriented programming was developed to deal cogently with events. SIMULA influenced not only Smalltalk, but also, notably, C++ and Java.
In the object-oriented modeling and design that became part of development culture in the 90s, events are featured in both state and activity diagrams.
There was a wave of interest in Complex Event Processing (CEP) in the 90s – the dynamic inference of ‘aggregate’ events by pattern detection across the ‘cloud’ of events ingested by the CEP system. Similar to the history of AI, while the early promise was not largely realized, in part because of what a big lift it was at the time, the technology continued to make inroads.
Now that the Cloud has made high-speed, high-volume, scalable processing ready-to-hand, there is renewed interest in CEP and large-scale event-driven architectures.
Apache Flink has a CEP pattern-matching library. But all stateful streaming systems such as Kafka Streams, Spark Streaming, etc., are capable of doing CEP.
We – the back office IT we – still largely think of events primarily as triggers, based on their history. But we are starting to see them more and more for their other value, as carriers of changes of state.
Here is a current (2018) definition of ‘events’ from Ben Stopford’s excellent ‘Designing Event Driven Systems’ (Stopford, who works for Confluent, is championing the use of Kafka Streams):
Events are both a fact and a notification. They represent something that happened in the real world but include no expectation of any future action. They travel in only one direction and expect no response (sometimes called “fire and forget”), but one may be “synthesized” from a subsequent event.
‘Designing Event Driven Systems‘
‘Event sourcing’, especially as part of Command Query Responsibility Segregation (CQRS) in the context of Microservice architectures, is attracting a lot of interest. Events are coming to be seen as the new fundamental atomic units of data, as opposed to the state snapshots that live in our databases. Given an indelible log of events, the current state – or any historical state – of an entity may be recreated at will.
In CQRS, rather than a command updating the same resource that a subsequent query reads, the command represents a state change – update Billy Bob’s address to 123 Round The Outside Lane – which is persisted as an event. One or more related databases, such as a row-wise efficient Aurora one and a column-wise efficient Redshift one, are updated near-real-time from the canonical event log. There is a well-understood level of complexity associated with CQRS, so its adoption is being done judiciously.
We’ve seen how our modern understanding of events has evolved from Discrete Event Simulation in the 1960s, Unix signals, and GUI programming.
Next time we will peel back some of that, and try to get a clear line of sight into the underlying conceptual space, to help put us in a better position to make good practical decisions about our own adoption of CEP, event-driven architecture, and CQRS.
Stay tuned.
Health IT Architect 