Introduction to Theory of Control
How control defines software development and limits AI
Theory of Control talks about control flow and is complimentary to the Theory of Constraints that talks about work flow.
Control has defined my engineering manager life - and I didn’t know it! My career was started early as a developer, new in the industry when the internet was young and the World Wide Web had just been invented. Because my boss made me hiring more developers, I became an engineering manager. Working in startups and enterprises, I encountered many challenges along the way. These included processes, culture, deadlines, quality. I later introduced Scrum in several companies to solve many of those challenges.
It was like with Platos Allegory of the cave. Someone sitting in a cave watching the shadows on a wall from the people that walk by, trying to understand the what is going on. I watched the shadows at the wall and tried to figure out what was happening. Those shadows we already know: Processes, culture, quality, deadline, incidents and meetings, meetings and meetings. I didn’t know for a long time, but most of those things were a reflection of control. Solving the challenges I structured and institutionalized control. One particular story comes to mind. In a fast growing startup that felt like a rocket ride, there was always disagreement on what to build and when everyone would get what they needed to keep the rocket going. My solution: Introducing a product council to discuss and align with the CEO, CPO, CMO, CRO and CFO what to build in what order. I didn’t know at that time, but what I did was institutionalize control that was running wild. Later, adding a bug process to decide what to fix and what to close, institutionalizing control.
Over the years, wherever I’ve looked, it was about structuring control. Necessary and accidental control. Control is central to software development. Software development is all about control! Software processes are the defining sign of control.
What is control? The first point for my understanding came from SOX (Sarbanes‑Oxley Act). When eBay bought the startup I was CTO at, I came into contact with the concept of control working on SOX compliance. In the context of SOX, a control is any policy, procedure, or activity that a company puts in place to reduce the risk of a material misstatement in its financial statements. In other words, a control is a safeguard that helps ensure financial data is complete, accurate, timely, and authorized. The introduction of SOX brought many discussions on how to control development from a SOX perspective: For example approvals and audit trails. It made developers angry, because tickets and commits where tightly controlled. It dawned on me then, that control is not only central to SOX but central to software development in general.
Control means defining the what and how of other peoples work.
Control in software engineering can take many different forms. Requirements, architecture, guidelines, coding standards, review meetings, planning meetings, feedback, quality assurance, checklists - all are some kind of control.
Looking through this lens, most of software development is about control. Software development processes make this explicit, in Scrum there is a product owner who owns the product and controls what features are build and how those features are build.
We want control so everyone gets what they want. We have review and planning meetings so the product manager gets what they want, requirements so business gets what they want, coding standards so the team gets what they want, architecture standards so the CTO gets what they want. Control is needed because we do not think that others do things the way we want them to be done. Because of this, we structure software development around control.
Mismatch of control leads to conflicts: Prioritization conflicts, acceptance conflicts.
My second point of understanding of control came from the fabulous book “The Goal” by Eliyahu M. Goldratt. It explains the story of production through the lens of bottlenecks, and forms a Theory of Constraints. The way bottlenecks in factories define production, I realized that controls defines software development.
My third point of understanding control came from Simon Wardley and Value Maps. Those Wardley maps are mapping (computer) systems and services into value chains and evolution. They help people understand the relationships and dependencies of systems and how they are evolving or should evolve. From the map you can find a strategy (“what to do”).
Can we map control to understand what is going on and what to do?
How would we map control? I’ve said processes are a sign of control in software development, let’s draw a process, where time flows from left to right.
Everyone knows this simple agile process. We start with Sprint planning, which is about what we are going to do and to some degree how. After Sprint planning, we have a Daily Standup, to align and discuss what, when and how to do things again, in a more detailed way. Off everyone goes and developers develop software (as they should!). In reality the Daily/Developing cycle loops several time, if we look at only one feature, it might be finished with one cycle. In this very simple model the sprint ends now and we have a Sprint Review. Everyone has done this, everyone has been there, everyone bought the t-shirt (although today the t-shirt feels a little filthy). Where is control in this, didn’t I say software development is defined by control?
Lets look at the 4 elements of that little process
Sprint Planning
Daily Standup
Developing
Sprint Review
We can ask ourselves for each element, is the element mainly about delivering work (what Lean would call value) or is it not delivering work (what Lean would call waste). Three of these elements are not delivering work: Sprint Planning, Daily Standup and Sprint Review. They talk about behavior of the application that should be coded, the UI of the app, how APIs should change, who works on what, in what order things are worked on, and who works with whom. Controlling the work of people.
Contrary to that, Developing is delivering work results. Through the lens of control, the three elements not delivering work are there to control the one element that does deliver work. How work is done and how the results look like. We can mark that in our map.
There are two different elements in our map: Control and work. Instead of marking them red to make control explicit, we can draw those elements on different lanes.
When we add an Y-axis as a direction, label it Control, and while all elements have some kind of control in them, those higher up have more control than those lower down.
To make it clearer, what is mostly work, and what is mostly control, we can add colors to our map. This way we can see on first glance what is mostly work and what is mostly control.
But doesn’t Sprint Planning influence the Sprint Review? And the Daily? Yes, control not only controls the work elements but also other control elements. What is discussed in Sprint Planning controls what is discussed and done in the Daily Standup. Sometimes we want to make that explicit in a map, if there is strong control from one control element to another, or one further down the flow, we can draw an arrow from the controlling to the controlled element. Here the sprint review controls sprint planning.
Is all control the same? Are there higher levels of control? More control, with a bigger scope and a smaller scope? Yes, each control is different from other control. There is a hierarchy of control based on how tight it is and how big the scope is.
As an example from our simple process: The way each elements exerts control over Developing is different for those elements. We place control elements with broader scope and tighter control higher up on the Y-axis of control, and elements with less and less-tight control lower on the Y-axis.
Sprint Planning, what to work on, has more control over what people are spending their time on, then the Daily (Standup). Business Initiatives have a broader scope but are less connected and less tight concerning Development. The level of control elements on the Y-axis is not a measurement but about their relative relationships.
What have we achieved? From the realization that software development is about control, we created a control map for a deeper and better understanding of control and work, what control exists, how control influences work and how control is setup within our organization. You can use this understanding and control mapping in your own organization - which will create a much bigger map that also includes DevOps, Product Management discovery, OKR planning and more.
At this point you have two questions: How does this help me (you!) and what has this to do with AI. We’ll explore the second question in the next installment, but the more you control AI the less you can get from it. To get the most of AI, we need to understand control, and the Theory of Control is your tool.
For the first question: Mapping out control with control maps helps you understand what role control plays in your development process. You can answer questions like: Do we have enough control? Do we control the right things? Do we have too much control? Is something “out of control”. For these questions and how to adjust control we’ll also take a deeper look in one of the next installments. Stay tuned!