Software development slows down over time.
I wrote a whole book to help leaders reverse this slowdown and the central point of the book is a process any engineering leader can apply.
I call this process Technical Coherence and you can mostly achieve it in a single meeting with your leaders. You can implement it in your org gradually or all at once.
The central idea is this:
- We identify the necessary user experience domains for our products
- We identify the shared product domains that underpin multiple user experience domains
- We organize engineering into three layers, the top two correspond to the above domains and the third provides infrastructure.
How to design an engineering org
There are some hard questions that every engineering leader I know struggles with (myself included):
- What proportion of engineers should work on infrastructure versus product?
- Should we pay engineers in infrastructure more? Less?
- Are there multiple hiring bars for teams working on different technologies?
- How do frontend infrastructure teams and backend infrastructure teams interact?
- Is Data Engineering a part of Engineering? How about Data Science?
- What is the ideal relationship between Security Engineering and Product Engineering? And can we get away from the consulting relationship where Security always feels brought in too late?
- How much do we pay down technical debt and who does it and which debt?
Answering these requires some kind of working theory of how engineering actually functions, otherwise we’re left copying the ratios and comp bands of other companies. No leader should have to be a copycat to answer basic questions about their org.
Knowing your sociotechnical system
Technical Coherence is a theory and a structure to answer these questions about the whole system full of people and software and data.
We start from the outside and work our way in, from the purpose of the product into a technical structure.
This post takes as a given that there are three layers of engineering. Check out Infrastructure Gravity and Domain Engineering if that’s unfamiliar.
Step 1: Identifying UX Domains
The first step to applying Technical Coherence is identifying UX domains. This term comes to us from the field of UX research but it’s similar to “bounded contexts” from Domain Drive Design. A UX domain is basically a bounded context of a user experience. Or, more helpfully, a UX domain is the whole set of things a person does while they’re in a specific role.
A common UX domain is ‘Onboarding’. A user signs up to your product and attempts to figure it out. Until they become a fully-onboarded user everything they encounter is in the ‘Onboarding’ UX domain.
Some of the internals of the system (both frontend and backend) are only relevant to this UX domain. Others are shared across UX domains.
This distinction is what gives Technical Coherence its power.
Mapping out your UX Domains
Consultants can make money with impressively detailed UML diagrams but in a real team we need something that our colleagues can make immediate sense of. So let’s draw a complex product’s UX domains using just five boxes.
Imagine you offer payroll services in the US. There will be a button somewhere called, perhaps, ‘Run Payroll’. Before the user can click that perhaps their boss needs to go through onboarding to set up the company finances. And before that someone at the company has to go through a flow that converts them from a potential account to a real one. Your internal Operations teams might have to manage the product and your CEO may need insights from a Business Intelligence interface.
In this example, you may have the following UX domains:
Each domain encompasses a broad set of user experiences —– any experience that a user has while they’re in some particular role. Someone accessing the Business Intelligence UX is a decision maker inside the company. Everything they need to observe trends and make decisions, across any number of tools, is a part of this UX.
We start developing Technical Coherence by drawing these UX domains. If it’s an experience we offer then it’s part of the product. The reason we do this is because the product has likely been built as if there were just one or two first-class experiences, with others bolted on later. Just because the Finance or Customer Support UX isn’t as urgent doesn’t mean it’s not as important. Your airplane may only need the landing gear to work at the very end of a journey but that doesn’t mean the landing gear is optional. Failing to calculate the financials correctly or failing to support users is as much a risk to the company as an underdeveloped product.
So if your CFO requires some way to download CSVs with correct financial data from the product then that is a non-optional UX domain. Put it in the diagram.
If Customer Support needs a way to reset a user’s password or access user data then Customer Support’s experience is a non-optional domain. Put it in the diagram.
Once we’ve identified all of the UX domains we’re finished with the hard part. And we’re finished with the first of three steps of pursuing Technical Coherence.
Step 2: Identify Shared Domains
The next step is to identify the domains in the product that are shared between UX domains.
The competitive advantage inside the product
I find it easy to identifying shared domains because they’re the things engineering talks about the most. Let’s find a couple examples from the payroll company we’ve been drawing.
We want to identify which domains underpin the payroll company’s UX domains. We’re looking for conceptual areas where there are limited inputs or outputs with a lot of internal complexity.
Like bank integrations.
Somewhere in a payroll system there has to be an encapsulation of the actual movement of money. This involves banking APIs, timing, status codes, encryption schemes, audit logs — all in a financially compliant way.
Or messaging to users.
The Conversion and Onboarding UX domains likely need to contact users. Operations may also need to contact users via email or text and there’s probably some automated email or text message that happens once money moves correctly. We message users multiple ways, for multiple reasons. We can have each product team implement the bare minimum messaging for their individual features but that’s going to result in slow development and a buggy, inconsistent messaging pattern with more overall code size than if we just did it correctly, centrally, once.
So let’s draw our domain chart again. This time we’ll leave out the dependency arrows and draw the UX domains as if they’re the surface of a deeper system. Let’s see how these two domains inside the product (‘Bank Integrations’ and ‘User Messaging’) connect to the surface area of the system.
User messaging happens at various parts of the user lifecycle, triggered by various actions like the user signing up, someone in the Operations dashboard writing to the user, and by automated payroll actions.
Bank integrations get set up during Onboarding, used during Run Payroll, and managed by Operations.
And all of it needs to be visible in the BI interfaces.
As we identify more shared domains and add them to the image the lines connecting UX domains to shared domains become impossibly messy — just a solid sheet of ink. A pattern emerges: The UX domains, as a set, depend on the shared domains, as a set.
We fill in a couple more shared domains and we get a better look.
Our product and design colleagues can help us plan the UX domains with a traditional product roadmap. The shared domains are less visible to non-engineers so they become solely engineering’s responsibility and require a different investment model along much longer timelines.
Step 3: Staffing the Breadth and Depth of Engineering
The 3 layers of engineering are distinct in how they work, how we staff them, who they serve, and how we incentivize the engineers.
Product Engineering
Product Engineering provides features. This is what most people think Engineering does. It’s creating and improving any of the various user experiences that the company offers. Whether the users are external or internal, if someone needs to use part of the product then Product Engineering enables that.
Domain Engineering
Domain Engineering provides what is unique to this company but shared across the company. This is how Technical Coherence delivers product acceleration.
This work is often unstaffed yet it’s extremely valuable. Senior people in both Product and Engineering yearn for more staffing here. The engineers want it because they know it improves the work of every engineer. The product leaders want it because they know it unlocks new roadmap possibilities.
Domain Engineering is the infrastructure for the Product function.
Domain Engineering enables the Product function to develop a more ambitious roadmap.
That sounds pretty hand-wavy, so consider this: When the Product leadership creates a roadmap it’s with an intuition about what’s possible, given historical engineering performance. They look backward at the previous year’s engineering output and then assume roughly the same capacity is available going forward. Product needs Domain Engineering teams to unlock new capabilities in order to make next year’s roadmap more ambitious than this year’s.
Infrastructure Engineering
Infrastructure Engineering provides what any company would need.
This is messaging systems, datastores, repository strategies, testing suites, the CI/CD flow, observability tools, a runtime environment, etc. Infrastructure Engineering delivers the large mass of useful tools that any modern company in the same industry would use. The technology at this layer must be absolutely product agnostic and there must be exactly one pattern for every solution type.
A company allowing a proliferation of duplicate patterns soon finds itself afoul of the most true principle of development tooling:
“Any development pattern is better than two.” — Me
This will require intense empathy with Product Engineering. Infrastructure engineers have big opinions on how to write software but, thanks to infrastructure gravity, they typically can’t write product software at their current company. And yet they need to provide one pattern for each problem type, ensure Product Engineers love it, and avoid adding a second.
The staffing plan
First we allocate the minimum number of the most skilled Infrastructure people to the Infrastructure layer. This layer, riddled with infrastructure gravity, requires some experience with systems. You can’t afford to let someone invent commodities like a service mesh or generalized persistence abstractions — those problems have been solved years ago. The minimum staffing here is also roughly the maximum: You want this staffed enough that the team can succeed and continually rebase the company onto the current year’s best technologies. Any more than that is a costly distraction.
Next, we add as many engineers as possible to the Domain Engineering layer. There aren’t that many people in Engineering who can work at this layer and who are also interested in it. This layer is near-impossible to overstaff because as these teams make improvements even your most junior new hires in Product Engineering become empowered. And the people qualified to work on Domain Engineering tend to enjoy switching back to creating features once the mess is cleaned up, so the risk of starving yourself of feature development is low.
Finally, Everyone else should be assigned to an appropriate Product Engineering team supporting a specific UX domain. Staffing domains — both UX domains and shared domains — is somewhat about interest and context so you’ll see individuals fit better in some domains than others.
Learning More
This post is a brief summary of Technical Coherence from Executive Engineering. The book explains much more about the metrics you’ll need to implement this and the planning models you’ll want to use to keep yourself honest.