Technology Decision Making (And Boring Technology)
—
I came across a link to Choose Boring Technology by Dan McKinley via one of my community Slack channels. It got me thinking about the interconnectedness of: complexity and tech debt, developer productivity and velocity, engineering culture and decision-making. Ultimately, I came to the conclusion that the culture of decision-making — how decisions are made — has a huge influence on the long-term productivity of the engineering organization.
First, some highlights from Boring Technology. The three key items that resonate with me:
1. The long-term maintenance cost of a technology dominates the total cost and far surpasses the short-term velocity benefit.
We tend to focus on the benefits of a new technology without always fully acknowledging the ongoing maintenance costs — which often dominate in the long run.
The way we behave really depends on what you believe about which term dominates this equation in the real world.
If technology is really expensive to operate, the costs dominate. If technology really makes a huge difference in how easy your job is, the benefits dominate.
2. People (often) get very opinionated over database choice…
I’m included in that list of people who are very opinionated, but more for reasons related to the fact that I understand intimately how much time and effort it takes to build the specialized knowledge required to really understand a particular RDBMS or data storage tech.
In my experience it is about here where the wheels come off. People lose their shit and start beating their polyglot programmer drums. There’s something about the idea of adding a new database that has people storming the Bastille, saying “you can’t stop us from using the best tool for the job, man.”
And when people succumb to this instinct they tell themselves that they’re giving developers freedom. And sure, it is freedom, but it’s a very narrow definition of what freedom is. #36
The point here is that opinion and the notion of freedom and/or satisfaction can dominate the narrative of technology choice if there isn’t a solid process for making decisions.
3. Ultimately, our goal in building and maintaining products (systems) is to support a successful business outcome. Not to build fancy technology.
Going back to my first SaaS start-up experience — we built Eloqua on boring technology — Visual Basic (not joking, it was the year 2000), SQL Server, IIS.
In those first few years, I built a few boring tools (you guessed it — also in Visual Basic) to automate deployments and perform basic endpoint monitoring. (It was pre-public-cloud, people!). My main goal was to save myself time and reduce errors in deployments — we had a very lean team, and these tools materially improved our efficiency.
In 2008, eight years into the company’s lifecycle, my nine person “Production Operations” team supported the data center, all hardware and infra, network and database expertise. Daily, we saw some tens of millions of transactions. In Q4 2008, we maintained 99.998% uptime, and we did so while reducing costs, on this really boring technology stack.
Obviously this boring stack at Eloqua would have looked a lot different if we’d built it with the technologies available in the last five to ten years. And, there would have been significant benefits we could have seen with some specific technologies. But with the current technologies available, the ease of spinning up a new managed service has to be weighed against the long-term total cost of ownership.
Therefore:
You should have a process for adding technology to your stack that involves talking to other humans. #86
How do you go about making these decisions at the scale of 20+ teams?
At FreshBooks, we spent a lot of time working through this questions, and ultimately worked toward bringing clarity and substance to the decision-making process using:
- Engineering documentation to clearly document the important, slowly changing information and processes that are important to the organization
- A decision-making framework to help teams understand what decisions can be taken at a team level vs. organizational level and where to go for documentation
- An internal Tech Radar that describes the tools, technologies that should and should not be adopted
- A Request for Comments (RFC) process for a framework for evaluating and recommending new approaches or technologies
- Architecture Decision Records (ADRs) to document decisions
Collectively, these approaches help all engineers and teams understand what technologies are in use and are recommended, provide a clear process for proposing and evaluating new technologies, and ensure that decisions are clearly documented for the sake of current and future engineers.
I’ve just recently decided to move on from FreshBooks, but I’m hopeful that these collective approaches will lead to durable decisions, reduced complexity and increased productivity. (FreshBooks folks — keep me honest and let me know how it pans out…)
Back to Eloqua. We made our boring technology choices out of necessity for the bulk of the company’s history. We created a new category — Marketing Automation — but had a culture that was perceived as (and probably was) low in technical innovation. We didn’t have a strong decision-making process, but our decisions were heavily influenced by the need to support an enterprise B2B revenue stream.
Ultimately, we sold Eloqua and its boring-tech-stack to Oracle for close to $900M. I’m pretty sure that the recurring revenue and revenue growth rate was their key interest in this deal… and our boring technology choices simply passed the due diligence.