Estimates are a controversial topic in software development, often leading to heated discussions and arguments. Many of us spend as much as 25% of our time on detailed planning and estimating, and in more than a few cases, these plans fail to meet reality, resulting in disappointment and frustration – from our customers, our managers and our developers. Then there are the non-obvious implications, both technical and personal, which increase entropy in our codebase and the organization. Why does that happen? Can we do better?
Why Estimate?
When asked why they estimate, engineering managers often state the need for predictability, coordination and control. Estimates are seen as an inseparable part of the planning process, and planning is at the core of many common Agile methodologies – whether we use Kanban, Scrum, or some other variant. However, many do not stop to reason about the value they gain from the planning process itself.
At the end of the day, it boils down to trust and accountability. We need to prove to our superiors or investors that We’re using the funding and time effectively. We need a way to trust that our developers are doing the right things, not wasting their time, or even that they are actually working. Maybe we have some junior engineers and need a way to give them small, achievable goals. But does the significant overhead incurred by the planning and estimation process really solve these problems? If we remove trust from the equation, do we still need to plan and estimate? Do we actually gain predictability and control by creating detailed plans?
Another reason given for estimating is alignment to hard deadlines that are not under our control – whether due to integration with another party that cannot (or will not) align itself to us (such as a bigger company), or a major client that will only onboard if they get a specific set of features. Note that internal hard deadlines (integrating with another team/group inside our organization) fall under coordination and can be solved by better means described below.
Estimate or Guesstimate?
The process of turning a list of requirements into a detailed work plan consists of several, seemingly simple, steps. We break down a feature into tasks achievable by individual developers. We ask developers to predict how long each task is going to take. We pick the tasks with the highest value-to-effort ratio, we add 10-20% buffer time for unplanned work and a-viola – we have a plan for the next sprint.
But in reality, this is not simple at all, and for several reasons. Many tasks cannot be deemed done (some cannot even start) until another task has completed. Then, there’s the cost of combining work done by several developers into a complete feature that can be defined as done – often incurring additional costs for integration and verification. Another problem might arise if developers don’t take all aspects of a task into consideration, such as writing tests, delivery overhead, refactoring, or bug fixes.
Ultimately, though, we engage in (hopefully) educated guesswork. Often we discover in mid-development that some assumptions were wrong, that the requirements were unclear, or that there are existing preconditions in current design or architecture that make implementing what should have been a trivial task much more difficult.
Even if nothing goes wrong during implementation, we simply cannot predict the future. A key developer might become sick, hurting their own delivery as well as others who might depend on them. A production crisis might end up consuming much of our developers’ time. Context switches due to meetings or company events might hurt more than we realize. And global events such as wars, pandemics or natural disasters can incapacitate our developers physically or mentally.
The Hidden Costs Of Missing Our Estimates
In addition to time lost to detailed planning, there are technical, individual and organizational implications to making (and missing) estimates that we might not be aware of.
No matter the cause, developers often engage in stressed work when facing unachievable goals, which results in technical debt being accumulated – missed refactoring opportunities, half-baked work, and – most often – lackluster test coverage. Over time, these lead to increased cost of change – the codebase becomes more difficult to change and our ability to make founded estimates diminishes even more.
A culture where estimates are often missed also hurts the developers. Frequently working overtime leads to chronic stress and eventual burnout. Burnt-out developers might leave, or worse – stay but work ineffectively. Another negative outcome is a loss of trust between developers and management, since a constant sense of urgency is tantamount to no sense of urgency at all. Managers might suspect that developers are not working as hard as they should, developers might learn to avoid stress by significantly beefing up their estimates, and those few who refrain from doing that are bound by guilt and frustration over missing objectives that might have never been realistic to begin with.
At the organizational level, these problems eventually lead to chaos, rivalry and attrition. Disgruntled developers and middle managers tire of chasing non-tangible goals and leave. Replacing them derails the remaining staff even more – and it is often the best developers who leave first. Frustrated by their inability to meet their objectives, engineering managers often oscillate between different approaches to task management and planning, contributing to confusion and chaos. Different groups inside the organization might be seen as more or less effective at meeting their goals, resulting in unhealthy rivalry and enmity.
Effective Progress Without Estimates
How can we make predictable, coordinated, and manageable progress without needing to create detailed plans that require estimates? A possible solution was surfaced by Woody Zuill back in 2012: #NoEstimates, a method of practicing Scrum without explicit detailed estimates. It claimed that forecasts are always wrong and offered several techniques to prioritize tasks and come up with a work plan without doing detailed planning and estimation, and without incurring the overhead usually associated with Scrum planning. However, #NoEstimate proponents are still actually estimating. In my opinion, the first questions that need to be asked are – do we really need to estimate? Do we need to come up with strict plans?
Trust
Developers are knowledge workers, and as Peter Drucker said back in 1999, they are the asset – much more than the software they deliver. We as engineering managers must cultivate a culture of trust and continuous learning, serving as facilitators rather than overlords. All aspects of planning that stem from our need for control have to be dismissed. We trust our developers by default – unless we have a good reason not to. By removing planning and estimation, we free up more time for 1:1 meetings with our developers – meetings during which we build trust relationships with them and get regular updates on their progress and how they perceive their effectiveness.
Information Flow
To prioritize between several competing tasks, we can use our intuition to roughly guesstimate the scope of each task – and accept that we might be wrong. It doesn’t matter if we are, though, because unforeseen complications might arise anyway. The important bit is to have our developers update us as soon as any new piece of information makes itself available, so that we can reconsider our options. For instance, we can start a task, but postpone its completion if it appears to take much longer than we initially thought.
Mentorship
If we learn that a developer struggles or feels ineffective, we can have them pair-program with senior developers for several weeks. Healthy software organizations have ICs at the Staff or Principal level who regularly spend their time pairing with more junior developers, thus helping preserve and spread their knowledge. Pair-programming is also a great way to onboard new developers and help them assimilate our culture.
Cooperation
If we have a feature that requires a coordinated effort with other groups inside the organization, we can do much better than plan together and work separately – because doing so defers integration until all subtasks are complete, and it is often only then that we discover problems, misunderstanding or incomplete requirements. A much better solution is to form an ad-hoc squad consisting of a developer from each group, pair (or mob)-programming the task, integrating early, then working independently to finalize implementation details and add more detail to the feature. Some added values of forming these ad-hoc squads are to help propagate knowledge between different groups, breaking down silos and increasing information flow throughout the grand organization, and to facilitate horizontal mobility – people who’re bored with one domain can move to a different team in a different domain rather than leave.
Meeting Hard Deadlines
How can we meet hard deadlines, set by major prospective clients or organizations we might want to integrate with? It starts with facing reality: there’s no way to predict the future. To retain our ability to consistently deliver at an effective pace, we must not compromise on quality, and so the only way to meet hard deadlines effectively is to limit the scope of work that has to be done to meet them, and then expand iteratively.
We discuss the scope of delivery with the other party who originated the deadline, we narrow down the delivery to the bare minimum, and we use terms such as forecast rather than commitment. When forecasting a delivery date, we use bracketing – for instance, we can predict that a task can be done in two to four weeks.
We use all tools at our disposal to minimize the risk of disappointment: we can make sure that our best developers are working on the most critical tasks under hard deadlines, preferably while pairing with other developers so that over time more developers can be trusted with these time-sensitive tasks. We periodically update our forecast, as soon as we discover new facts, and we reflect the updated forecast to the other party to make sure that expectations are aligned. We do that using terms such as “time left” to indicate that progress has been made and progress still has to be made. If push comes to shove and we really must work overtime towards a critical, hard deadline, we make sure everyone on the team understands why and share a common vision, and we make sure to compensate after the task has been done – in vacation time, in time dedicated for cleanup work, and, preferably, in bonuses or stock options.
In Summary
Estimates often arise from the need for control, coordination and predictability, but since the future cannot be predicted, they usually lead to significant waste, mistrust and frustration. By focusing on trust, information flow, mentorship and coordination, we can achieve better results, while cultivating a sustainable culture of accountability and professionalism, a healthier codebase and reducing attrition. If crunch time is unavoidable, we minimize its scope and its recurrence, and remember to compensate after it’s done.