10 Principles to Improve Decision-Making – The Async Review
10 Principles to Improve Decision-Making
An Insider’s View of how Netflix Makes Hard Decisions
By Gibson Biddle
Jun 23, 2022
An Insider’s View of how Netflix Makes Hard Decisions
In recent years, I’ve given a talk called “Wicked Hard Decisions” which has five “What would you do?” cases that provide an insider’s view of how Netflix made tough product and business decisions. I learned a lot while at Netflix and even more teaching these cases.
The skills can be learned. Below I list ten principles that will improve your odds of making great decisions. I also outline how you can practice decision-making everyday, whether you are CEO or have just begun your career as a product manager.
These principles are:
- Frame the stakes, process, and timeline
- Embrace consumer science
- Obsess over customers
- Employ strategy, tactics, and metrics
- Make decisions quickly
- Debate, decide, then do
- Learning is more important than knowing
- Embrace risk and shrug off failure
- Keep embracing risk
- Stay humble
1. Frame the stakes, process, and timeline
Jeff Bezos gets at the first principle in his 2016 Annual Report:
“Never use a one-size-fits-all decision-making process. Many decisions are reversible, two-way doors. Those decisions can use a light-weight process. For those, so what if you’re wrong?”
One of the first questions to ask is: “Is this a high stakes or low stakes decision?” If it’s a high stakes decision — like the launch of streaming in early 2007— you’ll need a deliberate approach by a cross-functional team composed of high judgment individuals.
But for low stakes decisions — like killing a feature used by only 2% of our members — a small team, close to both the problem and customers, can make a faster decision. And if they’re wrong, they can reverse or adapt quickly.
In looking at the variety of decisions companies make, most are reversible, so bias towards action helps you to execute and learn quickly. Unfortunately, many teams use heavyweight processes for all decisions.
I’m guilty of this, myself.
At Netflix, I belabored the decision to remove an early incarnation of sub-accounts (“Profiles”). We eventually killed the feature, but then were overwhelmed by the negative response of the 2% of our members who used the feature. (It turns out that most of our board members were Profile users and felt we were ruining their marriages by forcing them to share queues.)
We reversed course and all was well. But I’ll never get back the six months I struggled with the decision. I would have been far better off executing quickly, and then made adjustments based on member response.
2. Embrace consumer science
I define “consumer science” as the search for consumer insight using a mix of data sources:
- Qualitative. Focus groups, interviews, and usability sessions help to define the problem space, to get inside the heads of customers, and to learn how to package and position ideas in ways that resonate with consumers. But I don’t use these sources to make decisions; a few folks in a focus group does not adequately represent your total audience. Qualitative sources help frame how customers think about the product, let you see the product through a new customer’s eyes, and enable you to generate lots of new ideas.
- Surveys. I use this tool to understand the demographics of customers and to see what new features or attributes they might value. Again, I don’t use this source to predict behavior. What customers say and what they do aren’t always the same.
- Empirical or existing data. Increasingly, companies have tons of data — both financial and behavioral — that needs to be organized in meaningful ways. My focus here is to build a suite of metrics that helps form hypotheses about what drives customer and shareholder value. Later, I test these hypotheses via A/B tests.
- A/B Testing and Machine Learning. These are the most effective tools in the consumer science toolbox. The ability to push out a new feature to a small subset of members in an A/B test helps build a low-risk, “let’s test it!” mentality and predict outcomes when you release the change to all. Today, Netflix’ personalization effort relies on algorithmic innovation that combines offline machine learning experimentation with online A/B testing to better predict which movies members will love — or not.
Today, world-class consumer tech companies like Facebook, Amazon, and Netflix are quick to embrace the “let’s test it!” mentality and have made large investments in both A/B testing and machine learning. This makes decision-making straightforward. These companies already have a high confidence prediction about the expected result when they push a large-scale change to customers.
Netflix’s recent decision to switch from a five-star rating system to a thumbs-up/thumbs-down model was straightforward. They tested the new approach with hundreds of thousands of new members and discovered a 200% increase in ratings using the simpler, “thumbs up” system. Yes, there was grumbling from existing members like me (I hate change!), but Netflix knew via A/B test results that members would not cancel the service because of this change (I’m still a member!)
Most companies don’t operate at the scale of Netflix, Facebook, Google, and Amazon, but that doesn’t mean they can’t develop consumer insight.
With fast-growing startups, I have found focus groups outside the Bay Area (away from Silicon Valley freaks like me) to be very helpful. A well-executed Net Promoter Score survey can help startups to evaluate product-market fit. And, a well-developed set of e-staff metrics can help form hypotheses about potential drivers for customer delight.
These four sources of data — qualitative, survey, empirical, and A/B testing — help form a customer-focused mindset that ensures consumers are represented in key decisions. Today, the companies that fully embrace “consumer science” lead the pack in the consumer tech industry.
3. Obsess over customers
When I worked at Netflix, I defined my job using a simple model: to delight customers in hard-to-copy, margin-enhancing ways.
At the time, I thought the job was to balance customer and shareholder value. By stepping away and thinking about my good and bad decisions, I’ve learned to focus even more on customer delight. To do this, you have to be maniacally focused on delighting your customers.
There’s a project at Netflix I worked on that gnaws at me ten years after the fact.
During Netflix’s DVD era, I got excited about a new merchandising algorithm that took into account the cost of each DVD we delivered to customers. At the time, a new release DVD cost three dollars to deliver to a customer, while a DVD that was three to twelve months-old cost two dollars, and a title that was more than one year-old cost only a dollar to deliver.
By implementing a cost-based algorithm, we could add millions of dollars to our bottom line. But I got a tepid response from our CEO about the idea. His initial fear was that we would merchandise cheaper titles ahead of titles we knew customers would love. I assured him that we would only “nudge” a lower-cost title when the customer’s predicted rating for each movie was the same. We executed the algorithm and lowered content costs by millions of dollars — it felt like a lot of money at the time.
As I think back on the project and Reed’s lukewarm response and then spend time watching companies like Amazon obsess over its customers, I’ve learned to focus even more on customer delight. I think Reed’s ambivalence reflected his worry that I was too focused on building shareholder value and not enough on delighting customers. In hindsight, I wish I had focused nearly exclusively on the projects that would potentially deliver delight — bigger catalog, streaming on all devices, better video quality, voice control, interactive storytelling, and any number of things Netflix is likely testing now.
My guess is that one of Netflix’ most disastrous decisions — the decision to separate the streaming and DVD businesses via the launch of “Qwikster” — was due in part to the failure of Netflix to obsess over its customers. Splitting the service in two required that customers create and manage two separate accounts, resulting in customer outrage — the opposite of delight. (Yes, there was a price increase, too, which was hard for customers to stomach, especially as both the price increase and Qwikster changes were poorly communicated.)
Jeff Bezos describes the benefits of customer obsession in a way I more fully appreciate:
“There are many advantages to a customer-centric approach, but here’s the big one: customers are always beautifully, wonderfully dissatisfied, even when they report being happy and business is great. Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf.”
As you work to satisfy your inherently unsatisfiable customers and contemplate the effect your decisions will have on them, you will find yourself relentlessly driven down the path of innovation.
4. Employ strategy, tactics, and metrics
Do you have a well-articulated company and product strategy? If you do, you’re in a good place to make great decisions.
Product strategies are high-level hypotheses about what delights customers in hard-to-copy, margin-enhancing ways. In making decisions ask, “Is this decision on strategy?” If the answer is no, don’t do it.
In 2005, Netflix added a shipping hub in Hawaii and retention improved as disks took one day instead of three to arrive at a member’s mailbox. This retention improvement led to the hypothesis that faster delivery of DVDs nationwide would substantially improve retention. This strategy — deliver movies “instantly” — guided lots of projects to deliver disks faster to customers.
We measured our progress through a metric that described the percent of times members got their first choice disk the next day in the mail.
Our “instant” strategy, with measured progress against the metric, fueled lots of projects — doubling the number of shipping hubs in the US, tuning our merchandising algorithms so that only the disks in your local shipping hub were displayed on the site, and eventually led to the launch of streaming. Most of these decisions were straightforward as they were very much on strategy and we could measure progress via a precise metric.
This metric improved from 70 to 95 percent over the course of a few years and I’m confident it led to retention gains — Netflix’s ultimate measure of delight.
5. Make decisions quickly
Most decisions are low stakes and reversible, so your bias should be to make them quickly. But what about the high stakes decisions? What’s the right pace for these decisions?
When I was a college student, I got my pilot’s license. A lot of the flight safety training focused on high-risk, fast-paced decision-making. You are trained to make difficult decisions quickly, knowing you can adjust after the fact.
You also learn that doing nothing — when flying into a cloud without an instrument rating, for instance — is rarely the right decision.
In flying, you’re introduced to the concept of provisional decision making process. The concept: “With the data you have right now, what decision would you make?” Then ask yourself, “What additional data do I need to verify this decision? How and when can I get that data?”
In the instance of flying into a cloud, the common provisional decision is to make a 180-degree turn. The additional data required is more weather information about clouds at different altitudes and potential alternate routes.
In 2007, Blockbuster engaged in an assault on our business with their “Total Access” program.
They offered their DVD-by-mail customers unlimited disk swaps at their stores. We debated an appropriate response — lower prices or do nothing — but the additional piece of data we needed was, “How many disks are Blockbuster customers watching each month, and based on this, how long can they afford to continue the program?” We gave ourselves a month to get this data and found it.
The answer: Given a high rate of in-store “disk swaps” by their customers, Blockbuster could only afford to maintain the Total Access program for nine months. Our response: double down on streaming. By the time Blockbuster stopped “Total Access” (at nine months!), we had a substantially better streaming service, while Blockbuster had none.
A simple rule of thumb in decision-making: Get 70% of the data you need, then make a decision. If you have less data than this, you’re potentially reckless. If you have more data than this, you’re probably wasting too much time.
6. Debate, decide, then do
Growing up, my parents would say to me, “Good fights make good marriages.” (They’ve been married 55 years!)
This folksy, New England saying reinforces the idea that you need good debates to make great big decisions together. And once the decision is made, you need to quickly align to execute. Amazon calls this behavior, “disagree and commit.”
At Netflix, we worked to deliver overwhelming value to our customers.
Price was a big lever. We A/B tested pricing but there were many debates as we interpreted the results: “How to balance Wall Street’s expectation for profit against the growth that lower prices delivered? How much debt should we take on in order to fund future growth?” There were many other, potential long-term impacts of price changes that we couldn’t evaluate via A/B tests.
Every Monday morning Netflix had an executive team meeting and I can still remember Reed, Barry McCarthy (CFO) and Leslie Kilgore (CMO) modeling the behavior they expected of the broader executive team. Results of A/B price tests in hand, they would debate the more nuanced issues of pricing.
And from time to time, Reed would ask participants to flip their point of view and argue the opposite — an exercise that encourages not just debate, but careful listening.
A common question I get: “You had passionate debates at Netflix, but how did you ensure everyone got on board with the decision?” My response, “It was part of the culture.”
To this day, I can still remember phrases that were repeated to encourage the behavior: “disagree and commit,” “decide and do,” and the simple concept of “well-formed adults” who could switch seamlessly from candid, passionate debate to lockstep execution as a team.
7. Learning is more important than knowing
For many tough decisions there is no right or wrong choice. What is important is learning as much as you can about your customers, business, and industry to improve the odds in future decisions.
Netflix spent a year preparing for the launch of its DVD-by-mail service in the UK and then a week before launch cancelled the project due to a rumor that Amazon was launching its own DVD-by-mail service in the U.S.
Was it the right decision? It’s impossible to know. But we learned the importance of focus, of saying “no,” and ignoring sunk cost as we chose to focus on our core U.S. business.
We also knew that once streaming was ubiquitous, we’d be in a good position to expand worldwide.
8. Embrace risk and shrug off failure
The decision Netflix made to separate its DVD-by-mail and streaming service in 2011 was a huge mistake.
Netflix was lampooned on late night TV, lost hundreds of thousands of members, and their market cap dropped from $18B to $5B. It was a high stakes, poorly executed decision.
CEO Reed Hastings acknowledged the mistake. He advocated that self-inflicted wounds were much easier to fix than competitive damage and that if the company stayed focused on building customer value, Netflix would regain its health. Rather than beat themselves up for making the mistake — which discourages future risk-taking — the team learned from its mistake, then moved on.
Today, Netflix has a market cap of $80 billion and more than 221 million members worldwide.
9. Keep embracing risk
Today’s leading consumer internet tech companies — Google, Amazon, Facebook, and Netflix — are all about twenty years old.
Each of these companies got to this stage by embracing risk. On the other hand, there are dozens of companies that were a big deal on their tenth birthday, but are a shadow of their former selves today. Blackberry, Yahoo!, and MySpace come to mind.
I was so impressed by Facebook when they celebrated their eighth birthday. That year they went public, acquired WhatsApp for $19 billion dollars, experimented with drones to provide ubiquitous internet access, and made large investments in VR through their $3B acquisition of Oculus.
Facebook understood that in order to build a great company they needed to maintain the same level of risk-taking that established them as a startup. Playing it safe won’t get you there.
Today, I ask companies to describe the key projects they are engaged in. Then I ask which projects are optimizations and which are bold bets that might provide a 10x return.
Generally, it feels good if the optimization/innovation split is 80/20. When it’s time to make decisions about these bold bets, acknowledge they are high risk but carry a greater prospect for substantial reward. Lean in to the risk.
10. Stay humble
The commitment to quickly shift from decision to fast execution — to “decide and do” — requires focus and a degree of bravado as you navigate the unknown.
As great exec teams develop they invariably make better decisions. In some cases, however, the team grows overconfident and makes stupid mistakes.
In explaining his decision to separate the streaming and DVD services via the launch of the Qwikster DVD-by-mail service, Reed Hastings admitted, “I messed up,” then speaks of his lack of humility:
“It is clear from the feedback over the past two months that many members felt we lacked respect and humility in the way we announced the separation of DVD and streaming and the price changes. That was certainly not our intent, and I offer my sincere apology.”
Before the Qwikster debacle, Netflix made a series of great decisions. Two examples: they partnered with hardware manufactures to enable “Netflix Ready” TVs worldwide then made a bold, highly successful bet on original content through their $100M investment in “House of Cards.”
I think these highly successful decisions fueled overconfidence, however, leading to the disastrous decision to split the DVD and streaming product in two.
Putting decision-making principles into practice
Making great decisions — about product, people, and business — requires practice. But how can you practice decision-making in your everyday job?
Ask yourself these questions to make decisions day-to-day
In any meeting where a decision needs to be made, ask yourself, “What questions do I need to answer to make this decision?’ then ask the questions!
After that, ask yourself, “What should we do?” regardless of whether you are the decision maker. Your final step: bravely state your opinion and work to initiate meaningful debate.
Over time, you’ll develop the necessary decision-making skills and people will seek out your opinion. One day, you’ll find your CEO asking, “What would you do?” You’ll know at this point that you are on the way to making your own “wicked hard decisions.”
Aim for 70% right, over time
These ten decision-making principles provide a framework for making tough decisions, but there’s no one-size-fits-all approach.
The key is thinking about the principles and identifying which apply to your decision. In the long-term, you’ll develop more consumer insight with each decision you make, eventually leading to high-quality, high-velocity decision-making.
Newly-formed management teams get it right about half the time. A world-class management team will get it right 70% of the time. And if a team gets it right more than 70% of the time, there’s likely something wrong: they’re not taking on enough risk.
In his 2016 annual report letter, Bezos reinforces both the importance and challenge of the product leader’s job:
“[Being an innovative company] requires you to experiment patiently, accept failures, plant seeds, protect saplings, and double down when you see customer delight. A customer-obsessed culture best creates the conditions where all of that can happen… …but you, the product or service owner, must understand the customer, have a vision, and love the offering. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste…”
Product leadership is a tough job, with a diverse set of skills required. But with each decision you make the odds for success get higher, given your growing knowledge of consumer behavior. Jeff Bezos has been at this more than 25 years. You’ll get there, too.
More stories on:
Jun 16, 2022
By Hailey Friedman
Dec 14, 2021
By Darryn King