Adam Thomas | Product Management Expert

Category: Decision Science

  • Accuracy vs. Precision: what’s the difference?

    Accuracy vs. Precision: what’s the difference?

    Accuracy and precision may seem like the same thing, but understanding the distinction between them will let you level up your product management skills.

    red white and black round wheel
    Photo by Afif Ramdhasuma on Unsplash

    Would you prefer your work to be accurate or precise? This sounds like a trick question, right? Aren’t they the same thing? 

    Well, not exactly. Both are important to your product management practice. As a product leader, you’ll need to understand and leverage both concepts to make your strategy a reality.

    Accuracy vs. Precision

    Let’s use the concept of the bullseye to make the distinction between accuracy and precision clearer. 

    Imagine you’re an archer taking some practice shots by shooting at a bullseye. I’m judging your shots over the span of time, and I tell you I’m looking at both precision and accuracy. I would judge your performance the following way:

    ACCURACY

    Accuracy measures how close measurements are to an accepted value, or in our example, how close your shots are to the center of the bullseye. If you’re hitting the innermost circle of the target, then that shot is as accurate as possible. 

    In product management terms, consider your ability to deliver products that the market wants and how right you are when solving a problem. Remember, you can’t ever be completely accurate, but you can approach perfection.

    PRECISION

    Precision measures how close each of your measurements — or shots — are to each other. If all your arrows are hitting the same place, then you’re as precise as possible, even if the shots aren’t accurate. 

    In product management, consider how consistently you deliver useful results to the customers and bring value to the market. Are your releases consistent? 

    Both concepts of accuracy and precision are essential to executing your product strategy well. If you’re failing at both, you’ll find your team is powerless. When both are rolling, however, you’ll find that your team is full of trust and positioned for success over the long term.

    Comparing Accuracy and Precision

    Let’s look at accuracy and precision in a matrix since visualizing things makes understanding this principle even simpler:

    precision-accuracy
    The relationship between precision and accuracy. | Image: Adam Thomas

    Let’s talk about each one of these quadrants, where a “+” symbol means the precision or accuracy is high, and a “-” symbol means the precision or accuracy is low. 

    Each section below will define how these quadrant scenarios look in practice. If your product team isn’t in the top-left quadrant, you’re going to want to fix that. We’ll talk about the consequences of each of the other quadrants and then, in the end, I’ll leave you with solutions for each that will allow your team to move to the top-left.

    HIGH ACCURACY AND HIGH PRECISION 

    Let’s start with a look at the best-case scenario. What is the ideal state? 

    When your strategy is both accurate and precise, you solve customer problems and leverage your resources well. 

    This team can take many forms. Much like enlightenment, the path is different for everyone. But you’ll be able to tell you’re on the right track because things run smoothly.

    A team that’s both precise and accurate has the right stuff. Your team is curious, thoughtful and action-oriented. They know how to look at the marketplace and create big wins to make a splash. The team also knows how to keep the marketplace happy with consistent results.

    When you ship smaller releases, your customers are generally appreciative. Larger ones generate a buzz in the marketplace. Product usage makes clear that the customer needs what you’re giving them. For most releases, you’ll see over 40 percent usage. Those that aren’t at that level are usually specific add-ons for a particular segment of customers whose numbers may be small, but who are willing to pay for the exclusivity. 

    Product leaders: When a team is operating like this, your job is to get out of the way and let them cook.

    Now that we have a good baseline, let’s take a look at what happens when teams are missing a part of the equation and it’s time for you to help. 

    HIGH ACCURACY AND LOW PRECISION 

    Let’s go now to BobCo, whose product team has a precision problem. 

    Over the last few quarters, they seem to have found some things that work and work well. The only problem is that, out of the 10 items they ship, only one or two of the products are helpful to customers.

    Why is this a problem? Home runs are great and tend to bring better visibility to a company and a team. But delivering only big wins, punctuated by long stretches of not much at all, eventually leaves people wanting. After a while, teams expect to have some consistency in their work. 

    The result? The product team never knows how to justify resource requests to the broader company. Your lack of consistency means you don’t know what will be a hit and what will fall flat. Outside the company, the marketplace thinks you are great entertainment but not a serious player. 

    BobCo’s product team continues to work hard, but they can’t seem to make the case for growing the product function. The team starts to fade from importance in the organization, and as a result, loses influence. Product trades in influence, so this is a serious problem.  

    As a product leader, you’re responsible for the fact that the whole organization is stuck in the mud as other teams catch on to the issues. 

    How to Fix High Accuracy and Low Precision

    The culprit is likely your lack of iterating on the process itself. My guess is that the most significant fixes will come when your teams have an opportunity to adjust processes. You probably aren’t reflecting on anything. Take a moment to slow down delivery and see how you can get alignment on how you ship. 

    A useful tool to generate this alignment is a team retrospective workshop.

    LOW ACCURACY AND HIGH PRECISION 

    Let’s go now to HobCo, a team that has an accuracy problem. 

    Unlike BobCo, they seem to be able to get the team to provide value regularly. The problem is that they aren’t shaking up the marketplace. 

    Why is this a problem? Well, it’s basically the inverse of what we saw above. Being reliable is excellent, but over time, that consistency becomes less effective. It’s how you open room for new challengers to eat your lunch in the marketplace. You’ll run into issues with people feeling like the product is in the background. They aren’t making a splash, and people forget the team exists. 

    The result of all this? The product team recedes into the back of people’s minds. Always being reliable makes your product always a bridesmaid, never a bride. When a new player hits a home run, you’ll have to explain why you missed it. 

    Hobco’s product team continues to work hard but, just like we saw above, can’t manage to grow the team or its influence. 

    As a product leader, you’ll watch your team fade in importance. You lose influence, and much like your compatriots at BobCo, your team will be stuck in the mud.

    How to Fix Low Accuracy and High Precision

    When you find this is your problem, it’s an excellent time to take a look at your discovery processes. How are you talking to customers? Are you identifying who they are and what gets them emotionally invested or divested? 

    To do this, look at why customers are leaving your product, and find some big swings to take. Switch interviews are a great tool for making this change.

    LOW ACCURACY AND LOW PRECISION 

    Let’s finally look at SobCo, a team that has a problem with both accuracy and precision.

    Unlike either of the previous companies, this team is neither accurate nor precise. If you thought the rest of the company ignored the product teams before, well, you haven’t seen anything yet. 

    Why is this a problem? No one takes product seriously as a team.

    Because product is considered trivial, everyone bypasses the product team entirely. If you’re lucky, your company becomes a feature factory that can crank out some products. More often than not, though, everything related to your strategy becomes more scattered. People can’t track what you are building or why it’s important.

    Product development still happens, but you just won’t be involved. For a product team, this situation is apocalyptic.

    The team itself is now on the brink of extinction. As a product leader, you either have to start digging your way out or start looking for another job. 

    How to Fix Low Accuracy and Low Precision

    To start fixing this situation, you need to recognize that the biggest problem now is trust. Other teams are more than likely ignoring you, and for a good reason: They don’t trust that you can get things done. If you want to dig out of this hole, your best shot is to look for something small to tackle immediately. Find some quick wins to get some momentum on your team’s side. You can use those wins to begin re-establishing your team’s confidence in itself as well as earning back respect from the company at large.

    I’d recommend finding a coach if you are a product leader in this situation. Even if you’re a new hire, having a second brain here is helpful. 

    To find tools to facilitate this process, go to Google and pick any framework related to process or discovery. Most will be helpful as a way to find a solid base to build on. 

    Start small, and good luck. You’ve got work to do.

    MORE FROM OUR EXPERT CONTRIBUTORS

    How AI Is Shaking Up the Insurance World

    Why Track Both Accuracy and Precision

    Precision and accuracy are both important to your product development. 

    Missing one or the other can lead your team to fade into obscurity and lose influence across an organization. If you are missing both, well, things aren’t great, and you’ll need to work immediately to gain control of the situation. 

    It isn’t enough to just build a strategy, you’ll need to execute that strategy in a way that sells product to an organization. So, when someone asks whether you would like to be accurate or precise, the answer is both.

  • Great product managers leverage cognitive bias

    Great product managers leverage cognitive bias

    All humans are susceptible to cognitive biases. As a product manager, your job involves turning those weaknesses into strengths for the team.

    brown tree
    Photo by Neil Thomas on Unsplash

    roduct managers (PdM) don’t make anything. Well, at least they don’t in the traditional sense. When it comes to product development, the product manager, unless under duress due to a lack of resources, bad planning, or maybe both, isn’t writing production code or crafting customer interactions. What they are doing at their best, though, is managing the decision science of the team.

    PdMs spend their time making decisions that affect how the team decides … well, anything. They are busy prioritizing the needs of product development, managing the velocity of the decisions that the team is facing, or working on any number of projects that shape how the team makes decisions.

    Today, I’d like for us to take a trip with cognitive science and talk about two concepts: availability bias and recency bias. Whether you realize it or not, you’re susceptible to both of these biases. In fact, they can even shift how product teams operate. The good news is that you, as a product manager, can actually leverage them to improve your team’s decision making.

    THE BIASES

    Recency bias is exactly what its name implies: A bias that privileges things that have recently happened. Have you ever sat in a meeting and noticed that decisions seem to largely get made at the end of the meeting, and the players inside of that meeting only reference data from the last 15 minutes? If so, you, my friend, have seen recency bias in action.

    Availability bias is similar to its recency counterpart in that it privileges data based on ease of access rather than quality. It involves making decisions using whatever information is immediately available. You’ve also probably seen this one in practice as well. If you’ve ever been in a meeting and watched people choose the first idea that comes to mind, that’s it. Its main difference from recency is that availability bias isn’t time-locked. Someone may have research they rely on that they’ve collected months or even years ago.

    As product people, one of our roles in an organization is to improve the decision-making process for product development. Both of these biases affect our output, which we can think of as the company’s “decision fitness,” or how well we make the decisions that affect our outcomes. Without managing these biases, you’re lowering the fitness of the company and are on a sure road to shallow decision making.  So let’s ask how well we’re making decisions.

    WHERE DOES PRODUCT MANAGEMENT COME INTO PLAY?

    Since we product managers don’t have any tangible output, upgrading everyone’s ability to make decisions is a way for us to show our value and also functions as a metric for us to gauge ourselves. Our battlefield is often the research lab. For product managers, this isn’t the stereotypical science lab that comes to mind, but rather the place where we do research and one-on-ones with other stakeholders. We also ply our trade in the meeting room, and, in both spaces, we need to be vigilant in watching for the following red flags from our teams.

    On recency bias:

    • Meetings without any agenda or facilitation. If no one is managing the conversation, our brains will default to the last thing we heard.
       
    • The roadmap is just the highest-paid person’s opinion (HiPPO). The tactics you use to get to the outcomes you want should have a variety of sources. If you only have one source, you have a problem.

    On availability bias:

    • Quick decisions and buzzer logic. We’ve spent all the time in the meeting talking about potential strategies only to take the first one available.
       
    • Teams not using research. Research always has curveballs. For example, it might show that your customers are using features to solve unforeseen problems and maybe causing new ones. If the decisions you come to haven’t considered even the possibility of alternatives, you’ll have shallow outcomes.

    SO WE CAN IDENTIFY BIAS. NOW WHAT?

    Now that we know how to spot bias, let’s leverage it. We can’t change how the human brain works. Trust me, I’ve tried. You aren’t going to get rid of cognitive bias — but if you have, please, email me!. What we can do instead is put processes in place that understand these biases exists and use them to help our decision making.

    So, let’s talk tactics.

    • Strive for short decision-making cycles. Ask yourself, “How can I make the decision-making cycle in a meeting as small as possible?” Far too often, teams wait until the end of meetings to make decisions. That timing falls into the traps we saw earlier. This likely means structuring meetings in decision cycles instead of theme, which brings us to our next tactic.
       
    • Meeting discipline. We all hate meetings, and I get that. Despite their unpopularity, they’re a useful tool for finding alignment and making decisions. PdMs should own meeting management for product development and ensure that people walk into meetings with context. Those cycles should leverage availability bias by making the right artifacts available at the right time and recency bias by making decisions in each cycle instead of waiting until the end of the meeting. That way, you’re structuring the meeting to take advantage of the bias instead of letting it derail your process.
       
    • Research discipline. Much like meetings, your participants, whether they are internal or external, will fall into the trap of biased thinking. Make sure you have a few good questions and set the table for those who want to understand the research can access the information. If the team trusts the research, and it’s available, they will add it to what’s “available.”

    These strategies are just a few things you can implement immediately to manage those biases and improve decision fitness.

    WHEN WE LEVERAGE OUR BIASES, SOME THINGS CHANGE

    Large-scale shifts in team thinking take time. Some things will start to happen immediately, however.

    • You’ll have more collective insight into the customer. Shorter decision cycles with more “just-in-time” context that includes different voices, including some that often don’t get into the conversation, means more voices will feel comfortable joining the conversation.
       
    • Less HiPPO wrangling. As a shift occurs in the power structure, the HiPPO will see that there are plenty of voices that are contributing and feel less pressure to “fill air.”
       
    • More usage. People will feel comfortable with product artifacts (research, strategy). Teams will have better inputs for their own decision making since they know where it is.

    When a PdM manages this process, the team and org get better outcomes.

    BOTTOM LINE: FIND BIAS, UNDERSTAND IT, LEVERAGE IT

    Product managers, since they are usually outside of the solution business (read: making shit), are far more useful when they understand the mental pitfalls an organization may fall into. Since they have the mental space to look at things from both the customer and business P.O.V., as well as associate with the product makers, they should see where cognitive bias is helping or hurting an organization.

    The best work we can do as PdM’s is to understand the outcomes we’re going for and use the environment around us to give it the best chance of success. Cognitive bias is a part of that environment. In particular, recency and availability bias have some silent consequences that affect the day to day that can lead to sustained mediocrity in product development.

  • Identifying and addressing common biases in product management

    Identifying and addressing common biases in product management

    a man sitting in front of a laptop computer
    Photo by Priscilla Du Preez 🇨🇦 on Unsplash

    Product management is “thinky” work. 

    As a product person, you need to bring about organizational alignment through product strategy while also keeping things running smoothly with product operations. I don’t need to tell you this is a delicate balance. And without time to sit and think, you’ll end up making rash decisions based on your biases.

    So, what is bias? Well, our good pal Wikipedia defines bias as:

    “Disproportionate weight in favor of or against an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, or a belief. In science and engineering, a bias is a systematic error.”

    The first sentence of this article was about “thinky” work, so let’s add something to our bias definition that focuses on the brain: the word cognitive. 

    Cognitive biases

    Here is the definition of cognitive bias, once again from our friends at Wikipedia:

    “A cognitive bias is a systematic pattern of deviation from norm or rationality in judgment. Individuals create their own ‘subjective reality’ from their perception of the input. An individual’s construction of reality, not the objective input, may dictate their behavior in the world.”

    The word system is there, too.  In fact, we should discuss systems before we go any further (especially because product exists somewhere between science and engineering). Our bias is an error in a system. If you don’t think your work involves systems, it does. And if that took you by surprise, take some time to audit your product organization to see just what systems exist. Org charts and meeting lists are a great way to start, as are team retrospectives. 

    Once you have a basic understanding of the system(s) your product development team exists in, this article may be helpful. If not, bookmark this article and come back to it after you’ve got a clearer picture of the system(s) your team is part of. 

    If you’re still with me, I’d like to spend some time talking about three biases that are affecting your teams, how they affect your teams, and some ways to identify them. These biases — confirmation bias, availability bias, and recency bias — can leave a product team spinning its wheels and keep your organization from reaching its goals. 

    Confirmation bias

    Wikipedia defines confirmation bias as:

    “The tendency to search for, interpret, favor, and recall information in a way that confirms or supports one’s prior beliefs or values. It is an important type of cognitive bias that has a significant effect on the proper functioning of society by distorting evidence-based decision-making.”

    Confirmation bias is the most common bias within product teams. In fact, it’s the “patron bias” behind feature factories. Since every result seems to “make sense” to the product team, they end up proceeding, full speed ahead, with their plans. Feature factories are incentivized and tracked by “feature release velocity (FRV)” and “can’t afford” to stop shipping things as rapidly as possible.

    The problem? Your product strategy is about customer outcomes, not how many features you’ve created.

    Your team may be experiencing widespread confirmation bias if:

    • Research always “makes sense.” If user research always goes smoothly and never reveals anything surprising, then it’s worth questioning how deeply you are interrogating the world around you.
    • There is no tension in meetings. Teams come from different perspectives, and there should be a healthy level of conflict that comes from those perspectives.

    Recency bias

    “What just happened?”

    If we are always thinking about that, then we’re focused on recency bias:

    “Recency bias is a cognitive bias that favors recent events over historic ones. A memory bias, recency bias gives ‘greater importance to the most recent event,’ such as the final lawyer’s closing argument a jury hears before being dismissed to deliberate.”

    Recency bias means we aren’t actively listening to different perspectives  — we’re just checking boxes until we hear the last word. Too often, this ends up being the highest-paid person’s opinion (HiPPO). When product operations is very top-down or if there is no rigor behind your decisions, then your teams are really just waiting until the end of the meeting for a decision. The problem here is that the work you are doing ends up not mattering to the product strategy. And in the long run, teams can slowly lose their agency, which harms operations and slows down processes.

    Some symptoms of recency bias include:

    • Meetings not having any agenda or facilitation. If no one is managing the conversation, our brains will just remember the last thing we heard.
    • The “roadmap” being the list of the HiPPO’s thoughts. The tactics you use to get to the outcomes you want should come from a variety of sources. If there’s only one, you have a problem. 

    Availability bias

    Relying on “what’s on hand” might seem like a smart practice, but it can lead to trouble:

    “The availability heuristic, also known as availability bias, is a mental shortcut that relies on immediate examples that come to a given person’s mind when evaluating a specific topic, concept, method or decision.” 

    I left this one for last because it’s a bit complicated. While the other two biases are almost always bad, this is one the pros can use to their advantage. Unfortunately, many teams can’t, which means we get stuck with the downsides, including lower decision quality. Our work isn’t a game show. How quickly you hit that buzzer doesn’t really matter — outcomes do. This bias leads to shallow outcomes and releases that may get even worse when combined with the other two biases mentioned which is an operational waste.

    Your team might be struggling with availability bias if: 

    • You’re reliant on quick decisions/buzzer logic. For example, you spend entire meetings talking about potential outcomes, only to take the first option available. 
    • Research is being ignored. Research almost always comes with curveballs, and if the decisions you come to haven’t considered them, you’ll end up with shallow outcomes. 

    The importance of time

    Biases come into play when teams don’t have time or space to carefully consider their decisions. In short, they are acting too fast with little time to process. If you feel that the team just moves from project to project, you’re most likely running right into bias traps. As a result, you’re probably making lower-quality decisions that can hurt your team long-term, both in how product development operates and when it comes to strategy.

    The solution is often simple: SLOW DOWN. Identify what’s important to the system you operate in and spend time paring away any processes that don’t make sense to your product strategy and operation. This includes projects as well. More often than not, it means getting rid of things and prioritizing so the team has time to think.

    This is “thinky work,”  and without allowing time for real thought, you’ll ultimately need luck for success. And I’ll end with a word of caution — luck is biased toward the patient and the prepared.