Continuous Discovery Habits - Book Summary

**

Torres, Teresa. Continuous Discovery Habits: Discover Products that Create Customer Value and Business

At a minimum, weekly touchpoints with customers 

By the team building the product 

Where they conduct small research activities 

In pursuit of a desired outcome

Relentless focus on customers was a better strategy than obsessing about our competitors.

the work that you do to decide what to build as discovery and the work that you do to build and ship a product as delivery.

“product trio” will refer to a product manager, a designer, and a software engineer

Opportunity solution trees have a number of benefits. They help product trios: 

  • Resolve the tension between business needs and customer needs 

  • Build and maintain a shared understanding of how they might reach their desired outcome 

  • When a team takes the time to visualize their options, they build a shared understanding of how they might reach their desired outcome. If they maintain this visual as they learn week over week, they maintain that shared understanding, allowing them to collaborate over time.

  • Adopt a continuous mindset 

  • Unlock better decision-making

  • Instead of framing our decisions as “whether or not” decisions, this book will teach you to develop a “compare and contrast” mindset. Instead of asking, “Should we solve this customer need?” we’ll ask, “Which of these customer needs is most important for us to address right now?” We’ll compare and contrast our options. Instead of falling in love with our first idea, we’ll ask, “What else could we build?” or “How else might we address this opportunity?” 

  • Unlock faster learning cycles

  • By visually mapping out the opportunity space on an opportunity solution tree, a product trio is making their understanding of their customer explicit. When a solution fails, they can revisit this mapping to quickly revise that understanding. 

  • Build confidence in knowing what to do next 

  • Unlock simpler stakeholder management

Six mindsets underlying continuous discovery

  1. Outcome-oriented: you define success as the value that code creates for your customers and for your business (the outcomes). Rather than measuring value in features and bells and whistles, we measure success in impact—the impact we have had on our customers’ lives and the impact we have had on the sustainability and growth of our business.

  2. Customer-centric: The second mindset places the customer at the center of our world. It requires that we not lose sight of the fact (even though many companies have) that the purpose of business is to create and serve a customer. We elevate customer needs to be on par with business needs and focus on creating customer value as well as business value.

  3. Collaborative: The third mindset requires that you embrace the cross-functional nature of digital product work and reject the siloed model, where we hand off deliverables through stage gates. Rather than the product manager decides, the designer designs, and the engineer codes, we embrace a model where we make team decisions while leveraging the expertise and knowledge that we each bring to those decisions.

  4. Visual: The fourth mindset encourages us to step beyond the comfort of spoken and written language and to tap into our immense power as spatial thinkers. The habits in this book will encourage you to draw, to externalize your thinking, and to map what you know. Cognitive psychologists have shown in study after study that human beings have an immense capacity for spatial reasoning.

  5. Experimental: The fifth mindset encourages you to don your scientific-thinking hat. Many of us may not have scientific training, but, to do discovery well, we need to learn to think like scientists identifying assumptions and gathering evidence.

  6. Continuous: And finally, these habits will help you evolve from a project mindset to a continuous mindset. Rather than thinking about discovery as something that we do at the beginning of a project, you will learn to infuse discovery continuously throughout your development process.

customer needs, pain points, and desires collectively as “opportunities”—they represent opportunities to intervene in our customers’ lives in a positive way. (the opportunity space to represent the problem space as well as the desire space)

(we have many examples of products or services that don’t fix problems. Disneyland entertains me. Ice cream is delicious. Mountain biking is fun. These products address my desires.)

most important steps for reaching our desired outcome are first, how we map out and structure the opportunity space, and second, how we select which opportunities to pursue.

  • The root of the tree is your desired outcome—the business need that reflects how your team can create business value. 

  • Next is the opportunity space. These are the customer needs, pain points, and desires that, if addressed, will drive your desired outcome. 

  • Below the opportunity space is the solution space. This is where we’ll visually depict the solutions we are exploring. 

  • Below the solution space are assumption tests. This is how we’ll evaluate which solutions will help us best create customer value in a way that drives business value.

When a team takes the time to visualize their options, they build a shared understanding of how they might reach their desired outcome.

“Which of these customer needs is most important for us to address right now?” 

“What else could we build?” or “How else might we address this opportunity?”

However, most of the decisions that we make in discovery are reversible decisions. If we do the necessary work to test our decisions, we can quickly correct course when we find that we made the wrong decision. This gives us the luxury of moving quickly, rather than falling prey to analysis paralysis.

As they explore potential solutions, they learn more about the problem, and, as they learn more about the problem, new solutions become possible.

“Based on my current understanding of my customer, I thought this solution would work. It didn’t. What did I misunderstand about my customer?” We then need to revise our understanding of the opportunity space before moving on to new solutions.10 When we do this, our next set of solutions get better.

business outcomes, product outcomes, and traction metrics. A business outcome measures how well the business is progressing. A product outcome measures how well the product is moving the business forward. A traction metric measures usage of a specific feature or workflow in the product.

Business outcomes start with financial metrics (e.g., grow revenue, reduce costs), but they can also represent strategic initiatives (e.g., grow market share in a specific region, increase sales to a new customer segment). Many business outcomes, however, are lagging indicators. They measure something after it has happened. It’s hard for lagging indicators to guide a team’s work because it puts them in react mode, rather than empowers them to proactively drive results.

product trios will make more progress on a product outcome rather than a business outcome. Remember, product outcomes measure how well the product moves the business forward. By definition, a product outcome is within the product trio’s span of control. Business outcomes, on the other hand, often require coordination across many business functions.

Assigning product outcomes to product trios increases a sense of responsibility and ownership. If a product team is assigned a business outcome, it’s easy for the trio to blame the marketing or customer-support team for not hitting their goal. However, if they are assigned a product outcome, they alone are responsible for driving results.

However, we can increase the accountability of each team by assigning a metric that is relevant to their own work. In this example, we might ask the product team to increase the number of dogs who like the food (something within the product team’s span of control), whereas we might ask the marketing team to increase the transparency of the pricing after the trial ends, and we might ask the customer-support team to decrease their average response times. All three groups are contributing to the business outcome of increasing customer retention, but each is doing so in the way that they can best contribute.

The key is to use traction metrics only when you are optimizing a solution and not when the intent is to discover new solutions.

This research suggests that product trios, when faced with a new outcome, should first start with a learning goal (e.g., discover the opportunities that will drive engagement) before being tasked with a performance goal (e.g., increase engagement by 10%).

The purpose of customer interviewing is not to ask your customers what you should build. Instead, the purpose of an interview is to discover and explore opportunities.

The key to interviewing well is to distinguish what you are trying to learn (your research questions) from what you ask in the interview (your interview questions).

Our primary research question in any interview should be: What needs, pain points, and desires matter most to this customer?

the best way to learn about their needs, pain points, and desires is to ask them to share specific stories about their experience. You’ll need to translate your research questions into interview questions that elicit these stories. Memories about recent instances are more reliable than our generalizations about our own behavior or our answers to direct questions.

Instead of asking, “What criteria do you use when purchasing a pair of jeans?”—a direct question that encourages our participant to speculate about their behavior—we want to ask, “Tell me about the last time you purchased a pair of jeans.” The story will help us uncover what criteria our participant used when purchasing a pair of jeans, but because the answer is situated in a specific instance (an actual time when they bought jeans), it will reflect their actual behavior, not their perceived behavior.

you would like them to share their full story with you, to share as many details as possible, to leave nothing out, and that, when they are done with their story, you’ll ask for missing details.

An interview snapshot is a one-pager designed to help you synthesize what you learned in a single interview.

If the participant requests a specific feature or solution, ask about why they need that, and capture the opportunity (rather than the solution). A good way to do this is to ask, “If you had that feature, what would that do for you?” For example, if an interviewee says, “I wish I could just say the name of the movie I’m searching for,” that’s a feature request. If you ask, “What would that do for you?” they might respond, “I don’t want to have to type out a long movie title.” That’s the underlying need.

We don’t want to think about interviewing as a step in a linear process. Instead, our goal is to interview continuously.

  • Automate the Recruiting Process

  • Recruit Participants While They Are Using Your Product or Service

  • Ask Your Customer-Facing Colleagues to Recruit

  • If a customer calls to cancel their subscription, schedule an interview. 

  • If a customer has a question about feature x, schedule an interview. 

  • If a customer requests a customization, schedule an interview.

If you want to build a successful product, you need to understand your customers’ actual behavior—their reality—not the story they tell themselves.

We should compare and contrast the impact of addressing one opportunity against the impact of addressing another opportunity.

However, it can be hard to prioritize a flat list of opportunities, because opportunities come in different shapes and sizes. Some opportunities are interrelated, while others are subsets of others.

Instead of managing an opportunity backlog, we’ll use an opportunity solution tree (introduced in Chapter 2) to help us map out and understand the opportunity space. The tree structure will help us visualize and understand the complexity of the opportunity space.

Trees depict two key relationships—parent-child relationships and sibling relationships. Both will help us make sense of the messy opportunity space. The parent-child relationship will be used to represent subsets—a child opportunity (or sub-opportunity) is a subset of a parent opportunity.

Siblings should be similar to each other—in that they are each a subset of the same parent—but distinct in that you can address one without addressing another.

The value of breaking big opportunities into a series of smaller opportunities is twofold. First, it allows us to tackle problems that otherwise might seem unsolvable. And second, it allows us to deliver value over time. That second benefit is at the heart of the Agile manifesto and is a key tenet of continuous improvement. Rather than waiting until we can solve the bigger problem—“Is this show any good?”—we can deliver value iteratively over time.

We need each opportunity to be distinct from every other opportunity.

visualize what you already know about your customers’ experience. The output of this exercise was an experience map that showcases what your customers do to address their needs today. You learned that the map can help direct the stories that you collect in your interviews.

Is this opportunity framed as a customer need, pain point, or desire and not a solution? 

Is this opportunity unique to this customer, or have we seen it in more than one interview? 

If we address this opportunity, will it drive our desired outcome?

For example, “The interface is hard to use” is way too broad of an opportunity. We could boil the ocean trying to address every usability issue in our product. Instead, we want to get more specific. Where did this pain point show up in your customer’s story? This may turn into several opportunities: “I can’t figure out how to find a specific show that I have in mind,” “I don’t like entering a show name using the remote,” or “I can’t remember what episode I watched last.”

Start by grouping similar opportunities together. Similar opportunities might be siblings,

If your similar opportunities are siblings, look for a parent opportunity.

we need to frame opportunities from our customers’ perspective. No customer would ever say, “I wish I had more streaming-entertainment subscriptions.” But they might say, “I want access to more compelling content.” Review each opportunity on your tree and ask, “Have we heard this in interviews?” If you had to add opportunities to support the structure of your tree, you might ask, “Can I imagine a customer saying this?” Or are we just wishing a customer would say this?

If your top-level opportunities represent distinct moments in time, then no opportunity should have two parents. If you are finding that an opportunity should ladder up to more than one parent, it’s framed too broadly.

Opportunities that represent themes, design guidelines, or even sentiment, aren’t specific enough. “I wish this was easy to use,” “This is too hard,” and “I want to do everything on the go” are not good opportunities. However, if we make them more specific, they can become good opportunities: “I wish finding a show to watch was easier,” “Entering a movie title using the remote is hard,” and “I want to watch shows on my train commute” are great opportunities.

When we capture opportunities like “I’m frustrated” or “I’m overwhelmed,” we limit how we can help. We can’t fix feelings. But if we capture the cause of those feelings—“I hate typing in my password every time I purchase a show” or “I’m way behind on this show”—we can often identify solutions that address the underlying cause.

“The build trap is when organizations become stuck measuring their success by outputs rather than outcomes. It’s when they focus more on shipping and developing features rather than on the actual value those things produce.”

Product strategy happens in the opportunity space. Strategy emerges from the decisions we make about which outcomes to pursue, customers to serve, and opportunities to address.

Assessing a Set of Opportunities

  • Opportunity sizing helps us answer the questions: How many customers are affected and how often?

  • Market factors help us evaluate how addressing each opportunity might affect our position in the market. Depending on the competitive landscape, some opportunities might be table stakes, while others might be strategic differentiators.

  • Company factors help us evaluate the strategic impact of each opportunity for our company, business group, or team. We want to prioritize opportunities that support our company vision, mission, and strategic objectives. 

  • Customer factors help us evaluate how important each opportunity is to our customers. If we interviewed and opportunity mapped well, every opportunity on our tree will represent a real customer need, pain point, or desire. However, not all opportunities are equally important to customers.

When assessing and prioritizing the opportunity space, it’s important that we find the right balance between being data-informed and not getting stuck in analysis paralysis.

However, creativity research tells us that our first idea is rarely our best idea. Researchers measure creativity using three primary criteria: fluency (the number of ideas we generate), flexibility (how diverse the ideas are), and originality (how novel an idea is).

Similar research shows that fluency is correlated with both flexibility and originality. In other words, as we generate more ideas, the diversity and novelty of those ideas increases. Additionally, the most original ideas tend to be generated toward the end of the ideation session.

Study after study found that the individuals generating ideas alone outperformed the brainstorming groups. Individuals generated more ideas, more diverse ideas, and more original ideas.

In group brainstorming sessions, people lose ideas amid the chaos of everyone sharing ideas in rapid succession.

studies that showed alternating between individual ideation and group sharing of ideas can improve the quality of ideas generated in subsequent individual ideation sessions.41 Exposure to other people’s ideas did inspire new ideas.

Participants started by ideating on their own. Then they shared their ideas with the group. Then they went back to ideating on their own. They never ideated as a group, but they received the benefit of hearing each other’s ideas.

  1. Review your target opportunity. Make sure that everyone on your team knows what it means and is familiar with the necessary context. Make sure it’s distinct from the other opportunities that you’ve discussed and that it’s an appropriately sized leaf-node opportunity. If you need to revisit any of these concepts, review the prior chapter before continuing. 

  2. Generate ideas alone. Take some time to jot down as many ideas as you can. When you get stuck, take a break, and come back to it. If you are still stuck, try to find inspiration from your competitors and analogous products. For analogous products, think broadly. You aren’t limited to just the other players in your space. 

  3. Share ideas across your team. You can do this in a face-to-face, real-time meeting, or you can do it asynchronously in a digital chat channel (e.g., Slack or Microsoft Teams). The key is to take the time to describe each of your ideas, allow people to ask questions, and to riff on the ideas. 

  4. Repeat steps 2 and 3. Remember, the benefit of sharing your ideas is that hearing other people’s ideas will inspire even more ideas. So, don’t skip repeating step 2 to ensure that you reap the rewards. Repeat until you’ve generated between 15 and 20 ideas for your target opportunity. Remember, research shows that your first ideas are rarely your best ideas. The goal is to push your creative output to find the more diverse and more original ideas.

“Does this idea solve the target opportunity?” (dots)

expectations, teams competent in modern discovery techniques can generally test on the order of 10–20 iterations per week.”

The escalation of commitment46 is a bias in which the more we invest in an idea, the more committed we become to that idea.

Product teams are particularly susceptible to confirmation bias and the escalation of commitment. We tend to fall in love with our ideas.

Best product teams complete a dozen or more discovery iterations every week. This pace is possible only when we step away from the concept of testing ideas and instead focus on testing the assumptions that need to be true in order for our ideas to succeed.

Desirability assumptions: Does anyone want it? Will our customers get value from it? As we create solutions, we assume that our customers will want to use our solution, that they will be willing to do the things that we need them to do, and that they’ll trust us to provide those solutions. All of these types of assumptions fall into the desirability category.

Viability assumptions: Should we build it? There are many ideas that will work for our customers but won’t work for our business. If we want to continue to serve customers over time, we need to make sure that our solutions are viable—that they create a return for our business. This typically means that the idea will generate more revenue than it will cost to build, service, and maintain.

Feasibility assumptions: Can we build it? We primarily think about feasibility as technical feasibility. Is it technically possible? Feasibility assumptions, however, can also include, “What’s feasible for our business?” For example, will our legal or security team allow for it? Will our culture support it? Does it comply with regulations?

Usability assumptions: Is it usable? Can customers find what they need? Will they understand how to use it or what they need to do? Are they able to do what we need them to do? Is it accessible?

Ethical assumptions: Is there any potential harm in building this idea? This is an area that is grossly underdeveloped for many product trios. As an industry, we need to do a better job of asking questions like: What data are we collecting? How are we storing it? How are we using it? If our customers had full transparency to those answers, would they be okay with it?

One of the best ways to align as a team around what your ideas mean is to story map them. Story mapping is a popular technique in which teams map out each step end-users have to take to get value from a product or service. Story mapping forces you to get specific about how an idea will work and what you expect your end-users will do. While many teams use story mapping to align around product requirements, it’s also a great technique for helping us to surface underlying assumptions.

Use Your Story Maps to Generate Assumptions

Start by assuming the solution already exists. You aren’t story mapping what it will take to implement an idea. Instead, you are mapping what end-users will do to get value from the solution once it exists in the world.

Identify the key actors. Who needs to interact with whom for the idea to work?

Map out the steps each actor has to take for anyone to get value from the solution. Be specific. What does each actor need to do in order for someone to get value from the solution?

Sequence the steps horizontally over time. Lay out the steps horizontally one after the other. Sequence them in the order they need to happen.

If an end-user can choose multiple paths, map out the successful path. If there are multiple successful paths, map them out sequentially.

Suppose we are working at our streaming-entertainment company, and we are exploring three different solutions (setting up a good compare-and-contrast decision) for the target opportunity “I want to watch live sports.” Integrate local networks (e.g., ABC, CBS, NBC) into our service License broadcast rights directly from the different sports leagues and serve the sporting events up ourselves Bundle our streaming service with a partner who streams live sports To story map our first idea—integrating local networks—

Throughout your story map, every time you assume that an end-user will do something, you are making desirability assumptions (i.e., that your user wants to do what you are asking them to do and that they are willing to do it), usability assumptions (i.e., your user understands what needs to be done and is able to do it), and feasibility assumptions (i.e., you can build whatever is required to support each step of the map). You can literally go step by step through your story map and generate dozens of assumptions.

From our first step of the story map, “Our subscriber comes to our platform to watch sports,” we can generate the following assumptions: 

  • Desirability: Our subscriber wants to watch sports. 

  • Desirability: Our subscriber wants to watch sports on our platform. 

  • Usability: Our subscriber knows they can watch sports on our platform. 

  • Usability: Our subscriber thinks of our platform when it’s time to watch sports. 

  • Feasibility: Our platform is available when our subscriber wants to watch sports.

From our second step, “Our platform has live sports options for our subscriber to choose from,” we can generate the following assumptions: 

  • Desirability: Our platform has the sports our subscriber wants to watch. 

  • Usability: Our subscriber can find where to go on our platform to find sports. 

  • Feasibility: We know what sports are available right now. 

  • Feasibility: We can display what sports are available right now.

time to generate many assumptions, we’ll increase the likelihood that we’ll uncover the risky ones.

Conduct a Pre-Mortem

“Imagine it’s six months in the future; your product or initiative launched, and it was a complete failure. What went wrong?” As you generate reasons for why your product or service might fail, you are exposing assumptions that your idea depends upon that may not be true.

Another way to generate assumptions, particularly viability assumptions, is to use your opportunity solution tree to work backwards from your solution back to your outcome. You can start by generating assumptions using the following starters: This solution will address the target opportunity because… Addressing the target opportunity will drive the desired outcome because…

Why will your solution address the target opportunity? Your answer will contain one or more assumptions that you’ll want to capture. For example, “Adding local channels will allow our subscribers to watch live sports” because “The sports that our subscribers want to watch are on local channels” because “Most of the major sports are on local channels” and “Our subscribers are more likely to want to watch major sports.” Each phrase in quotes is an assumption we can test.

When you are generating assumptions, always phrase your assumptions such that you need them to be true: “Customers will remember their passwords.” For many assumptions, you’ll find that this positive framing will make them easier to test.

TESTING ASSUMPTIONS, NOT IDEAS

We run countless tests with little impact. We forget to clearly define what we are trying to learn and what success looks like, leaving us with ambiguous results.

Instead, we want to systematically collect evidence about our assumptions underlying all three ideas. The more we learn about each idea, the more likely we are to compare and contrast the ideas against each other. This helps us make better decisions about which ideas are most promising. Remember—we are looking for a clear front runner. As you read through the rest of this chapter, remember: We aren’t testing one idea at a time. We are testing assumptions from a set of ideas.

A strong assumption test simulates an experience, giving your participant the opportunity to behave either in accordance with your assumption or not. This behavior is what allows us to evaluate our assumption. To construct a good assumption test, you’ll want to think carefully about the right moment to simulate. You don’t want to simulate any more than you need to. This is what allows you to iterate quickly through several assumption tests.

” It’s common for ideas to share assumptions. It’s one of the reasons why assumption testing is faster than idea testing. Assumption tests don’t merely give us a go/no-go decision for an individual idea; they help us evaluate sets of ideas.

criteria. Instead of saying, “Some people choose sports,” we want to say, “At least 3 out of 10 people choose sports.” We want to define both how many people we’ll test with and how many people need to exhibit the behavior that we expect to see.

Now remember, you aren’t trying to prove that this assumption is true. The burden of truth is too much. You are simply trying to reduce risk.

team on the smallest assumption test you can design that still gets you results that the team will feel comfortable acting

You might be tempted to test with larger pools of people to help get buy-in. But this strategy comes at a cost—it takes more time. We don’t want to invest the time, energy, and effort into an experiment if we don’t even have an early signal that we are on the right track.

We stop testing when we’ve removed enough risk and/or the effort to run the next test is so great that it makes more sense to simply build the idea.

A false positive is when our test gives us data suggesting that our assumption is true, when it isn’t.

“false negative.” Our test is providing data that indicates our assumption is faulty when it may not be.

Our goal as a product team is not to seek truth but to mitigate risk. We need to do just enough research to mitigate the risk that our companies cannot bear and no more.

Many assumptions can be tested with quick answers to a single question. This is where one-question survey tools can be tremendously helpful. If we wanted to test the “Our subscribers want to watch sports” assumption in more than one way, we could launch a one-question survey asking them, “When was the last time you watched a sporting event?” We could use their answers to triangulate with our prototype simulation.

Before you dive into the data, be sure to define your evaluation criteria upfront. How many search queries will you sample? How many need to be related to sports? How will you determine “related to sports”? Remember, aligning around success criteria upfront guards against confirmation bias and ensures that your team agrees on what the results mean.

The world is full of good ideas that will succeed on some level. However, an outcome-focused product trio needs to stay focused on the end goal—driving the desired outcome. We need to remember to measure not just what we need to evaluate our assumption tests, but also what we need to measure impact on our outcome.

Start by instrumenting what you need to collect to evaluate your assumption tests. As you build your live prototypes54, consider what you need to measure to support your evaluation criteria. Don’t worry about measuring too much beyond that. For example, in the story that opens this chapter, we had several assumptions we needed to test: Students will start more searches if we ask them easier questions. Students will view jobs that we recommend. Students will apply to jobs that we recommend.

250 out of 500 visitors will start their search using our new interface. (Remember, we were seeing only 180 out of 500, or 36%, start their search on our old interface. We wanted to see a big jump in search starts to warrant such a different interface.) At least 63 of our 500 students will view at least one job. (Our current interface was performing at 81 out of 500. We set our initial criteria lower, because we knew our canned searches weren’t perfect, and we were confident we could improve them over time.) At least 7 of our 500 students would apply for a job. (Our current interface was performing at 12 out of 500. Again, we set our initial criteria lower because we knew our results would get better over time.)

product outcome was to improve search starts, and we succeeded at doing that. But our business outcome was to increase the number of students

Overcommitting to an opportunity. Throughout your discovery, you will uncover opportunities that are important to your customers that you won’t be able to deliver on.

Avoiding hard opportunities. Some teams interpret continuous delivery to mean continuous delivery of easy solutions. Quick wins have a time and a place in our work. If we can deliver impact this week, we should. However, many of the opportunities we uncover will take time to address adequately. Don’t confuse quick testing and iterative delivery with easy solutions.

Drawing conclusions from shallow learnings. As you learned in Chapter 2, discovery requires strong critical-thinking skills. Otherwise, it’s easy to draw fast conclusions from shallow learnings.

Giving up before small changes have time to add up. While you do want to measure the impact of your product changes, don’t expect to see large step-function results from every change. Oftentimes it takes a series of changes to move the needle on our outcome.

**

Enjoy this post? Buy me a Coffee at ko-fi.com

Notes mentioning this note