How Much Discovery is Enough-
- Published Date: 2022-07-07
- Never launch anything solely based on opinions
- It’s ok to release minor tweaks based on assessment
- Having supporting data (without testing) is enough to launch only very small, low-risk, easy-to-revert changes
- Everything else should be validated through tests and experiments.
How much validation you need depends on:
- the cost of building the idea in full
- how risky it is
- your risk tolerance.
Opinions (Confidence Level <= 0.1 out of 10)
This form of evidence is based on the self conviction of a single person or a small group of like-minded people (who aren’t the intended user).
Assessment (0.1 < Confidence Level ≦ 0.5)
- ICE analysis
- Projections and business models — Using back-of the envelope calculations, business model canvas, and other forms of business/tech analysis to refine our estimates of impact and ease.
- Risk analysis — For example through assumption mapping (for reference consider Cagan’s Four Big Risks)
- Reviews — Including peer reviews, expert reviews, management reviews, and stakeholder reviews.
Data (0.5 < Confidence Level ≦ 3)
- Customer/user requests
- Customer/user interviews
- Field studies
- Log and usage data
- Competitive research
- Market research
- Other sources
- Anecdotal data (Confidence Level of 1.0 or less) comes from a small set of data points: 1-3 users/customers expressing
- Market data (Confidence Level between 1.0 and 3.0) comes from larger datasets.
- 13 out of 20 potential customers we interviewed rated HDR-support as very important in their choice of video platform * 68% of consumer devices support HDR video * 3 out of 11 competitors have HDR support; 4 more have it on their roadmap
- Anecdotal data shows that at least someone outside the building
- We should use anecdotal evidence to OK just very low-effort, very low-risk ideas.
- but if we don’t have the capacity, these are examples of ideas where it might be ok to take the risk:
- Incremental features that add new functionality (rather than change or subtract), and are unlikely to annoy anyone, for example HDR support
- Redesigns of parts of the UI that users don’t visit often — e.g. the settings page
- Short-duration, in-product promos and calls-to-action
- We definitely don’t want to bloat the product with features no one uses.
Tests and Experiments (3 < Confidence Level)
- Testing means putting a version of the idea in front of users/customers and measuring the reactions.
- Early-stage tests use a “fake product” or a prototype before the product is built, for example landing pages, wizards of Oz, concierge services, and prototype usability studies.
- Mid-stage tests, for example internal team dogfood, early adopter programs, alpha, real product-based usability tests, use a rough and incomplete version of the idea with a small group of users.
- Late-stage tests (e.g. betas, labs, previews) use an almost-complete version of the idea with a larger group of users
- Split tests (e.g. a/b, multivariate) test complete versions of the idea with lots of users, and include a control element to reduce the odds of false results.
- Launch tests use partial roll-outs to test the finished idea on a large scale.
- questions about value in use, usability, and adoption and usage patterns. These are more expensive tests, and they’re enough to greenlight most medium risk and/or medium cost ideas.
- a new core feature, removing or replacing an existing feature, a major redesign, minor new product.
- 1) when the idea is going to affect all/most users in a big way
- and you want to ensure it’s bullet-proof, or 2) when some assumptions
- can only be validated with large amounts of data.
- use them for cases where you need quantitative data to validate the assumptions,
- Will people choose this option? Will we earn more money?
- Will retention not degrade?
- At a minimum we’d like to ensure that our most critical assumptions, the ones that may make the idea useless, are validated.
- We start with cheaper modes of validation (assessment, data gathering, early-stage tests), and move to more expensive ones only if the idea still looks good.
- Every time we collect new evidence we should reevaluate the idea and decide what to do next.