Metrics Hygiene

Thanks again to Jonathan Yeo for sharing this great article on how to think about metrics.

Call it a mission or a raison d’être or a value statement or whatever. It should be summable in a single sentence. There should be one not just for the entire company, but also for every team and every project within that company. When people hear this, they should feel inspired. Like it’s something they’d be excited to get and make happen every morning.

If your team, org, or company doesn’t have this, and instead has a bunch of metrics, it might be time to take a step back and think about how to better articulate the “why do we do what we do?” more clearly.

Then, you can start executing on the mission by breaking it down into its requisite milestones, each with its own set of metrics or measurements to help serve as guideposts for how well things are going.

Some rules of thumb for good metrics hygiene:

These are some of my learnings in my quest to become more and more disciplined about the tactics of good goal setting and measurement.

To assess for product-market fit, look at retention. Do not look at the sheer number of people using your product or feature (which can be skewed by things like how aggressively you promote it.) Retention best correlates with whether your product is valuable because it tells you whether people who tried it liked it enough to return and use it again.
To optimize for growth, understand your funnel. In order for people to become regular users of your product, they have to pass through a bunch of hurdles. First, they have to be aware of your product. Second, they have to be interested enough to check it out. Third, they have to convert (download an app, fill out a form, confirm e-mail, etc.) Fourth, they have to do enough within your product to understand why it might be valuable in their lives. Fifth, they have to remember to come back. At each of these steps, you will lose people. If you can track and measure what that rate of loss is, you can then start to figure out where to focus your efforts to make your funnel less leaky.
Figure out which metrics are truly important, and focus on those. It’s tempting to get into the state where you track everything (because you can), and you have a dashboard filled with numbers that all feel like they should be green. Recognize that most things don’t matter, and that only a small handful actually do. Don’t waste time talking about the unimportant stuff, and don’t sweat letting some of the less important metrics go up or down.
To figure out the best metric to track, use the magic-wand technique. Ask yourself: “If I could wave a magic wand and know anything about my users in the world, what would I most want to know to tell me whether my app will be successful?” Even if your answer is not something you can actually measure (“Is my app suggesting recommendations that my users find valuable?”), it is a helpful starting point to work from. (“Okay… so I can’t ask every user if the recs were valuable… but if it were valuable, I’d probably see them saving or sharing recs more, and they’d probably spend more time reading recs, and…etc, etc.”)
Don’t just accept a metrics goal without understanding it. I can’t emphasize this enough: the goals you and your team agree to will be hugely impactful to your work, so make sure you buy into them. Do not accept metric goals at face value. Ask why. Ponder whether or not they make sense, and what behaviors they will incentivize. Are there situations where something will feel like a good decision but the metric doesn’t move? Conversely, are there situations where you could imagine the metric going up a lot but not be convinced that the product is actually better? If so, would another metric (or set of metrics) do a better job of tracking what actually matters?
View data skeptically by suggesting countermetrics. If the data is showing you what look like good results, ask yourself: “What else can I look at to convince me that these results aren’t as good as they seem?” These are called countermetrics, and every success metric should have some. (For example, don’t look at click-through rate without looking at the number of fast bounces back, don’t look at the sales numbers of a product without looking at how many returns or cancellations there are, etc.) It’s much better to be paranoid about interpreting data so you can quickly catch your mistakes and adjust your strategy. Don’t fall into the trap of confirmation bias where you’re just looking for signals that prove your intuitions are right.
Use qualitative research to get at the why. Quantitative data that tells you what people did is best paired with qualitative research that give you insight into how people felt. Conduct usability testing, utilize focus groups, and run surveys to get at the why behind the behavior you’re seeing.