When I started out in product management, it was still the era of packaged software. We literally shipped products — a package with a CD and a printed manual — to each customer. Since all software was used and installed onsite with our clients, we had almost no visibility into how it was used. I resorted to leading training classes just to have some insight into how users interacted with the product.
Today, the story is different. SaaS and mobile applications throw off tons of data, and as a result, there are lots and lots of things that can be measured. The first data mistake is, of course, not measuring anything at all. But just because something can be measured doesn’t mean that it should. This article takes a quick look at some of the common pitfalls that product teams run into when thinking through how to measure product performance in a world awash in data.
-
Not Measuring Actual Engagement
There is a lot of data that can be captured from user behavior in digital products. However, some of the easiest things to measure are also the least actionable. Measurements such as total registered users, raw page views, or app downloads, are all easy to capture. Yet none of these is particularly helpful to product teams, or a relevant measure of success without also understanding the manner and frequency of users engagement with a product.
Whether someone has registered or downloaded your app doesn’t tell you if they’re realizing value. They may be responding to a marketing promotion or new free trial. Consider taking these baseline measurements a bit further – looking not just a total breadth of users, but also the frequency with which they use the product (i.e. after signing up do they continue to use it), and how much (what features) of the product they use.
-
Ignoring Qualitative Feedback
Whatever you’re measuring, don’t stop talking to customers. With the relative ease — and in many cases low expense — of capturing behavioral data in digital products, it’s easy to rely on this data at the expense of actual customer feedback. But, behavioral data is often missing important context that can help clarify specific decisions about your product.
Consider a simple example of an underused feature: You ship a new feature, and measure adoption to find that it’s not being picked up nearly as much as you expected. Now what? Without additional context, it’s really hard to decide what to do next. It could be that users simply aren’t aware of the feature, that they can’t understand how to use it, or that they don’t perceive value. What you would do to the product is very different for each of those scenarios. Qualitative feedback provides that all-important “why” to the user behavior that you observe, and it helps you make much better prioritization decisions.
-
Over-reliance on Vocal Customers
Customer feedback is a critical piece of product data, but not all feedback is equal. Every product has that specific set of users who are all too happy to provide vocal feedback to any question. These users are valuable, but they often aren’t representative of the entire population. In fact, the very things that make them vocal are often the same things that make them a poor source of feedback. They may be using your product for unique or edge use cases which causes them to run into more issues than the majority of customers. They may be less tech-savvy and struggle more with usability, or conversely, power users who are stretching the boundaries of product APIs and integrations.
Because these users readily provide feedback, it’s tempting to reach out to them over and over again rather than try to coax information out of your less vocal customers. This over-reliance distorts feedback and prevents you from accurately gauging how your products provide value to the majority of users.
-
Focusing on Downstream Metrics Only
Most product teams report using revenue as one of their primary metrics. This, of course, makes total sense, since product revenue is the ultimate benchmark of success.
Revenue and customer retention are downstream metrics. They are the result of a product experience that has (or hasn’t) delivered value. By the time a bad experience shows up as lost revenue or customer churn, it’s too late for the product team to do anything about it – those customers are already gone.
Revenue and retention should never be ignored, but they should be paired with more upstream metrics that track user engagement on a shorter time horizon. If login frequency or depth of features used decreases for a particular user cohort, you can react before those customers are lost, and engage with them to identify why they’re not realizing value from the product.
-
Measuring Too Many Things
There’s plenty of data that you can collect about your products. Web and mobile apps can be instrumented in lots of ways, and that data can be paired with customer surveys, satisfaction ratings, and a bevy of downstream financial metrics. All of this data can be valuable but can result in kind of an “analysis paralysis” if too many of them are tracked at a time. Too many conflicting metrics make it hard to identify trends or improvement opportunities and can result in a lot of time spent simply gathering and reporting on data.
Ultimately, you want to find the measures that are most indicative of the current health of the product. Start small and scale your KPIs. Over time you will learn what measures matter, and which don’t provide actionable insight. If a metric isn’t proving helpful – stop collecting it. There’s no point to collect data for data’s sake. A tight, proven set of product metrics will make you more effective, and help you better promote your successes to your leadership team and board.
Building an Effective Product Data Strategy
As a product manager, the first product I ever launched failed horribly. Ultimately, over several releases, we were able to add enough value-delivering features to make the product a success, but it was a painful process. Today, with the right sets of measurements, it’s much easier to diagnose and address a product’s shortcomings. With the right KPIs in place, I’d like to think that our product would have been successful much more quickly. For product teams today, it’s much easier to capture and leverage product data to launch, improve, and refine digital products. By carefully capturing the right metrics (and avoiding some of the pitfalls discussed here), you can end up with an effective product data strategy.
