Debate

Which Data Yields Better Product Insights?

Data. We struggle because we don’t have enough of it. We struggle because we have so much and we don’t know how to interpret it. Product managers wrangle an incredible amount of data, of different kinds and from different sources: behavioral analytics captured in the app, NPS, conversion funnels, heat maps, customer interviews, cohort analysis. And the list goes on. If it’s all quantitative we don’t trust it to reflect real humans. If it’s all qualitative, we worry that it’s too anecdotal. We make the case for more data-driven decisions, but we also talk a lot about the role of intuition in product design.

If it’s delight we’re trying to achieve among users, can quantitative data really get us there? Can a slew of customer interviews really yield a plan of action or do those just obfuscate the path to the next feature?

This week in our poll, we asked you which type of data yields better product insights. 66% of you thought that qualitative data is the way to go, but among our debaters, qualitative data was treated somewhat more cautiously. What has your experience been? Tell us in the comments.

Qualitative

The best product insights come from running qualitative research methods and data analysis. From my experience, best insights came from being first grounded in data, before structuring the qualitative analysis in the most efficient way. Ultimately, though, the real breakthroughs are not purely data-driven. You won't disrupt anything or innovate much if data is your only guide.

Maria Thomas
VP Products, Insightly
Combined Qual+Quant

Quantitative data is very powerful in analyzing users' responses to your-already developed products mainly in the form of A/B tests, BI analysis, and counting support calls and feature requests. The challenge is that in some cases you need lots of users and time to get significant results. Personally, I hate surveys (and hardly know a case were they help). And I have little trust in quantitive future product predictions (we all know the sad story of the survey of how much people are willing to spend on mobile phones). Qualitative -- in the form of talking to users -- is powerful (among other things) in understanding how do users feel about things, which can be very helpful with prioritization. It's really best to combine approaches. For example, use quant to analyze your funnel to identify drops, and then use qual to ask why. Talk to a few users, get some ideas, and then go back to quant and run an A/B test to make sure that qualitative answers didn't mislead you.

David Schwartz
VP Product, Wix
It depends

Well, of course, the answer is quite easy: it depends what you're looking for. Qualitative data is critical to uncover market knowledge, patterns, and trends; however, by itself is anecdotal. This data must be coupled with quantitative data to validate the pervasiveness of that pattern or trend. So, if I am looking to discover market patterns and trends, gathering qualitative data rules the day, but if I want to ensure I build scalable solutions that cater to markets full of customers vs. one customer at a time, quantitative data is your best friend.

Kirsten Butzow
Product Coach, Pragmatic Marketing
User Comments