Data-Informed
Quantitative vs Qualitative Tools. When should you use each?
Being a product manager means having a diverse toolkit. That’s a good thing: different situations call for different types of analysis. Often at the same time.
On the other hand, having too many tools can lead to a kind of paralysis. Which tool to use when? Which approach best fits the scenario you’re in?
Knowing which analytics strategy to adopt is a kind of art. But it’s an art that can definitely be learned. We’re here to help!
So which should you use - quantitative tools (like product analytics), or qualitative tools (like session replay or heatmaps)? Let’s find out.
First things first: focus on Product-Market-Fit
A key outcome for every product team is Product-Market Fit (PMF). As you’re likely aware, PMF can be a fickle, moving target. The point where a product satisfies real customer needs and gets them willing to pay can change as the product evolves. The sweet spot is constantly shifting.
In general, teams have two research methods that help find PMF: the qualitative and the quantitative.
Qualitative product research involves talking with customers and listening to their stories and experiences. It focuses on the needs of a few users, which it takes as representing the whole of the user base. Qualitative research helps us understand what customers really want in a deep and meaningful way.
Quantitative product research, on the other hand, is all about using data to measure and analyze customer behaviors and preferences. It’s great for analyzing trends and seeing what different groups of users do. Unlike qualitative research, quantitative tools give us a more objective picture of how well a product is doing.
Qualitative and Quantitative Research Frameworks
The challenge is knowing when to use each method and how to combine their insights to find Product-Market Fit. In the past, this was tough, but now we have plenty of thought leadership and tools to help product managers.
We recently interviewed Dan Olsen, a product management expert, to learn more about these approaches.
Check out the full interview between Dan Olsen and Heap CEO, Ken Fine.
According to Dan, “There are two fundamental ways to do research to minimize that uncertainty and learn.” Dan describes qualitative tools as the 'Oprah' approach, and quantitative tools as the 'Spock' approach.
Image courtesy of Dan Olsen
Qualitative - The Oprah Approach
In Dan’s words:
“I want to describe at a high level what each of these is really all about. So qualitative research or learning is when you actually care about what each person has to say. You want to sit down one-on-one with them and talk to them and have an interactive conversation with them to understand the nuances of what their problems are, what they care about, and what's important to them.
You're not going to hit statistical significance when you're doing qualitative research, even if you talk to 10 or 20 or 30 people.
If you think about someone from television who's really good at sitting down with someone one-on-one and staring into their eyes and interviewing them and getting them to open up, it's Oprah Winfrey. There are people that have interviewed with her and cried. There are people who say it’s like she was staring into their souls. And they cried in a good way because she got them to open up.
So that's kind of the qualitative approach, personified by Oprah. Each user is a snowflake! Of course, we want a pattern match across multiple users. But with qualitative research, you want that rich nuance of data.”
Quantitative - The Spock Approach
Dan continues:
“In contrast, for quantitative research, you don't care about any one user. You want thousands of data points, or hundreds of thousands, or millions! The point isn’t to be crass - it’s that with quantitative research your goal is to reach statistical significance.
It's are very different mindset. In this case, the fictional character from tv to think about is the master of logic and analysis: Spock. Spock wants data. He wants a lot of data! Spock’s goal is to prove things. He’ll use math and analytics to get to the answer.”
The key for Product Managers is to know when to use each approach and how to combine the insights to drive key aspects of the product life cycle.
Choosing the right mix, for the right product phase
Using both qualitative and quantitative research methods throughout the product life cycle has big benefits for finding Product-Market fit. Before hitting the market, qualitative research digs deep into customer pain points and desires through interviews and surveys. This helps shape the product to match what customers really want.
After launching, quantitative research takes over, crunching numbers to track user engagement, retention, and conversions. This data-driven approach guides product managers in making smart decisions to keep the product successful and competitive.
Combining these two methods gives a complete picture and leads to better products and long-term success.
Choosing the right platform for your research needs
As we circle back to the key question a product manager has to answer on their journey to Product-Market fit, we turn to Dan again.
“If we broaden our perspective, Christen Rohrer, a UX leader, presents an excellent framework that combines quantitative and qualitative aspects with the additional dimensions of attitudinal and behavioral factors.
The difference between the two can be demonstrated in this example. If I ask you how often you go to the gym you might say “once a week” (attitudinal), but if I observe your behavior I’ll learn it’s more like “once a month” (behavioral).
Low left corner: This is strictly qualitative, one on one interviews, Voice of the Customer (VOC) feedback - sharing their opinion directly.
Upper left: This is still qualitative but it is behavioral. Here you are observing people one at a time, and listening to what they say. This is where pure usability testing and session replay would live. The reality is any time you’re going to do usability testing - you’re going to get both attitudinal and behavioral elements.
Lower right: We switch to the quantitative realm. Now we’re not talking to each user one on one like Oprah - we are in the “Attitudinal” space. This means employing a way of getting many customers' opinions at scale. This is where surveys live.
For example, NPS scores are a good example for this. There’s a reason it’s so popular because there’s one question - with a scale. “How likely are you to recommend this product to a friend?” and then you can pick values between 1-10. I’ve seen how surveys can be misused in the past, but NPS score helps clear the noise. But then you want to get the WHY - why did you give us a 10 or 5?
Upper right: Still quantitative, this time behavioral. Here you are collecting data at scale and this is where analytics, A/B testing, and experimentation live. You’re looking at real product usage data, at scale, and derive statistically significant insights.”
This is an effective way to break down the different tools your team needs to carry out the different types of research, across different product phases. In a phase, you would need to lean in on different methodologies and techniques to find PMF and validate your hypothesis.
Recap
As you explore the various research methodologies needed for each phase of your product development, remember that the real power lies in combining both quantitative and qualitative data.
When seeking tools and platforms to support your team's agile research, opt for those that seamlessly integrate both approaches. By doing so, you'll gain a more comprehensive understanding of what's happening and why, allowing you to uncover valuable insights that pave the way for faster and better product development.