Unlock 2025 Benchmark data → Access insights to stay ahead in the digital experience race.

Get the Report
skip to content
Loading...
    • Why Product Analytics And what can it do for you?
    • How Heap Works A video guide
    • How Heap Compares Heap vs. competitors
    • Product Analytics + Digital Experience Analytics A deeper dive
    • The Future of Insights A comic book guide
    Watch a Demo
  • Data Insights

    • Journeys Visual maps of all user flows
    • Sense AI Analytics for everyone
    • Web Analytics Integrate key web metrics
    • Session Replay Complete context with a single click
    • Heatmaps Visualize user behavior instantly
    • Heap Illuminate Data science that pinpoints unknown friction

    Data Analysis

    • Segments User cohorts for actionable insights
    • Dashboards Share insights on critical metrics
    • Charts Analyze everything about your users
    • Playbooks Plug-and-play templates and analyses

    Data Foundation

    • Capture Automatic event tracking and apis
    • Mobile Track and analyze your users across devices
    • Enrichment Add context to your data
    • Integrations Connect bi-directionally to other tools

    Data Management

    • Governance Keep data clean and trusted
    • Security & Privacy Security and compliance made simple
    • Infrastructure How we build for scale
    • Heap Connect Send Heap data directly to your warehouse
  • Solutions

    • Funnel Optimization Improve conversion in user flows
    • Product Adoption Maximize adoption across your site
    • User Behavior Understand what your users do
    • Product Led Growth Manage PLG with data

    Industries

    • SaaS Easily improve acquisition, retention, and expansion
    • Retail and eComm Increase purchases and order value
    • Healthcare Build better digital patient experiences
    • Financial Services Raise share of wallet and LTV

    Heap For Teams

    • Product Teams Optimize product activation, conversion and retention
    • Marketing Teams Optimize acquisition performance and costs
    • Data Teams Optimize behavioral data without code
  • Pricing
  • Support

    • Heap University Video Tutorials
    • Help Center How to use Heap
    • Heap Plays Tactical how-to guides
    • Professional Services

    Resources

    • Down the Funnel Our complete blog and content library
    • Webinars & Events Events and webinar recordings
    • Press News from and about Heap
    • Careers Join us

    Ecosystem

    • Customer Community Join the conversation
    • Partners Technology and Solutions Partners
    • Developers
    • Customers Stories from over 9,000 successful companies
  • Free TrialRequest Demo
  • Log In
  • Free Trial
  • Request Demo
  • Log In

All Blogs

Data-Informed

Building a Retention Strategy, Part 3: How to test the strength of your ideas with an Impact Calculator

January 5, 20246 min read
  • Facebook
  • Twitter
  • LinkedIn

If you’re concerned about developing an effective retention strategy, this series is for you. In this and future posts, we’ll walk through the key ways to use data to make sure you’re building a smart, scalable retention motion. Most will also include worksheets you can download and use with your team.

In Part 1, we showed you how to establish Active Usage Metrics for the value of your product equate to value for your business.

In Part 2, we showed you how to connect these metrics to your ultimate goal—revenue—by designing a metrics tree.

-------------------------

Today, our next step is to figure out what kinds of interventions will make the biggest difference. In the past, you would have just guessed at this randomly. But there are smarter ways to improve your accuracy! At Heap, we use this Impact Calculator. It’s a simple tool for quickly understanding the potential impact of any project so you can compare the value of different initiatives you’re considering.

PRO TIP: This is also a great exercise for making sure you’re tracking all your metrics, both general and local.

As a PM, you have difficult choices to make!

These days, everything is about business impact. When faced with the prospect of moving the needle on important topics like revenue and retention, PMs are typically overwhelmed (and even strong-armed) with suggestions and requests on what paths they should pursue. But you only have so much time, energy and budget at your disposal, so it’s critical to know how to evaluate the best choices in order to invest their team's time and energy.

Our goal for you: To have hypotheses at the ready about which interventions will move the needle most effectively and efficiently for the greatest ROI.

There are two main questions that you need to ask:

1) “If we make a specific change, how many users will be exposed to it?”

2) “How much will that change impact our users’ behavior?”

Imagine the advantages if you could answer these questions before taking on a project.

  • What approaches would seem most profitable?

  • What adventurous new steps might you take?

  • What old tactics would you abandon?

  • What new ideas might light up in your team’s imaginations?

Meet your new favorite tool: The Impact Calculator

Very simply, impact calculators assess the potential rewards and consequences of different actions and scenarios. Decision makers can now make informed choices about optimizing resources by understanding the risks and benefits associated with various decisions.

In our application, the Impact Calculator evaluates your ideas for changes in a user flow by measuring their effect on a higher-level KPI. You start by analyzing the current baseline measurements for your flow, and then predict how your proposed changes will impact those baseline measurements. The calculator does this by tracking how the two most important functions of the change will affect the baseline metrics.

There are many ways to move a metric, but the success of any initiative at improving a KPI ultimately depends on these key factors—Scope and Magnitude. 

Scope = Scale of the audience

An improvement that impacts 1,000 users will move the overall KPI more than a similar improvement that only impacts 100 users. Makes sense, right? In general, the larger the audience, the better.

Magnitude = Scale of the action

An improvement that impacts the user experience more deeply and meaningfully will make a larger impact. It sounds obvious, but every PM has encountered proposals for changes that minimally affect the user experience, but are presented as something that will make a significant impact, like color changes and button placements. Will that be true?

The Impact Calculator has three basic components:

  1. Baseline metrics for the overall experience.

  2. Baseline metrics for the part of the experience you wish to change.

  3. Estimated guess for the new rate.

How to use it

Note: looking for a more detailed walkthrough? We have a full post on that.

1. Start with your metrics tree. What have you identified as most important?

2. Now let’s look at either the core metrics (breadth, depth, etc) or the active usage metrics for your product. What you choose will depend on the size of your product, your product area, etc.

3. Brainstorm potential ways to move these metrics! There are lots of things to think about here. They can be tiny and incremental or a whole new approach. You don’t have to assume that minor change = minor impact. Remember, that’s what the calculator is for!

What are some of the approaches you can try?

Send more emails

Alter flows in the product

Change messaging in the product

Add more push notifications

Make a tour video

Just like your other brainstorming efforts, the sky's the limit! Go nuts here.

4. Now you have a list of great ideas. It’s time to take the most promising ones and run them through the calculator!  

NOTE: You will be doing this more than once! The calculator is simply a way to call attention to your most viable ideas and weed out your least viable approaches.

An example

Let's look at a hypothetical situation and see how the team might use an Impact Calculator to determine which intervention they should attempt.

You start with an overall retention rate of 80%. You have 10,000 customers.

Instead of changing notifications in the product to nudge people to use a certain feature, you’re considering sending an email campaign to a select group of people.

Let’s imagine that 500 people will receive the email campaign, and 1000 will see the nudge in the product. 

  • Of those 500, the overall retention rate is 40%

  • Of those 1000, the retention rate is 60%

(You should know this from your Heap data, your Marketo data, or wherever you pull data from.)

For the last step of the Impact Calculator, you’ll need to make an educated guess.

So you make these hypotheses:

  • With the group of 500, you can raise the retention rate from 40% to 55%

  • With the group of 1000, you can raise the retention rate from 60% to 65%.

You can base this assumption on prior experiments and your product knowledge. (When in doubt, it’s best to be conservative with your estimate.) You can also try different values to see how sensitive your outcome is to the assumptions you are making.

Using the calculator, you can see the overall increase is 0.45% vs 0.2%. It makes more sense to implement the email campaign.

Some caveats

As a good PM, you know there are an infinite amount of changes you can make to a digital experience. So don’t just settle on your first idea. Come up with many approaches, make new estimates, and see which idea performs best. Accuracy through volume is a good approach here!  

You’re always making estimates of what you think the change in retention rate will be. Feel free to be flexible with your numbers and see how they compute.

You may receive a directional signal instead of a definitive answer. But that in itself can be super useful! 

The overarching goal is just to be smart about what you’re doing. Build the habit of paying attention to your markers, and notice what they mean.

Use the Impact Calculator to measure the difference in performance between your ideas so you can make the best investment in terms of not only capital but your team’s time, energy, focus and motivation.

Collect your measurements in advance, make hypotheses, and see what works! And have fun with this and push your boundaries, because at this stage there’s no risk to getting anything wrong.

In our upcoming Retention Strategy Posts, we’ll cover prioritizing your book of business, and orchestrating your scaled CS strategies.

Was this helpful?
PreviousNext

See Heap in Action

Explore our on demand demo videos to learn more about Heap and see how modern data analytics can unlock better insights, faster.

Watch Now

Related Stories

See All

  • Creative visualization of AI CoPilot capability
    article

    Heap announces new generative AI CoPilot

    Heap, the leader in product analytics, unveils AI CoPilot’s open beta today.

  • Heap.io
    article

    What’s Next in Experience Analytics?

    What does the future of analytics hold, and what does it mean for you?

  • Heap.io
    article

    Building a Retention Strategy, Part 2: Connecting Activities to Revenue with a Metrics Tree

    If you read one post from this series, it should be this one.

Better insights. Faster.

Request Demo
  • Platform
  • Capture
  • Enrichment
  • Integrations
  • Governance
  • Security & Privacy
  • Infrastructure
  • Heap Illuminate
  • Segments
  • Charts
  • Dashboards
  • Playbooks
  • Use Cases
  • Funnel Optimization
  • Product Adoption
  • User Behavior
  • Product Led Growth
  • Customer 360
  • SaaS
  • Retail and eComm
  • Financial Services
  • Why Heap
  • Why Product Analytics
  • How Heap Works
  • How Heap Compares
  • ROI Calculator
  • The Future of Insights
  • Resources
  • Blog
  • Content Library
  • Events
  • Topics
  • Heap University
  • Community
  • Professional Services
  • Company
  • About
  • Partners
  • Press
  • Careers
  • Customers
  • DEI
  • Support
  • Request Demo
  • Help Center
  • Contact Us
  • Pricing
  • Social
    • Twitter
    • Facebook
    • LinkedIn
    • YouTube

© 2025 Heap Inc. All Rights Reserved.

  • Legal
  • Privacy Policy
  • Status
  • Trust