🍀 The Art and Science of Subscription Experimentation

I work with the smartest people I’ve ever met. And, today, I’d like for you to meet my coworker and friend, Juan Cruz Giusto. There is nobody, and I mean it, nobody that I trust more for CRO and ecommerce experimentation. If you’ve met him, you feel the same way. 

I asked begged Juan to take over Go-to-Millions today. Because it’s about time we talked about subscription ecommerce testing at length.

LOVE YOU JUAN. THANK YOU!!!! You can find him on X here, LinkedIn here, and please do yourself a favor and subscribe to Juan’s newsletter now.

Without further ado, please read Juan’s words (titled The Art and Science of Subscription Experimentation) and don’t you dare try to steal him from me. ♾️ ♾️ ♾️ :

THE ART AND SCIENCE OF SUBSCRIPTION EXPERIMENTATION 

If you’re reading this, you’re probably familiar with CRO & A/B testing:

Test a new landing page or product page change, measure conversion rate or ARPU, implement the winner.

Pretty straightforward.

However, subscription businesses are trickier.

Consider this scenario

You run a test trying to bust subscriptions versus one-time purchases, making the latter more appealing:

Here are the results:

• Conversion Rate: +1.25%
• AOV: -7%
• ARPU: -5.75%
• Subscription rate: 50%+

In a typical test, you’d immediately declare this a failure—you’re generating almost 6% less revenue per website visitor.

This means your ROAS takes a hit.

But with subscription eCommerce, what matters is the longer-term impact of acquiring more subscribers.

So you pause the experiment and monitor it for 2 months.

After 60 days, the variant’s LTV exceeds the control by +7%.

BINGO!

And the numbers will continue improving since subscribers generate higher LTV than one-time buyers.

When testing on subscription eCommerce, conversion rate and ARPU tell only part of the story.

The crucial metric is Average Revenue per User after 30/60/90 days, which factors in:

• Conversion Rate
• AOV
• Retention Rates
• And a definite time window (60 days, for example)

HOW TO MAKE IT HAPPEN

To test subscriptions effectively, you need:

1. Integrate your A/B testing platform with your eCommerce store

At Sharma Brands we use Intelligems and Shopify. Very easy to use and if you have your COGS uploaded, you can evaluate profit too.

2. Tag customers based on which test version they encountered

With Flows and Intelligems, you can do it pretty easily. Just follow this simple tutorial and you are good to go.

3. Use an analytics tool to track each cohort’s LTV progression.

If you have a Data Analytics team on your brand, you can create a Looker Studio that pulls data from Shopify automatically and filters customers based on tags.

If you don’t have that luxury, you can always export the data into an Excel or Google Sheets and calculate each cohort’s total revenue.

4. Conduct statistical tests to compare variant and control performance

Since revenue is a continuous metric – contrary to conversion rate that is binary – the usual statistics calculator won’t suffice.

At Sharma Brands, we use this one created by Andrea Corvi, that lets you analyze the test results in few steps.

The only extra element you’ll need to calculate is each cohort’s standard deviation, which can be done through an Excel Formula or uploading your data to ChatGPT and asking it to calculate it for you.

That’s it!

Hopefully, you are now armed to start experimenting with subscriptions and explore other venues than conversion rate and AOV.

And if you want to really take this to the next level, you can measure contribution margin per cohort by discounting COGS and CPA, but that should probably be a topic for another send.

In short, subscription experimentation is exciting but complex.

Get it wrong, and you risk implementing changes that harm your long-term bottom line.

Get it right, and watch your retention rate, LTV, and profit soar.

————-

I told you, Juan is so fucking smart. This is the tip of the iceberg. Watching him do this for 40 hours a week isn’t enough for me.

Follow him on X, on LinkedIn, and definitely read his newsletter. If you do – you’ll know so much more about subscription ecom, experimentation, CRO, and honestly, any subject Juan sets his mind to.