Shopify’s Archie Abrams on why churn matters, goal-setting without metrics, the “hundred-year mindset,” and integrating sales for a product-led growth model
I loved this conversation between Lenny and Archie—Shopify’s approach to experimentation is both fascinating and incredibly unique. I think it’s worth highlighting how Shopify’s dynamics likely mean that their high rate of false positives on tests wouldn’t apply to most SaaS companies.
Two key factors make Shopify’s situation so different:
1) High churn due to entrepreneurial failure rates: As Archie pointed out, Shopify’s mission to lower the barriers to entrepreneurship means that many merchants try but don’t succeed, regardless of Shopify’s value. This high churn introduces variability that can skew the results of experiments—short-term wins may just reflect temporary behaviors rather than lasting impact.
2) Power-law dynamics in cohort success: Shopify’s cohorts are driven by a few merchants who generate the majority of GMV and profits. Archie’s analogy to venture capital was spot on: these big winners create outsized results that can make short-term metrics unreliable.
For most SaaS businesses, this isn’t the case. They tend to have more consistent retention and less variability between their most and least valuable customers—particularly those customers that reach the "aha moment." Because of that, they’re less likely to see as many false positives from statistically significant tests.
I love how Shopify’s long-term holdout experiments align with their unique model, but for other companies, this level of scrutiny may not be as necessary. Thanks for such an insightful discussion—this really got me thinking about how growth strategies should be tailored to a company’s specific dynamics!
I loved this conversation between Lenny and Archie—Shopify’s approach to experimentation is both fascinating and incredibly unique. I think it’s worth highlighting how Shopify’s dynamics likely mean that their high rate of false positives on tests wouldn’t apply to most SaaS companies.
Two key factors make Shopify’s situation so different:
1) High churn due to entrepreneurial failure rates: As Archie pointed out, Shopify’s mission to lower the barriers to entrepreneurship means that many merchants try but don’t succeed, regardless of Shopify’s value. This high churn introduces variability that can skew the results of experiments—short-term wins may just reflect temporary behaviors rather than lasting impact.
2) Power-law dynamics in cohort success: Shopify’s cohorts are driven by a few merchants who generate the majority of GMV and profits. Archie’s analogy to venture capital was spot on: these big winners create outsized results that can make short-term metrics unreliable.
For most SaaS businesses, this isn’t the case. They tend to have more consistent retention and less variability between their most and least valuable customers—particularly those customers that reach the "aha moment." Because of that, they’re less likely to see as many false positives from statistically significant tests.
I love how Shopify’s long-term holdout experiments align with their unique model, but for other companies, this level of scrutiny may not be as necessary. Thanks for such an insightful discussion—this really got me thinking about how growth strategies should be tailored to a company’s specific dynamics!
Thanks for sharing Archie. Loved the really long hold out and the 100 year vision! Go Shopify!
Love it that you optimise for helping entrepreneurs get started time and time again.
Also agree the 4 Minute Mile is a great book. My old office was next to where the track was in Oxford and you could feel the magic in the air!