Intro to A/B Testing
- Start here: Statistics for A/B testing
Guilherme covers the core statistical concepts of a/b testing: overall evaluation criterion (OEC), null hypothesis (Ho), p-value, significance level, standard deviation, minimum sample size, and confidence intervals.
How To Conduct an A/B Test
- Experiments at Airbnb
This article uses case studies to explain common pitfalls in A/B tests: hitting "significance" early (solution: calculate sample size for a treatment effect ahead of time), tracking the impact of changes in different contexts (example: different browsers), and making sure your A/B testing system works (using A/A dummy tests).
- Conservation of Intent: The Hidden Reason Why A/B Tests Aren’t as Effective as They Look
Andrew helps you figure out where to focus your a/b testing energy: on high-intent buyers toward the bottom of the funnel.
- Inside Growth at Wistia: The Process Behind Our A/B Tests
This article details the processes, documents, and tools that can be used to establish a culture of experimentation and A/B testing. It describes how to maintain a master ideas log, how to communicate an A/B testing roadmap, and how to log the details of a specific test.
What NOT To Test
Communicating A/B Test Results
- How We Talk About A/B Test Results
Laura provides two A/B testing case studies, one successful and one not, and then explains how to communicate the results. The key is to derive learnings from successes and failures.
A/B Test Case Studies
- The Tenets of A/B Testing from Duolingo’s Master Growth Hacker
This article covers 4 A/B test case studies that increased product engagement: delaying the sign-up screen, encouraging behavior via streaks, adding badges, and an encouraging mascot.
Technical Considerations of A/B Testing
- It’s All A/Bout Testing: The Netflix Experimentation Platform
This article provides a technical overview of how Netflix's homegrown A/B testing platform works, including the system architecture and all workflow steps between the A/B server and client.
- Under the Hood of Uber’s Experimentation Platform
Eva and her team go behind the scenes of Uber's experimentation platform that powers more than 1,000 tests per month. They explain various components of the system, including its statistics engine, statistics methodology, metric recommendation engine, and its sequential testing capabilities.