A/B Testing
Intro to A/B Testing
Start here: Statistics for A/B testingGuilherme covers the core statistical concepts of a/b testing: overall evaluation criterion (OEC), null hypothesis (Ho), p-value, significance level, standard deviation, minimum sample size, and confidence intervals.
How To Conduct an A/B Test
Guidelines For Ab TestingEmily provides the do's and don'ts of A/B testing, with lessons learned from dozens of experiments from Etsy.
Experiments at AirbnbThis article uses case studies to explain common pitfalls in A/B tests: hitting "significance" early (solution: calculate sample size for a treatment effect ahead of time), tracking the impact of changes in different contexts (example: different browsers), and making sure your A/B testing system works (using A/A dummy tests).
Conservation of Intent: The Hidden Reason Why A/B Tests Aren’t as Effective as They LookAndrew helps you figure out where to focus your a/b testing energy: on high-intent buyers toward the bottom of the funnel.
Inside Growth at Wistia: The Process Behind Our A/B TestsThis article details the processes, documents, and tools that can be used to establish a culture of experimentation and A/B testing. It describes how to maintain a master ideas log, how to communicate an A/B testing roadmap, and how to log the details of a specific test.
What NOT To Test
The Agony and Ecstasy of Building with DataJulie provides 3 pitfalls when working with data, and 4 pitfalls when performing a/b tests.
Communicating A/B Test Results
How We Talk About A/B Test ResultsLaura provides two A/B testing case studies, one successful and one not, and then explains how to communicate the results. The key is to derive learnings from successes and failures.
A/B Test Case Studies
The Tenets of A/B Testing from Duolingo’s Master Growth HackerThis article covers 4 A/B test case studies that increased product engagement: delaying the sign-up screen, encouraging behavior via streaks, adding badges, and an encouraging mascot.
Technical Considerations of A/B Testing
It’s All A/Bout Testing: The Netflix Experimentation PlatformThis article provides a technical overview of how Netflix's homegrown A/B testing platform works, including the system architecture and all workflow steps between the A/B server and client.
Under the Hood of Uber’s Experimentation PlatformEva and her team go behind the scenes of Uber's experimentation platform that powers more than 1,000 tests per month. They explain various components of the system, including its statistics engine, statistics methodology, metric recommendation engine, and its sequential testing capabilities.
