Interpret an A/B test result for a new ranking model
Interview insights are lockedSee how often this is asked and which companies asked it with Buncho Pro.You A/B test a new ranking model for two weeks. Each group has 50,000 users.
| Users | Conversions | Conversion rate | |
|---|---|---|---|
| Control (old ranker) | 50,000 | 2,000 | 4.00% |
| Treatment (new ranker) | 50,000 | 2,100 | 4.20% |
A colleague says: "p = 0.11, so there is an 89% chance the new model is better," and wants to ship. Another says: "Let's check daily and stop when it turns significant."
- Interpret the result correctly.
- What is wrong with both colleagues' reasoning?
- How should the experiment have been designed, and what would you recommend now?