A/B Test Planning
Planning an A/B test for an educational technology company.
Purpose
The client planned a multi-phase A/B test to evaluate new features on their educational technology platform. The initial phase involved exploratory qualitative research, including user interviews and surveys, to identify key features of interest. This input was then triangulated with established learning theories to prioritize which features would be included in the subsequent structured A/B test.
My Role
I led the statistical design and planning for the A/B test. A key challenge was determining the unit of randomization, as randomizing at the classroom level (rather than the student level) could be logistically easier but risked compromising the study’s integrity. I developed and ran statistical power simulations to quantify this trade-off. The simulations demonstrated that randomizing at the classroom level would have drastically reduced statistical power, making it difficult to detect meaningful effects.
Due to implementation, the randomization needed to occur at the classroom level and the number of classrooms recruited were adjusted to account for the lower statistical power. We adjusted our analysis using classroom-clustered robust standard errors. This approach allowed us to account for the correlation of outcomes within classrooms and ensure the accuracy of our statistical inferences.