Testing hypotheses cleanly — statistics, sample sizes, lessons. Tools: GrowthBook, Statsig, Optimizely.
Category · Data & Analytics
Hypotheses, not gut feel.
An A/B test randomly splits users into variants and measures which one moves a predefined metric better. Experimentation is the framework around it: a clean hypothesis, a sufficient sample, a clear significance level. Tools like GrowthBook, Statsig and Optimizely provide the infrastructure.
The point isn't to be proven right, but to be able to disprove assumptions before they ship as a feature into production.
How we set tests up.
We calculate the required sample size before the start and fix the end before we see any data. We like to use GrowthBook when the analysis should dock onto your own warehouse — the data then stays in your own stack.
When a test is worthless.
With too little traffic, a valid test takes months — then you're better off deciding by judgement and qualitative feedback. And whoever stops the test the moment the numbers look briefly good is measuring chance, not effect. An aborted test is worse than none, because it creates false certainty.