Statistical significance is a determination that a relationship between two or more variables in a dataset is caused by something other than chance. A result is typically considered statistically significant when the p-value is less than the chosen significance level (alpha), most commonly 0.05 (5%). In practice, it means your observed difference is unlikely to have occurred due to random variation alone.

A p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis (no difference) is true. A p-value of 0.03 means there is only a 3% probability the observed difference occurred by chance. A p-value below 0.05 is the conventional threshold for declaring a result statistically significant. Critically, a low p-value does not tell you the size or practical importance of the effect — only that it is unlikely to be random.

A one-tailed test checks for an effect in one specific direction only — for example, testing whether a variant is better than the control. A two-tailed test checks for any difference in either direction (better or worse). Two-tailed tests are more conservative, requiring stronger evidence to declare significance, and are the standard choice for most A/B tests. Use a one-tailed test only if you have a strong, pre-registered reason to expect a change in just one direction and are not concerned about effects in the other direction.

The required sample size depends on four factors: your baseline conversion rate, the minimum detectable effect (MDE) you care about, your desired statistical power (typically 80%), and your significance level (typically 5%). As a rough guide: detecting a 10% relative improvement on a 5% baseline conversion rate requires approximately 15,000 visitors per variant. Smaller effects require dramatically larger samples. Always calculate sample size before starting the test, not after.

Statistical power is the probability that your test correctly rejects the null hypothesis when a real effect actually exists — in other words, the probability of detecting a true difference. A power of 80% means there is an 80% chance of finding a significant result if the effect is real. Low power tests risk false negatives (missing real effects). Power increases with larger sample sizes, larger effect sizes, or higher significance levels.

For most business A/B tests, a 95% confidence level (α = 0.05) is the industry standard. Higher-stakes decisions — such as major pricing changes or medical interventions — may warrant 99% confidence. For fast-moving iterative tests with low downside risk, some teams accept 90% confidence to reach decisions more quickly. The key rule: set your confidence level before running the test, not after seeing the results, to avoid p-hacking.

A z-score measures how many standard deviations the observed difference is from zero (the null hypothesis of no difference). For a two-tailed test at 95% confidence, a z-score beyond ±1.96 indicates statistical significance. For a one-tailed test at 95% confidence, the critical z-value is 1.645. Larger absolute z-scores correspond to lower p-values and stronger evidence against the null hypothesis.

No. Statistical significance only tells you that an observed difference is unlikely to be due to random chance. It does not, by itself, establish causation. However, in a properly randomised and controlled A/B test — where users are randomly assigned to control or variant and all other variables are held constant — a statistically significant difference in conversion rates does provide strong evidence of a causal relationship between the change and the outcome. Observational studies require much more caution in causal interpretation.