A/B Testing Guide: 15 Experiments Every Website Should Run

Most A/B tests produce no result. Not because the hypothesis was wrong, but because there was never enough traffic to produce one.

The maths nobody does beforehand

To detect a 10% improvement from a 2% conversion rate, you need roughly 30,000 visitors per variant. For a 5% improvement, four times that.

If your page gets 2,000 visitors a month, a correct test would take over two years. Any conclusion drawn earlier is noise.

Work this out before the test. If the number is impossible, don't run it — make a reasoned change and move on.

What to test when you don't have traffic

Big changes, not small ones. A completely new page versus the old one can produce 30–50% differences, which are detectable with far fewer visitors than 5%.

High-volume steps. Don't test the thank-you page. Test the page everyone lands on.

Measure differently. Customer interviews, session recordings, exit surveys. They don't give statistical certainty, but they give direction — and direction is what you're actually missing.

Mistakes that invalidate the test

Stopping when it looks good. Conversion rates swing wildly in the first days. Stop whenever variant B is ahead and you'll always "win" and never grow.

Testing five things at once. Change the headline, image, button and price, get a good result, and you don't know what worked — so you can't repeat it.

Running under a week. Behaviour differs by weekday. A Tuesday-to-Thursday test measures Tuesday to Thursday.

Ignoring segments. A variant can lose overall and clearly win on mobile. The average hides that.

What to do with an inconclusive result

Accept it. "No difference" is a valid result and tells you the hypothesis didn't matter. Don't convert it into "nearly significant" to justify the change.

Where to start

Not with the test. With the question "where do most people drop out". Look at the funnel, find the biggest leak, and test there. Tests born of curiosity don't produce growth.