How long should an A/B test run?
How long an A/B test should run, planned before it starts, in whole weeks, with the smallest change your traffic can detect worked out for 2, 4 and 8 weeks.
Read the guideA/B test calculator
Enter how many people would see the test each week and how often they convert. The table shows the smallest change a test of each length can find.
How the calculator works
An A/B test splits people 50/50 between the original (A) and one change (B). The calculator asks how big a difference between them a test can tell apart from chance, with a two-sided test at 95 % confidence and 80 % power.
It uses the arcsine method, the same maths as the planner in MIRA FIVE: B must reach a rate of sin²(asin√p + 1.4 × √(2 / n)), where p is today’s rate and n the people in each variant. The smallest change is that rate against p.
At 4 % and 1,000 people a week, a 2-week test finds only a change of about +70 %; 4 weeks about +48 %; 8 weeks about +33 %. Finding +10 % would take about 39,400 people per variant. Plan the length before you start, run whole weeks, and decide once at the end.
The calculator assumes every person is new each week. MIRA FIVE’s planner projects from your own last 28 days, so returning people count once.
Plan it, run it, decide once
In MIRA FIVE you pick 2, 3, 4, 6 or 8 weeks, see what the test can find before it starts, and get one verdict at the planned end plus its window, on the goal you chose.
Questions and answers
Why not stop the test as soon as B looks better?
Checking a running test and stopping at the first good-looking day finds winners that are not there. Plan the length first and read the result once at the end.
Why whole weeks?
People buy differently on a Monday and a Saturday. A test that runs whole weeks gives each weekday the same weight in both variants.
How does this compare with MIRA FIVE’s planner?
Same maths: two-sided 95 %, 80 % power, the arcsine method. The planner projects people from your own last 28 days; this calculator takes a weekly number you enter.
Is anything I type sent anywhere?
No. It calculates in your browser; nothing is sent or stored.