A/B test

An A/B test puts a change to the test before you keep it. Half the people see the original (A), half see the change (B), and the test compares how often each group does what you want: buys, signs up, clicks the button. Chance decides who sees which version, so a clear difference between the groups can be put down to the change.

Two habits decide whether the answer holds. Plan the length before you start, from the smallest change worth finding. Then read the result once, after the planned end: checking every day and stopping at the first good-looking number turns noise into winners that were never there.

In MIRA FIVE

A/B tests live under Experiments, on the same data and script as your analytics. Each one is the original against one change, split 50/50 and assigned by browser or by signed-in person; a share of traffic and conditions can narrow who is in it. One goal or action of the project decides it, and buyers and revenue are shown beside it.

The length is planned up front: 2, 3, 4, 6 or 8 weeks (at most 3 on Free) plus a window of 1, 7, 14 or 30 days, and a planner shows the smallest change that length can find. The verdict is taken once, after the planned end plus the window, never while counting, and needs at least 100 people per variant. It reads Variant B wins, worse, no big difference or no clear winner.

A page experiment swaps elements marked in your HTML through a snippet in the page head. A code experiment is a feature flag your app reads, in the browser or on your server. In the browser, only people who consented are counted, and nothing is stored or counted before they do.

Example

The Acme Shop tested its checkout button for 4 weeks plus a 7-day window, decided by “Order paid”. 49 of 1,204 people who saw the original reached it (4.1%), and 71 of 1,188 who saw Variant B (6.0%). The result: Variant B wins.