A/B test
noun
An A/B test is an experiment in which two groups each see a different variant. Comparing their behaviour shows which variant performs better against a predefined goal.
We used an A/B test to find out which wording encouraged more visitors to submit the form.
An A/B test compares two versions of a web page, button or feature. One group of users sees variant A and the other sees variant B. You then measure how each variant affects a chosen goal, such as registrations, purchases or completed tasks.
How do you run an A/B test?
Decide in advance what you want to investigate and which metric you will use to assess the result. Preferably change one clearly defined element. If you change a button's wording, colour and position at the same time, you will not know which adjustment caused the difference.
Users are assigned to the variants at random. This reduces the chance that differences between the groups affect the result. Both variants run over the same period, so a public holiday or temporary campaign does not affect just one of them. A Glossary · In brieffeature flagA feature flag is a setting that enables or disables software functionality without another code deployment. It can apply to all users or a particular group.Read more can determine which variant a user receives.
When are the results useful?
A difference in the measurements does not immediately mean that variant A or B is truly better. A small group can produce an unusual result by chance. The number of participants and the time needed depend on factors such as the expected difference, the current conversion rate and the required level of statistical confidence.
Set the duration, target group and success criterion in advance. Stopping as soon as one variant takes the lead can produce a misleading conclusion. A well-designed A/B test helps inform a specific decision, but does not automatically explain why users behave differently. That may require further user research.