Is there a way to check the result of A/B test to determine the better one.
Suppose,
Group A has 100 views and 10 clicks.
Group B has 10 views and 2 clicks.
How to chose which one produces more clicks ?
$\begingroup$ $\endgroup$
3 
- 3$\begingroup$ Of course. Many ways. One is to do a likelihood ratio test (if you have to choose one). Another is to do a significance test (if you don't; maybe the difference is due to chance). In open-ended scenarios you can use Thompson sampling to sequentially exploit the best option to the degree that it is performing better. $\endgroup$Emre– Emre2018-01-05 02:06:59 +00:00Commented Jan 5, 2018 at 2:06
- $\begingroup$ @Emre what do you consider the difference between a likelihood ratio test and a significance test? $\endgroup$Dave– Dave2020-02-08 03:45:24 +00:00Commented Feb 8, 2020 at 3:45
- $\begingroup$ The former compares multiple hypotheses. The latter potentially rejects a single hypothesis. Two approaches, reflecting different philosophies, were developed back in the day, and became somewhat conflated. A reading of the history might bring clarity. $\endgroup$Emre– Emre2020-02-11 01:17:22 +00:00Commented Feb 11, 2020 at 1:17
Add a comment |
1 Answer
$\begingroup$ $\endgroup$
There are ways of understanding which of these is the right one to go with. Unforunately 10 views means no calculation is going to be great for you, but i'll try to explain both.
T-tests and Z-tests are used by a lot of AB Testing tools, as described here, to calculate winners. You'll be able to input views and clicks, and receieve back stats that describe if Group B's treatment is to win. You get back a yes/no answer, based on tolerances provided.
Bayesian calculations, on the other hand, rely on a percentage likelihood of being a winner, after which you can make a judgement.
If possible, you should run this for longer and try to drive more views to Group B. If the interaction rate of 20% persists, you'll see an eventual winner there (100% lift over the 10% rate for Group A).
