Following this paper from VWO I have implemented the following model for revenue in a subscription business:
The revenue generated by user $i$ is given by:
$$ \alpha_i \leftarrow Bernoulli\left(\lambda\right) $$ $$ r_i \leftarrow Expon\left(\theta\right) %$$ $$ v_i \leftarrow \alpha_i r_i $$
where $ \alpha_i \in \{0, 1\}$ is an r.v. representing whether user $i$ bought anythign and $r_i \in \mathrm{R}^+$ is the size of the purchase. Finally $v_i$ is the size of the purchase.
In a given A/B test there $n_A$ users in variation A, and $c_A$ sales.
$$ s_A = \frac{1}{c_a}\sum_{k=1}^{c_A} s_A^K $$
Is the revenue per sale in variation A
We assume a $\mathrm{Beta}\left(a, b\right)$ prior on $\lambda$ and a $\mathrm{Gamma}\left(k, \Theta\right) $ on $\theta$. Then the total posterior on $\left(\lambda, \theta\right) $ is
$$ P\left(\lambda, \theta \mid n, c, s\right) = \mathrm{Beta}\left(a+c, b+n-c\right) \mathrm{Gamma}\left(k+c, \frac{\Theta}{1+\Theta c s}\right) $$
From this posterior you can calculate things like:
$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$
by direct simulation.
However, I am interested in comparing multiple hypotheses. In this case, whether
$$ P\left( \lambda_A > \lambda_B \right) $$
and
$$ P\left( \theta_A > \theta_B\right) $$
and
$$ P\left( \frac{\lambda_A}{\theta_A} > \frac{\lambda_B}{\theta_B}\right) $$
Simultaneously.
Andrew Gelman in this paper suggest using a heirarchical regression model to soove the multiple comparisons problem. So would a model where I put a hyperprior over $(\lambda_A, \theta_A)$ and $(\lambda_B, \theta_B)$ be sufficient?
Something like:
The revenue generated by user $i$ in group $j$ is given by:
$$ \alpha_{ij} \leftarrow Bernoulli\left(\lambda_j\right) $$ $$ r_{ij} \leftarrow Expon\left(\theta_j\right) $$ $$ v_{ij} \leftarrow \alpha_{ij} r_{ij} $$ $$ \lambda_j \sim \mathcal{D}\left(\lambda, \sigma_{\lambda}\right) $$ $$ \theta_j \sim \mathcal{D}\left(\theta, \sigma_{\theta}\right) $$
where $ \mathcal{D} $ is some suitable distribution I haven't figured out yet.
