0
$\begingroup$

I am testing splitting phone calls to a service between two IVR menu's to determine which messaging is more effective. 80% to the main IVR menu (Group a) and 20% to an alternative (Group b).

The data I have is Group a: Total calls, Transfer rate (10,000, 15%) and the same for group b (2000, 10%).

I want to determine whether the result is statistically significant.

So H0: No change H1: the change has been effective

As I have two proportions, should I be using a z-score? and can I assume the data is normally distributed with only these proportions?

My working so far is:

p1 = 15%, n = 10,000, sd:SQRT(0.15(1-0.15)/10,000) = 0.00357

p2 = 10%, n = 2,000, sd = 0.00671

z = (0.1-0.15)/0.00671 = -7.4535, pvalue = 3.4^-E13

Is this approach correct? If not, what should I be doing to determine whether the change has been effective?

Thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ "Group a: Total calls, Transfer rate (10,000, 15%) and the same for group b (2000, 10%). I want to determine whether the result is statistically significant. " ... if by "result" you mean to ask whether the proportions are more different than could be accounted for by random variation, plainly they are. That much difference with that large a sample size? of course the population proportions are different (unless your significance level is extremely tiny, I guess). Effectiveness would seem to at least be in part a matter of effect size, rather than significance -- I'd focus on tat $\endgroup$ Commented Jun 2, 2017 at 6:56

1 Answer 1

Reset to default
0
$\begingroup$

Your Z value is wrong (the denominator should be the square root of the pooled variance), but it hardly matters ... if by "result" you mean to ask whether the proportions are more different than could be accounted for by random variation, plainly they are. That much difference with that large a sample size? of course the population proportions are different! What would be the purpose in testing such a plain difference?

Effectiveness would seem to at least be in part a matter of effect size, rather than significance -- I'd focus on that (though the size seems like it would be practically important as well).

Improve this answer
$\endgroup$
2

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.