8
$\begingroup$

The first section of this article provides some intuition on base rate fallacy with p-values. In particular, it uses as example a cancer test.

Suppose I am testing a hundred potential cancer medications. Only ten of these drugs actually work, but I don’t know which; I must perform experiments to find them. In these experiments, I’ll look for p<0.05 gains over a placebo, demonstrating that the drug has a significant benefit.

It then calculates a hundred hypothesis tests and concludes that

Because the base rate of effective cancer drugs is so low – only 10% of our hundred trial drugs actually work – most of the tested drugs do not work, and we have many opportunities for false positives.

The whole argument makes sense to me but I am not sure if I entirely understand how it relates to a single hypothesis test. I.e. how does the base rate fallacy creep in a single hypothesis test? Say we have setup a hypothesis test to check if the average height differs between males and females for a specific sample we collected.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
4
$\begingroup$

The positive predictive value (PPV; the probability that a drug actually working, given that we rejected the null hypothesis that it had no effect—i.e. the probability that we made a true rejection) is sensitive to the base rate of cancer drugs that actually work.

Consider the $2\times2$ table below, where testing positive or negative corresponds to rejecting or not rejecting H$_{0}$, and the truth being positive or negative means that H$_{0}$ is false or true, respectively. Each quadrant contains the counts of the four possibilities under these conditions: the number of true positive tests, number of true negative tests, number of false positive tests, and number of false negative tests. The margins sum the rows and columns, and the sum of row margins equals the sum of column margins equals the total number of tests.

2 by 2 table

PPV is the number of true positives over the total testing positive. If you imagine that the area in each quadrant of the table is proportional to the number in each quadrant, and further, imagine that the vertical line down the center of the $2 \times2$ table represents the base rate (e.g. prevalence), then the table above shows half of tests of cancer drugs truly rejecting H$_{0}$. If the base rate is lowered (that vertical line shifts left), you can see that true positives shrink relative to false positives and therefore the PPV gets smaller (i.e. just because you rejected the null hypothesis for a drug means that you still probably made a false rejection).

By contrast, the $p$ value is the probability of observing your data, if in fact the null hypothesis is true. In the table, the null hypothesis being true is the left column, and $\alpha$ (your willingness to reject the null when the null is true) is the number of false negatives over the total truly negative (or one minus the specificity of the test).

So, if the null hypothesis is true, and the base rate is low, the $p$ value being small enough to reject, even if it is very small, means that you are probably seeing a false positive.

Suggested further reading

Altman, D. G. and Bland, J. M. (1994). Diagnostic tests 2: predictive values. British Medical Journal, 309:102.

Altman, D. G. and Bland, J. M. (1994). Diagnostic tests 1: sensitivity and specificity. British Medical Journal, 308:1552.

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8):0696–0701.

Improve this answer
$\endgroup$
14
  • $\begingroup$ In general, what do each of the boxes contain? The samples? I.e. lowering the prevalence lowers also the number of samples that turn out to be True Positives? I.e. how does this apply to a single hypothesis test performed on a single sample? $\endgroup$ Commented Apr 28, 2014 at 18:18
  • 1
    $\begingroup$ I have clarified the contents of the table in a new paragraph. A lower prevalence (of drugs with true effects out of all drugs) will decrease the number of true positives relative to the number of false positives, and therefore decrease PPV. The PPV tells you the probability of a drug truly having an effect given that you have rejected the null hypothesis for that drug. $\endgroup$ Commented Apr 28, 2014 at 18:27
  • 1
    $\begingroup$ See my correction to the paragraph following the table. In the case of a single hypothesis test: (1) Reject H$_{0}$ height of men equals height of women; (2) pose the questions (i) what is the prevalence of true rejections of H$_{0}$ (how could one know this? One can't: it's an assumption for which there may or may not be evidence), and (ii) how does this affect PPV?; (3) if PPV is low, feel comfortable expecting that your rejection of H$_{0}$ with a low $p$-value is likely a false positive. $\endgroup$ Commented Apr 28, 2014 at 19:57
  • 1
    $\begingroup$ @redblackbit I believe the intuition you may be missing regarding individual hypothesis tests is to think about your prior probabilities regarding which of the hypotheses is true. If before collecting your data you believe it is extremely unlikely that your alternative hypothesis is true, then it's ok to still be skeptical of the alternative even after seeing a low p-value. $\endgroup$ Commented Apr 28, 2014 at 22:12
  • 2
    $\begingroup$ @redblackbit As an example, suppose I am interested in trying to determine whether or not my friend has ESP. I am skeptical, so I think there is an extremely small possibility that my friend has ESP. Suppose I flip a fair coin 10 times and he correctly guesses every time, a p-value of about .001. Now, one of two things happened. Either my friend has ESP, which is why he was able to correctly predict all 10 flips, or my friend doesn't have ESP and was lucky. Confronted with this data, I still believe there is a low chance that my friend has ESP because my prior probability was so low. $\endgroup$ Commented Apr 28, 2014 at 22:21

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.