2
$\begingroup$

I've seen this stated Why are p-values uniformly distributed under the null hypothesis? and https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/basic-statistics/inference/supporting-topics/basics/type-i-and-type-ii-error/#:~:text=The%20probability%20of%20making%20a,you%20reject%20the%20null%20hypothesis.&text=The%20probability%20of%20rejecting%20the,is%20equal%20to%201%E2%80%93%CE%B2 and essentially in my other resources.

The probability of rejecting the null hypothesis is the significance level $\alpha$, which we typically take as $\alpha = 0.05$. When we reject a null hypothesis, there are two possibilities: (1) the null hypothesis was in fact correct, so we made a type 1 error (2) the null hypothesis was in fact wrong, so made a correct choice.

(1) and (2) are disjoint events, and if we denote $p_1$ and $p_2$ to correspond to their probabilities, we have

$$ p_1 + p_2 = \alpha $$

So how is it that $\alpha$ is the probability of making a type 1 error? That would mean $p_1 = \alpha$? I guess I feel like the proper way of the phrase should be the probability of a type 1 error is at most $\alpha$?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ What justifies "$p_1+p_2=\alpha$"?? In fact, what justifies associating a probability with any hypothesis? $\endgroup$ Commented Apr 9, 2021 at 21:00
  • $\begingroup$ @whuber Let $a$ denote the event that we reject the null hypothesis, then $Pr(a) = \alpha$. When the event $a$ occurs, there are two possibilities, $b_1$ and $b_2$, corresponding to (1) and (2) above. So $Pr(a) = Pr(a|b_1)Pr(b_1) + Pr(a|b_2)Pr(b_2)$ where above I defined $p_i = Pr(a|b_i)P(b_i)$ $\endgroup$ Commented Apr 9, 2021 at 21:19
  • 4
    $\begingroup$ Your initial statement is invalid: you simply don't know the chance of rejecting the null hypothesis. All you know is that the chance of a rejection conditional on the null equals $\alpha.$ $\endgroup$ Commented Apr 9, 2021 at 21:30
  • 3
    $\begingroup$ Yes, because everything else you write is a reaction to or consequence of that error. (BTW, I believe you meant to write that the p-values are uniformly distributed under the null.) $\endgroup$ Commented Apr 9, 2021 at 21:38
  • 1
    $\begingroup$ @whuber Oops yes, I meant uniformly distributed. Ahh I get it now. Basically, my $p_2$ above, under the assumption that the null hypothesis is true, is 0. $\endgroup$ Commented Apr 9, 2021 at 21:47

2 Answers 2

Reset to default
3
$\begingroup$

So how is it that α is the probability of making a type 1 error?

It is not. The significance level $\alpha$ is not the probability of making a type 1 error. @whuber answered this in the comments already. However, I think it is worth providing an extended answer to this questions, since the statement that "the probability of a type 1 error is the significance level $\alpha$" appears on some answers to questions related to p values on cross validated. (Here, here I think, authors make this statement in the context of an assumed null hypothesis but they do not say so explicitly. Hence the confusion.

The significance level is the probability of rejecting $H_0$ conditional on the null before you sample. (By "conditional on" the null I mean if the null were true and by probability I mean the frequentist viewpoint, that is the fraction of repeated experiments in which we reject $H_0$).

Edit: I removed a statement to illustrate why the significance level is not the probability of a type 1 error using notation like $P(p < \alpha~|~H_0)$, since the notation is misleading.

See this answer that clarifies such notation.

I find This answer very good in describing the relationship between p values and type 1 errors.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Please be careful about using "conditional on" language and notation, because many readers will understand that to mean a conditional probability, which it is not. $\endgroup$ Commented Dec 26, 2024 at 14:00
  • $\begingroup$ @whuber thank you for the hint. I attempted to clarify the statements. $\endgroup$ Commented Dec 26, 2024 at 15:46
2
$\begingroup$

"The probability of rejecting the null hypothesis is the significance level 𝛼". That is false, or at best badly incomplete. That probability is unknown and depends on (i) whether the null hypothesis is true; (ii) the power of the experiment and analysis to correctly discard a false null hypothesis. Item (ii) in turn depends on (iii) the size of the actual discrepancy between the null hypothesis and the true value of parameter of interest; (iv) the sample size; (iv) the value chosen for $\alpha$ in advance of the analysis; (v) the amount of noise and variation in the data; (vi) how well the statistical model captures the nature of the actual data generating system and experimental setup. And there are probably other things as well.

"𝑝1+𝑝2=𝛼" That is false and I do not see how it would follow from what you wrote above it.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.