Timeline for Hypothesis Testing: Permutation Testing Justification
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 11, 2017 at 23:30 | comment | added | Coder47 | Let us assume that randomly sampling w/ replacement from two independent and equally distributed populations. Under this assumption, if I can somehow show that this implies that most samples will have a p-dist that is similar to our original sample's p-dist, then I will be convinced that the true null distribution is roughly the same as our original p-dist, since the null distribution is an average of the p-dist of all possible samples. Unfortunately, I cannot prove this implication to myself, and I feel it is strange to just assume the resultant of the implication. | |
| Feb 11, 2017 at 23:24 | comment | added | Coder47 | Another concern: I can't think of a specific example of a set of reasonable assumptions to make about the sampling procedure and populations that would ensure the sampling distribution of the test statistic (will refer to this distribution as p-dist from now on) would be roughly identical to the true null distribution, so as to avoid errors in conclusions. In the case that sample sizes are essentially the size of the original populations, then I am convinced that the p-dist will be roughly equal to the null distribution, but in the case that the sample size is small, I am not. | |
| Feb 11, 2017 at 16:29 | comment | added | Coder47 | Also, if possible, could you please answer the side question, too? | |
| Feb 11, 2017 at 16:27 | comment | added | Coder47 | To summarize: We make additional "hidden" assumptions about our sampling procedure (e.g. random sample w/ replacement) and populations (e.g. independent and equally distributed) to narrow down what the null distribution could look like. This seems fair given that after all we are trying to reason under uncertainty. But this makes me wonder: am I correct in saying that the more these hidden assumptions stray from reality of the population and sampling procedure, the more prone we are to making type I and type II errors? | |
| Feb 11, 2017 at 7:10 | history | answered | Glen_b | CC BY-SA 3.0 |