1
$\begingroup$

So, supposing I'm carrying out a field experiment (agricultural data), and I'm interested to understand how four plant genotypes affect three dependent variables (yield, height, and number of leaves).

I understand that first we have to conduct an ANOVA for each of the dependent variables. After that, we conduct a post-hoc test (in my case, Tukey), which I believed (up to now) control for type I error inflation. But, besides the innate control in the Tukey test, is it necessary to add another p-value adjustment because I'm conducing three evaluations (yield, height, and number of leaves) in my experiment?

Improve this question
$\endgroup$
3
  • $\begingroup$ Related: stats.stackexchange.com/q/449111/97844 $\endgroup$ Commented Aug 11 at 0:24
  • $\begingroup$ Are you making inferences about some more abstract thing X (e.g. "tree growth") on the basis of all three? Or are you making separate inferences of the three (i.e. , "yield is affected by genotype" is a separate inference from "height is affected by genotype" etc.)? $\endgroup$ Commented Aug 11 at 7:38
  • $\begingroup$ Instead of performing 1 regression for each outcome, consider a multivariate regression that models all 3 outcomes together. The point estimates will be the same as for 3 separate models, but the (co)variances will take the correlations among outcomes into account. This document shows how to do that in R. $\endgroup$ Commented Aug 11 at 14:09

1 Answer 1

Reset to default
2
$\begingroup$

You're not likely to get a consensus answer on this because the word necessary begs more information. Indeed, this answer makes the excellent point that control of error rate is across some set of tests / procedures. If you designed the study in this particular way, you are free to choose what set of tests belong together in terms of needing to control Type I error rate. Using Tukey's HSD for each ANOVA is controlling the familywise error rate for that specific set of tests (presumably at the nominal $\alpha = .05$). One could argue that since you intended to run ANOVAs on each dependent variable, that you aren't doing those tests post hoc, so among the set of ANOVAs, you would not need to further control the error rate.

I think the main thing to remember is that in frequentist inference, we acknowledge that the decision-making procedure inherent in hypothesis testing is error prone. We are free to choose and to justify our choices with respect to our power, test statistic, error-controlling procedure, etc, if we do so a priori. Using post-hoc procedures following an ANOVA follows from the fact that you probably wouldn't have done those pair-wise comparisons if the ANOVA $p$-value wasn't first under some threshold. Either way, I think in your instance, you could either do some further easy adjustment like the simple Bonferonni correction, which will be overly conservative, or just argue, as most might, that you have planned these tests in advance, and presumably designed your study to have decent power, and so no adjustment is necessary in your case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ (+1) A couple of thoughts. First, there's probably no need even to perform the preliminary ANOVA if the post-modeling comparisons were specified in advance. See this page. Second, the scenario in this question seems to require a combined multivariate regression model that evaluates correlations among the 3 outcomes. See this page. Using 3 separate regressions could miss a lot, and the (hidden) correlations among their results will affect attempts at multiple-comparison correction. $\endgroup$ Commented Aug 11 at 14:05
  • $\begingroup$ Good points, I’ll edit in a few minutes $\endgroup$ Commented Aug 11 at 14:42

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.