Timeline for Interaction not significant, but one simple effect significant: linear mixed model with lmer() in R
Current License: CC BY-SA 4.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 20, 2020 at 21:05 | comment | added | kurtosis | Let us continue this discussion in chat. | |
| Aug 20, 2020 at 19:42 | comment | added | Meg | 2/2 You’ve lost this level of detail (i.e., the simple effect). The only way to get this detail is with the interaction in the model. However, when an interaction is not significant, it is generally argued there is no need to leave it in the model, and now your chance to capture that simple effect is gone. Is there something I’m missing? | |
| Aug 20, 2020 at 19:35 | comment | added | Meg | 1/2 In theory, no significant interaction tells you that there is no need to differentiate between the four groupings - the main effect for Sex and/or Intervention (here, just Intervention) is "enough.” So, now you collapse (get rid of the interaction), and conclude that pre- and post-interventions differ, but not that – specifically - females post-intervention differ from females pre-intervention. | |
| Aug 20, 2020 at 19:24 | comment | added | kurtosis | I'm not sure why you think an insignificant interaction means no significant simple effects. It just means one of your simple effects (Sex1:Intervention1) is insignificant. Does that help? | |
| Aug 20, 2020 at 19:10 | comment | added | Meg | Thanks for clarifying everything is identifiable. And yes, I want to compare back to baseline so I can get simple effects here. And I agree - the one significant simple effect is probably driving the significant Intervention main effect. What I can't get over is this: I would have missed this simple effect altogether had I never "stumbled upon" it (again, when estimating ORs when treating the outcome as binary), because an insignificant interaction should generally indicate no need to go on to look for significant simple effects. So this goes back to my question: How is this happening? | |
| Aug 20, 2020 at 18:53 | comment | added | kurtosis | Ah, no. Everything you have is identifiable. It's just what gets reported. That is how the significance gets affected: how something is coded often implies a basis for comparison. So coding as $\pm$1 would be comparing to 0 while the usual treatment contract compares to the baseline. I suspect your significant simple effect is (most of but not all of) what is driving the significant InterventionPre effect. | |
| Aug 20, 2020 at 18:00 | comment | added | Meg | Oh, yes, I know that not all simple effects will be output, but they can subsequently be calculated (with algebra using 0s and 1s, or using something like emmeans). p-values can be obtained by refitting the model with different baselines, or, again, using emmeans. I thought by aliasing you meant some things may never be estimable, but all simple effects in this 2x2 situation should be estimable, just by using algebra/changing baseline/using emmeans. It is unclear to me if/how this is related to the lack of a significant interaction despite a significant simple effect. | |
| Aug 20, 2020 at 17:48 | comment | added | kurtosis | See here: bbolker.github.io/stat4c03/notes/contrasts.pdf Note that aliasing happens in your case; that is why you do not see an effect estimated for each of the simple effects. So Sex0:Intervention0, Sex1:Intervention0, and Sex0:Intervention1 would all not be reported and the model summary would instead report the intercept, Sex effect, and Intervention effect. Only Sex1:Intervention1 would not be aliased and reported as Sex1:Intervention1. | |
| Aug 20, 2020 at 17:24 | comment | added | Meg | My data are coded as dummy variables (0/1), which is the natural way to then estimate simple effects, and I have not seen an example where aliasing has caused an issue with doing so. Indeed, I have seen the 2x2 ANOVA case as a straightforward example for the sake of illustrating simple effects, so it's unclear why it would be an issue here. (Note I have only seen aliasing discussed with effects coding (say, -1/1).) I have spent some time now searching various terms, and have not found an indication that aliasing would be at play here. Do you have a source for the 2x2, dummy-coded case? | |
| Aug 20, 2020 at 17:04 | comment | added | kurtosis | Aliasing is a direct result of identifiability. It happens but how it happens depends on how your factors are coded, not on the size of groups and treatments. So yes, you should expect this in a 2x2 setup. Most discussions of contrasts in $R$ discuss how this happens. | |
| Aug 20, 2020 at 16:58 | comment | added | Meg | * Typo two comments above: "and, indeed, the Sex main effect is significant here" should be, "and, indeed, the Intervention main effect is significant here". | |
| Aug 20, 2020 at 16:46 | comment | added | Meg | Would I expect aliasing in a 2x2 ANOVA (fit as a linear mixed model, so random effects could be incorporated)? | |
| Aug 20, 2020 at 16:26 | comment | added | kurtosis | Good to hear that a logit transform performed similarly; and, yes, the random effects sometimes make a difference. One of your simple effects can be significant if the others are incredibly noisy. Also, presumably one of those simple effects is aliased with your baseline, so that is likely an issue with looking at the simple effects. | |
| Aug 20, 2020 at 16:10 | comment | added | Meg | 4/4 I also think you’re misunderstanding main vs. simple effects. I understand that you can readily have a significant main effect without a significant interaction (and, indeed, the Sex main effect is significant here). What I’m asking about are the simple effects: All four combinations of Sex and Intervention (change from male to female when holding intervention at “pre,” e.g.). It makes less (no?) sense to me how one of these simple effects can be significant if the interaction is not. | |
| Aug 20, 2020 at 16:10 | comment | added | Meg | 3/4 As I mentioned in my post, I had no evidence of heteroskedasticity in the residuals after I fit the model. For completeness, however, I had already also tried an empirical logit transformation on the proportions, and the results are equivalent. But, as I said, I don’t think a transformation is necessary here because the proportions are already so well-behaved (as in my post). | |
| Aug 20, 2020 at 16:10 | comment | added | Meg | 2/4 I am not insisting the interaction be significant. I originally stumbled upon the significant simple effect in light of the insignificant interaction because I have also considered a similar logistic model (using 0/1 data instead of the proportions), and wanted estimates of the odds ratios between all groups as measures of effect size to report anyway, despite lack of statistical significance. That is, until I saw one of the simple effects was significant. That’s what led to me questioning if/how/when this could happen. | |
| Aug 20, 2020 at 16:09 | comment | added | Meg | 1/4 Thanks, @kurtosis. Although in theory the random effects hopefully don’t change our results much, this is not always true. In a logistic mixed model of these data, the results are sensitive to which random effects are/are not included. There could be other issues with the model leading to this, but one cannot assume that any old random effects will do. As a matter of fact, there are many conflicting opinions in the literature on how to best choose random effects so as to balance the type I error rate and power. | |
| Aug 20, 2020 at 15:59 | history | edited | kurtosis | CC BY-SA 4.0 | Fixed analogy+typo |
| Aug 20, 2020 at 15:54 | history | answered | kurtosis | CC BY-SA 4.0 |