2
$\begingroup$

I would like to statistically test the impact of an experimental design that opposes a within-subject to a between-subject manipulation of the independent variable. In other words, does the design type interacts with the effect of my independent variable.

There is an article from Erlebacher (1977, see also 1978) that addresses this question. However, I have a hard time understanding the formulas or even reproducing the example they provide.

I was also wondering if there is a more recent version of such a test (since the article is quite old), and also if there is a R package/script to implement it.

Ultimately, it would really help me to have a concrete example in R. Below is the data they used in their 1977 article:

X.within <- data.frame( id = 1:20, LCS = c(60, 73, 93, 10, 90, 80, 83, 37, 83, 70, 77, 7, 100, 70, 100, 43,43, 83, 40, 73), SCS = c(36, 53, 66, 0, 73, 43, 20, 10, 26, 40, 60, 3, 53, 26, 63, 6, 3, 30, 7, 10) ) X.between <- data.frame( id = 21:60, cond = rep(c("LCS", "SCS"), each = 20), score = c( c(53, 77, 2, 38, 68, 92, 3, 15, 67, 53, 58, 20, 17, 40, 85, 60, 25, 3, 82, 67), c(62, 0, 57, 42, 3, 55, 22, 28, 45, 47, 52, 75, 38, 45, 65, 50, 2, 0, 10, 60) ) )

The basic idea is to test whether the X.within$LCS - X.within$SCS is different from X.between$score[X.between$cond == "LCS"] - X.between$score[X.between$cond == "SCS"].

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
+50
$\begingroup$

Edited in response to mat's comment. Thanks for the clarification! Following idea:

First bring also the X.within in a 'long' format 

library(reshape2) X.within_long = melt(X.within, id.vars = c("id"),value.name="score") colnames(X.within_long)[which(colnames(X.within_long)=="variable")] = "cond"

now mark the experiment in each data.frame 

X.within_long$design = "within" X.between$design = "between"

and put both data.frames together as follows

X_all = rbind(X.between,X.within_long) > head(X_all) id cond score design 1 21 LCS 53 between 2 22 LCS 77 between 3 23 LCS 2 between 4 24 LCS 38 between 5 25 LCS 68 between 6 26 LCS 92 between > tail(X_all) id cond score design 75 15 SCS 63 within 76 16 SCS 6 within 77 17 SCS 3 within 78 18 SCS 30 within 79 19 SCS 7 within 80 20 SCS 10 within

Then do a simple mixed-effect model with condition, experimantel design and the interaction of both (plus individual random-effects to model possible individual heterogenity in the within design).

library(lme4) library(lmerTest) lmer_model = lmer(score ~ cond + design + cond*design + (1 | id), REML = FALSE, data = X_all) anova(lmer_model) rand(lmer_model) > anova(lmer_model) Analysis of Variance Table Df Sum Sq Mean Sq F value cond 1 10793.4 10793.4 79.9210 design 1 123.7 123.7 0.9162 cond:design 1 1148.5 1148.5 8.5042 > rand(lmer_model) ANOVA-like table for random-effects: Single term deletions Model: score ~ cond + design + (1 | id) + cond:design npar logLik AIC LRT Df Pr(>Chisq) <none> 6 -363.18 738.37 (1 | id) 5 -372.79 755.57 19.207 1 0.00001173 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The interaction is statistically significant let aside the discussion whether you should use statistical significance or p-values, indicating that the experimental design indeed matters. This is based on the identity link function, but if you plot your scores, it will indicate an excess of low scores which you might want to model separately.

The random-effects are highly significant, meaning that individual heterogeneity between participants seems to make a differences. Whether this is good or bad, depends on the context but often this is interpreted as a rejection of the between design which ignores individual heterogeneity (implicating you should go for the within approach).

Please be aware that there is no consensus about how to calculate p-values for lmer models, see here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am not sure how this test the effect of experimental-design x cond. Shouldn't we add a design variable to the database and test design*cond? $\endgroup$ Commented Jun 30, 2019 at 17:48
  • $\begingroup$ Thanks for the clarification (and sorry for my misunderstand), I modified my answer accordingly. Hope I this corresponds to what you want. $\endgroup$ Commented Jul 1, 2019 at 20:15

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.