I just completed a training on Quantitative Research for UX.
The instructor uses the χ² (chi-squared) test throughout the course to determine if patterns can be determined in the study results, if they are significant.
As I understand it, the function always needs two distributions (number of people who chose an option): the actual one, and an expected one, if people chose arbitrarily. Is that correct?
It will then calculate (more or less) how much the actual results align with arbitrary results.
In the example of a first-click-test, which groups participants into success and failure categories, depending on where they clicked, the instructor simply used this categorisation with a fifty-fifty chance to succeed as the expected distribution for 40 participants:
| Category | Participants |
|---|---|
| Success | 20 |
| Failure | 20 |
This seems wrong to me, as testers don’t decide whether to fail or succeed, so if they clicked arbitrarily, the distribution wouldn’t be fifty-fifty.
Their decision is limited to the element they click on to solve the task. So shouldn’t the expected distribution be a function of the number of valid and invalid elements to be clicked?
| Category | Participants |
|---|---|
| Success | portion of valid elements in prototype × participants |
| Failure | portion of invalid elements in prototype × participants |
To make an example: The prototype contains 10 links and buttons. Only 2 of those lead to success. The test is run with 40 participants.
| Category | Participants |
|---|---|
| Success | 8 (20% of 40 participants) |
| Failure | 32 (80% of 40 participants) |
Am I mistaken?
