2
$\begingroup$

If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using:

Ps <- c(0.04,0.02,0.8,0.9,0.004) stats::p.adjust(Ps, method = "BH", n = length(Ps))

Tied p-values are adjusted in the same way:

Ps <- rep(c(0.0001,0.02,0.8,0.9,0.004), 2) stats::p.adjust(Ps, method = "BH", n = length(Ps))

If our p values are calculated as

P = 2*((1 + sum(observed_efects <= null_effects))/(N+1))

Where N is the number of samples in the null distribution, our minimum P value is limited as

min(P) == 1/(N+1)

If we have a collection of P values in which all P values are equal to 1/(N+1) then is the FDR appropriate given all P values are given the same rank? e.g.

Ps <- rep(0.002, 555) stats::p.adjust(Ps, method = "BH", n = length(Ps))
Improve this question
$\endgroup$
2
  • $\begingroup$ Yes, the FDR calculation works for any set of independent p-values, as you have already said yourself. It would be easier to give a more elaborate answer if you explained why you think there might be problem. There is no problem in BH's theorem or in the p.adjust calculation with tied p-values. The result given by p.adjust in this case seems perfectly natural --- it says we that expect no more than 1 false discovery amongst these 555 tests. $\endgroup$ Commented Aug 5, 2022 at 10:36
  • $\begingroup$ Ignoring the false negative probability may mean that the results are unreliable. $\endgroup$ Commented Aug 9, 2022 at 12:21

1 Answer 1

Reset to default
5
+50
$\begingroup$

The Benjamini-Hochberg FDR calculation performed by p.adjust seems perfectly appropriate in your situation. There is no theoretical problem with tied p-values in Benjamini and Hochberg's 1995 theorem on which the p.adjust code is based.

For your data, p.adjust returns a FDR value of 0.002 for all tests. Since you did 555 tests and $0.002\times 555=1.11$, this says that you expect no more than 1 false discovery if you reject all 555 null hypotheses. This seems to me to be a very reasonable and useful result. It agrees with intuition because clearly you could not have not have got all the p-values at the lowest possible value unless virtually all the null hypotheses were false.

The BH algorithms works on an ordered list of p-values and each FDR value applies to all the tests with p-values less than or equal to the current value. If all the p-values are equal, then all the FDR values will be equal and will apply to the entire ensemble of tests.

The BH algorithm works in such a way that the expected FDR for the whole set of tests is equal to the largest p-value. If all the p-values are equal, as they are in your case, then only the overall FDR is relevant.

By contrast, FWER multiple testing adjustments would not work so well. Both the Bonferroni and Holm multiple testing adjustments would give no significance at all, with all adjusted p-values equal to 1.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.