Multiple testing - CytePool

Testing 20,000 genes at a p-value of 0.05 calls about 1,000 of them even if none changed. An adjustment for multiple testing turns each p-value into one that controls an error rate over the family of tests. adjust_p_values computes the adjustments of R’s p.adjust. Every differential-expression result adjusts its p-values by the method of Benjamini and Hochberg over the genes of one cell type and contrast, as limma’s topTable does; deseq2_wald_test and deseq2_likelihood_ratio_test adjust over the genes that pass DESeq2’s independent filter on the mean count Bourgon et al., 2010.

The methods¶

With $n$ tests in a family and the sorted p-values $p_{(1)} \le \dots \le p_{(n)}$ , the adjusted value of $p_{(i)}$ is, capped at 1:

Method	Controls	Adjusted p-value
`"bonferroni"`	family-wise error rate	$n\, p_{(i)}$
`"holm"` Holm, 1979	family-wise error rate	$\max_{j \le i} (n - j + 1)\, p_{(j)}$
`"hochberg"` Hochberg, 1988	family-wise error rate	$\min_{j \ge i} (n - j + 1)\, p_{(j)}$
`"hommel"` Hommel, 1988	family-wise error rate	the largest Simes p-value Simes, 1986 over the sets of tests that contain test $i$
`"bh"` Benjamini & Hochberg, 1995	false discovery rate	$\min_{j \ge i} n\, p_{(j)} / j$
`"by"` Benjamini & Yekutieli, 2001	false discovery rate under any dependence	the `"bh"` value times $\sum_{k=1}^{n} 1/k$

The family-wise error rate is the chance of any false call; the false discovery rate is the expected share of false calls among all calls. Genome-wide screens usually use the false discovery rate: a list of 200 genes at an adjusted p-value of 0.05 is expected to hold at most about 10 false ones. Bonferroni’s and Holm’s methods hold whatever the dependence between the tests; Benjamini and Yekutieli’s method controls the false discovery rate under any dependence, at a cost in power Benjamini & Yekutieli, 2001.

Adjusting within or across cell types¶

limma_voom, limma_trend and dream_voom adjust over the genes of one cell type and contrast, so a call is judged within its cell type. With adjust_across_groups=True they add a column adjusted over every gene of every cell type for the same contrast, one family per contrast. In simulations, the adjustment over all cell types lowered both the false discovery rate and the power, and so was very conservative Crowell et al., 2020. The adjustment across cell types is not larger for every test: a family with more small p-values can lower an adjusted value.

Other families¶

Gene-set tests adjust over the sets (camera, fry, mroast); romer’s p-values are not adjusted, as in limma, and adjust_p_values adjusts them. A composition test adjusts over the cell types, and a meta-analysis over the genes of each cell type and contrast.

References¶

Bourgon, R., Gentleman, R., & Huber, W. (2010). Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences, 107(21), 9546–9551. 10.1073/pnas.0914005107
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802. 10.1093/biomet/75.4.800
Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383–386. 10.1093/biomet/75.2.383
Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73(3), 751–754. 10.1093/biomet/73.3.751
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. 10.1111/j.2517-6161.1995.tb02031.x
Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4), 1165–1188. 10.1214/aos/1013699998
Crowell, H. L., Soneson, C., Germain, P.-L., Calini, D., Collin, L., Raposo, C., Malhotra, D., & Robinson, M. D. (2020). muscat detects subpopulation-specific state transitions from multi-sample multi-condition single-cell transcriptomics data. Nature Communications, 11(1), 6077. 10.1038/s41467-020-19894-4