Multiple testing

Multiple testing#

Adjustment of p-values for multiple testing.

The procedures follow R’s p.adjust exactly (R Core Team 2025), including its conventions for missing values and for families declared larger than the p-values observed in them, so adjusted p-values can be compared with R output directly.

References

Holm S (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65-70.

Hochberg Y (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800-802. doi:10.1093/biomet/75.4.800

Hommel G (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 383-386. doi:10.1093/biomet/75.2.383

Simes RJ (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73, 751-754. doi:10.1093/biomet/73.3.751

Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289-300. doi:10.1111/j.2517-6161.1995.tb02031.x

Benjamini Y, Yekutieli D (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 29, 1165-1188. doi:10.1214/aos/1013699998

R Core Team (2025). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. https://www.R-project.org/

cytepool.stats.multiple_testing.adjust_p_values(p, method='bh', *, n=None, axis=None)#

Adjust p-values for multiple testing, as R’s p.adjust does.

Bonferroni, Holm (1979), Hochberg (1988) and Hommel (1988) control the family-wise error rate; Benjamini and Hochberg (1995) and Benjamini and Yekutieli (2001) control the false discovery rate. With n tests in a family and sorted p-values p_(1) <= ... <= p_(n), the adjusted value of p_(i) is the following, capped at 1:

  • "bonferroni": n p_(i).

  • "holm": max over j <= i of (n - j + 1) p_(j).

  • "hochberg": min over j >= i of (n - j + 1) p_(j).

  • "hommel": the largest Simes (1986) p-value, min over j of k p_(j:S) / j, over all sets S of k tests that contain test i, where p_(j:S) is the j-th smallest p-value in S.

  • "bh": min over j >= i of n p_(j) / j.

  • "by": the "bh" value times 1 + 1/2 + ... + 1/n.

  • "none": p_(i).

Every method takes O(m log m) time for m p-values, Hommel’s included.

Parameters:

  • p – p-values in [0, 1]. NaN marks a missing value; it stays NaN and is not counted as a test.

  • method – The procedure, case-insensitive: "bonferroni", "holm", "hochberg", "hommel", "bh" (also "fdr" and "benjamini-hochberg"), "by" (also "benjamini-yekutieli") or "none". The default is "bh"; R’s default is "holm".

  • n – Number of tests in each family, when larger than the number of non-missing p-values. As in R, the unobserved p-values are taken to be larger than all observed ones for Bonferroni and Holm, and equal to 1 for the other methods. Defaults to the number of non-missing p-values in each family.

  • axis – None adjusts all p-values together as one family. An integer adjusts each slice along that axis as its own family; for a matrix of features by contrasts, axis=0 adjusts each column.

Returns:

The adjusted p-values as float64, with the shape of p.

Raises:

  • TypeError – If p is not real-valued, method is not a string, n or axis is not an integer, or p, n or axis is a numpy masked array.

  • ValueError – If a p-value is outside [0, 1], method is unknown, n is smaller than the number of non-missing p-values in a family or larger than 2**53, or axis is out of range.

Used in: Quick start

Explained in: Choosing a method, Multiple testing