Combined p-values

Combined p-values#

combine_p_values() combines, for every gene, the p-values of the studies that tested it into one p-value of the hypothesis that no study has an effect: by Fisher’s, Stouffer’s (with weights, such as the square roots of the studies’ sample sizes), Tippett’s, Wilkinson’s, Pearson’s, Mudholkar and George’s or Simes’s method, as scipy’s combine_pvalues and the R package metap compute them. With the directions of the effects, two-sided p-values become one-sided ones, and effects in opposite directions count against each other. partial_conjunction() gives the p-value that at least u of the studies have an effect, by combining the largest p-values (Benjamini and Heller 2008). A study that did not test a gene is left out of its combination.

Combined p-values of several studies, and partial conjunction tests of replication.

combine_p_values() combines, for every feature (gene), the p-values of the studies that tested it into one p-value of the global null hypothesis that no study has an effect, by the methods of Fisher (1932), Stouffer et al. (1949), Tippett (1931), Wilkinson (1951), Pearson (1933), Mudholkar and George (George and Mudholkar 1983) and Simes (1986), as scipy’s combine_pvalues and the R package metap compute them. partial_conjunction() gives the p-value that at least u of the n studies have an effect (Benjamini and Heller 2008): a gene that one study finds is not replicated by the others, and the global null hypothesis, which one study’s effect rejects, cannot say so.

The methods:

For the p-values p_1, …, p_k of the k studies that tested a feature, in order p_(1) <= … <= p_(k):

  • "fisher": X = -2 sum log p_i against the chi-squared distribution with 2k degrees of freedom (metap’s sumlog).

  • "stouffer": Z = sum w_i z_i / sqrt(sum w_i^2), with z_i = Phi^-1(1 - p_i), against the standard normal distribution (Stouffer et al. 1949; Lipták 1958; metap’s sumz). The weights w_i are equal by default; the square roots of the studies’ sample sizes are the usual choice (Zaykin 2011).

  • "tippett": the smallest p-value, with P(p_(1) <= s) = 1 - (1 - s)^k (metap’s minimump).

  • "wilkinson": the r-th smallest p-value, against the beta distribution with parameters r and k - r + 1 (metap’s wilkinsonp; r = 1 is Tippett’s method and r = k metap’s maximump, p_(k)^k). It tests the global null hypothesis: that p_(k)^k is small says that every study has some evidence, not that every study has an effect, which partial_conjunction() tests.

  • "pearson": X = -2 sum log(1 - p_i), whose chi-squared distribution function with 2k degrees of freedom is the p-value: small when every p-value is small, and near 1 when any p-value is near 1 (scipy’s "pearson", whose statistic is -X).

  • "mudholkar_george": the logit statistic T = sum log((1 - p_i) / p_i), times sqrt(3 / k) / pi sqrt((5k + 4) / (5k + 2)), against Student’s t distribution with 5k + 4 degrees of freedom (metap’s logitp, whose statistic this is; scipy reports T).

  • "simes": min over j of k p_(j) / j, a p-value of the global null hypothesis that is valid for independent and positively dependent p-values (Simes 1986).

The statistics are monotone in each p-value, and each p-value is exact for independent studies whose p-values are uniform under the null hypothesis. Fisher’s and Tippett’s methods respond to the strongest study; Stouffer’s, the logit and Pearson’s to consistent evidence across studies; a p-value near 1 weighs most on Pearson’s.

One-sided and two-sided p-values:

The methods are defined for one-sided p-values, all for the same direction of effect, which effects in the other direction push towards 1. Two-sided p-values combined as they are test whether some study has an effect in either direction, and effects in opposite directions do not cancel. directions, the signs of the studies’ effects (such as their log2 fold changes), turns two-sided p-values into one-sided ones, as metap’s two2one: p / 2 for an effect above zero and 1 - p / 2 for one below, and 1/2 for an effect of exactly zero, for the effects above zero; their complements, for the effects below. Each direction is combined, giving p_value_up and p_value_down, and the p-value is min(1, 2 min(p_up, p_down)), the Bonferroni bound over the two directions (Owen 2009): for Stouffer’s method it is the two-sided p-value of Z, 2 (1 - Phi(|Z|)), and effects in opposite directions cancel. The one-sided p-value near 0 is kept exactly: Stouffer’s z_i is the normal quantile of p / 2 with the sign of the effect, and each method reads log(1 - p_i) from the complement p / 2 where p_i is 1 - p / 2.

Partial conjunction:

The hypothesis that fewer than u of the n studies that tested a feature have an effect is tested by combining the n - u + 1 largest p-values, p_(u), …, p_(n), by a method valid for n - u + 1 independent p-values (Benjamini and Heller 2008): Fisher’s, Stouffer’s (without weights, as the paper defines it) or Simes’s, min over i = u, …, n of (n - u + 1) p_(i) / (i - u + 1). For u = 1 it is the global null hypothesis; for u = n, every method gives p_(n). With directions, the effects must also share their direction: the one-sided p-values of each direction are combined as above, and the p-value is min(1, 2 min(p_up, p_down)). A feature that fewer than u studies tested has no p-value (NaN). Adjusting the p-values of the features by Benjamini and Hochberg’s procedure controls the false discovery rate of the claims that a feature has an effect in at least u studies, for independent features (Benjamini and Heller 2008), as adjust_p_values(result.p_value, "bh") gives it.

Studies that did not test a feature:

NaN marks them. Each feature is combined over the k studies that tested it, which n_studies gives; a feature that no study tested has NaN. A single study’s p-value is its own combined p-value, by every method but the logit, whose t distribution approximates the distribution of a sum of logistic variables, also for one.

p-values of exactly 0 and 1:

They are limits that every method reaches: a p-value of 0 makes Fisher’s, Tippett’s, Simes’s, Stouffer’s and the logit p-values 0, and a p-value of 1 makes Pearson’s 1, and Stouffer’s and the logit 1 unless another study’s p-value is 0, where the limit does not exist and the combined p-value is NaN.

Deliberate differences from scipy and metap:

validation/combined_p_values_and_replication.py reports them.

  • metap leaves out p-values of 0 (and of 1 for sumz and logitp), with a warning that “some studies omitted”, and stops with fewer than two studies. Here they are combined as limits, and one study gives its own p-value.

  • scipy’s combine_pvalues gives NaN for a feature with a missing p-value unless asked to omit it (nan_policy="omit"), which this module does.

References

Fisher RA (1932). Statistical Methods for Research Workers, 4th edition. Oliver and Boyd, Edinburgh.

Stouffer SA, Suchman EA, DeVinney LC, Star SA, Williams RM (1949). The American Soldier, volume 1: Adjustment during Army Life. Princeton University Press, Princeton.

Lipták T (1958). On the combination of independent tests. Magyar Tudományos Akadémia Matematikai Kutató Intézetének Közleményei 3, 171-197.

Tippett LHC (1931). The Methods of Statistics. Williams and Norgate, London.

Wilkinson B (1951). A statistical consideration in psychological research. Psychological Bulletin 48, 156-158. doi:10.1037/h0059111

Pearson K (1933). On a method of determining whether a sample of size n supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random. Biometrika 25, 379-410. doi:10.1093/biomet/25.3-4.379

George EO, Mudholkar GS (1983). On the convolution of logistic random variables. Metrika 30, 1-13. doi:10.1007/BF02056895

Simes RJ (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73, 751-754. doi:10.1093/biomet/73.3.751

Zaykin DV (2011). Optimally weighted Z-test is a powerful method for combining probabilities in meta-analysis. Journal of Evolutionary Biology 24, 1836-1841. doi:10.1111/j.1420-9101.2011.02297.x

Owen AB (2009). Karl Pearson’s meta-analysis revisited. Annals of Statistics 37, 3867-3892. doi:10.1214/09-AOS697

Benjamini Y, Heller R (2008). Screening for partial conjunction hypotheses. Biometrics 64, 1215-1222. doi:10.1111/j.1541-0420.2007.00984.x

class cytepool.stats.combined_p_values.CombinedPValues(p_value, statistic, p_value_up, p_value_down, direction, n_studies, method, at_least, rank)#

Combined p-values of many features, from combine_p_values() or partial_conjunction().

The per-feature arrays have the shape of the p-values without their first (study) axis: one value per feature, or 0-d arrays for the p-values of one feature given as a vector. They are read-only, and NaN where fewer than at_least studies tested a feature.

Variables:

  • p_value – The combined p-value: with directions, min(1, 2 min(p_up, p_down)), two-sided.

  • statistic – The method’s statistic: Fisher’s and Pearson’s X, Stouffer’s Z, the smallest or the r-th smallest p-value (Tippett, Wilkinson), the logit method’s t, or the Simes p-value; of the n - u + 1 largest p-values for a partial conjunction; with directions, that of the effects above zero, so that Stouffer’s Z is positive for effects mostly up.

  • p_value_up – With directions, the combined one-sided p-value for effects above zero; None without.

  • p_value_down – Likewise, for effects below zero.

  • direction – With directions, 1 where p_value_up is the smaller, -1 where p_value_down is, and 0 where they are equal; None without.

  • n_studies – The number of studies that tested each feature (whose p-value is not NaN).

  • method – The method, such as "fisher".

  • at_least – u, the fewest studies with an effect that the alternative hypothesis claims: 1 for combine_p_values().

  • rank – r of Wilkinson’s method; None for the others.

Explained in: Meta-analysis

cytepool.stats.combined_p_values.combine_p_values(p_values, *, method='fisher', weights=None, directions=None, rank=None)#

Combine the p-values of several studies, feature by feature.

Every feature’s p-values, over the studies that tested it, are combined into one p-value of the hypothesis that no study has an effect, by the methods that scipy’s combine_pvalues and metap offer; the module description gives each method, the handling of one-sided and two-sided p-values and of p-values of 0 and 1.

Parameters:

  • p_values – (n_studies, n_features) p-values in [0, 1], or (n_studies,) for one feature; NaN where a study did not test a feature. One-sided p-values for the same direction of effect, or two-sided p-values, which directions turns into one-sided ones.

  • method – "fisher", "stouffer", "tippett", "wilkinson", "pearson", "mudholkar_george" or "simes".

  • weights – For Stouffer’s method only: the weight of every study, (n_studies,), or of every study and feature, of the shape of p_values; positive and finite where a p-value is given, such as the square roots of the studies’ sample sizes. Equal by default.

  • directions – The signs of the studies’ effects, such as their log2 fold changes, of the shape of p_values, for two-sided p-values: each direction is combined from one-sided p-values, and the p-value is two-sided. Not NaN where a p-value is given; zero for an effect of exactly zero.

  • rank – r, for Wilkinson’s method only: the r-th smallest p-value is combined, at least 1. A feature that fewer than r studies tested has NaN.

Returns:

The combined p-value and statistic of every feature, with the number of studies that tested it, and with directions the one-sided p-values and the direction of the combined effect.

Raises:

  • TypeError – If an argument does not hold real numbers, is a numpy masked array, or an option has the wrong type.

  • ValueError – If a p-value lies outside [0, 1], the shapes do not match, there is no study, method is unknown, weights are given for a method other than Stouffer’s or are not positive and finite where a p-value is given, a direction is NaN where a p-value is given, or rank is missing for Wilkinson’s method, given for another, or less than 1.

Example

>>> result = combine_p_values([0.01, 0.2, 0.5])
>>> round(float(result.statistic), 4), round(float(result.p_value), 6)
(13.8155, 0.031766)

Two studies (rows) of two genes (columns), with two-sided p-values: the effects of the first gene agree, those of the second do not.

>>> two_sided = [[0.01, 0.01], [0.04, 0.04]]
>>> signed = combine_p_values(
...     two_sided, method="stouffer", directions=[[1.0, 1.0], [1.0, -1.0]]
... )
>>> signed.direction.tolist(), signed.p_value.round(4).tolist()
([1.0, 1.0], [0.0011, 0.712])

Used in: Replication across studies

Explained in: Meta-analysis

cytepool.stats.combined_p_values.partial_conjunction(p_values, at_least, *, method='simes', directions=None)#

The p-value that at least u of the studies have an effect, feature by feature.

Benjamini and Heller’s (2008) partial conjunction test: the hypothesis that fewer than u = at_least of the n studies that tested a feature have an effect is tested by combining its n - u + 1 largest p-values by Simes’s, Fisher’s or Stouffer’s method. With at_least=2 a small p-value says that the effect is replicated: one study’s effect alone does not make it small. With directions, the effects must share their direction. The module description gives the method; adjust the p-values over the features with adjust_p_values(result.p_value, "bh") to control the false discovery rate of the replication claims.

Parameters:

  • p_values – (n_studies, n_features) p-values in [0, 1], or (n_studies,) for one feature; NaN where a study did not test a feature.

  • at_least – u, the fewest studies with an effect, at least 1. A feature that fewer studies tested has NaN.

  • method – "simes", "fisher" or "stouffer".

  • directions – The signs of the studies’ effects, for two-sided p-values, as for combine_p_values().

Returns:

The partial conjunction p-value of every feature, with the number of studies that tested it, and with directions the one-sided p-values and the direction of the effect.

Raises:

  • TypeError – If an argument does not hold real numbers, is a numpy masked array, or an option has the wrong type.

  • ValueError – If a p-value lies outside [0, 1], the shapes do not match, there is no study, method is unknown, at_least is less than 1, or a direction is NaN where a p-value is given.

Example

>>> found_once = [[1e-8], [0.6], [0.9]]
>>> partial_conjunction(found_once, 1).p_value.round(12).tolist()
[3e-08]
>>> partial_conjunction(found_once, 2).p_value.tolist()
[0.9]

Used in: Replication across studies

Explained in: Meta-analysis