Multiple contrast tests#

Dunnett’s, Williams-type and optimal contrasts, single-step tests of several contrasts of every gene, and stage-wise calls of the contrasts across genes; Several contrasts of one gene explains them.

Multiple contrast tests: which treatments or doses differ from the control, with one error rate for the contrasts of each feature.

A dose series asks, for each gene, which doses differ from the vehicle. Testing each contrast on its own and correcting by Bonferroni ignores that the contrasts share the vehicle, which makes their t-statistics correlated, and loses power; an F-test says whether any dose differs, not which. multiple_contrast_test() tests every contrast of every feature and adjusts each p-value for the contrasts of its feature by the single-step max-t method (Hothorn, Bretz and Westfall 2008): the adjusted p-value of contrast j is the chance that the largest of the feature’s correlated t-statistics is at least as extreme as t_j, from the multivariate t distribution (multivariate_t_cdf()). It takes a moderated fit, or estimates and their covariance as multcomp’s glht does with parm(). dunnett_contrasts() builds Dunnett’s contrasts, each treatment against the control (Dunnett 1955), and williams_contrasts() the Williams-type contrasts, the control against the size-weighted mean of the highest one, two or more doses, for means that change in dose order (Bretz 2006), as multcomp’s contrMat; optimal_contrasts() builds the contrasts that best detect each of several candidate shapes, for the MCP step, the multiple comparison procedure, of MCP-Mod (multiple comparison procedure and modelling; Bretz, Pinheiro and Branson 2005; Pinheiro et al. 2014), as DoseFinding’s optContr. https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains the tests.

Across features, the contrasts of the features that a screen calls are called by stage_wise_adjusted_p_values(), stageR’s stage-wise procedure (Heller et al. 2009; Van den Berge et al. 2017): Holm’s step-down within each feature (Holm 1979), with Shaffer’s (1986) first step, at the level that the screen leaves (https://docs.nygen.io/CytePool/statistics/multiple-testing#stage-wise-testing).

Differences from multcomp:
  • df need not be an integer, as moderated degrees of freedom are not, and has no default with estimates; multcomp’s default of 0 gives z-tests, here np.inf.

  • Williams-type contrasts are named by the doses they pool, such as “mid to high - vehicle”, where contrMat names them “C 1”, “C 2”, …; with two groups they are Dunnett’s single contrast, which contrMat refuses.

  • Each adjusted p-value carries a standard error of at most relative_error of it (1% by default) at every magnitude, where mvtnorm’s default allows an absolute error of 1e-3.

References

Dunnett CW (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association 50, 1096-1121. doi:10.1080/01621459.1955.10501294

Williams DA (1971). A test for differences between treatment means when several dose levels are compared with a zero dose control. Biometrics 27, 103. doi:10.2307/2528930

Bretz F (2006). An extension of the Williams trend test to general unbalanced linear models. Computational Statistics and Data Analysis 50, 1735-1748. doi:10.1016/j.csda.2005.02.005

Hothorn T, Bretz F, Westfall P (2008). Simultaneous inference in general parametric models. Biometrical Journal 50, 346-363. doi:10.1002/bimj.200810425

Bretz F, Pinheiro JC, Branson M (2005). Combining multiple comparisons and modeling techniques in dose-response studies. Biometrics 61, 738-748. doi:10.1111/j.1541-0420.2005.00344.x

Pinheiro J, Bornkamp B, Glimm E, Bretz F (2014). Model-based dose finding under model uncertainty using general parametric models. Statistics in Medicine 33, 1646-1661. doi:10.1002/sim.6052

Holm S (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65-70.

Shaffer JP (1986). Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association 81, 826-831. doi:10.1080/01621459.1986.10478341

Heller R, Manduchi E, Grant GR, Ewens WJ (2009). A flexible two-stage procedure for identifying gene sets that are differentially expressed. Bioinformatics 25, 1019-1025. doi:10.1093/bioinformatics/btp076

Van den Berge K, Soneson C, Robinson MD, Clement L (2017). stageR: a general stage-wise method for controlling the gene-level false discovery rate in differential expression and differential transcript usage. Genome Biology 18, 151. doi:10.1186/s13059-017-1277-0

class cytepool.stats.multiple_contrasts.MultipleContrastTest(names, estimate_names, contrasts, estimate, standard_error, t, p_value, p_value_error, correlation, df, alternative, relative_error)#

Multiple contrast tests of every feature, adjusted for the feature’s contrasts.

For each feature and contrast c: the estimate c’b, its standard error sqrt(c’ V c), t, and the single-step adjusted p-value, the chance that the largest of the feature’s t-statistics on its df is at least as extreme as t (Hothorn, Bretz and Westfall 2008). The arrays are read-only. The repr counts the features whose adjusted_p_value of table() is below 0.05, and, as p_values_short_of_relative_error, the p-values whose standard error exceeds relative_error of the smaller of p and 1 - p, which happens only when the computation stopped at its limit of 2^20 points: use them with their p_value_error, or ask for a larger relative_error.

Variables:
  • names – The contrast names, which label the columns of estimate, t and the other (n_features, n_contrasts) arrays.

  • estimate_names – The names of the estimates, which label the rows of contrasts.

  • contrasts – (n_estimates, n_contrasts) weights of each contrast.

  • estimate – (n_features, n_contrasts) estimates of the contrasts.

  • standard_error – (n_features, n_contrasts) their standard errors.

  • t – (n_features, n_contrasts) t-statistics.

  • p_value – (n_features, n_contrasts) p-values adjusted for the contrasts of the feature (single-step): P(max_l |T_l| >= |t|) two-sided, P(max_l T_l >= t) for "greater" and P(min_l T_l <= t) for "less". Within a feature they never decrease as t becomes less extreme.

  • p_value_error – (n_features, n_contrasts) the standard error of each p-value (the quadrature’s error bound with two contrasts, 0 with one).

  • correlation – (n_features, n_contrasts, n_contrasts) the correlation of each feature’s t-statistics, which multivariate_t_quantile() turns into the critical value of simultaneous confidence intervals.

  • df – (n_features,) degrees of freedom of the t-statistics.

  • alternative – "two-sided", "greater" or "less".

  • relative_error – The requested standard error of the smaller of each p-value and 1 minus it, as a fraction of it.

Explained in: Multiple testing

table(contrast=None, *, feature_ids=None)#

Return the tests as a table, one row per feature.

Without contrast, the test of each feature as a whole, the max-t test of its contrasts: feature_id; contrast, the contrast with the most extreme t, whose adjusted p-value is the feature’s; its estimate, standard_error, t, p_value and p_value_error; and adjusted_p_value, Benjamini and Hochberg’s adjustment of p_value over the features. With contrast: its estimate, standard_error, t, p_value and p_value_error. This p_value is adjusted for the feature’s contrasts, unlike the p-values of other CytePool tables; across features, call the features from the table without contrast first. https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains the two levels.

Parameters:
  • contrast – A contrast, by name or position; None for the test of each feature as a whole.

  • feature_ids – One id per feature, as text; by default feature_0, feature_1, …

Raises:
  • KeyError – If no contrast has the name.

  • IndexError – If the position is out of range.

  • ValueError – If feature_ids does not give one id per feature.

  • TypeError – If an id is not text, contrast is neither a name nor a position, or an argument is or holds a numpy masked array.

Explained in: Multiple testing

cytepool.stats.multiple_contrasts.dunnett_contrasts(groups, *, control=0)#

Dunnett’s contrasts: each treatment or dose against the control.

Contrast “{group} - {control}” weighs the group 1 and the control -1, as multcomp’s contrMat(n, type = "Dunnett") names and weighs them; the contrasts follow the order of the groups. Dunnett’s test (Dunnett 1955) is multiple_contrast_test() of these contrasts. https://docs.nygen.io/CytePool/statistics/multiple-testing#dunnett-and-williams-contrasts explains them.

Parameters:
  • groups – The group names, in order, as a sequence, or the keys of a mapping such as the group sizes that williams_contrasts() takes (Dunnett’s weights do not depend on the sizes), such as the coefficient names of a fit with one coefficient per group.

  • control – The control group, by name or 0-based position; the first group by default.

Returns:

The contrasts, with the groups as the columns they weigh.

Raises:
  • ValueError – If there are fewer than two groups, a name is empty or repeated, a group is a model’s intercept, or control is not one of the groups.

  • TypeError – If groups is a single string, a set (which has no order), or names a group with something other than text, or an argument is a numpy masked array.

Example

>>> dunnett_contrasts(["vehicle", "low", "high"]).names
('low - vehicle', 'high - vehicle')

Explained in: Choosing a method, Multiple testing

cytepool.stats.multiple_contrasts.multiple_contrast_test(estimates, covariances=None, contrasts=None, *, df=None, estimate_names=None, alternative='two-sided', relative_error=0.01, rng=None, n_threads=None)#

Test several contrasts of every feature, with one error rate per feature.

For each feature, the contrasts C’ b of its estimates b, their standard errors from C’ V C for its covariance V, and their t-statistics, which are correlated through V; each p-value is adjusted for the feature’s contrasts by the single-step max-t method (Hothorn, Bretz and Westfall 2008), from the multivariate t distribution on the feature’s df with the correlation of C’ V C (multivariate_t_cdf()), as multcomp’s summary(glht(parm(b, V, df), linfct = t(C))). With dunnett_contrasts() this is Dunnett’s test of each dose against the vehicle; with williams_contrasts(), the Williams-type test of a change that grows in dose order. https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains the tests.

A ModeratedFit, such as limma_voom(...).fit(group).moderated, gives everything but the contrasts: its coefficients (or contrasts), their covariance, the posterior variance times the fit’s unscaled covariance, the df of its moderated t-statistics and its coefficient names. Contrasts that name the estimates they weigh, such as those of dunnett_contrasts() and williams_contrasts(), are matched to the estimate names by name, never by position; other weights are taken in the order of the estimates. A feature with a missing or infinite estimate or covariance, or a contrast of zero variance, gets NaN for every contrast.

Parameters:
  • estimates – A ModeratedFit; or (n_features, n_estimates) estimates, such as group means or the coefficients of a fit, or (n_estimates,) for one feature, with covariances and df.

  • covariances – The covariance matrices of the estimates, (n_features, n_estimates, n_estimates), or one (n_estimates, n_estimates) matrix for every feature; symmetric and positive semidefinite. Left out with a ModeratedFit.

  • contrasts – The contrasts to test: a Contrasts, a mapping of name to weights, or a (n_estimates, n_contrasts) matrix; None tests the estimates themselves, such as the contrasts of a fit.

  • df – Degrees of freedom of the t-statistics: one number, or one per feature, positive; np.inf for z-tests. Need not be integers, as moderated degrees of freedom are not. Left out with a ModeratedFit.

  • estimate_names – The names of the estimates, in order, which named contrasts are matched to; required by them. Left out with a ModeratedFit.

  • alternative – "two-sided", "greater" (the contrasts are larger than 0) or "less".

  • relative_error – The largest standard error of the smaller of each adjusted p-value and 1 minus it, as a fraction of it, between 0 and 1.

  • rng – A numpy random generator, an integer seed, or None; a seed gives the same p-values whatever the number of threads. Two or fewer contrasts use no random numbers.

  • n_threads – The most threads to use; None uses numba’s setting. The results do not depend on it.

Returns:

The tests, with one table per contrast and one of each feature as a whole.

Raises:
  • ValueError – If the shapes disagree, named contrasts do not weigh exactly the estimates or come without their names, a contrast is all zero, df is not positive or is not one number per feature, the covariance of a feature is not symmetric or gives its contrasts a covariance that is not positive semidefinite, alternative is unknown, relative_error is not between 0 and 1, a ModeratedFit comes with covariances, df or estimate_names, or its fit has limma’s approximate standard errors of contrasts.

  • TypeError – If covariances or df is missing with estimates, an argument has the wrong type or is a numpy masked array.

Example

Dunnett’s test of group means, with the covariance of the means of a one-way model: the pooled variance over each group’s replicates, on N - K degrees of freedom (moderated ones need not be integers).

>>> import numpy as np
>>> groups = ["vehicle", "low", "high"]
>>> replicates = np.array([4, 4, 4])
>>> means = np.array([[5.0, 5.2, 6.1]])
>>> pooled_variance = 0.2
>>> test = multiple_contrast_test(
...     means, pooled_variance * np.diag(1 / replicates),
...     dunnett_contrasts(groups), df=replicates.sum() - replicates.size,
...     estimate_names=groups)
>>> test.p_value[0].round(4).tolist()  # low and high against the vehicle
[0.7587, 0.0127]

Used in: Bulk RNA-seq

Explained in: Choosing a method, Multiple testing, Dose response

cytepool.stats.multiple_contrasts.optimal_contrasts(shapes, covariance, *, groups=None)#

The contrasts of group means that best detect each of several shapes: the optimal contrasts of the MCP step, the multiple comparison procedure, of MCP-Mod (multiple comparison procedure and modelling).

For a shape mu, the means it expects in k groups, and the covariance S of the groups’ estimates up to a factor, the contrast c that maximises the non-centrality c’mu / sqrt(c’ S c) of its t-statistic among contrasts (weights that sum to 0) is proportional to S^-1 (mu - m 1), with m = (1’ S^-1 mu) / (1’ S^-1 1) (Bretz, Pinheiro and Branson 2005; Pinheiro et al. 2014). Each contrast is scaled to unit length; c’mu is then positive, so the contrast points the way the shape rises. For independent groups of n_i samples, S = diag(1 / n_i) and c_i is proportional to n_i (mu_i - the size-weighted mean of mu), DoseFinding’s optContr(models, w = n); a general S gives optContr(models, S = S). A shape and the same shape scaled by a positive factor or shifted give the same contrast, so only the shapes’ directions matter. Test the contrasts with multiple_contrast_test(). https://docs.nygen.io/CytePool/statistics/dose-response#dose-response-contrasts explains them.

Parameters:
  • shapes – The shapes, a mapping of name to the k means each expects; for CandidateShapes, dict(zip(shapes.names, shapes.means(doses))).

  • covariance – (k, k) the covariance of the groups’ estimates, up to a factor: symmetric and positive definite, such as np.diag(1 / n).

  • groups – The k group names, which the contrasts weigh; "1" to "k" by default, as dunnett_contrasts() names unnamed groups.

Returns:

One contrast per shape, named after it, with the groups as the columns it weighs.

Raises:
  • ValueError – If there are fewer than two groups or no shape, the shapes or groups disagree with covariance in size, a value is not finite, a shape is constant (its values within 1e-8 of their largest magnitude of each other), which no contrast detects, the covariance is not symmetric or not positive definite, or a name is empty or repeated.

  • TypeError – If shapes is not a mapping, an argument has the wrong type, or an argument is a numpy masked array.

Example

Groups of 2, 1 and 1 samples and a shape rising by 1 per group: the size-weighted mean is 0.75, so the contrast is proportional to 2 (0 - 0.75), 1 (1 - 0.75) and 1 (2 - 0.75), (-1.5, 0.25, 1.25), here of unit length.

>>> import numpy as np
>>> shapes = {"linear": [0.0, 1.0, 2.0]}
>>> contrasts = optimal_contrasts(shapes, np.diag(1 / np.array([2.0, 1, 1])))
>>> contrasts.matrix[:, 0].round(3).tolist()
[-0.762, 0.127, 0.635]

Explained in: Multiple testing, Dose response

cytepool.stats.multiple_contrasts.stage_wise_adjusted_p_values(screen_p_values, p_values, *, alpha=0.05, method='holm')#

Adjust several p-values per feature in two stages, after a screen of the features, so that the overall false discovery rate of the features holds.

Stage-wise testing (Heller et al. 2009; Van den Berge et al. 2017), as stageR’s stageWiseAdjustment: the screen calls the features whose Benjamini-Hochberg adjusted screen p-value is at most alpha, R of the G features with a screen p-value; then, in each called feature, its p-values are adjusted for the feature’s hypotheses and scaled by G / R, the level of R alpha / G that the screen leaves. With the feature’s m p-values sorted, p_(1) <= … <= p_(m), the adjusted p-value of p_(j) is min(1, (G / R) max over i <= j of c_i p_(i)), with c = (m - 1, m - 1, m - 2, …, 1) for "holm", Holm’s step-down with Shaffer’s first step (Holm 1979; Shaffer 1986), or c = 1 for "none", for p-values already adjusted within each feature. Shaffer’s first multiplier is m - 1, not m, since the screen has rejected the hypothesis that every hypothesis of the feature holds; with one hypothesis it is 1, Bonferroni, where stageR’s 0 would call the hypothesis on the screen alone. A feature or hypothesis is called when its adjusted p-value is at most alpha, as stageR counts. The calls control the overall false discovery rate (OFDR) at alpha: the expected share of called features with at least one false call, the screen’s or a hypothesis’s. Heller et al. (2009) proved it under independence between features, and Van den Berge et al. (2017) expect it wherever Benjamini-Hochberg controls the false discovery rate. The share of false calls among all feature-hypothesis calls is not controlled. The values hold only at the alpha given. https://docs.nygen.io/CytePool/statistics/multiple-testing#stage-wise-testing explains the procedure.

Parameters:
  • screen_p_values – (n_features,) p-values of the screen, a test of every hypothesis of a feature at once, in [0, 1]; NaN leaves a feature out of the screen’s family.

  • p_values – (n_features, n_hypotheses) p-values of the hypotheses, each unadjusted ("holm") or adjusted within its feature ("none"), in [0, 1]; NaN leaves a hypothesis out of its feature’s family. Shaffer’s first step needs the screen to have tested the feature’s other hypotheses only: when it also tested those missing here, adjust each feature by Holm’s procedure first, adjust_p_values(p_values, "holm", axis=1), and pass the result with method="none".

  • alpha – The level of the calls, between 0 and 1.

  • method – "holm" (default) or "none".

Returns:

(n_features, n_hypotheses) stage-wise adjusted p-values, NaN for a feature that the screen does not call and for a missing p-value.

Raises:
  • ValueError – If the shapes disagree, a p-value is outside [0, 1] or infinite, alpha is not between 0 and 1, or method is unknown.

  • TypeError – If an argument has the wrong type or is a numpy masked array.

Example

>>> import numpy as np
>>> screen = np.array([1e-6, 1e-4, 0.3, 0.8])
>>> doses = np.array([[1e-5, 0.01, 0.2], [0.003, 0.003, 0.6],
...                   [0.01, 0.5, 0.5], [0.5, 0.5, 0.5]])
>>> stage_wise_adjusted_p_values(screen, doses).tolist()[:2]
[[4e-05, 0.04, 0.4], [0.012, 0.012, 1.0]]

Explained in: Choosing a method, Multiple testing, Dose response

cytepool.stats.multiple_contrasts.williams_contrasts(sizes)#

Williams-type contrasts: the control against the size-weighted mean of the highest one, two or more doses.

The first group is the control and the doses follow in increasing order. Contrast j = 1, …, K - 1 weighs the control -1 and each of the j highest doses n_i / (the sum of their sizes), as multcomp’s contrMat(n, type = "Williams") (Bretz 2006). Tested with multiple_contrast_test(), they detect a change that grows in dose order: two-sided for genes, whose direction is not known, which tests rises and falls at one error rate, and one-sided ("greater" or "less") when the direction is known. A significant contrast means that the doses it pools differ from the control on average, not that each of them does; which doses changed is the question of dunnett_contrasts(). They differ from Williams’ (1971) trend test, which compares each dose with the control after fitting means that change in dose order. https://docs.nygen.io/CytePool/statistics/multiple-testing#dunnett-and-williams-contrasts explains them.

Parameters:

sizes – The replicates of each group, as a mapping of group name to count, in dose order with the control first.

Returns:

The contrasts, named by the doses they pool, such as “high - vehicle”, “mid to high - vehicle” and “low to high - vehicle”, with the groups as the columns they weigh. With two groups, the one contrast is Dunnett’s.

Raises:
  • ValueError – If there are fewer than two groups, a count is not a positive integer, a name is empty or repeated, or a group is a model’s intercept.

  • TypeError – If sizes is not a mapping, a count is not a number, or an argument is a numpy masked array.

Example

>>> williams_contrasts({"vehicle": 4, "low": 3, "high": 3}).names
('high - vehicle', 'low to high - vehicle')

Explained in: Choosing a method, Multiple testing