Multiple contrast tests#
Dunnett’s, Williams-type and optimal contrasts, single-step tests of several contrasts of every gene, and stage-wise calls of the contrasts across genes; Several contrasts of one gene explains them.
Multiple contrast tests: which treatments or doses differ from the control, with one error rate for the contrasts of each feature.
A dose series asks, for each gene, which doses differ from the vehicle. Testing each
contrast on its own and correcting by Bonferroni ignores that the contrasts share the
vehicle, which makes their t-statistics correlated, and loses power; an F-test says
whether any dose differs, not which. multiple_contrast_test() tests every
contrast of every feature and adjusts each p-value for the contrasts of its feature
by the single-step max-t method (Hothorn, Bretz and Westfall 2008): the adjusted
p-value of contrast j is the chance that the largest of the feature’s correlated
t-statistics is at least as extreme as t_j, from the multivariate t distribution
(multivariate_t_cdf()). It takes a moderated fit,
or estimates and their covariance as multcomp’s glht does with parm().
dunnett_contrasts() builds Dunnett’s contrasts, each treatment against the
control (Dunnett 1955), and williams_contrasts() the Williams-type contrasts,
the control against the size-weighted mean of the highest one, two or more doses, for
means that change in dose order (Bretz 2006), as multcomp’s contrMat;
optimal_contrasts() builds the contrasts that best detect each of several
candidate shapes, for the MCP step, the multiple comparison procedure, of MCP-Mod
(multiple comparison procedure and modelling; Bretz, Pinheiro and Branson 2005;
Pinheiro et al. 2014), as DoseFinding’s optContr.
https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains
the tests.
Across features, the contrasts of the features that a screen calls are called by
stage_wise_adjusted_p_values(), stageR’s stage-wise procedure (Heller et al.
2009; Van den Berge et al. 2017): Holm’s step-down within each feature (Holm 1979),
with Shaffer’s (1986) first step, at the level that the screen leaves
(https://docs.nygen.io/CytePool/statistics/multiple-testing#stage-wise-testing).
- Differences from multcomp:
dfneed not be an integer, as moderated degrees of freedom are not, and has no default with estimates; multcomp’s default of 0 gives z-tests, herenp.inf.Williams-type contrasts are named by the doses they pool, such as “mid to high - vehicle”, where
contrMatnames them “C 1”, “C 2”, …; with two groups they are Dunnett’s single contrast, whichcontrMatrefuses.Each adjusted p-value carries a standard error of at most
relative_errorof it (1% by default) at every magnitude, where mvtnorm’s default allows an absolute error of 1e-3.
References
Dunnett CW (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association 50, 1096-1121. doi:10.1080/01621459.1955.10501294
Williams DA (1971). A test for differences between treatment means when several dose levels are compared with a zero dose control. Biometrics 27, 103. doi:10.2307/2528930
Bretz F (2006). An extension of the Williams trend test to general unbalanced linear models. Computational Statistics and Data Analysis 50, 1735-1748. doi:10.1016/j.csda.2005.02.005
Hothorn T, Bretz F, Westfall P (2008). Simultaneous inference in general parametric models. Biometrical Journal 50, 346-363. doi:10.1002/bimj.200810425
Bretz F, Pinheiro JC, Branson M (2005). Combining multiple comparisons and modeling techniques in dose-response studies. Biometrics 61, 738-748. doi:10.1111/j.1541-0420.2005.00344.x
Pinheiro J, Bornkamp B, Glimm E, Bretz F (2014). Model-based dose finding under model uncertainty using general parametric models. Statistics in Medicine 33, 1646-1661. doi:10.1002/sim.6052
Holm S (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65-70.
Shaffer JP (1986). Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association 81, 826-831. doi:10.1080/01621459.1986.10478341
Heller R, Manduchi E, Grant GR, Ewens WJ (2009). A flexible two-stage procedure for identifying gene sets that are differentially expressed. Bioinformatics 25, 1019-1025. doi:10.1093/bioinformatics/btp076
Van den Berge K, Soneson C, Robinson MD, Clement L (2017). stageR: a general stage-wise method for controlling the gene-level false discovery rate in differential expression and differential transcript usage. Genome Biology 18, 151. doi:10.1186/s13059-017-1277-0
- class cytepool.stats.multiple_contrasts.MultipleContrastTest(names, estimate_names, contrasts, estimate, standard_error, t, p_value, p_value_error, correlation, df, alternative, relative_error)#
Multiple contrast tests of every feature, adjusted for the feature’s contrasts.
For each feature and contrast c: the estimate c’b, its standard error sqrt(c’ V c), t, and the single-step adjusted p-value, the chance that the largest of the feature’s t-statistics on its
dfis at least as extreme as t (Hothorn, Bretz and Westfall 2008). The arrays are read-only. The repr counts the features whoseadjusted_p_valueoftable()is below 0.05, and, asp_values_short_of_relative_error, the p-values whose standard error exceedsrelative_errorof the smaller of p and 1 - p, which happens only when the computation stopped at its limit of 2^20 points: use them with theirp_value_error, or ask for a largerrelative_error.- Variables:
names – The contrast names, which label the columns of
estimate,tand the other(n_features, n_contrasts)arrays.estimate_names – The names of the estimates, which label the rows of
contrasts.contrasts –
(n_estimates, n_contrasts)weights of each contrast.estimate –
(n_features, n_contrasts)estimates of the contrasts.standard_error –
(n_features, n_contrasts)their standard errors.t –
(n_features, n_contrasts)t-statistics.p_value –
(n_features, n_contrasts)p-values adjusted for the contrasts of the feature (single-step): P(max_l |T_l| >= |t|) two-sided, P(max_l T_l >= t) for"greater"and P(min_l T_l <= t) for"less". Within a feature they never decrease as t becomes less extreme.p_value_error –
(n_features, n_contrasts)the standard error of each p-value (the quadrature’s error bound with two contrasts, 0 with one).correlation –
(n_features, n_contrasts, n_contrasts)the correlation of each feature’s t-statistics, whichmultivariate_t_quantile()turns into the critical value of simultaneous confidence intervals.df –
(n_features,)degrees of freedom of the t-statistics.alternative –
"two-sided","greater"or"less".relative_error – The requested standard error of the smaller of each p-value and 1 minus it, as a fraction of it.
Explained in: Multiple testing
- table(contrast=None, *, feature_ids=None)#
Return the tests as a table, one row per feature.
Without
contrast, the test of each feature as a whole, the max-t test of its contrasts:feature_id;contrast, the contrast with the most extreme t, whose adjusted p-value is the feature’s; itsestimate,standard_error,t,p_valueandp_value_error; andadjusted_p_value, Benjamini and Hochberg’s adjustment ofp_valueover the features. Withcontrast: itsestimate,standard_error,t,p_valueandp_value_error. Thisp_valueis adjusted for the feature’s contrasts, unlike the p-values of other CytePool tables; across features, call the features from the table withoutcontrastfirst. https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains the two levels.- Parameters:
contrast – A contrast, by name or position; None for the test of each feature as a whole.
feature_ids – One id per feature, as text; by default
feature_0,feature_1, …
- Raises:
KeyError – If no contrast has the name.
IndexError – If the position is out of range.
ValueError – If
feature_idsdoes not give one id per feature.TypeError – If an id is not text,
contrastis neither a name nor a position, or an argument is or holds a numpy masked array.
Explained in: Multiple testing
- cytepool.stats.multiple_contrasts.dunnett_contrasts(groups, *, control=0)#
Dunnett’s contrasts: each treatment or dose against the control.
Contrast “{group} - {control}” weighs the group 1 and the control -1, as multcomp’s
contrMat(n, type = "Dunnett")names and weighs them; the contrasts follow the order of the groups. Dunnett’s test (Dunnett 1955) ismultiple_contrast_test()of these contrasts. https://docs.nygen.io/CytePool/statistics/multiple-testing#dunnett-and-williams-contrasts explains them.- Parameters:
groups – The group names, in order, as a sequence, or the keys of a mapping such as the group sizes that
williams_contrasts()takes (Dunnett’s weights do not depend on the sizes), such as the coefficient names of a fit with one coefficient per group.control – The control group, by name or 0-based position; the first group by default.
- Returns:
The contrasts, with the groups as the columns they weigh.
- Raises:
ValueError – If there are fewer than two groups, a name is empty or repeated, a group is a model’s intercept, or
controlis not one of the groups.TypeError – If
groupsis a single string, a set (which has no order), or names a group with something other than text, or an argument is a numpy masked array.
Example
>>> dunnett_contrasts(["vehicle", "low", "high"]).names ('low - vehicle', 'high - vehicle')
Explained in: Choosing a method, Multiple testing
- cytepool.stats.multiple_contrasts.multiple_contrast_test(estimates, covariances=None, contrasts=None, *, df=None, estimate_names=None, alternative='two-sided', relative_error=0.01, rng=None, n_threads=None)#
Test several contrasts of every feature, with one error rate per feature.
For each feature, the contrasts C’ b of its estimates b, their standard errors from C’ V C for its covariance V, and their t-statistics, which are correlated through V; each p-value is adjusted for the feature’s contrasts by the single-step max-t method (Hothorn, Bretz and Westfall 2008), from the multivariate t distribution on the feature’s
dfwith the correlation of C’ V C (multivariate_t_cdf()), as multcomp’ssummary(glht(parm(b, V, df), linfct = t(C))). Withdunnett_contrasts()this is Dunnett’s test of each dose against the vehicle; withwilliams_contrasts(), the Williams-type test of a change that grows in dose order. https://docs.nygen.io/CytePool/statistics/multiple-testing#correlated-tests explains the tests.A
ModeratedFit, such aslimma_voom(...).fit(group).moderated, gives everything but the contrasts: its coefficients (or contrasts), their covariance, the posterior variance times the fit’s unscaled covariance, the df of its moderated t-statistics and its coefficient names. Contrasts that name the estimates they weigh, such as those ofdunnett_contrasts()andwilliams_contrasts(), are matched to the estimate names by name, never by position; other weights are taken in the order of the estimates. A feature with a missing or infinite estimate or covariance, or a contrast of zero variance, gets NaN for every contrast.- Parameters:
estimates – A
ModeratedFit; or(n_features, n_estimates)estimates, such as group means or the coefficients of a fit, or(n_estimates,)for one feature, withcovariancesanddf.covariances – The covariance matrices of the estimates,
(n_features, n_estimates, n_estimates), or one(n_estimates, n_estimates)matrix for every feature; symmetric and positive semidefinite. Left out with aModeratedFit.contrasts – The contrasts to test: a
Contrasts, a mapping of name to weights, or a(n_estimates, n_contrasts)matrix; None tests the estimates themselves, such as the contrasts of a fit.df – Degrees of freedom of the t-statistics: one number, or one per feature, positive;
np.inffor z-tests. Need not be integers, as moderated degrees of freedom are not. Left out with aModeratedFit.estimate_names – The names of the estimates, in order, which named contrasts are matched to; required by them. Left out with a
ModeratedFit.alternative –
"two-sided","greater"(the contrasts are larger than 0) or"less".relative_error – The largest standard error of the smaller of each adjusted p-value and 1 minus it, as a fraction of it, between 0 and 1.
rng – A numpy random generator, an integer seed, or None; a seed gives the same p-values whatever the number of threads. Two or fewer contrasts use no random numbers.
n_threads – The most threads to use; None uses numba’s setting. The results do not depend on it.
- Returns:
The tests, with one table per contrast and one of each feature as a whole.
- Raises:
ValueError – If the shapes disagree, named contrasts do not weigh exactly the estimates or come without their names, a contrast is all zero,
dfis not positive or is not one number per feature, the covariance of a feature is not symmetric or gives its contrasts a covariance that is not positive semidefinite,alternativeis unknown,relative_erroris not between 0 and 1, aModeratedFitcomes withcovariances,dforestimate_names, or its fit has limma’s approximate standard errors of contrasts.TypeError – If
covariancesordfis missing with estimates, an argument has the wrong type or is a numpy masked array.
Example
Dunnett’s test of group means, with the covariance of the means of a one-way model: the pooled variance over each group’s replicates, on N - K degrees of freedom (moderated ones need not be integers).
>>> import numpy as np >>> groups = ["vehicle", "low", "high"] >>> replicates = np.array([4, 4, 4]) >>> means = np.array([[5.0, 5.2, 6.1]]) >>> pooled_variance = 0.2 >>> test = multiple_contrast_test( ... means, pooled_variance * np.diag(1 / replicates), ... dunnett_contrasts(groups), df=replicates.sum() - replicates.size, ... estimate_names=groups) >>> test.p_value[0].round(4).tolist() # low and high against the vehicle [0.7587, 0.0127]
Used in: Bulk RNA-seq
Explained in: Choosing a method, Multiple testing, Dose response
- cytepool.stats.multiple_contrasts.optimal_contrasts(shapes, covariance, *, groups=None)#
The contrasts of group means that best detect each of several shapes: the optimal contrasts of the MCP step, the multiple comparison procedure, of MCP-Mod (multiple comparison procedure and modelling).
For a shape mu, the means it expects in k groups, and the covariance S of the groups’ estimates up to a factor, the contrast c that maximises the non-centrality c’mu / sqrt(c’ S c) of its t-statistic among contrasts (weights that sum to 0) is proportional to S^-1 (mu - m 1), with m = (1’ S^-1 mu) / (1’ S^-1 1) (Bretz, Pinheiro and Branson 2005; Pinheiro et al. 2014). Each contrast is scaled to unit length; c’mu is then positive, so the contrast points the way the shape rises. For independent groups of n_i samples, S = diag(1 / n_i) and c_i is proportional to n_i (mu_i - the size-weighted mean of mu), DoseFinding’s
optContr(models, w = n); a general S givesoptContr(models, S = S). A shape and the same shape scaled by a positive factor or shifted give the same contrast, so only the shapes’ directions matter. Test the contrasts withmultiple_contrast_test(). https://docs.nygen.io/CytePool/statistics/dose-response#dose-response-contrasts explains them.- Parameters:
shapes – The shapes, a mapping of name to the k means each expects; for
CandidateShapes,dict(zip(shapes.names, shapes.means(doses))).covariance –
(k, k)the covariance of the groups’ estimates, up to a factor: symmetric and positive definite, such asnp.diag(1 / n).groups – The k group names, which the contrasts weigh;
"1"to"k"by default, asdunnett_contrasts()names unnamed groups.
- Returns:
One contrast per shape, named after it, with the groups as the columns it weighs.
- Raises:
ValueError – If there are fewer than two groups or no shape, the shapes or
groupsdisagree withcovariancein size, a value is not finite, a shape is constant (its values within 1e-8 of their largest magnitude of each other), which no contrast detects, the covariance is not symmetric or not positive definite, or a name is empty or repeated.TypeError – If
shapesis not a mapping, an argument has the wrong type, or an argument is a numpy masked array.
Example
Groups of 2, 1 and 1 samples and a shape rising by 1 per group: the size-weighted mean is 0.75, so the contrast is proportional to 2 (0 - 0.75), 1 (1 - 0.75) and 1 (2 - 0.75), (-1.5, 0.25, 1.25), here of unit length.
>>> import numpy as np >>> shapes = {"linear": [0.0, 1.0, 2.0]} >>> contrasts = optimal_contrasts(shapes, np.diag(1 / np.array([2.0, 1, 1]))) >>> contrasts.matrix[:, 0].round(3).tolist() [-0.762, 0.127, 0.635]
Explained in: Multiple testing, Dose response
- cytepool.stats.multiple_contrasts.stage_wise_adjusted_p_values(screen_p_values, p_values, *, alpha=0.05, method='holm')#
Adjust several p-values per feature in two stages, after a screen of the features, so that the overall false discovery rate of the features holds.
Stage-wise testing (Heller et al. 2009; Van den Berge et al. 2017), as stageR’s
stageWiseAdjustment: the screen calls the features whose Benjamini-Hochberg adjusted screen p-value is at mostalpha, R of the G features with a screen p-value; then, in each called feature, its p-values are adjusted for the feature’s hypotheses and scaled by G / R, the level of R alpha / G that the screen leaves. With the feature’s m p-values sorted, p_(1) <= … <= p_(m), the adjusted p-value of p_(j) is min(1, (G / R) max over i <= j of c_i p_(i)), with c = (m - 1, m - 1, m - 2, …, 1) for"holm", Holm’s step-down with Shaffer’s first step (Holm 1979; Shaffer 1986), or c = 1 for"none", for p-values already adjusted within each feature. Shaffer’s first multiplier is m - 1, not m, since the screen has rejected the hypothesis that every hypothesis of the feature holds; with one hypothesis it is 1, Bonferroni, where stageR’s 0 would call the hypothesis on the screen alone. A feature or hypothesis is called when its adjusted p-value is at mostalpha, as stageR counts. The calls control the overall false discovery rate (OFDR) atalpha: the expected share of called features with at least one false call, the screen’s or a hypothesis’s. Heller et al. (2009) proved it under independence between features, and Van den Berge et al. (2017) expect it wherever Benjamini-Hochberg controls the false discovery rate. The share of false calls among all feature-hypothesis calls is not controlled. The values hold only at thealphagiven. https://docs.nygen.io/CytePool/statistics/multiple-testing#stage-wise-testing explains the procedure.- Parameters:
screen_p_values –
(n_features,)p-values of the screen, a test of every hypothesis of a feature at once, in [0, 1]; NaN leaves a feature out of the screen’s family.p_values –
(n_features, n_hypotheses)p-values of the hypotheses, each unadjusted ("holm") or adjusted within its feature ("none"), in [0, 1]; NaN leaves a hypothesis out of its feature’s family. Shaffer’s first step needs the screen to have tested the feature’s other hypotheses only: when it also tested those missing here, adjust each feature by Holm’s procedure first,adjust_p_values(p_values, "holm", axis=1), and pass the result withmethod="none".alpha – The level of the calls, between 0 and 1.
method –
"holm"(default) or"none".
- Returns:
(n_features, n_hypotheses)stage-wise adjusted p-values, NaN for a feature that the screen does not call and for a missing p-value.- Raises:
ValueError – If the shapes disagree, a p-value is outside [0, 1] or infinite,
alphais not between 0 and 1, ormethodis unknown.TypeError – If an argument has the wrong type or is a numpy masked array.
Example
>>> import numpy as np >>> screen = np.array([1e-6, 1e-4, 0.3, 0.8]) >>> doses = np.array([[1e-5, 0.01, 0.2], [0.003, 0.003, 0.6], ... [0.01, 0.5, 0.5], [0.5, 0.5, 0.5]]) >>> stage_wise_adjusted_p_values(screen, doses).tolist()[:2] [[4e-05, 0.04, 0.4], [0.012, 0.012, 1.0]]
Explained in: Choosing a method, Multiple testing, Dose response
- cytepool.stats.multiple_contrasts.williams_contrasts(sizes)#
Williams-type contrasts: the control against the size-weighted mean of the highest one, two or more doses.
The first group is the control and the doses follow in increasing order. Contrast j = 1, …, K - 1 weighs the control -1 and each of the j highest doses n_i / (the sum of their sizes), as multcomp’s
contrMat(n, type = "Williams")(Bretz 2006). Tested withmultiple_contrast_test(), they detect a change that grows in dose order: two-sided for genes, whose direction is not known, which tests rises and falls at one error rate, and one-sided ("greater"or"less") when the direction is known. A significant contrast means that the doses it pools differ from the control on average, not that each of them does; which doses changed is the question ofdunnett_contrasts(). They differ from Williams’ (1971) trend test, which compares each dose with the control after fitting means that change in dose order. https://docs.nygen.io/CytePool/statistics/multiple-testing#dunnett-and-williams-contrasts explains them.- Parameters:
sizes – The replicates of each group, as a mapping of group name to count, in dose order with the control first.
- Returns:
The contrasts, named by the doses they pool, such as “high - vehicle”, “mid to high - vehicle” and “low to high - vehicle”, with the groups as the columns they weigh. With two groups, the one contrast is Dunnett’s.
- Raises:
ValueError – If there are fewer than two groups, a count is not a positive integer, a name is empty or repeated, or a group is a model’s intercept.
TypeError – If
sizesis not a mapping, a count is not a number, or an argument is a numpy masked array.
Example
>>> williams_contrasts({"vehicle": 4, "low": 3, "high": 3}).names ('high - vehicle', 'low to high - vehicle')
Explained in: Choosing a method, Multiple testing