Cell-type composition

Cell-type composition#

propeller() tests whether the proportions of cell types differ between conditions, as speckle’s propeller, propeller.ttest and propeller.anova do. It reads the number of cells of every sample in every group of a cube (or a table of them), transforms the proportions (logit or arcsine square root), fits a linear model with a Design matched to the samples by sample_id, moderates the variances of the cell types by empirical Bayes, and tests the design’s named contrasts by moderated t-statistics and the levels of a factor by a moderated F-statistic. The result holds speckle’s columns for every cell type, the samples and cell types left out, the proportions, and a record that identifies the cell numbers by the cube’s fingerprint.

Tests of differences in cell-type composition between conditions, by propeller.

propeller() tests, for every group (cell type) of a cube, whether the share of a sample’s cells that belong to the group differs between conditions, as speckle’s propeller does (Phipson et al. 2022). It reads only the number of cells of every sample in every group (Cube.n_cells), or a table of them, and fits one linear model to all groups at once:

  1. Proportions. Sample j, with x_ij cells of group i and N_j cells in all, has the proportion p_ij = x_ij / N_j of group i.

  2. Transform. The logit transform, log(q_ij / (1 - q_ij)), takes the proportions after half a cell is added to every count, q_ij = (x_ij + 0.5) / sum_i (x_ij + 0.5), so that a group absent from a sample has a finite value; the arcsine square-root transform, arcsin(sqrt(p_ij)), takes the proportions as they are (Phipson et al. 2022, supplementary section 1).

  3. Linear model. The transformed proportions of every group are fitted with the design by least squares (limma’s lmFit), and the contrasts are formed (contrasts.fit).

  4. Moderation. The residual variances of the groups are moderated by empirical Bayes, robustly by default, as speckle calls limma’s eBayes(robust = TRUE) (Smyth 2004; Phipson et al. 2016).

  5. Tests. Each contrast is tested by its moderated t-statistic, as propeller.ttest does. f_test names a factor whose levels are tested for equal means by the moderated F-statistic, as propeller.anova does, and propeller with more than two groups. The p-values of a test are adjusted by the method of Benjamini and Hochberg over the groups (speckle’s FDR).

Samples and groups:

The design’s sample table must have a sample_id column; its samples are matched to those of the data by sample_id, never by position, and may be some of them, in any order. A sample of the design without cells in any group has no proportions and is left out, and the result records why. Proportions are taken over the groups of the data, so to test the composition of one lineage, select its groups first (Cube.select(groups=...)). A group without cells in any sample used is left out too: it is not observed in the experiment, and speckle, which reads the label of every cell, never sees it. At least two groups must remain.

Descriptive columns:

Each table also gives speckle’s descriptive columns, whose definitions were found by probing speckle 1.6.0. The baseline proportion of a group is its share of all the cells of the samples used (speckle’s BaselineProp). The mean proportion of a level of a factor is the mean of p_ij over the samples at that level (PropMean), taken over the samples as they are, whatever the other terms of the design. A contrast compares the levels of a factor when it is a combination, with weights d_l that sum to zero, of the differences between the levels of a factor that enters the formula as a main effect only; for example conditionstim in ~ donor + condition, conditionstim - conditionctrl in ~ 0 + condition, or (groupA + groupB)/2 - groupC, but not groupA + groupB. The proportion ratio of such a contrast is the product of the mean proportions m_l of its levels raised to their weights, prod_l m_l^d_l (PropRatio): m_stim / m_ctrl for stim against ctrl, and sqrt(m_A m_B) / m_C for (groupA + groupB)/2 - groupC. The table of such a contrast gives the mean proportion of each level with a weight other than zero, and the ratio; the table of another contrast, such as a covariate’s slope, gives neither.

Deliberate differences from speckle:

validation/composition_tests.py reports each of them.

  • speckle’s PropMean are the coefficients of the proportions regressed on the columns of the design matrix that the contrast weighs, and its PropRatio is their product raised to the contrast’s weights (found by probing). Where those columns indicate groups of samples, as in a design without an intercept whose first columns are the groups, the coefficients are the groups’ mean proportions and the ratio is the one above. Otherwise they are not: for groupA + groupB, which compares nothing, speckle reports the product m_A m_B as a ratio; for (Intercept) + groupB in ~ group + age, the mean of the samples outside B and a difference of means; and for a contrast that weighs a single column, such as conditionstim in ~ condition or a covariate, it stops with an error. Here the levels a contrast compares are found from the design’s coding, so every parametrisation of the same comparison gives the same table, and a contrast that compares no levels is tested without descriptive columns.

  • A group without cells in any sample is left out. speckle keeps an unused level of a factor of cell labels, tests it, and with the logit transform its half cells change the transformed proportions of every other group (found by probing).

  • A sample without cells is left out. speckle keeps an unused level of a factor of sample labels: with the arcsine transform its proportions are NaN and limma leaves it out too, but with the logit transform every group of that sample has the proportion 1 / (number of groups), which enters the fit as data (found by probing).

  • Data that the design fits exactly in every group, up to rounding, as when the samples of each condition are copies of one another, raise an error: no variance can be estimated. speckle returns statistics of the rounding errors (t-statistics near 1e15, for example) or NaN.

Accuracy:

The transforms are computed as log((x_ij + 0.5) / (N_j - x_ij + 0.5 (I - 1))) for I groups and as arctan2(sqrt(x_ij), sqrt(N_j - x_ij)): the formulas above, without the cancellation in 1 - q_ij and the loss of digits of arcsin near 1. They agree with speckle’s values to rounding, and stay within a few units in the last place where those lose digits, for a group that holds nearly all the cells of a sample. Cell numbers are summed exactly, in float64 up to 2**53.

References

Phipson B, Sim CB, Porrello ER, Hewitt AW, Powell J, Oshlack A (2022). propeller: testing for differences in cell type proportions in single cell data. Bioinformatics 38, 4720-4726. doi:10.1093/bioinformatics/btac582

Smyth GK (2004). Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology 3, article 3. doi:10.2202/1544-6115.1027

Phipson B, Lee S, Majewski IJ, Alexander WS, Smyth GK (2016). Robust hyperparameter estimation protects against hypervariable genes and improves power to detect differential expression. Annals of Applied Statistics 10, 946-963. doi:10.1214/16-AOAS920

Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289-300. doi:10.1111/j.2517-6161.1995.tb02031.x

class cytepool.composition.CompositionResult(contrasts, tables, f_test, summary, samples, proportions, provenance)#

Tests of differences in cell-type composition, from propeller().

Tables hold one row per group (cell type) tested, in the order of the data’s groups; speckle sorts them by p-value, as table.take(numpy.argsort( table["p_value"], kind="stable")) does. The table of a contrast (tables, table) has the columns

  • group_id;

  • baseline_proportion: the group’s share of all the cells of the samples used (speckle’s BaselineProp);

  • for a contrast that compares the levels of a factor (see cytepool.composition), mean_proportion_<level> for each level it weighs, in level order: the mean proportion of the group over the samples at that level (PropMean), and proportion_ratio: the product of these means raised to the weights of their levels, such as m_stim / m_ctrl (PropRatio);

  • t and p_value: the moderated t-statistic of the contrast on the transformed proportions and its two-sided p-value (Tstatistic, P.Value);

  • adjusted_p_value: Benjamini-Hochberg over the groups (FDR).

The F-test table (f_test) has group_id, baseline_proportion, mean_proportion_<level> for every level of the factor, f_statistic, p_value and adjusted_p_value (speckle’s Fstatistic, P.Value and FDR).

The summary has one row per group of the data: group_id; n_cells, its cells in the samples used; tested; reason, why it was not tested (empty when tested); df_residual; and prior_df, the prior degrees of freedom of the moderation (each group has its own with robust moderation; NaN for a group not tested). The sample table has one row per design sample, in the data’s sample order: sample_id, n_cells (in all groups), used and reason. The proportions table has one row per group tested and sample used: group_id, sample_id, n_cells, proportion and transformed_proportion, the value the linear model was fitted to.

Built by propeller() from the tables and record it computed. Immutable.

Raises:

  • TypeError – If contrasts is not a Contrasts, tables not a mapping of contrast name to Table, f_test not a Table or None, or summary, samples or proportions not a Table, or provenance not a mapping that JSON can write.

  • ValueError – If provenance holds a NaN or an infinity.

Explained in: Composition tests

property contrasts#

The contrasts tested by t-tests, over the columns of the design.

property tables#

The table of every contrast, by contrast name.

property f_test#

The table of the F-test across the levels of f_test, or None.

property summary#

Every group of the data, with its cells, whether it was tested, and its df.

property samples#

Every design sample, used or left out, and why.

property proportions#

The proportion and transformed proportion of every group in every sample.

property settings#

The settings of the analysis, as a new dict (also in the provenance).

property provenance#

A JSON-serialisable record of what produced every number, as a new dict.

It holds the method; the CytePool, numpy, numba and Python versions; what the data were ("input": "cube" or "table"); the cube ("cube"): its fingerprint, the digests it is built from and its own provenance, where a table of cell numbers is recorded as the cube without features that holds them, whose samples, groups and n_cells digests identify the table; the design (formula, columns, levels, the values of its variables for every sample, and the weights of every contrast); the settings; the hypothesis of the F-test (the factor, its levels and the contrasts between them), or null; the samples used, in the order fitted, and those left out with the reason; and the groups tested and those left out with the reason. The fingerprint, and the digest of n_cells, say whether a cube at hand holds the cell numbers behind the tables (Cube.verify_fingerprint).

table(contrast)#

Return the table of one contrast.

Parameters:

contrast – A contrast name.

Raises:

KeyError – If the contrast was not tested; the message lists those that were.

Explained in: Composition tests

cytepool.composition.propeller(data, design, *, contrasts=None, f_test=None, transform='logit', robust=True, trend=False)#

Test whether the cell-type composition of samples differs between conditions.

For every group (cell type), the proportion of each sample’s cells in the group is transformed, a linear model is fitted to the transformed proportions with the design, and its contrasts are tested by moderated t-statistics, and the levels of a factor by a moderated F-statistic, after empirical Bayes moderation of the residual variances across groups: speckle’s propeller, propeller.ttest and propeller.anova. The module documentation describes the steps, the descriptive columns and the deliberate differences from speckle.

As in speckle, propeller(clusters, sample, group) with two groups tests the first level against the second: Design(samples, "~ 0 + group", contrasts={"A_vs_B": "groupA - groupB"}); with more groups it is f_test="group". propeller.ttest with a design matrix of group columns followed by covariates is a design such as ~ 0 + group + age with a contrast between groups, and propeller.anova the F-test of group in it. The same comparisons in another parametrisation, such as ~ group + age with the contrast groupB, give the same tables.

Provenance:

The result records the cube’s fingerprint, the digests it is built from and its own provenance, as limma_voom() does, although the test reads only the cell numbers. For a cube opened with read_cube() this reads nothing, as the file stores the digests; a cube in memory hashes its layers once, at about 1 s per GB of values where the processor has SHA instructions, and keeps the digests.

Parameters:

  • data – The number of cells of every sample in every group: a Cube that records them (n_cells), or a table with one row per sample, a text column sample_id and one column of cell numbers (integers) per group, named by its group id: a Table, a mapping of columns, or a pandas DataFrame. A DataFrame without a sample_id column is read with its index as the sample ids, as pandas.crosstab(cells["sample_id"], cells["cell_type"]) gives it.

  • design – A Design whose sample table has a sample_id column with sample ids of the data; its rows may be some of the samples, in any order. A design with random effects is refused: the fit has fixed effects only.

  • contrasts – The contrasts to test by t-tests, as accepted by make_contrasts() and matched to the design’s columns by name, such as {"stim_vs_ctrl": "conditionstim"}. By default the design’s named contrasts; an empty list tests none.

  • f_test – The name of a factor of the design, entering its formula as a main effect only, whose levels are tested for equal means by an F-test (speckle’s ANOVA), or None for no F-test. To compare some of its levels only, test their differences as contrasts.

  • transform – "logit" (speckle’s default), with half a cell added to every count, or "asin" for the arcsine square root.

  • robust – Whether to estimate the prior of the moderation robustly, as speckle does by default (eBayes(robust = TRUE)), which protects against groups whose variance is unusually large.

  • trend – Whether the prior variance follows the mean transformed proportion of the groups (eBayes(trend = TRUE)).

Returns:

The table of every contrast and of the F-test, a summary of every group, the samples used and left out, the proportions, the settings and a provenance record that identifies the data by their fingerprint.

Raises:

  • TypeError – If data is not a Cube or a table of cell numbers, a column of cell numbers does not hold integers, an id of the data or the design’s sample_id is not text, design is not a Design, an option has the wrong type, or data or contrasts is or holds a numpy masked array.

  • ValueError – If the cube records no cell numbers; the table has no rows or no group columns, or an id that is missing, empty or repeated; a cell number is negative or a sample has 2**53 cells or more; the design has random effects or no sample_id column, names samples the data do not have or names a sample twice; a contrast is invalid, or there is nothing to test; transform is neither "logit" nor "asin"; f_test does not name a factor that enters the formula as a main effect only; fewer than two groups have cells; the design is not of full column rank or leaves no residual degrees of freedom on the samples used; or it fits the transformed proportions of every group exactly.

Explained in: Choosing a method, Composition tests