Pathway and regulator activity

Pathway and regulator activity#

ulm(), mlm() and weighted_aggregate() score every source of a network, a transcription factor or a pathway, in every profile: the samples of an expression matrix or of a cell type of a cube, or the contrasts of a differential-expression result, as decoupler’s mt.ulm, mt.mlm and mt.waggr compute them. The activities, an Activities, are profiles x sources, so that fit_linear_model() tests them with a design built from their profiles; plot_activities() draws them as a heatmap and plot_top_activities() the top sources of a profile as bars. The networks come from cytepool.networks.

Activities of transcription factors and pathways, inferred from the expression of their target genes, as decoupler computes them: the univariate and multivariate linear models (ULM, MLM) and the weighted aggregate.

A network (Network), such as CollecTRI’s regulons or PROGENy’s pathway signatures (cytepool.networks), gives each source, a transcription factor or a pathway, a signed weight on each of its target genes. A source is active in a profile, the expression of a sample or the statistics of a contrast, when its targets are high where their weights are positive and low where they are negative. Each function scores every source in every profile, as decoupler 2.2.0’s mt.ulm, mt.mlm and mt.waggr do (Badia-i-Mompel et al. 2022), and returns an Activities, profiles x sources, which the linear models take as they take pathway scores, and which plot_activities() draws as a heatmap and plot_top_activities() as bars.

The profiles:

  • A samples x genes array, one profile per row, with one gene id per column.

  • A group (cell type) of a Cube: its samples with at least min_cells cells and some counts, over the genes that edgeR’s filterByExpr keeps, as log2 counts per million after TMM normalization, the values that gsva() scores.

  • The statistics of a DifferentialExpressionResult: one profile for every group and contrast tested, by default the moderated t-statistics, as decoupler’s guides score the statistics of a contrast; or of a table of results, a pandas Series or a mapping of gene id to statistic, one profile.

Each profile’s genes are those with an id and a value: NaN marks a missing value, and a gene that a group did not test has none in that group’s profiles. Genes whose values are zero in every profile are left out, as decoupler leaves them out: in sparse data they are genes that were not detected.

The network’s targets:

Targets are matched to the genes’ ids exactly. An edge whose weight is zero is no edge. A source with fewer than min_targets targets among a profile’s genes has no score there (NaN), and a source with fewer in every profile is left out, as decoupler’s tmin leaves it out; n_targets gives each source’s targets among each profile’s genes.

ULM (ulm()):

For each source and profile, a linear model of the profile’s n genes on the source’s weights, zero for the genes that are not its targets:

y_g = b_0 + b_1 x_g + e_g,

fitted by least squares. The activity score is the t-statistic of the slope, t = b_1 / se(b_1) = r sqrt((n - 2) / (1 - r^2)), with r the correlation of y and x, and its p-value is two-sided, from Student’s t distribution on n - 2 degrees of freedom. estimates are the slopes b_1.

MLM (mlm()):

One linear model of each profile’s genes on the weights of all its sources at once, with an intercept, y_g = b_0 + sum_s b_s x_gs + e_g, so that each source’s score, the t-statistic of b_s on n - m - 1 degrees of freedom for m sources, is that of its targets beyond what the other sources explain. The model is fitted by a QR decomposition. Sources whose weights over the genes are linear combinations of the weights of other sources, such as two sources with the same targets and weights, cannot be told apart, and are refused: the message names them.

Weighted aggregate (weighted_aggregate()):

The weighted mean of the values of a source’s targets, sum_g w_g y_g / sum_g |w_g|, or their weighted sum, sum_g w_g y_g, the score of the original PROGENy method (Schubert et al. 2018). With n_permutations random permutations of the genes, each permutation giving every gene the value of another, as decoupler draws them (its seed is rng), the score is normalized by the scores of the permutations, (ES - mean) / sd, the standard deviation with n - 1 in its denominator, and the p-value doubles the smaller of the two one-sided permutation p-values, (1 + b) / (1 + n_permutations), with b the permutations whose score is at least, or at most, the observed one (Phipson and Smyth 2010). estimates keep the weighted means or sums.

Adjusted p-values:

adjusted_p_values are Benjamini and Hochberg’s (1995) over the sources of each profile, as decoupler adjusts them.

Deliberate differences from decoupler 2.2.0:

  • decoupler reports MLM’s p-values without adjusting them, under the name of its adjusted p-values; here they are p_values, and adjusted_p_values are adjusted as for the other methods.

  • decoupler keeps adjusted p-values in single precision, and computes MLM’s p-values as one minus the distribution function, which gives 0 below about 1e-16; here both are in double precision, from the upper tail.

  • decoupler counts the permutations of the weighted aggregate whose absolute score exceeds the absolute observed score, and then folds that share as it would fold a one-sided one, so that a score near zero, which most permutations exceed, gets a p-value near 2 / n_permutations; here the p-value doubles the smaller one-sided p-value. When every permutation scores alike, decoupler’s normalized score is infinite, or 0 for a score of 0; here it is NaN.

  • decoupler adds 2.2e-16 to 1 - r and 1 + r in ULM’s t-statistic; here it is exact, which changes the score only for |r| within about 1e-6 of 1.

  • decoupler refuses missing values and leaves out samples whose values are all zero; here each profile uses its observed genes, and a profile without genes has NaN scores. Where MLM cannot separate sources, decoupler stops with a singular matrix; here the message names them.

References

Badia-i-Mompel P, Velez Santiago J, Braunger J, Geiss C, Dimitrov D, Mueller-Dott S, Taus P, Dugourd A, Holland CH, Ramirez Flores RO, Saez-Rodriguez J (2022). decoupleR: ensemble of computational methods to infer biological activities from omics data. Bioinformatics Advances 2, vbac016. doi:10.1093/bioadv/vbac016

Schubert M, Klinger B, Kluenemann M, Sieber A, Uhlitz F, Sauer S, Garnett MJ, Bluethgen N, Saez-Rodriguez J (2018). Perturbation-response genes reveal signaling footprints in cancer gene expression. Nature Communications 9, 20. doi:10.1038/s41467-017-02391-6

Phipson B, Smyth GK (2010). Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology 9, article 39. doi:10.2202/1544-6115.1585

Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289-300. doi:10.1111/j.2517-6161.1995.tb02031.x

Welford BP (1962). Note on a method for calculating corrected sums of squares and products. Technometrics 4, 419-420. doi:10.1080/00401706.1962.10490022

class cytepool.activity.Activities(scores, estimates, p_values, adjusted_p_values, n_targets, profiles, network, method, settings)#

Activities of the sources of a network in every profile.

Rows of the arrays are profiles, in the order of profiles, and columns are sources, in the order of network.sources, so that fit_linear_model() takes scores as its values, with a design built from profiles. The arrays are read-only.

Variables:

  • scores – (n_profiles, n_sources) activity scores: the t-statistics of ULM and MLM, and of the weighted aggregate the scores normalized by permutations, or, without them, the weighted means or sums. NaN where a source has fewer than min_targets targets among a profile’s genes.

  • estimates – The slopes (ULM), coefficients (MLM) or weighted means or sums (weighted aggregate).

  • p_values – Two-sided p-values: of the t-tests (ULM, MLM), or of the permutations (weighted aggregate; NaN without them).

  • adjusted_p_values – Benjamini and Hochberg’s, over the sources of each profile.

  • n_targets – (n_profiles, n_sources) int64: the targets of each source among each profile’s genes.

  • profiles – One row per profile: profile_id and, for a cube’s group, the columns of the cube’s sample table, or, for a differential-expression result, group_id and contrast.

  • network – The sources scored, with their targets among the genes and the record of the network given.

  • method – "ulm", "mlm" or "weighted_aggregate".

  • settings – The options of the method and how the profiles were read.

Explained in: Gene-set tests

property sources#

The sources scored, in the order of the columns.

table()#

Return the activities as a table with one row per source and profile.

Returns:

A table with source, profile_id, n_targets, score, estimate, p_value and adjusted_p_value, the rows of each source together, in the order of sources, and its profiles in the order of profiles.

Explained in: Gene-set tests

cytepool.activity.mlm(data, network, *, gene_ids=None, sample_ids=None, group=None, contrast=None, min_cells=MIN_CELLS, expression_filter=True, id_column='feature_id', statistic_column='t', min_targets=_MIN_TARGETS)#

Score the sources of a network in every profile by one multivariate linear model (MLM).

As decoupler’s mt.mlm (Badia-i-Mompel et al. 2022): the values of a profile’s genes are regressed on the weights of all the sources at once, with an intercept, and each source’s score is the t-statistic of its coefficient, which measures its targets beyond what the other sources explain. The module docstring gives the profiles, the matching of targets and the deliberate differences from decoupler.

Parameters:

  • data – The profiles, as for ulm().

  • network – The network, as for ulm().

  • gene_ids – For an array or a cube, as for ulm().

  • sample_ids – For an array, as for ulm().

  • group – For a cube or a differential-expression result, as for ulm().

  • contrast – For a differential-expression result, as for ulm().

  • min_cells – For a cube, as for ulm().

  • expression_filter – For a cube, as for ulm().

  • id_column – For a differential-expression result or a table, as for ulm().

  • statistic_column – For a differential-expression result or a table, as for ulm().

  • min_targets – The fewest targets among a profile’s genes for a source to be in its model, at least 1 (decoupler’s tmin).

Returns:

The activities, with the t-statistics as scores and the coefficients as estimates.

Raises:

  • TypeError – If an argument has the wrong type, values are not real numbers, gene ids are not strings, or an argument is or holds a numpy masked array.

  • ValueError – If the values are not 2-D or hold an infinite value, the ids do not match the values, an id names two genes, an option is out of range or applies to another kind of data, a table lacks a column, no source has min_targets targets among the genes, a profile has no more genes than sources, or the weights of some sources are linear combinations of those of others.

  • KeyError – If the cube has no such group or features column, or the result has no such group or contrast.

Used in: Pathway and regulator activity

Explained in: Choosing a method, Gene-set tests

cytepool.activity.ulm(data, network, *, gene_ids=None, sample_ids=None, group=None, contrast=None, min_cells=MIN_CELLS, expression_filter=True, id_column='feature_id', statistic_column='t', min_targets=_MIN_TARGETS, n_threads=None)#

Score the sources of a network in every profile by univariate linear models (ULM).

As decoupler’s mt.ulm (Badia-i-Mompel et al. 2022): for each source, the values of a profile’s genes are regressed on the source’s weights, zero for the genes that are not its targets, and the score is the t-statistic of the slope. The module docstring gives the profiles, the matching of targets and the deliberate differences from decoupler.

Parameters:

  • data – (n_profiles, n_genes) values, such as log-expression values of samples or statistics of contrasts, NaN where missing; a Cube, whose group is scored; a DifferentialExpressionResult, whose statistics of every group and contrast are scored; or a table of results (a Table or pandas DataFrame), a pandas Series indexed by gene id or a mapping of gene id to statistic, one profile. A samples x genes DataFrame, as decoupler takes it, is given as frame.to_numpy() with gene_ids and sample_ids.

  • network – A Network, such as collectri() or progeny() returns, or what Network takes: a table with the columns source, target and weight, or a mapping of source to targets.

  • gene_ids – For an array, the genes’ ids, one per column (None, NaN or an empty string for a gene without one); for a cube, the name of a column of its features table, "feature_id" by default or "symbol", or one id per feature.

  • sample_ids – For an array, one id per row; by default "1", "2", …

  • group – For a cube, the group (cell type) to score; it may be left out when the cube has one. For a differential-expression result, the group whose statistics are scored; by default every group.

  • contrast – For a differential-expression result, the contrast whose statistics are scored; by default every contrast.

  • min_cells – For a cube, the fewest cells a sample needs in the group to be scored, at least 0, as for limma_voom().

  • expression_filter – For a cube, whether to score only the genes that filter_by_expression() keeps.

  • id_column – For a differential-expression result or a table, the column of gene ids, such as "feature_id" or "symbol".

  • statistic_column – For a differential-expression result or a table, the column of statistics, such as "t" or "log2_fold_change".

  • min_targets – The fewest targets among a profile’s genes for a source to be scored, at least 1 (decoupler’s tmin).

  • n_threads – The most threads to use; None uses numba’s current setting. The result does not depend on it.

Returns:

The activities, with the t-statistics as scores and the slopes as estimates.

Raises:

  • TypeError – If an argument has the wrong type, values are not real numbers, gene ids are not strings, or an argument is or holds a numpy masked array.

  • ValueError – If the values are not 2-D or hold an infinite value, the ids do not match the values, an id names two genes, an option is out of range or applies to another kind of data, a table lacks a column, or no source has min_targets targets among the genes.

  • KeyError – If the cube has no such group or features column, or the result has no such group or contrast.

Example

>>> import numpy as np
>>> from cytepool.networks import Network
>>> rng = np.random.default_rng(1)
>>> genes = [f"g{j}" for j in range(40)]
>>> y = rng.normal(size=(2, 40))
>>> y[1, :6] += 3.0  # the targets of A are up in the second sample
>>> network = Network({"A": genes[:6], "B": genes[10:16]})
>>> ulm(y, network, gene_ids=genes).scores.round(2).tolist()
[[0.71, -0.1], [7.94, -0.85]]

Used in: Pathway and regulator activity

Explained in: Choosing a method, Gene-set tests

cytepool.activity.weighted_aggregate(data, network, *, statistic='mean', n_permutations=_N_PERMUTATIONS, rng=None, gene_ids=None, sample_ids=None, group=None, contrast=None, min_cells=MIN_CELLS, expression_filter=True, id_column='feature_id', statistic_column='t', min_targets=_MIN_TARGETS, n_threads=None)#

Score the sources of a network in every profile by the weighted mean or sum of their targets’ values.

As decoupler’s mt.waggr (Badia-i-Mompel et al. 2022): the weighted mean of the values of a source’s targets, sum w y / sum |w|, or their weighted sum, sum w y, which is how PROGENy scores its pathways (Schubert et al. 2018); with permutations of the genes, normalized by the scores of the permutations, with a permutation p-value. The module docstring gives the permutations, the p-values and the deliberate differences from decoupler.

Parameters:

  • data – The profiles, as for ulm().

  • network – The network, as for ulm().

  • statistic – "mean", the weighted mean (decoupler’s "wmean"), or "sum", the weighted sum ("wsum").

  • n_permutations – The random permutations of the genes, 0 for none or at least 2 (decoupler’s times).

  • rng – A numpy.random.Generator to draw the permutations from, an integer seed for one (decoupler’s seed, 42 by default there), or None for a fresh, unseeded generator.

  • gene_ids – For an array or a cube, as for ulm().

  • sample_ids – For an array, as for ulm().

  • group – For a cube or a differential-expression result, as for ulm().

  • contrast – For a differential-expression result, as for ulm().

  • min_cells – For a cube, as for ulm().

  • expression_filter – For a cube, as for ulm().

  • id_column – For a differential-expression result or a table, as for ulm().

  • statistic_column – For a differential-expression result or a table, as for ulm().

  • min_targets – The fewest targets among a profile’s genes for a source to be scored, at least 1 (decoupler’s tmin).

  • n_threads – The most threads to use; None uses numba’s current setting. The result does not depend on it.

Returns:

The activities, with the normalized scores as scores (the weighted means or sums without permutations) and the weighted means or sums as estimates.

Raises:

  • TypeError – If an argument has the wrong type, values are not real numbers, gene ids are not strings, or an argument is or holds a numpy masked array.

  • ValueError – If the values are not 2-D or hold an infinite value, the ids do not match the values, an id names two genes, an option is out of range or applies to another kind of data, a table lacks a column, or no source has min_targets targets among the genes.

  • KeyError – If the cube has no such group or features column, or the result has no such group or contrast.

Used in: Pathway and regulator activity

Explained in: Choosing a method, Gene-set tests