Designs and contrasts

Designs and contrasts#

A Design joins a sample table, the fixed effects as an R formula, optional random effects in lme4 notation and named contrasts. Its model matrix has the columns and column names that R’s model.matrix gives for the same level order. Without explicit levels, factor levels are sorted by code point, as R sorts them with LC_COLLATE=C; R’s default follows the session’s locale. check() reports problems such as aliased columns before anything is fitted.

Design matrices, contrasts and design checks.

class cytepool.design.Contrasts(matrix, names, columns)#

Named contrasts: weights on the columns of a design.

A read-only mapping of contrast name to weights: indexing by name returns one contrast’s weights, len is the number of contrasts, iterating yields their names, and dict(contrasts) gives a plain dict.

The weight vectors that indexing, values(), items() and dict give follow the order of columns and carry no names, so other functions read them by position. Wherever the weights meet a fit or another design, pass the Contrasts itself, or select(...) for some of them: those are matched by name.

Variables:

matrix – float64 array of shape (n_columns, n_contrasts), read-only; column j holds the weights of contrast names[j].
names – The contrast names.
columns – The design columns that the rows of matrix refer to. Rows are matched to the coefficients of a fit, or to the columns of another design, by these names.

Raises:

TypeError – If a field is or holds a numpy masked array.
ValueError – If matrix does not have one row per column and one column per name.

Explained in: Designs, linear models and contrasts

select(names)#

Return the given contrasts, in the given order, over the same columns.

Unlike the vectors that indexing returns, the result keeps the names of the columns, so contrasts_fit and make_contrasts match it by name.

Parameters:

names – Contrast names; a single string selects one contrast.

Raises:

KeyError – If a name is not a contrast.
TypeError – If names is a numpy masked array.
ValueError – If a name is repeated.

Explained in: Designs, linear models and contrasts

keys()#: The contrast names.

values()#: The weights of each contrast, in order.

items()#: Pairs of contrast name and weights, in order.

get(name, default=None)#: The weights of a contrast, or default if there is none by that name.

class cytepool.design.Design(samples, formula, *, random=None, contrasts=None, levels=None)#

A sample table with its design matrix, random effects and named contrasts.

Every model in the package takes a design: matrix and columns are what a linear model is fitted to, random describes the random effects of a mixed model, and contrasts holds the comparisons to test. check() looks for the problems that silently ruin an analysis before anything is fitted. fit_mixed_model estimates random intercepts by REML; random slopes are not estimated by any function yet. fit_linear_model fits one random intercept with a known correlation, given block= that groups the samples as the intercept does, and refuses a design whose random effects it would otherwise drop.

Parameters:

samples – The sample table: a Table, a mapping of column name to 1-D array, or a pandas DataFrame, with one row per sample, in the order of the samples of the data the design will be fitted to. The categories of pandas categorical columns give their level order.
formula – The fixed effects as an R formula, such as "~ condition + batch" or "~ 0 + group"; see model_matrix.
random – Random-effect terms in lme4 notation, such as "(1 | donor)" or "(1 | donor) + (1 | plate)", or a list of such strings. check() examines them, and fit_mixed_model estimates random intercepts.
contrasts – Named contrasts over the design columns, as accepted by make_contrasts: for example {"stim_vs_ctrl": "conditionstim"}.
levels – Level order of factors, such as {"condition": ["ctrl", "stim"]}; the first level is the reference. Names must be columns of the sample table; levels of columns the design does not use are ignored.

Raises:

TypeError – If an argument has the wrong type, or is or holds a numpy masked array.
ValueError – If the formula or random effects are invalid, use columns that do not exist or have missing values, levels does not match the data, or a contrast is invalid.

Example

>>> design = Design(
...     {"condition": ["ctrl", "stim"] * 3, "donor": ["D1", "D2", "D3"] * 2},
...     "~ condition",
...     random="(1 | donor)",
...     contrasts={"stim_vs_ctrl": "conditionstim"},
... )
>>> design.columns
('(Intercept)', 'conditionstim')

Used in: Quick start, Differential expression per cell type, Bulk RNA-seq

Explained in: Designs, linear models and contrasts

property samples#: The sample table, one row per sample.

property n_samples#: The number of samples.

property formula#: The fixed-effects formula.

property model_matrix#: The fixed-effects model matrix with its terms and coding.

property matrix#: The design matrix, float64 of shape (n_samples, n_columns), read-only.

property columns#: The column names, as R names them.

property terms#: The term labels, in column order.

property assign#: The term of each column, 0 for the intercept, i for terms[i-1].

property levels#: The levels of each factor of the fixed effects; the first is the reference.

property random#: The random-effect terms, in the order given.

property contrasts#

The named contrasts; index by name for one contrast’s weights.

Pass them, or contrasts.select(...), to contrasts_fit, which matches them to the fit’s coefficients by name; the vectors that indexing returns carry no names and are read in order.

with_contrasts(contrasts)#

Return the same design with these contrasts in place of the current ones.

Parameters:

contrasts – Named contrasts, as accepted by make_contrasts.

Raises:

TypeError – If a contrast has the wrong type, or is or holds a numpy masked array.
ValueError – If a contrast is invalid.

Explained in: Designs, linear models and contrasts

check()#

Check the design for problems that silently ruin an analysis.

Finds rank deficiency (naming the aliased columns and the combination each equals), zero residual degrees of freedom, factors confounded with each other or with a covariate, constant variables, empty levels, levels with a single sample, samples fitted exactly or nearly so, numeric columns that look like batch or plate codes, labels that look like missing values or differ only in case or spaces, contrasts that are not estimable, and random effects whose variances cannot be estimated. See cytepool.design.checks for the list.

Returns:: The errors and warnings found, with the rank, the aliased columns and the leverage of every sample.

Used in: Quick start

Explained in: Choosing a method, Designs, linear models and contrasts

class cytepool.design.DesignIssue(kind, message)#

One finding of Design.check.

Variables:

kind – A short code for the kind of problem, such as "rank_deficient" or "confounded" (see cytepool.design.checks).
message – What is wrong and how to fix it.

Explained in: Designs, linear models and contrasts

class cytepool.design.DesignReport(errors, warnings, n_samples, n_columns, rank, aliased, leverage)#

The result of Design.check: errors, warnings and a summary of the rank.

Variables:

errors – Problems that make a coefficient, contrast or variance impossible to estimate.
warnings – Designs that can be fitted but deserve a second look.
n_samples – Number of samples.
n_columns – Number of design columns.
rank – Rank of the design matrix.
aliased – Columns that are linear combinations of earlier columns, in column order: the columns whose coefficients R’s lm.fit (and limma’s lmFit) reports as NA, found by the same pivoting rule.
leverage – The leverage of each sample, the diagonal of the hat matrix X (X'X)^- X': 1 for a sample the model fits exactly.

Explained in: Designs, linear models and contrasts

property residual_df#: Residual degrees of freedom, the number of samples minus the rank.

property ok#: Whether the check found no errors.

kinds()#

The kinds of all findings, errors first.

Explained in: Designs, linear models and contrasts

class cytepool.design.FormulaTerms(formula, terms, intercept, variables)#

The terms of the right-hand side of a formula, as R’s terms() gives them.

Variables:

formula – The formula as given.
terms – Each term as a tuple of its variables, in R’s term order: by the number of variables, then by order of appearance. The variables of a term follow variables.
intercept – Whether the formula includes an intercept.
variables – Every variable named in the formula, in order of first appearance, including variables of terms that were removed. This is the order that decides the order of variables within terms.

Explained in: Designs, linear models and contrasts

property labels#: The term labels, such as "condition" and "condition:batch".

property used_variables#: The variables that appear in at least one term, in formula order.

class cytepool.design.ModelMatrix(matrix, columns, assign, terms, intercept, formula, levels, variables, coding)#

A design matrix with R’s column names and the terms behind each column.

Variables:

matrix – float64 array of shape (n_samples, n_columns), read-only.
columns – Column names, as R’s colnames(model.matrix(...)).
assign – For each column, the term it belongs to: 0 for the intercept and i for terms[i - 1], as R’s attr(, "assign").
terms – Term labels in column order, as R’s attr(terms, "term.labels").
intercept – Whether the first column is the intercept.
formula – The formula the matrix was built from.
levels – For every factor in the formula, its levels in order; the first is the reference level of treatment contrasts.
variables – The variables the columns are built from, in formula order.
coding – For each column, the (variable, level) pairs it multiplies, with level None for a covariate; the intercept has none. For example (("condition", "stim"), ("age", None)) for conditionstim:age.

Raises:

TypeError – If a field other than intercept and formula, which are stored as given, is or holds a numpy masked array.
ValueError – If matrix is not 2-D, or columns, assign or coding does not have one entry per column.

Explained in: Designs, linear models and contrasts

property shape#: (n_samples, n_columns).

class cytepool.design.RandomEffect(effects, grouping, correlated=True)#

One random-effect term, such as (1 | donor), in lme4 notation.

Variables:

effects – The left side of the bar: what varies between groups. 1 is a random intercept; 1 + age (or age) adds a random slope for age; 0 + age is a slope without an intercept.
grouping – The variables whose combinations define the groups, in the order of lme4’s name for the grouping factor. A nested term (1 | donor/sample) gives two effects, the outer one grouped by ("donor",) and the nested one by ("sample", "donor"), which lme4 names sample:donor.
correlated – False for the double-bar form (age || donor), whose effects are uncorrelated.

Explained in: Designs, linear models and contrasts

property group#

The name of the grouping factor as lme4 gives it, such as "sample:donor".

A single grouping variable keeps its name as it is (cell type); a combination of variables is written as in a formula, with backticks around names that need them (found by probing lme4).

cytepool.design.make_contrasts(contrasts, columns)#

Turn named linear expressions in design column names into a contrast matrix.

Parameters:

contrasts – A mapping of contrast name to an expression such as "groupB - groupA" or "(groupB + groupC)/2 - groupA", or to weights: a vector with one entry per column, in the order of columns, or a pandas Series indexed by column name (one with the default index 0, 1, …, n - 1 is read in order, and other indexes that do not hold strings are an error). A list of expressions (or a single expression) is also accepted; each expression is then its own name, as in limma. Contrasts made for other columns are re-expressed on these columns by name, so they must weigh the same columns, in any order.
columns – The design column names, or an object with a columns attribute such as a ModelMatrix or Design. Contrasts weigh the fixed effects, so the random effects of a Design play no part.

Returns:

The contrasts, with a (n_columns, n_contrasts) weight matrix.

Raises:

TypeError – If an argument has the wrong type, or is or holds a numpy masked array.
ValueError – If a name is not a column, an expression is not linear (a product or ratio of columns, a constant term, a power or a function), a contrast is zero, names repeat, columns repeat, or weights that name their columns do not name exactly columns.

Example

>>> make_contrasts({"b_vs_a": "groupB - groupA"},
...                ["groupA", "groupB", "groupC"]).matrix[:, 0]
array([-1.,  1.,  0.])

Explained in: Designs, linear models and contrasts

cytepool.design.model_matrix(data, formula, *, levels=None)#

Build the design matrix of a formula, column for column as R’s model.matrix.

Parameters:

data – The sample table: a Table, a mapping of column name to 1-D array, or a pandas DataFrame, with one row per sample.
formula – A right-hand-side formula such as "~ condition + batch", "~ 0 + group", "~ condition * sex" or "~ donor + condition". See parse_formula for the grammar.
levels – Level order of factors, for example {"condition": ["ctrl", "stim"]}; the first level is the reference. Each list must contain every value the column holds. For a boolean column, give [True, False] (or ["TRUE", "FALSE"]) to make TRUE the reference. Names must be columns of the table; levels of columns the formula does not use are ignored, so one mapping can serve several formulas.

Returns:

The model matrix with R’s column names, column order and coding.

Raises:

TypeError – If an argument has the wrong type or is or holds a numpy masked array, or a used column holds datetimes or complex numbers.
ValueError – If the formula is invalid, names a column that does not exist, uses a column with missing or infinite values, a factor has a single level, or levels does not match the data.

Example

>>> samples = {"condition": ["ctrl", "stim", "ctrl", "stim"],
...            "age": [31.0, 45.0, 52.0, 38.0]}
>>> model_matrix(samples, "~ condition + age").columns
('(Intercept)', 'conditionstim', 'age')

Explained in: Designs, linear models and contrasts

cytepool.design.parse_formula(formula)#

Expand the right-hand side of an R model formula into its terms.

Parameters:

formula – A formula without a response, such as "~ condition + batch", "~ 0 + group" or "~ condition * sex". Names that are not syntactic in R are quoted with backticks: "~ `cell type` + age".

Returns:

The terms, the intercept flag and the variables of the formula.

Raises:

TypeError – If formula is not a string.
ValueError – If the formula is not valid, has a response, contains a function call, a random-effect term or another unsupported construct; the message says where and how to fix it.

Example

>>> parse_formula("~ a * b").labels
('a', 'b', 'a:b')

Explained in: Designs, linear models and contrasts

cytepool.design.parse_random_effects(random)#

Parse lme4-style random-effect terms, such as "(1 | donor) + (1 | plate)".

Parameters:

random – One or more parenthesized terms (effects | grouping) joined by +, optionally after ~. grouping is a variable, an interaction g1:g2, or a nesting g1/g2, which expands to (effects | g1) + (effects | g2:g1) as in lme4. || instead of | makes the effects uncorrelated.

Returns:

One RandomEffect per term, in the order written; a nesting gives one effect per level of nesting, outer first. Grouping factors are named as lme4 names them.

Raises:

TypeError – If random is not a string.
ValueError – If the text is not a sum of parenthesized bar terms.

Explained in: Designs, linear models and contrasts

cytepool.design.INTERCEPT = '(Intercept)'#: The name of the intercept column of a model matrix, as R names it.

Column names and term labels write a variable name that is not syntactic in R between backticks, as R does: a factor cell type gives columns such as `cell type`B, and a contrast refers to them by the same names.

cytepool.design.formula.quote_name(name)#

Return a variable name as R writes it in term labels and column names.

Syntactic R names (letters, digits, . and _, not starting with a digit, _ or . followed by a digit, and not a reserved word) are returned as they are; other names are quoted with backticks: quote_name("cell type") returns "`cell type`".

Raises:: TypeError – If name is not a string.

Explained in: Designs, linear models and contrasts

Designs and contrasts

Contents

Designs and contrasts#