AnnData and MuData

AnnData and MuData#

scverse keeps pseudobulk profiles in two layouts, and CytePool reads and writes both. to_anndata() and from_anndata() convert a cube to and from a long AnnData, with a row per sample and cell type, as scanpy.get.aggregate and decoupler.pp.pseudobulk make it; to_mudata() and from_mudata() convert it to and from a MuData with a modality per cell type and the samples as its rows, as LIANA’s adata_to_views makes it. An import takes only sums and counts of cells, so a cube made from aggregated profiles holds what one made from the cells would hold, and a cube exported and read back has the same fingerprint. anndata and mudata are optional: install CytePool’s anndata or mudata extra.

Converting cubes to and from AnnData and MuData, the data structures of scverse.

A cube holds a samples x features matrix for every group, and an AnnData (Virshup et al. 2024) holds 2-D matrices only (since anndata 0.13, X and every layer must be 2-D). scverse therefore keeps pseudobulk profiles in one of two layouts, and CytePool reads and writes both:

  • a long AnnData, with one row per sample and group, as scanpy.get.aggregate (Wolf, Angerer and Theis 2018) and decoupler.pp.pseudobulk (Badia-i-Mompel et al. 2022) make it: to_anndata and from_anndata;

  • a MuData (Bredikhin, Kats and Stegle 2022) with one modality per group and the samples as its observations, as LIANA’s liana.multi.adata_to_views (Dimitrov et al. 2024) makes it: to_mudata and from_mudata.

anndata and mudata are optional dependencies: install CytePool’s anndata or mudata extra, or the packages themselves. Only to_anndata and to_mudata import them; from_anndata and from_mudata read the object they are given.

What an import takes:

A cube’s layers are sums over the cells of each sample and group, so an import takes only what is such a sum or gives one exactly: summed counts, the number of cells detecting each feature, and sums of normalized or log-normalized values. The source of a cube layer is X or a layer of the AnnData (of every modality, for a MuData), read as sums, except where its name says what scanpy or decoupler stored in it:

  • count_nonzero (scanpy) counts the cells with a nonzero value, and gives detected only;

  • psbulk_props (decoupler) is the fraction of cells with a nonzero value. It gives detected with the cell numbers: each fraction times its number of cells must be a whole number, to within the precision the fractions are stored in, and rounding it recovers the number of cells exactly;

  • mean, var and median hold means, variances and medians, which are refused. A variance or a median gives no sum, and a mean times its number of cells gives the sum exactly only where the sum is a whole number; every tool that gives means gives sums too: aggregate with func="sum" (scanpy) or mode="sum" (decoupler).

Counts and detected become integers, as from_cells makes them: uint32 when every value fits, uint64 otherwise, and float64 for counts that are not whole numbers; sums of whole numbers that a tool stored as floats become integers. Counts are checked against the cells that detected counts: a sum of counts is positive exactly where a cell has a positive count, and a sum of whole counts (counts that come as integers, or from a recognised layout) is at least the number of such cells, which means, medians and counts of other cells break.

A combination of sample and group without a row is an entry without cells, which only cell numbers can record: an import with such a combination and no cell numbers is refused. A row removed after pooling, as by decoupler’s filter_samples or LIANA’s min_cells, is such a combination too, so its cells are then missing from merged groups and pooled samples of the cube: import the profiles before filtering them, and leave out samples with few cells in the analysis instead (limma_voom has min_cells).

Recognised layouts:

The arguments left as None take their values from the layout of the data; a long AnnData that CytePool did not make still needs sample= and group=, the columns of its labels.

  • CytePool’s own export, which uns["cytepool"] marks (see “The record in uns” below): every layer, the cell numbers, the tables, the normalization and the label columns, as they were exported.

  • decoupler.pp.pseudobulk (an obs column psbulk_cells, or psbulk_n_cells as older versions name it): counts from X, cell numbers from that column, and detected from psbulk_props. Whatever layers gives, X must be a sum of counts: positive exactly where psbulk_props is, which medians are not; where whole, at least the number of detecting cells that the fractions bound even without cell numbers, which maxima and means seldom are; and, row by row, at most the total of the pooled counts that decoupler computes, whatever its mode, as psbulk_counts (less after a filter of the features, such as decoupler.pp.filter_by_expr), without being that total over the cells, as means are.

  • scanpy.get.aggregate (an obs column n_obs_aggregated): counts from the layer sum, detected from count_nonzero when present, and cell numbers from n_obs_aggregated.

  • liana.multi.adata_to_views, for a MuData whose every modality has a psbulk_props layer: counts from X, checked as decoupler’s, and cell numbers, and so detected, from uns["psbulk_stats"] when it holds them. LIANA keeps cell numbers only when its keep_stats finds decoupler’s column psbulk_n_cells, which decoupler 2 names psbulk_cells, so they are usually missing: pass n_cells, for example pandas.crosstab of the cells’ sample and group labels.

Counts that a recognised layout gives must be whole numbers; give layers to import counts that are not, or to read a sum of normalized values. Data of no recognised layout need layers.

Export:

to_anndata writes a row for every entry with cells (every entry of a cube without cell numbers), the rows of each group together, groups and samples in the cube’s order. Entries without cells have no row, as in scanpy.get.aggregate. obs holds sample_id and group_id, as categorical columns, the other columns of the sample and group tables, and n_cells; obs_names are <sample id>_<group id>, or the row positions when those repeat. var holds the feature table, indexed by feature id, and each layer of the cube is a layer of the same name. X is left empty, so that no layer is stored twice: adata.X = adata.layers["counts"].copy() gives tools that read X the counts.

to_mudata writes a modality per group, named by the group id, whose observations are the samples with cells in that group (every sample, without cell numbers), named by sample id, with n_cells in its obs, every layer under its own name, and X empty. Its var_names are <group id>:<feature id>, as LIANA writes them, so that they are unique across modalities; var holds the other columns of the feature table. The columns of the sample table are on the MuData’s obs. A MuData file cannot store a modality whose name holds / or NUL, or is . or ..: rename such a group first with Cube.regroup.

anndata writes only some columns, so table columns are written in a form it stores and reads back as they were: datetimes as ISO 8601 text (ISO 2019), NaT where missing, timedeltas as int64 counts of their unit, and a text column without any value as float64 NaN. Column names have _ in place of / and NUL, and ., .. and _index, which anndata reserves, end in _. A sample or group column whose name is taken is prefixed with sample_ or group_.

The record in uns:

uns["cytepool"] (of the AnnData, or of the MuData) holds format ("cytepool.cube.anndata" or "cytepool.cube.mudata"), format_version (1), the cube’s fingerprint, and as JSON text its provenance and normalization, its layers with the type of each, whether it has cell numbers (n_cells), its samples and groups tables, with the samples and groups that have no row, and the name, var column and type of each column of its features table. Only text and integers are stored, which every anndata writer keeps.

Reading the record, an import gives back the cube: the same tables (column types included), layers of the same types and the same values, so the same fingerprint, unless the data were changed in between. A layer converted to another type is converted back when that keeps its values; a layer whose values its type no longer holds was changed, and is refused (give layers to read it as what it holds). Every argument overrides what the record gives. The tables of samples and groups come from the record, so obs columns added later are not read; features come from var, so a subset of features gives a cube of those features. The provenance is the exported cube’s, followed by a record of the import with the fingerprint that the export stored: cube.fingerprint() == adata.uns["cytepool"]["fingerprint"] says whether the AnnData still holds the exported cube.

Example

>>> import numpy as np
>>> from cytepool.cube import from_cells
>>> cube = from_cells(
...     np.array([[1, 0], [3, 2], [0, 4]]),
...     samples=["S1", "S1", "S2"],
...     groups=["B", "T", "B"],
...     layers=["counts", "detected"],
... )
>>> adata = to_anndata(cube)
>>> adata.obs_names.tolist()
['S1_B', 'S2_B', 'S1_T']
>>> from_anndata(adata).fingerprint() == cube.fingerprint()
True

References

Virshup I, Rybakov S, Theis FJ, Angerer P, Wolf FA (2024). anndata: access and store annotated data matrices. Journal of Open Source Software 9, 4371. doi:10.21105/joss.04371

Bredikhin D, Kats I, Stegle O (2022). MUON: multimodal omics analysis framework. Genome Biology 23, 42. doi:10.1186/s13059-021-02577-8

Wolf FA, Angerer P, Theis FJ (2018). SCANPY: large-scale single-cell gene expression data analysis. Genome Biology 19, 15. doi:10.1186/s13059-017-1382-0

Badia-i-Mompel P, Velez Santiago J, Braunger J, Geiss C, Dimitrov D, Mueller-Dott S, Taus P, Dugourd A, Holland CH, Ramirez Flores RO, Saez-Rodriguez J (2022). decoupleR: ensemble of computational methods to infer biological activities from omics data. Bioinformatics Advances 2, vbac016. doi:10.1093/bioadv/vbac016

Dimitrov D, Schaefer PSL, Farr E, Rodriguez-Mier P, Lobentanzer S, Badia-i-Mompel P, Dugourd A, Tanevski J, Ramirez Flores RO, Saez-Rodriguez J (2024). LIANA+ provides an all-in-one framework for cell-cell communication inference. Nature Cell Biology 26, 1613-1622. doi:10.1038/s41556-024-01469-w

ISO (2019). ISO 8601-1:2019, Date and time: representations for information interchange, part 1: basic rules. International Organization for Standardization, Geneva.

cytepool.io.anndata.to_anndata(cube)#

Return a cube as a long AnnData, with a row per sample and group with cells.

The rows of each group come together, groups and samples in the cube’s order; a cube without cell numbers gives a row for every entry. obs holds sample_id and group_id (categorical), the other columns of the sample and group tables, and n_cells; var the feature table, indexed by feature id; each layer of the cube is a layer of the same name, and X is empty. uns["cytepool"] holds the cube’s fingerprint and provenance, and what from_anndata needs to give back the cube with the same fingerprint. The module documentation describes the layout (“Export”) and the record (“The record in uns”).

Parameters:

cube – The cube. A lazy cube is read one group at a time.

Returns:

An anndata.AnnData.

Raises:

  • TypeError – If cube is not a Cube.

  • ImportError – If anndata is not installed.

  • ValueError – If a lazy layer gives a value that breaks a constraint (see Cube).

  • OverflowError – If a sum of a derived lazy layer does not fit.

Explained in: The cube and its additive layers

cytepool.io.anndata.from_anndata(adata, *, sample=None, group=None, layers=None, n_cells=None, sample_table=None, group_table=None, normalization=None, source=None)#

Build a cube from a long AnnData of pooled profiles, a row per sample and group.

Each row is the profile of the cells of one sample and group, as scanpy.get.aggregate and decoupler.pp.pseudobulk make them; a combination without a row is an entry without cells. With the defaults, the layout of the AnnData chooses the source of each layer and of the cell numbers, and an AnnData that to_anndata made gives back its cube with the same fingerprint. The module documentation says what each source may hold (“What an import takes”) and which layouts are recognised.

Parameters:

  • adata – The AnnData.

  • sample – The obs column with the sample of each row, as text or integers (which become their decimal text), as from_cells takes labels. Required unless CytePool made the AnnData.

  • group – The obs column with the group, usually the cell type, of each row. Required unless CytePool made the AnnData.

  • layers – Cube layer name to its source, "X" or the name of a layer, for example {"counts": "sum", "detected": "count_nonzero"}. counts is required. By default the sources of the recognised layout.

  • n_cells – The cells pooled into each row: the name of an obs column, or a table with samples as rows and groups as columns, aligned by its labels, such as pandas.crosstab(cells.obs["donor"], cells.obs["cell_type"]) for the cells that were pooled. By default the column of the recognised layout, or none.

  • sample_table – Sample metadata with a text column sample_id that holds every sample label, as from_cells takes it: its rows set the order of the samples, and a sample without rows has no cells. By default the samples are the sorted distinct labels, with the obs columns that are constant within every sample, over the rows with cells.

  • group_table – Group metadata, used in the same way. By default the groups are the sorted distinct labels, with the other obs columns that are constant within every group.

  • normalization – How the normalized values were computed, required with log_sum, log_sum_sq or linear_sum (see Cube).

  • source – A record of where the profiles came from, as a JSON-serialisable mapping, kept in the cube’s provenance as given (see from_cells).

Returns:

A cube. Its provenance holds source and how the AnnData was read; for an AnnData that CytePool made, it is the provenance of the exported cube, followed by a record of this import (cytepool.cube.provenance documents both).

Raises:

  • TypeError – If adata is not an AnnData, an argument has the wrong type, a source does not hold numbers, a column of var cannot be a column of the feature table, or an argument is or holds a numpy masked array.

  • KeyError – If a column or layer it names is missing.

  • ValueError – If a source holds means, variances or medians, or values that are not the sums, counts or fractions it is read as, such as counts that are not sums over the cells that detected counts; a layer that CytePool exported was changed; a row repeats an entry; a label is missing or not in a given table; a combination has no row and there are no cell numbers to mark it as an entry without cells; a value breaks a constraint of Cube; or uns["cytepool"] is not a record this version reads.

Explained in: The cube and its additive layers, Cube files and provenance records

cytepool.io.anndata.to_mudata(cube)#

Return a cube as a MuData with a modality per group, the samples as rows.

Each modality is named by its group id and holds the samples with cells in that group (every sample, for a cube without cell numbers), named by sample id, with n_cells in its obs, every layer of the cube under its own name, X empty, and the feature table in var, whose names are <group id>:<feature id> as LIANA writes them. The columns of the sample table are on the MuData’s obs, and uns["cytepool"] holds the cube’s fingerprint, provenance and what from_mudata needs to give back the cube with the same fingerprint (see “Export” and “The record in uns” in the module documentation).

Parameters:

cube – The cube. A lazy cube is read one group at a time.

Returns:

A mudata.MuData.

Raises:

  • TypeError – If cube is not a Cube.

  • ImportError – If anndata or mudata is not installed.

  • ValueError – If the cube has no groups (to_anndata keeps such a cube), or a lazy layer gives a value that breaks a constraint (see Cube).

  • OverflowError – If a sum of a derived lazy layer does not fit.

Explained in: The cube and its additive layers

cytepool.io.anndata.from_mudata(mdata, *, layers=None, n_cells=None, sample_table=None, group_table=None, normalization=None, source=None)#

Build a cube from a MuData with a modality per group and the samples as rows.

Each modality is a group, named by the modality’s name, and each of its rows the profile of a sample, named by its observation name, as LIANA’s liana.multi.adata_to_views makes them; a sample without a row in a modality has no cells in that group. Every modality must hold the same features: its var_names, without the prefix <modality name>: that LIANA adds. The feature table is the var of the first modality, and the samples’ metadata the columns of the MuData’s obs. With the defaults, the layout chooses the sources, and a MuData that to_mudata made gives back its cube with the same fingerprint (see the module documentation).

Parameters:

  • mdata – The MuData.

  • layers – Cube layer name to its source in every modality, "X" or the name of a layer. counts is required. By default the sources of the recognised layout.

  • n_cells – The cells pooled into each row: the name of an obs column of every modality, or a table with samples as rows and groups as columns, aligned by its labels, such as pandas.crosstab of the cells’ sample and group labels. By default the cell numbers of the recognised layout, or none.

  • sample_table – Sample metadata with a text column sample_id that holds every sample, as in from_anndata. By default the samples are the sorted distinct observation names, with the columns of the MuData’s obs.

  • group_table – Group metadata with a text column group_id that holds every modality name. By default the groups are the sorted modality names.

  • normalization – How the normalized values were computed, required with log_sum, log_sum_sq or linear_sum (see Cube).

  • source – A record of where the profiles came from, kept in the cube’s provenance as given (see from_cells).

Returns:

A cube, with a provenance as from_anndata gives it.

Raises:

  • TypeError – If mdata is not a MuData, an argument has the wrong type, a source does not hold numbers, or an argument is or holds a numpy masked array.

  • KeyError – If a column or layer it names is missing from a modality.

  • ValueError – If the modalities hold different features, or for the reasons from_anndata gives.

Explained in: The cube and its additive layers, Cube files and provenance records