Pseudobulk cubes and tables

Pseudobulk cubes and tables#

A cube holds additive statistics of the cells of each sample and group (such as a cell type), for every feature: counts, detected cells, and sums of normalized and log-normalized expression. Because the statistics add up, groups can be merged and samples pooled without going back to the cells. from_cells() pools a cell-level count matrix in one pass, and from_bulk() wraps a samples x features count matrix as a cube with one group. Sample, group and feature metadata are Table objects.

Pseudobulk cubes: additive statistics per sample, group and feature.

class cytepool.cube.Cube(layers, *, samples, groups, features, n_cells=None, normalization=None, provenance=None, copy=True)#

Additive statistics per sample, group and feature.

A cube pools the cells of each sample and group (usually a cell type) and keeps, for every feature, sums over those cells. Every layer is a sum, so merging groups or pooling samples is an exact sum and never needs the cells again. A cube is immutable: methods that change something return a new cube.

Layout:

Layers have shape (n_groups, n_samples, n_features), so one group is a contiguous samples x features matrix, which is what a per-group model fits. Quantities with one value per sample and group, n_cells and library_sizes(), have shape (n_samples, n_groups), like a table of cell type composition.

Layers:

  • counts – Sum of raw counts, the one required layer. An integer dtype, or float64 for non-integer counts such as transcript-level estimates.

  • detected – Number of cells with a positive count. An integer dtype.

  • log_sum – Sum over cells of the log-transformed normalized value, log1p for the usual natural log with pseudocount 1 (normalization records the transform). float64.

  • log_sum_sq – Sum over cells of the squares of those values. float64.

  • linear_sum – Sum over cells of the normalized value. float64.

Parameters:

  • layers – Layer name to an array of shape (n_groups, n_samples, n_features), or to a lazy array-like (see below).

  • samples – Sample metadata, as a Table or anything Table accepts, with a unique text column sample_id.

  • groups – Group metadata with a unique text column group_id, and optional text columns cell_ontology_id and parent.

  • features – Feature metadata with a unique text column feature_id, an optional text column symbol and an optional boolean column measured.

  • n_cells – Integers of shape (n_samples, n_groups): the cells pooled into each entry. None for bulk data, where cells are unknown. Required with detected, log_sum, log_sum_sq or linear_sum. A pandas DataFrame indexed by sample ids with group ids as columns, such as the output of pandas.crosstab, is aligned by those labels.

  • normalization – How normalized values were computed, as a JSON-serialisable mapping with a "method" string, for example {"method": "library_size", "target_sum": 10000.0, "log": "natural", "pseudocount": 1.0}. Required with log_sum, log_sum_sq or linear_sum.

  • provenance – A JSON-serialisable mapping that says where the data came from and how the cube was made (cytepool.cube.provenance documents the keys CytePool writes). Operations that derive a cube append a record of the operation, its input and what it kept or pooled to the list under "operations".

  • copy – If False, a layer given as a C-contiguous numpy array with an accepted native dtype is stored without a copy and made read-only in place, so it must not be changed through other references afterwards. Other layers are always copied.

Raises:

  • TypeError – If an argument has the wrong type or dtype (a table column of a type below included), or is or holds a numpy masked array.

  • ValueError – If shapes disagree, a required argument is missing, an id is missing or repeated, or a value breaks a constraint below.

Constraints:

Table columns hold text, booleans, integers, float16 to float64, complex64, complex128, datetimes or timedeltas: the types cube files store, and that a fingerprint encodes alike on every platform (longdouble, for example, is refused).

counts, detected, log_sum_sq and linear_sum are non-negative; integer layers and n_cells are at most 2**63 - 1, the largest int64, whatever their dtype; float layers are finite; detected is at most n_cells; every layer is zero in entries where n_cells is zero; and, because both moments sum over the same cells, log_sum**2 is at most n_cells * log_sum_sq (Cauchy-Schwarz), up to a relative 1e-6 for rounding. log_sum may be negative, which happens with a pseudocount below 1. Arrays are stored read-only.

Sums:

Library sizes, merged groups and pooled samples are sums. Integer layers and n_cells are summed in int64, and every sum is exact: one that would exceed 2**63 - 1 raises OverflowError, naming the layer, the entry and the exact sum, instead of wrapping around. Float layers are summed in float64, and a sum beyond the float64 range raises OverflowError too.

Lazy layers:

A layer may be any object with shape, dtype and a __getitem__ that accepts an integer on the first axis and returns one group, such as an HDF5 or zarr array; a numpy memmap is also kept lazy. Construction checks its shape and dtype and reads nothing. Each read puts what it needs in memory the cube owns, copying what the layer returns unless the layer is a ZarrArray (see below), and checks those values against the constraints, so a bad value raises when it is first used; the moment constraint is checked when variance() or to_memory() reads both moments. Selecting, regrouping and collapsing a lazy cube give a lazy cube, which computes its sums, and raises any OverflowError, when a group is read; to_memory() reads every layer.

A lazy cube asks a layer only for the groups, samples and features it needs, such as those that select() kept. A layer that also has an oindex attribute for orthogonal indexing, as zarr-python arrays have, is read with layer.oindex[group, samples, features], where group is an integer and samples and features are each slice(None) or a 1-D integer numpy array of positions in increasing order without repeats; it must return an array of shape (len(samples), len(features)), counting a slice as the whole axis. Any other layer is read one whole group at a time, and the cube keeps the part it needs.

The layers of read_cube are ZarrArray objects. They read only the chunks that hold the selected features, take positions in any order and return new arrays, which the cube keeps without a copy, so a read holds about the values it returns and one chunk (with zstd, also the decoder’s window, 2 MiB at the default level).

A read of a ZarrArray layer that covers a whole group, in any order (a group read whole, merged or with its samples pooled, or a selection that reorders samples or features), also hashes the group, to check it against its digest: stored with the cube file, or kept from the first such read. A read of a whole group of a layer derived from one hashes that group too, to keep its digest for fingerprint(). SHA-256 runs at about 2 GB/s on processors with SHA instructions and 0.5 GB/s without, which adds about a fifth to the time of reading a gzip-compressed group on the former. A selection of some samples or features reads part of a group, which cannot be checked on its own; verify_fingerprint() reads the groups it comes from whole.

Fingerprint and provenance:

fingerprint() identifies the cube’s values, tables and cell numbers included, and provenance records how the cube was made: the caller’s source record, the function that built it with its settings, and each operation that derived it, with the fingerprint of its input. history() reads the record back in plain terms; cytepool.cube.fingerprints and cytepool.cube.provenance document both. To record the fingerprint of its input, the first operation applied to a cube in memory hashes its layers, once, at about 1 s per GB of values on processors with SHA instructions.

Used in: Export a cube from another tool

Explained in: Cube files and provenance records

property n_samples#

The number of samples.

property n_groups#

The number of groups.

property n_features#

The number of features.

property layer_names#

The names of the layers this cube holds, in LAYER_NAMES order.

property samples#

The sample table, one row per sample.

property groups#

The group table, one row per group.

property features#

The feature table, one row per feature.

property n_cells#

Cells pooled per entry, int64 of shape (n_samples, n_groups), or None.

Explained in: Composition tests

property normalization#

A copy of the normalization mapping, or None.

property provenance#

A copy of the provenance mapping (see cytepool.cube.provenance).

Explained in: The cube and its additive layers, Cube files and provenance records

layer(name, group=None)#

Return a layer for one group or for all groups.

Parameters:

  • name – The layer name.

  • group – A group id or position; None returns every group.

Returns:

A read-only array of shape (n_samples, n_features) for one group, or (n_groups, n_samples, n_features). A lazy layer is read, and its values checked, on every call.

Raises:

  • KeyError – If the cube has no such layer, or group is an unknown id.

  • IndexError – If group is a position out of range.

  • TypeError – If group is neither a string nor an integer.

  • ValueError – If a lazy layer gives a value that breaks a constraint.

  • OverflowError – If a lazy layer that regroup() or collapse_samples() derived has a sum that does not fit.

Explained in: The cube and its additive layers

mean(group=None, scale='log')#

Return the mean normalized value over the cells of each entry.

Parameters:

  • group – A group id or position; None returns every group.

  • scale – "log" for log_sum / n_cells or "linear" for linear_sum / n_cells.

Returns:

float64 of shape (n_samples, n_features) for one group, or (n_groups, n_samples, n_features); NaN where n_cells is zero.

Raises:

  • ValueError – If scale is unknown or the cube lacks the layer it needs.

  • KeyError, IndexError, TypeError – For an invalid group, as in layer().

  • OverflowError – As in layer(), for a derived lazy layer.

Explained in: The cube and its additive layers

variance(group=None)#

Return the sample variance over the cells of each entry, log scale.

The variance comes from the raw moments, (log_sum_sq - log_sum**2 / n) / (n - 1) with n = n_cells. Its relative rounding error is about eps * (1 + mean**2 / variance), with eps = 2.2e-16 for float64: far below 1e-10 for log-normalized expression, but large when the values of an entry are nearly constant and far from zero (about 2e-6 at a mean-to-standard-deviation ratio of 1e5). For the same reason, the numerator of a constant entry comes out as zero give or take rounding (exactly zero when every value is zero); a numerator that rounding pushed below zero is returned as a variance of 0. A numerator below -1e-6 * log_sum_sq cannot come from rounding, so it raises.

Parameters:

group – A group id or position; None returns every group.

Returns:

float64 of shape (n_samples, n_features) for one group, or (n_groups, n_samples, n_features); NaN where n_cells < 2.

Raises:

  • ValueError – If the cube lacks log_sum or log_sum_sq, or the two layers break the moment constraint (they do not sum over the same cells).

  • KeyError, IndexError, TypeError – For an invalid group, as in layer().

  • OverflowError – As in layer(), for a derived lazy layer.

Explained in: The cube and its additive layers

fraction_detected(group=None)#

Return the fraction of cells with a positive count, detected / n_cells.

Parameters:

group – A group id or position; None returns every group.

Returns:

float64 of shape (n_samples, n_features) for one group, or (n_groups, n_samples, n_features); NaN where n_cells is zero.

Raises:

  • ValueError – If the cube lacks the detected layer.

  • KeyError, IndexError, TypeError – For an invalid group, as in layer().

  • OverflowError – As in layer(), for a derived lazy layer.

Explained in: The cube and its additive layers

library_sizes(group=None)#

Return the counts of each entry summed over features.

Parameters:

group – A group id or position; None returns every group.

Returns:

Shape (n_samples,) for one group, or (n_samples, n_groups); int64, and exact, for integer counts and float64 otherwise.

Raises:

  • OverflowError – If a library size exceeds 2**63 - 1 (integer counts) or the float64 range (float counts), or as in layer() for a derived lazy layer.

  • KeyError, IndexError, TypeError – For an invalid group, as in layer().

Explained in: The cube and its additive layers

to_memory()#

Return a cube with every lazy layer read into memory.

Values are checked as they are read, the moment constraint included, and so are the digests of a cube file’s groups. A cube without lazy layers is returned as it is. The result has the same values, so the same fingerprint and provenance.

Raises:

  • ValueError – If a value breaks a constraint, or a group of a cube file does not match its digest.

  • OverflowError – If a sum of a derived lazy layer does not fit.

Explained in: The cube and its additive layers

fingerprint()#

Return the fingerprint of the cube’s values.

The fingerprint identifies what an analysis reads: the sample, group and feature tables (ids included), n_cells, the normalization and every layer. It leaves out the provenance, so cubes with the same values have the same fingerprint however they were made, and a result can name the cube it was computed from. It is "sha256:" followed by 64 hexadecimal digits; cytepool.cube.fingerprints documents how it is computed, so another program can compute it too.

Cost:

The fingerprint is built from a digest of each table, of n_cells, of the normalization and of each group of each layer, and the cube keeps each digest once it is known.

  • A cube in memory hashes its values once.

  • A cube read from a cube file takes the digests stored in the file and reads no layer; the tables, n_cells and the normalization were checked against theirs when the file was opened, and each group is checked whenever a read covers all of it. A file written by an earlier version, without digests, is read whole once, a group at a time.

  • A lazy cube derived from a cube file with select(), regroup() or collapse_samples() reads, once, the groups whose values the operation changed and that were not read whole before, in every layer: none for a selection of groups only, the merged groups for regroup(), and every group for collapse_samples() and for a selection of samples or features. Groups it keeps unchanged take the digests of the cube it was derived from.

  • A lazy layer of another array-like, which can change, is read on every call.

Raises:

  • ValueError – If a lazy layer gives a value that breaks a constraint, or a group of a cube file does not match its digest.

  • OverflowError – If a sum of a derived lazy layer does not fit.

Explained in: The cube and its additive layers, Cube files and provenance records

digests()#

Return the digests of the cube’s parts and the fingerprint they make.

An analysis that reads some groups can record their digests, and check them later against a cube file, which checks a group whenever a read covers all of it. The digests are computed as fingerprint() computes them, at the same cost.

Returns:

A new JSON-serialisable dict in the form cube files store, which cytepool.cube.fingerprints documents. It holds the fingerprint version and the cube’s shape; the digest of each table, of n_cells and of the normalization (None without them); for each layer, its type and the digest of each group, by position; and the fingerprint.

Raises:

ValueError, OverflowError – As fingerprint().

Explained in: Cube files and provenance records

verify_fingerprint()#

Compute the fingerprint from the values, checking every digest known.

Every group of every layer is read whole and hashed, and so are the tables, n_cells and the normalization. A lazy layer derived from another is computed from whole reads of the groups it comes from, so that what a selection of some samples or features reads is checked too. Each digest must equal the one the cube holds: stored with the cube file it was read from, or computed before. Use it to check that a cube file still holds what was written, and that a cube is the one a result was computed from: read_cube(path).verify_fingerprint() == fingerprint_the_result_recorded.

Returns:

The fingerprint, computed from the values.

Raises:

  • ValueError – If a part does not match its digest (the message names it), or a lazy layer gives a value that breaks a constraint.

  • OverflowError – If a sum of a derived lazy layer does not fit.

Used in: Export a cube from another tool

Explained in: Cube files and provenance records

history()#

Describe how the cube was made, in plain terms, for printing.

The text is describe_provenance(cube.provenance) (see cytepool.cube.provenance): the caller’s source record, how the cube was built, and each operation in order, with the fingerprint of the cube it was applied to. A last line gives the cube’s own fingerprint when it is known without reading lazy layers (see fingerprint()).

Used in: Export a cube from another tool

Explained in: The cube and its additive layers, Cube files and provenance records

select(samples=None, groups=None, features=None)#

Return a cube with a subset of samples, groups and features.

Ids and positions also set the order of the result. A single id or position selects one entry. The provenance record lists the kept sample, group and feature ids, in order, and null for an axis given as None, and names this cube by its fingerprint (see “Fingerprint and provenance” in Cube). Selecting from a lazy cube reads nothing. Reads of the result ask its layers only for the kept groups, samples and features: a cube file reads only the chunks that hold them, and a layer without oindex is read one whole kept group at a time (see “Lazy layers” in Cube).

Parameters:

  • samples – Sample ids, a boolean mask or integer positions; None keeps every sample.

  • groups – Group ids, a boolean mask or integer positions; None keeps every group.

  • features – Feature ids, a boolean mask or integer positions; None keeps every feature.

Raises:

  • KeyError – If an id is unknown.

  • IndexError – If a position is out of range.

  • TypeError – If a selection is not ids, a boolean mask or integers, or is a numpy masked array.

  • ValueError – If a mask has the wrong length or a selection repeats an entry.

Explained in: The cube and its additive layers, Composition tests

regroup(mapping)#

Return a cube with groups merged.

Every layer and n_cells of a merged group is the sum over its members. Groups that are not listed are kept as they are. A merged group takes the position of its member that comes first in the current group order. In the group table, a merged row keeps a column’s value where all members agree and is missing otherwise; an integer or boolean column that needs a missing value becomes float64 with NaN. Integer layers of the result are int64. The provenance record gives each merged or renamed group with its members, in the current group order, and names this cube by its fingerprint (see “Fingerprint and provenance” in Cube).

Parameters:

mapping – New group id to the existing group ids it merges. A group id may appear in one merge only, and a new id must not be the id of a group that is kept. A single member renames a group.

Raises:

  • KeyError – If a member is not a group id.

  • TypeError – If mapping is not a mapping of strings to strings, or holds a numpy masked array.

  • ValueError – If a new id is empty, a merge is empty, a group is listed twice or a new id collides with a kept group.

  • OverflowError – If a merged n_cells or a merged value of an in-memory layer does not fit (see “Sums” in Cube). A lazy layer raises it when the merged group is read.

Explained in: The cube and its additive layers

collapse_samples(by)#

Return a cube with the samples that share a value of a column summed.

Use it to pool, for example, the tissues of one donor. The new sample_id is the shared value, in the order the values first appear. Sample columns that are constant within every pooled set are kept and the others are dropped. The column n_pooled_samples holds the number of original samples behind each new sample and comes last; if it already exists, it keeps its place and is summed, so collapsing twice counts the original samples. Integer layers of the result are int64. The provenance record gives each new sample with the ids of the samples it sums, in the current order, and names this cube by its fingerprint (see “Fingerprint and provenance” in Cube).

Parameters:

by – A text or integer sample column without missing values.

Raises:

  • KeyError – If there is no such sample column.

  • TypeError – If the column is not text or integer.

  • ValueError – If the column has a missing or empty value, or an existing n_pooled_samples column does not hold non-negative integers.

  • OverflowError – If a pooled n_cells, n_pooled_samples or value of an in-memory layer does not fit (see “Sums” in Cube). A lazy layer raises it when the group is read.

Explained in: The cube and its additive layers

cytepool.cube.from_bulk(counts, *, samples=None, features=None, source=None)#

Build a cube with one group from bulk counts.

Parameters:

  • counts – Counts with one row per sample and one column per feature, as a 2-D array; a sparse matrix with toarray() is made dense. Counts are non-negative and finite, whole numbers or not (transcript-level estimates, for example).

  • samples – Sample metadata with a text column sample_id and one row per row of counts (a Table or anything Table accepts), or a sequence of sample ids. By default the ids are the index of a pandas DataFrame counts when it holds text, and otherwise sample_0, sample_1, and so on.

  • features – Feature metadata with a text column feature_id and one row per column of counts, or a sequence of feature ids. By default the ids are the columns of a pandas DataFrame counts when they are text, and otherwise feature_0, feature_1, and so on.

  • source – A record of where the counts came from, as a JSON-serialisable mapping, stored as JSON in the cube’s provenance as it is given (see from_cells).

Returns:

A cube with the single group "bulk", only the counts layer, and n_cells None. Counts are uint32 when every count is a whole number up to 2**32 - 1, uint64 for larger whole numbers (up to 2**63 - 1), and float64 when a count is not a whole number (or, for float input, is at least 2**53). Its provenance holds source and where the sample and feature ids came from (cytepool.cube.provenance documents the record).

Raises:

  • TypeError – If counts is not numeric, source is not a mapping or holds a value JSON cannot hold, or an argument is or holds a numpy masked array.

  • ValueError – If counts is not 2-D, has a negative or non-finite value or an integer above 2**63 - 1, the largest a cube holds, its shape disagrees with samples or features, or source holds NaN, infinity or a lone surrogate.

Example

>>> cube = from_bulk(np.array([[5, 0, 12], [7, 1, 9]]))
>>> cube.layer("counts", "bulk").dtype
dtype('uint32')

Explained in: Choosing a method, The cube and its additive layers, Cube files and provenance records

cytepool.cube.from_cells(counts, *, samples, groups, sample_table=None, group_table=None, features=None, target_sum=10000.0, library_sizes=None, layers=None, n_threads=None, source=None)#

Pool the cells of every sample and group into a cube.

For every sample, group and feature, the cube sums over the cells of that sample and group: the raw counts (counts), the cells with a positive count (detected), and the normalized values x = count * target_sum / L, where L is the library size of the cell: x itself (linear_sum), log1p(x) (log_sum) and log1p(x)**2 (log_sum_sq). L is the total count of the cell, or its value in library_sizes. One pass over the stored counts computes every layer.

Parameters:

  • counts – Counts with one row per cell and one column per feature: a 2-D numpy array, or a sparse matrix in CSR layout with data, indices, indptr and shape, such as scipy’s csr_matrix or csr_array. Counts are non-negative and below 2**53, whole numbers or not. The column indices of a CSR row may be in any order, and repeated entries of one row and column are summed, as scipy does.

  • samples – The sample label of every cell, as text or integers (integers become their decimal text), for example a list, a numpy array or a pandas Series or Categorical.

  • groups – The group label, usually the cell type, of every cell, in the same forms.

  • sample_table – Sample metadata, as a Table or anything Table accepts, with a text column sample_id that holds every sample label. Its rows set the order of the samples, and a sample without cells gets zeros. By default the samples are the sorted distinct labels (integers in numeric order).

  • group_table – Group metadata with a text column group_id, used in the same way.

  • features – Feature metadata with a text column feature_id and one row per column of counts, or a sequence of feature ids. By default the ids are the columns of a pandas DataFrame counts when they are text, and otherwise feature_0, feature_1, and so on.

  • target_sum – The total that normalization scales every cell to.

  • library_sizes – One positive library size per cell to use instead of the total count of the cell, for example totals over all features when counts holds only some of them.

  • layers – The layers to compute: counts, which every cube needs, and any of detected, log_sum, log_sum_sq and linear_sum. By default all five.

  • n_threads – The most threads to use. None uses numba’s setting for the calling thread (see numba.set_num_threads).

  • source – A record of where the cells came from, as a JSON-serialisable mapping, stored as JSON in the cube’s provenance as it is given: for example a dataset identity, references to the artifacts of the tool that exported the cells (such as their frozen sample and group memberships, which stay in that tool’s files) and its settings. CytePool does not interpret it.

Returns:

A cube with n_cells of shape (n_samples, n_groups). Its normalization is {"method": "library_size", "target_sum": target_sum, "log": "natural", "pseudocount": 1.0, "library_sizes": "cell_totals"}, with "library_sizes": "given" when library_sizes is passed, and None without a normalized layer. Its provenance holds source and the settings: the number of cells, the layers, target_sum, where library sizes came from, how samples and groups were ordered and where feature ids came from (cytepool.cube.provenance documents the record).

Raises:

  • TypeError – If counts is neither a numeric array nor a CSR matrix (a CSC matrix, for example), an argument has the wrong type, source is not a mapping or holds a value JSON cannot hold, or an argument is or holds a numpy masked array.

  • ValueError – If a count is negative, not finite or at least 2**53 (alone or summed over repeated entries), labels are missing or absent from a table, lengths disagree, a layer name is unknown, the normalized values of a cell are undefined (a library size of 0) or beyond the float64 range, or source holds NaN, infinity or a lone surrogate.

  • OverflowError – If, over the cells of an entry, the whole-number counts of a feature sum to more than 2**63 - 1 (the most a cube’s integer layers hold) or its normalized values sum beyond the float64 range, or if detected is computed for an entry with more than 2**32 - 1 cells.

Dtypes:

counts is uint32 when every count is a whole number and every sum is at most 2**32 - 1, uint64 when a sum is larger (up to 2**63 - 1), and float64 when a count is not a whole number. Float input that holds whole numbers, as most h5ad files do, gives integer counts. detected is uint32, and the other layers are float64.

Cells without counts:

A cell with a library size of 0 adds nothing to counts and detected and is still counted in n_cells. Its normalized values would be 0 / 0, so a normalized layer cannot be computed while it is present: remove such cells first, pass library_sizes, or compute only counts and detected.

Memory:

Every layer holds n_groups x n_samples x n_features values: 4 bytes each for counts (8 when it is uint64 or float64) and detected, and 8 bytes for each other layer, 32 bytes per value for all five. For example, 30 groups x 100 samples x 30,000 features is 9e7 values: 2.9 GB for all five layers, 0.36 GB for counts alone. Beyond the result, pooling uses a few arrays with one value per cell, and each thread a few with one value per feature. The counts are not copied, except a dense array that is not C-contiguous and values of a dtype other than float64, float32, int64 and int32.

Example

>>> cube = from_cells(
...     np.array([[1, 0], [3, 2], [0, 4]]),
...     samples=["S1", "S1", "S2"],
...     groups=["T", "T", "T"],
...     layers=["counts", "detected"],
... )
>>> cube.layer("counts", "T").tolist()
[[4, 2], [0, 4]]

Used in: Export a cube from another tool, Differential expression per cell type

Explained in: Choosing a method, The cube and its additive layers, Cube files and provenance records

An immutable table of named, equal-length columns.

CytePool keeps sample, group and feature metadata, and later tidy results, in a Table because pandas is not a dependency. A table holds 1-D numpy arrays of equal length, one per column name. Columns are copied in and read-only, and every method that changes something returns a new table.

Column types:

  • Text columns are object arrays whose values are str or None (missing). Lists of strings, numpy str_ arrays, numpy StringDType arrays and object arrays are converted to this form; in an object array, None, NaN and pandas’ NA and NaT mark missing values. Comparisons follow Python: a missing value equals another missing value and nothing else. (numpy’s StringDType with None as its missing value treats a missing value as equal to "" in array comparisons, which is why it is not used.) A text column that holds None cannot be sorted, because Python cannot order None and str.

  • Numeric, boolean and datetime columns keep their dtype. NaN (or NaT) is the missing value of float (or datetime) columns. Integer and boolean columns have no missing value, so a sequence of numbers or booleans that contains None becomes float64 with NaN.

  • Bytes, structured and other object columns are rejected, and so are numpy masked arrays, whose mask numpy would drop: mark missing values with None or NaN instead.

class cytepool.table.Table(data=None)#

An immutable store of named, equal-length 1-D numpy columns.

Parameters:

data – A mapping of column name to 1-D array-like, another Table, or a pandas DataFrame. A DataFrame’s index is kept as leading columns when it is named and dropped otherwise. None gives an empty table.

Raises:

  • TypeError – If data has an unsupported type, a column name is not a string, or a column holds unsupported values or is a numpy masked array.

  • ValueError – If a column name is empty or repeated, a column is not 1-D, or the columns have unequal lengths.

Example

>>> table = Table({"sample_id": ["S1", "S2"], "age": [34.0, None]})
>>> table["age"]
array([34., nan])

Used in: Differential expression per cell type

Explained in: The cube and its additive layers

property columns#

The column names, in order.

take(indices)#

Return the rows at the given positions, in that order.

Parameters:

indices – 1-D integer positions; negative positions count from the end.

Raises:

  • TypeError – If indices are not integers, or are a numpy masked array.

  • ValueError – If indices are not 1-D.

  • IndexError – If a position is out of range.

filter(mask)#

Return the rows where mask is true.

Parameters:

mask – A boolean array with one value per row.

Raises:

  • TypeError – If mask is not boolean, or is a numpy masked array.

  • ValueError – If mask does not have one value per row.

select(names)#

Return a table with the given columns, in the given order.

Parameters:

names – Column names; a single string selects one column.

Raises:

  • KeyError – If a name is not a column.

  • TypeError – If names is a numpy masked array.

  • ValueError – If a name is repeated.

with_columns(columns=None, /, **named)#

Return a table with columns added or replaced.

A replaced column keeps its position; new columns are appended in the order given. Names that are not valid Python identifiers can be passed in the columns mapping.

Parameters:

  • columns – A mapping of column name to 1-D array-like.

  • **named – More columns, given as keyword arguments.

Raises:

  • TypeError – If columns is not a mapping, or a name or column has an unsupported type or is a numpy masked array.

  • ValueError – If a column is given twice, or a new column’s length differs from the table’s.

rename(mapping)#

Return a table with columns renamed.

Parameters:

mapping – Old column name to new column name.

Raises:

  • KeyError – If an old name is not a column.

  • ValueError – If renaming would repeat a column name.

to_dict()#

Return a new dict of column name to read-only array.

to_pandas()#

Return a pandas DataFrame with the same columns.

Text columns become object columns of str and None (pandas 3 stores them in its string dtype, with NaN as the missing value). Passing the DataFrame back to Table gives an equal table, except that pandas stores datetimes in seconds or finer units, so a coarser unit such as days comes back as seconds.

Raises:

ImportError – If pandas is not installed.

equals(other)#

Return whether other is a table with the same columns and values.

Column names, their order, dtypes and values must match. Missing values (None, NaN, NaT) are equal to each other.

Fingerprints and provenance#

A cube’s fingerprint identifies its values, tables and cell numbers included, so a result can name the cube it was computed from, and its provenance records how the cube was made: the caller’s record of the source, the settings of the function that built it, and every operation that derived it, with what it kept or pooled and the fingerprint of the cube it was applied to. history() reads the record back in plain terms.

Content fingerprints of cubes: what they cover and how they are encoded.

A cube’s fingerprint identifies the values an analysis reads: the sample, group and feature tables (ids included), the cells pooled into each entry (n_cells), the normalization that defines the normalized layers, and every layer. It leaves out the provenance, which records how the cube was made, so two cubes with the same values have the same fingerprint however they were made, and a result can name the cube it was computed from by its fingerprint. Cube.fingerprint() returns it and Cube.digests() the digests it is built from.

Parts and digests:

The fingerprint is built from the digests of the cube’s parts: each table, the cell numbers, the normalization, and each group of each layer. write_cube computes the digests while it writes the cube and stores them with the cube file, so the fingerprint of a saved cube needs no pass over its data. read_cube checks the tables, cell numbers and normalization against their digests when it opens the file, and every read that covers a whole group of a layer, in any order, against the group’s digest. One group can thus be checked without reading the others, and an analysis that reads whole groups checks exactly what it read. (A read of some samples or features of a group cannot be checked on its own; Cube.verify_fingerprint() reads such groups whole.) A single digest of the whole cube would need every layer read to check any of it.

Encoding, version 1 (FINGERPRINT_VERSION):

A digest is the SHA-256 hash (National Institute of Standards and Technology 2015) of a byte string, written as sha256: and 64 lowercase hexadecimal digits. Numbers in byte strings are little endian: u64 and i64 are unsigned and signed 8-byte integers. Text is UTF-8 (Yergeau 2003), written as its length in bytes (u64) followed by its bytes; a lone surrogate, which a Python string can hold but a cube file cannot, is written in its 3-byte form, as Python’s surrogatepass error handler writes it. Strings are taken as they are, without Unicode normalization.

Tables (samples, groups and features): the number of rows and of columns (u64 each), then for each column, in table order, its name (text), its type (text) and its values:

  • text, type text: the length in bytes of each value (i64, -1 for a missing value), then the bytes of the values present, one after another;

  • bool: one byte per value, 0 or 1;

  • int8 to int64 and uint8 to uint64: the values in that type;

  • float16, float32 and float64: the values in that type, the binary formats of IEEE 754 (IEEE 2019), with -0.0 written as 0.0 and every NaN as the quiet NaN (0x7E00, 0x7FC00000 or 0x7FF8000000000000);

  • complex64 and complex128: the real and imaginary parts of each value, as floats of half the size written as above; a value with a NaN part is missing, and both its parts are written as the quiet NaN;

  • datetimes and timedeltas, of type datetime64[unit] or timedelta64[unit] (numpy’s name, such as datetime64[ms]): the values as i64, NaT being -2**63.

The type is numpy’s name for the column’s dtype. These are the column types cube files store, and the only ones a cube’s tables accept. Two tables have the same digest exactly when Table.equals finds them equal: the same names in the same order, the same types, and equal values, with missing values equal.

Cell numbers: the number of samples and of groups (u64 each), then n_cells as i64, sample by sample.

Normalization: the mapping as a JSON value (Bray 2017), each value written as a tag byte and a payload: n for null; f and t for false and true; i and the decimal digits of the number as text (with a leading - when negative) for a number with a whole value, such as 10000 or 10000.0; d and its 8 bytes as a float64 for any other number; s and the text for a string; a, the number of items (u64) and the items for an array; o, the number of members (u64) and, in the order of the UTF-8 bytes of their keys, each key (text) and value for an object. A number is thus taken by its value, however a JSON writer spells it.

Layers: one digest for each group of each layer, of the group’s samples x features block: the number of samples and of features (u64 each), then the values sample by sample, as i64 for an integer layer and as float64, with -0.0 written as 0.0, for a float layer. An integer layer holds the same numbers whatever the width of its dtype (Cube accepts integers from 0 to 2**63 - 1, and merging or pooling makes them int64), so a uint32 group and an int64 group with the same values have the same digest.

Manifest: UTF-8 text with one line per part, each ending with a line feed (\n):

cytepool cube fingerprint 1
shape <groups> <samples> <features>
samples <digest>
groups <digest>
features <digest>
n_cells <digest, or none without cell numbers>
normalization <digest, or none without normalization>
layer <name> <int64 or float64> <group position> <digest>

with a layer line for each group of each layer: layers in the order of LAYER_NAMES (counts, detected, log_sum, log_sum_sq, linear_sum), and groups in the cube’s order, counted from 0. The fingerprint is the digest of the manifest.

Digest records:

Cube.digests() returns, and a cube file stores as the root attribute digests, a JSON object with everything the manifest lists (the order of its members does not matter):

{"version": 1, "shape": [<groups>, <samples>, <features>],
 "samples": <digest>, "groups": <digest>, "features": <digest>,
 "n_cells": <digest or null>, "normalization": <digest or null>,
 "layers": {<name>: {"type": "int64" or "float64",
                     "groups": [<digest of each group>]}},
 "fingerprint": <digest of the manifest>}

References

National Institute of Standards and Technology (2015). Secure Hash Standard (SHS). Federal Information Processing Standards Publication 180-4. doi:10.6028/NIST.FIPS.180-4

Yergeau F (2003). UTF-8, a transformation format of ISO 10646. RFC 3629, RFC Editor. doi:10.17487/RFC3629

Bray T (2017). The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259, RFC Editor. doi:10.17487/RFC8259

IEEE (2019). IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2019. doi:10.1109/IEEESTD.2019.8766229

cytepool.cube.fingerprints.FINGERPRINT_VERSION = 1#

The version of the encoding that this module documents and computes.

Provenance records: how a cube was built and what it pooled.

A result traced back to its data passes through the cube it was computed from. The cube’s fingerprint (cytepool.cube.fingerprints) names its values, and its provenance, Cube.provenance, says how it was made: a JSON object whose keys below are written by the functions and methods that build and derive cubes. Any other key that a caller gives (Cube(provenance=...)) is kept as it is. Cube.history() and describe_provenance read a record back in plain terms.

source:

The caller’s record of where the data came from, given to from_cells, from_bulk, from_anndata or from_mudata (cytepool.io.anndata) as source: for example a dataset identity, references to the exporter’s artifacts (such as frozen cell memberships, which stay in the exporter’s files) and the exporter’s settings. CytePool stores it as JSON, as it was given, does not interpret it, and reports its references as unresolved: following them needs the tool that wrote them. Null when no record was given.

construction:

How the cube was built: function ("from_cells", "from_bulk", "from_anndata" or "from_mudata"), cytepool_version, and the function’s settings. from_cells records

  • n_cells: the number of cells pooled (the cells of each sample and group are the cube’s n_cells);

  • layers: the layers computed;

  • target_sum: the total that normalization scales every cell to;

  • library_sizes: "cell_totals" (the total count of each cell) or "given" (the library_sizes argument), as in the normalization;

  • samples and groups: {"order": ..., "labels": ...}, where order is "sample_table" or "group_table" (the rows of that table) or "sorted_labels" (the distinct labels, text sorted by code point and integers numerically), and labels is "text" or "integers" (integer labels became their decimal text);

  • features: where the feature ids came from: "table", "ids" (a sequence), "data_frame_columns" or "positions" (feature_0, feature_1, …).

from_bulk records samples and features, where their ids came from: "table", "ids", "data_frame_index" (samples) or "data_frame_columns" (features), or "positions".

from_anndata and from_mudata, reading data that CytePool did not export, record

  • layout: the layout that CytePool recognised in the data, which chose the sources that layers did not name: "scanpy.get.aggregate", "decoupler.pp.pseudobulk" or "liana.multi.adata_to_views", or "given" for data of no layout it recognises;

  • rows: the number of profiles read, and for from_mudata modalities, the modality names;

  • layers: each cube layer with its source, "X" or a layer name;

  • n_cells: where the cell numbers came from: an obs column, "table" (a table of samples x groups), "uns['psbulk_stats']" (LIANA’s statistics) or null (none);

  • samples and groups, as from_cells records them;

  • obs_columns: the metadata read into the tables, null when the tables were given. from_anndata gives the obs columns placed in the sample table and in the group table, and those left out, as {"samples": [...], "groups": [...], "left_out": [...]}, and records the label columns as sample_column and group_column; from_mudata lists the columns of the MuData’s obs placed in the sample table.

operations:

The operations that derived the cube, oldest first. Each is an object with operation, the method, and input, the fingerprint of the cube it was applied to, so that a chain of operations can be checked link by link: apply each operation to the cube with its input fingerprint, and compare the result’s fingerprint with the next record’s input, or with the cube’s own fingerprint after the last.

input is recorded when it is known without reading lazy layer values. That is so for a cube in memory: the first operation applied to it hashes its layers, once, at about 1 s per GB of values where the processor has SHA instructions. It is so for a cube read from a cube file, which stores its digests, and for a lazy cube derived from one when every digest it needs is known: kept unchanged from the file, or kept from whole reads of its groups, as fingerprint() makes them. It is null for a lazy cube derived from another whose groups were not read whole, such as a cube file regrouped and then collapsed: call fingerprint() on the intermediate cube before deriving from it to record its fingerprint, or check the operations after the last recorded fingerprint together. It is always null for a cube whose lazy layers are other array-likes (a memmap or an HDF5 dataset), which can change: their fingerprint is computed anew on every call and never kept.

  • select: samples, groups and features, the ids kept, in the order of the result, or null for an axis that was given as None.

  • regroup: membership, the id of each merged or renamed group with the ids of the groups it sums, in the input’s order. The groups not listed are kept as they are.

  • collapse_samples: by, the sample column, and membership, the id of each new sample with the ids of the samples it sums, in the input’s order.

  • from_anndata and from_mudata: the import of an AnnData or MuData that CytePool exported, whose provenance the cube continues. input is the fingerprint that the export stored, which the cube’s own fingerprint equals unless the data were changed in between, and source the caller’s record, or null.

cytepool.cube.provenance.describe_provenance(provenance)#

Describe a provenance record in plain terms.

The source record comes first, then how the cube was built, then each operation in order with what it kept, merged or pooled; long lists are shortened. Entries the schema in this module does not describe, such as those of cubes saved by earlier versions, are shown as JSON. The references that CytePool cannot follow on its own are reported as unresolved: those of the source record, and the fingerprints of the cubes that operations were applied to, which name those cubes without saying where they are.

Parameters:

provenance – A provenance record, such as Cube.provenance or the record of a cube that a result keeps.

Returns:

Text with one item per line, for printing.

Raises:

  • TypeError – If provenance is not a mapping, holds a value JSON cannot hold, or is or holds a numpy masked array.

  • ValueError – If it holds NaN, infinity or text that is not valid Unicode (a lone surrogate), or its operations is not a list.

Example

>>> pooled = {"operation": "collapse_samples", "by": "donor"}
>>> pooled["membership"] = {"D1": ["S1", "S3"], "D2": ["S2"]}
>>> print(describe_provenance({"operations": [pooled]}))
1. collapse_samples by 'donor'
  D1: S1 + S3
  D2: S2

Explained in: Cube files and provenance records