Assays and metadata API reference

Assays and metadata API reference#

Each assay opens as the class of the preset that its store declares in assayTypes, such as RNA, ATAC, ADT, HTO, GeneActivity, or Assay for a generic assay; see Assay types. DataStore(assay_types=...) accepts only these presets and only names of assays in the store, and a read-only open accepts only the types that the store records. An open assay carries the preset that its DataStore resolved as assayType, which merge, subset, HTO demultiplexing, and DataStore.summary() read. An assay that measured only some cells of its store marks them in the Boolean cell column <assay>_I (Assay membership), which MetaData refuses to write or, while it carries the membership role, to drop.

Assay.score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed) is computation-only. It computes full-row-order averages blockwise in memory and is safe on read-only counts; it does not plan artifacts or write metadata. With log_transform=True it scores the values of normed(log_transform=True), so only normalizers that take log_transform accept it (see the table below). Persistent cell-cycle outputs belong to DataStore.scores.cell_cycle.

Feature-count percentages belong to DataStore.qc.feature_percentage(cell_selection, features). It derives the assay from the exact feature-selection ref and returns an assay-scoped quality_metric ref whose values array is the per-cell percentage. It does not add a metadata column. A read-only DataStore returns an identical existing result but raises PermissionError instead of computing a new one.

Persisted normalization belongs to DataStore.features.normalize(cell_selection, features), which returns an immutable artifact ref. It saves the values of normed with the configured normalizer. A flag left as None takes the operation’s default only for the RNA library-size normalizers norm_lib_size and norm_lib_size_log, whose scale CyteArc knows, as cytearc.assay.normalization.default_normalization_flags reports: features.normalize applies the flags they take by default. The values of every other normalizer, norm_dummy, CLR, TF-IDF, and custom normalizers, may be signed, already logged, or computed over chosen totals, so their flags stay False unless passed. The artifact records the resolved flags. Assay normalization methods are computation-only and require explicit feature indexes where they do not accept an ArtifactRef.

Every normalized path applies the assay’s configured normMethod: normed, saved normalization, HVG statistics, feature summaries, scores, marker search, pseudotime markers and aggregations, and marker heatmaps. Paths that compute library-size values straight from the counts, such as the subset writer of features.normalize and the RNA feature streams, run only when cytearc.assay.normalization.uses_library_size_normalization(assay) holds: the normalizer is norm_lib_size and sf is set.

The two normalization flags have one meaning on every path. log_transform=True takes log1p of the configured normalizer’s own output, in float64. renormalize_subset=True hands the normalizer each cell’s total over the selected features instead of over every feature. A normalizer accepts only the flags it can apply, as cytearc.assay.normalization.applicable_normalization_flags reports; a flag it cannot apply defaults to False, and passing True raises ValueError:

Assay and normalizer

log_transform

renormalize_subset

RNA norm_lib_size (default) and custom normalizers

yes

yes, through assay.scalar

RNA norm_lib_size_log

no, its values are already logarithms

yes

norm_dummy (the generic default) of RNA, ADT, and generic assays, and custom ADT and generic normalizers

yes

no

ATAC norm_tf_idf (default) and custom normalizers

no, ATAC values are never logged

yes, as the term-frequency denominator

norm_clr (the ADT default) on any assay, built-in normalizers of another assay class (norm_tf_idf outside ATAC, library-size normalizers outside RNA), and norm_dummy of ATAC

no

no

A custom RNA normalizer is a callable (assay, counts) that may read each cell’s total from assay.scalar while it builds its result; normed(log_transform=True) returns the log1p of what it returns. Saved results need a module-level function or a callable with an artifact_identity attribute, which provenance records. Pass log_transform=True to take logarithms of the counts that norm_dummy keeps as they are.

RNA library-size normalization divides each cell by its total, the <assay>_nCounts value or, with renormalize_subset=True, its sum over the selected features, and a cell whose total is zero normalizes to zeros. cytearc.assay.normalization.library_size_divisors holds this rule for every library-size path: normed with norm_lib_size or norm_lib_size_log, normalization artifacts, HVG statistics, feature-group scores, marker search, compare.compute_bulk means, WAGGR, and mapping and doublet projections. Each raises ValueError naming the totals when a total of a selected cell is negative or not finite. normed with a custom normMethod, which need not read the totals, does not check them and replaces a zero total with 1.

class cytearc.assay.Assay(z, workspace, name, cell_data, nthreads, matrix_root=None, resources=None, storageIo=None, *, assay_type=None)#

A generic Assay class that contains methods to calculate feature level statistics and stream normalized values for downstream computation.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • workspace (str | None) – Workspace name when assays live under matrices/ (None for a root datastore)

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • nthreads (int) – number of threads to use for parallel computations

  • assay_type (str | None) – Preset type that the store declares for the assay, such as HTO.

name#

A label for the assay instance

z#

Zarr group that contains the assay

cells#

A Metadata class object for cell attributes

nthreads#

number of threads to use for computations

rawData#

chunked array containing the raw data

feats#

a MetaData class object for feature attributes

attrs#

Zarr attributes for the zarr group of the assay

normMethod#

normalization method to use.

sf#

scaling factor for doing library-size normalization

property assayType: str | None#

Preset type declared for this assay, such as HTO, or None.

iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, scratch_itemsize=0, resident_bytes=0, **norm_params)#

Iterate over explicitly selected normalized features in batches.

Parameters:

  • cell_idx (ndarray) – Ordered physical cell indices to include.

  • feat_idx (ndarray) – Ordered physical feature indices to include.

  • batch_size (int | None) – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget.

  • msg (str | None) – Message to be displayed in the progress bar

  • as_dataframe (bool) – If true (default) then the yielded matrices are pandas dataframe

  • scratch_itemsize (int) – Bytes of working memory the caller needs per yielded value while it processes one batch. Batches are sized so that this scratch also fits the memory budget.

  • resident_bytes (int) – Bytes the caller keeps allocated for the whole iteration, such as an output buffer.

  • **norm_params (Any) – Extra keyword arguments forwarded to normed.

Return type:

Generator[DataFrame | tuple[ndarray, ndarray], None, None]

Returns:

Generator yielding DataFrames or (matrix, feature index) tuples.

normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False)#

This function normalizes the raw and returns a delayed chunked array of the normalized data.

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • renormalize_subset (bool) – Must be False.

  • log_transform (bool) – Return log1p of the normalized values.

Return type:

ChunkedArray

Returns: A chunked array (delayed matrix) containing normalized data.

Raises:

ValueError – If the configured normalizer cannot apply a requested flag.

score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed, *, log_transform=False)#

Calculates the scores (mean values) of selection of features over a randomly sampled selected feature set in given cells (as marked by cell_key)

Parameters:

  • feature_names (list[str]) – Names (as in ‘names’ column of the feature attribute table) of features to be used for scoring

  • cell_key (str) – Name of the key (column) from cell attribute table.

  • ctrl_size (int) – Number of reference features to be sampled from each bin.

  • n_bins (int) – Number of bins for sampling.

  • rand_seed (int) – The seed to use for the random number generation.

  • log_transform (bool) – Score log1p of the normalized values.

Return type:

ndarray

Returns: Numpy array of the calculated scores

Raises:

ValueError – If log_transform is True and the normalizer cannot apply it.

class cytearc.assay.RNAassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for feature selection and normalization of scRNA-Seq data.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • **kwargs (Any) – kwargs to be passed to the Assay class

normMethod#

A pointer to the function to be used for normalization of the raw data

sf#

scaling factor for doing library-size normalization

scalar#

This is used to cache the library size of the cells. It is set to None until normed method is called.

iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, scratch_itemsize=0, resident_bytes=0, **norm_params)#

Iterate over explicitly selected normalized features in batches.

Parameters:

  • cell_idx (ndarray) – Ordered physical cell indices to include.

  • feat_idx (ndarray) – Ordered physical feature indices to include.

  • batch_size (int | None) – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget.

  • msg (str | None) – Message to be displayed in the progress bar

  • as_dataframe (bool) – If true (default) then the yielded matrices are pandas dataframe

  • scratch_itemsize (int) – Bytes of working memory the caller needs per yielded value while it processes one batch. Batches are sized so that this scratch also fits the memory budget.

  • resident_bytes (int) – Bytes the caller keeps allocated for the whole iteration, such as an output buffer.

  • **norm_params (Any) – Extra keyword arguments forwarded to normed.

Return type:

Generator[DataFrame | tuple[ndarray, ndarray], None, None]

Returns:

Generator yielding DataFrames or (matrix, feature index) tuples.

normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False)#

This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the normed method in the generic Assay class this method is optimized for scRNA-Seq data and supports renormalize_subset with norm_lib_size (default normalization method for this class).

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • renormalize_subset (bool) – If true, normalize using only feat_idx rather than total expression across all features in a cell. (Default value: False)

  • log_transform (bool) – If True, then the normalized data is log-transformed (Default value: False).

Return type:

ChunkedArray

Returns:

A chunked array (delayed matrix) containing normalized data.

class cytearc.assay.ATACassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for feature selection and normalization of scATAC-Seq data.

normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False)#

This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the normed method in the generic Assay class this method is optimized for scATAC-Seq data. This method uses the normalization indicated by attribute self.normMethod which by default is set to norm_tf_idf. Document frequency is learned from cell_idx. The returned matrix contains feat_idx, while term frequency uses total ATAC counts unless subset renormalization is requested.

Parameters:

  • cell_idx (ndarray | None) – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table)

  • feat_idx (ndarray | None) – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis.

  • renormalize_subset (bool) – If True, use counts among feat_idx as the term-frequency denominator.

  • log_transform (bool) – Must be False.

Return type:

ChunkedArray

Returns: A chunked array (delayed matrix) containing normalized data.

class cytearc.assay.ADTassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs)#

This subclass of Assay is designed for normalization of ADT/HTO (feature-barcodes library) data from CITE-Seq experiments.

Parameters:

  • z (Group) – Zarr hierarchy where raw data is located

  • name (str) – A label/name for assay.

  • cell_data (MetaData) – Metadata class object for the cell attributes.

  • **kwargs (Any)

normMethod#

Pointer to the function to be used for normalization of the raw data

cytearc.assay.norm_dummy(_, counts)#

A dummy normalizer. Doesn’t perform any normalization. This is useful when the ‘raw data’ is already normalized.

Parameters:

  • _ (Assay)

  • counts (ChunkedArray) – A chunked array with ‘raw’ counts data

Return type:

ChunkedArray

Returns: A chunked array

class cytearc.metadata.MetaData(zgrp)#

Metadata table for cells and features backed by one Zarr group.

Changes made through this class are synchronized with the backing store.

active_index(key)#

Return global row indices selected by a boolean column.

Return type:

ndarray

property columns: list[str]#

Return all metadata column names.

default_block_rows(column='I')#

Prefer the Zarr chunk length of column for row iteration.

Return type:

int

drop(column)#

Delete an unprotected metadata column.

Return type:

None

fetch(column, key='I')#

Return stored column values for rows selected by key.

Rows flagged by the column’s linked missing mask hold stored placeholders; to_pandas_dataframe shows them as missing.

Return type:

ndarray

fetch_all(column)#

Return all stored values from a metadata column.

Rows flagged by the column’s linked missing mask hold stored placeholders; to_pandas_dataframe shows them as missing.

Return type:

ndarray

fetch_all_columns(columns)#

Return several whole columns; object stores read them concurrently.

Return type:

list[ndarray]

get_dtype(column)#

Return the dtype of a metadata column.

Return type:

dtype[Any]

get_index_by(value_targets, column, key=None)#

Return row indices for requested values in a metadata column.

Values match as case-insensitive text, as in CaseInsensitiveIndex, and every row that matches a target is returned, in target order. With key, indices are positions among the rows that key selects. A target that matches no row adds no index and is counted in a warning. Rows flagged by the column’s missing mask never match.

Return type:

ndarray

grep(pattern, only_valid=False)#

Return feature names matching a case-insensitive regex.

Return type:

list[str]

head(n=5)#

Return the first n rows of all columns, with masked rows missing.

Return type:

DataFrame

index_to_bool(idx, invert=False)#

Convert row indices into a table-sized boolean array.

Return type:

ndarray

insert(column_name, values, fill_value=None, key='I', overwrite=False, force=False)#

Insert a column into the table.

Raises:

ValueError – If column_name is protected, already exists without overwrite, or contains / or \, which Zarr reads as path separators.

Return type:

None

iter_row_blocks(*, cell_key='I', columns=None, block_rows=None)#

Yield contiguous row blocks over this table.

Each block covers a half-open global index range [start, stop). active_global_indices lists rows in that range that pass cell_key. Column arrays are aligned to those active indices only.

Return type:

Iterator[MetaDataRowBlock]

multi_sift(columns, lows, highs, keep_bounds=False)#

Return a boolean mask where all column filters are satisfied.

Return type:

ndarray

reset_key(key)#

Set every value in a boolean metadata key to true.

Return type:

None

sift(column, min_v=-inf, max_v=inf, keep_bounds=False)#

Return rows whose values fall within the requested bounds.

Rows flagged by the column’s linked missing mask never pass.

Return type:

ndarray

to_pandas_dataframe(columns, key=None)#

Return requested columns as a DataFrame, optionally filtered by key.

Rows flagged by a column’s linked missing mask are shown as missing: NaN for numeric columns, which become float64, a nullable boolean for boolean columns, and a missing value for other columns.

Return type:

DataFrame

update_key(values, key)#

Restrict a boolean metadata key using the supplied values.

Return type:

None