Source: https://docs.nygen.io/CyteArc/api/assays.html Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe Assays and metadata API reference ********************************* Each assay opens as the class of the preset that its store declares in "assayTypes", such as "RNA", "ATAC", "ADT", "HTO", "GeneActivity", or "Assay" for a generic assay; see Assay types. "DataStore(assay_types=...)" accepts only these presets and only names of assays in the store, and a read-only open accepts only the types that the store records. An open assay carries the preset that its "DataStore" resolved as "assayType", which merge, subset, HTO demultiplexing, and "DataStore.summary()" read. An assay that measured only some cells of its store marks them in the Boolean cell column "_I" (Assay membership), which "MetaData" refuses to write or, while it carries the membership role, to drop. "Assay.score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed)" is computation-only. It computes full-row-order averages blockwise in memory and is safe on read-only counts; it does not plan artifacts or write metadata. With "log_transform=True" it scores the values of "normed(log_transform=True)", so only normalizers that take "log_transform" accept it (see the table below). Persistent cell-cycle outputs belong to "DataStore.scores.cell_cycle". Feature-count percentages belong to "DataStore.qc.feature_percentage(cell_selection, features)". It derives the assay from the exact feature-selection ref and returns an assay-scoped "quality_metric" ref whose "values" array is the per-cell percentage. It does not add a metadata column. A read-only "DataStore" returns an identical existing result but raises "PermissionError" instead of computing a new one. Persisted normalization belongs to "DataStore.features.normalize(cell_selection, features)", which returns an immutable artifact ref. It saves the values of "normed" with the configured normalizer. A flag left as None takes the operation’s default only for the RNA library-size normalizers "norm_lib_size" and "norm_lib_size_log", whose scale CyteArc knows, as "cytearc.assay.normalization.default_normalization_flags" reports: "features.normalize" applies the flags they take by default. The values of every other normalizer, "norm_dummy", CLR, TF-IDF, and custom normalizers, may be signed, already logged, or computed over chosen totals, so their flags stay False unless passed. The artifact records the resolved flags. Assay normalization methods are computation-only and require explicit feature indexes where they do not accept an "ArtifactRef". Every normalized path applies the assay’s configured "normMethod": "normed", saved normalization, HVG statistics, feature summaries, scores, marker search, pseudotime markers and aggregations, and marker heatmaps. Paths that compute library-size values straight from the counts, such as the subset writer of "features.normalize" and the RNA feature streams, run only when "cytearc.assay.normalization.uses_library_size_normalization(assay)" holds: the normalizer is "norm_lib_size" and "sf" is set. The two normalization flags have one meaning on every path. "log_transform=True" takes "log1p" of the configured normalizer’s own output, in float64. "renormalize_subset=True" hands the normalizer each cell’s total over the selected features instead of over every feature. A normalizer accepts only the flags it can apply, as "cytearc.assay.normalization.applicable_normalization_flags" reports; a flag it cannot apply defaults to False, and passing True raises "ValueError": +-----------------------------------+-----------------------------------+-----------------------------------+ | Assay and normalizer | "log_transform" | "renormalize_subset" | |===================================|===================================|===================================| | RNA "norm_lib_size" (default) and | yes | yes, through "assay.scalar" | | custom normalizers | | | +-----------------------------------+-----------------------------------+-----------------------------------+ | RNA "norm_lib_size_log" | no, its values are already | yes | | | logarithms | | +-----------------------------------+-----------------------------------+-----------------------------------+ | "norm_dummy" (the generic | yes | no | | default) of RNA, ADT, and generic | | | | assays, and custom ADT and | | | | generic normalizers | | | +-----------------------------------+-----------------------------------+-----------------------------------+ | ATAC "norm_tf_idf" (default) and | no, ATAC values are never logged | yes, as the term-frequency | | custom normalizers | | denominator | +-----------------------------------+-----------------------------------+-----------------------------------+ | "norm_clr" (the ADT default) on | no | no | | any assay, built-in normalizers | | | | of another assay class | | | | ("norm_tf_idf" outside ATAC, | | | | library-size normalizers outside | | | | RNA), and "norm_dummy" of ATAC | | | +-----------------------------------+-----------------------------------+-----------------------------------+ A custom RNA normalizer is a callable "(assay, counts)" that may read each cell’s total from "assay.scalar" while it builds its result; "normed(log_transform=True)" returns the "log1p" of what it returns. Saved results need a module-level function or a callable with an "artifact_identity" attribute, which provenance records. Pass "log_transform=True" to take logarithms of the counts that "norm_dummy" keeps as they are. RNA library-size normalization divides each cell by its total, the "_nCounts" value or, with "renormalize_subset=True", its sum over the selected features, and a cell whose total is zero normalizes to zeros. "cytearc.assay.normalization.library_size_divisors" holds this rule for every library-size path: "normed" with "norm_lib_size" or "norm_lib_size_log", normalization artifacts, HVG statistics, feature-group scores, marker search, "compare.compute_bulk" means, WAGGR, and mapping and doublet projections. Each raises "ValueError" naming the totals when a total of a selected cell is negative or not finite. "normed" with a custom "normMethod", which need not read the totals, does not check them and replaces a zero total with 1. class cytearc.assay.Assay(z, workspace, name, cell_data, nthreads, matrix_root=None, resources=None, storageIo=None, *, assay_type=None) A generic Assay class that contains methods to calculate feature level statistics and stream normalized values for downstream computation. Parameters: * **z** ("Group") – Zarr hierarchy where raw data is located * **workspace** ("str" | "None") – Workspace name when assays live under "matrices/" (None for a root datastore) * **name** ("str") – A label/name for assay. * **cell_data** ("MetaData") – Metadata class object for the cell attributes. * **nthreads** ("int") – number of threads to use for parallel computations * **assay_type** ("str" | "None") – Preset type that the store declares for the assay, such as "HTO". name A label for the assay instance z Zarr group that contains the assay cells A Metadata class object for cell attributes nthreads number of threads to use for computations rawData chunked array containing the raw data feats a MetaData class object for feature attributes attrs Zarr attributes for the zarr group of the assay normMethod normalization method to use. sf scaling factor for doing library-size normalization property assayType: str | None Preset type declared for this assay, such as "HTO", or None. iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, scratch_itemsize=0, resident_bytes=0, **norm_params) Iterate over explicitly selected normalized features in batches. Parameters: * **cell_idx** ("ndarray") – Ordered physical cell indices to include. * **feat_idx** ("ndarray") – Ordered physical feature indices to include. * **batch_size** ("int" | "None") – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget. * **msg** ("str" | "None") – Message to be displayed in the progress bar * **as_dataframe** ("bool") – If true (default) then the yielded matrices are pandas dataframe * **scratch_itemsize** ("int") – Bytes of working memory the caller needs per yielded value while it processes one batch. Batches are sized so that this scratch also fits the memory budget. * **resident_bytes** ("int") – Bytes the caller keeps allocated for the whole iteration, such as an output buffer. * ****norm_params** ("Any") – Extra keyword arguments forwarded to "normed". Return type: "Generator"["DataFrame" | "tuple"["ndarray", "ndarray"], "None", "None"] Returns: Generator yielding DataFrames or (matrix, feature index) tuples. normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False) This function normalizes the raw and returns a delayed chunked array of the normalized data. Parameters: * **cell_idx** ("ndarray" | "None") – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table) * **feat_idx** ("ndarray" | "None") – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis. * **renormalize_subset** ("bool") – Must be False. * **log_transform** ("bool") – Return "log1p" of the normalized values. Return type: "ChunkedArray" Returns: A chunked array (delayed matrix) containing normalized data. Raises: **ValueError** – If the configured normalizer cannot apply a requested flag. score_features(feature_names, cell_key, ctrl_size, n_bins, rand_seed, *, log_transform=False) Calculates the scores (mean values) of selection of features over a randomly sampled selected feature set in given cells (as marked by cell_key) Parameters: * **feature_names** ("list"["str"]) – Names (as in ‘names’ column of the feature attribute table) of features to be used for scoring * **cell_key** ("str") – Name of the key (column) from cell attribute table. * **ctrl_size** ("int") – Number of reference features to be sampled from each bin. * **n_bins** ("int") – Number of bins for sampling. * **rand_seed** ("int") – The seed to use for the random number generation. * **log_transform** ("bool") – Score "log1p" of the normalized values. Return type: "ndarray" Returns: Numpy array of the calculated scores Raises: **ValueError** – If "log_transform" is True and the normalizer cannot apply it. class cytearc.assay.RNAassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs) This subclass of Assay is designed for feature selection and normalization of scRNA-Seq data. Parameters: * **z** ("Group") – Zarr hierarchy where raw data is located * **name** ("str") – A label/name for assay. * **cell_data** ("MetaData") – Metadata class object for the cell attributes. * ****kwargs** ("Any") – kwargs to be passed to the Assay class normMethod A pointer to the function to be used for normalization of the raw data sf scaling factor for doing library-size normalization scalar This is used to cache the library size of the cells. It is set to None until normed method is called. iter_normed_feature_wise(cell_idx, feat_idx, batch_size, msg, as_dataframe=True, scratch_itemsize=0, resident_bytes=0, **norm_params) Iterate over explicitly selected normalized features in batches. Parameters: * **cell_idx** ("ndarray") – Ordered physical cell indices to include. * **feat_idx** ("ndarray") – Ordered physical feature indices to include. * **batch_size** ("int" | "None") – Number of genes loaded at a time. When None, selected features are grouped into chunk-aligned blocks that fit the operation memory budget. * **msg** ("str" | "None") – Message to be displayed in the progress bar * **as_dataframe** ("bool") – If true (default) then the yielded matrices are pandas dataframe * **scratch_itemsize** ("int") – Bytes of working memory the caller needs per yielded value while it processes one batch. Batches are sized so that this scratch also fits the memory budget. * **resident_bytes** ("int") – Bytes the caller keeps allocated for the whole iteration, such as an output buffer. * ****norm_params** ("Any") – Extra keyword arguments forwarded to "normed". Return type: "Generator"["DataFrame" | "tuple"["ndarray", "ndarray"], "None", "None"] Returns: Generator yielding DataFrames or (matrix, feature index) tuples. normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False) This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the *normed* method in the generic Assay class this method is optimized for scRNA-Seq data and supports "renormalize_subset" with *norm_lib_size* (default normalization method for this class). Parameters: * **cell_idx** ("ndarray" | "None") – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table) * **feat_idx** ("ndarray" | "None") – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis. * **renormalize_subset** ("bool") – If true, normalize using only "feat_idx" rather than total expression across all features in a cell. (Default value: False) * **log_transform** ("bool") – If True, then the normalized data is log-transformed (Default value: False). Return type: "ChunkedArray" Returns: A chunked array (delayed matrix) containing normalized data. class cytearc.assay.ATACassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs) This subclass of Assay is designed for feature selection and normalization of scATAC-Seq data. normed(cell_idx=None, feat_idx=None, renormalize_subset=False, log_transform=False) This function normalizes the raw and returns a delayed chunked array of the normalized data. Unlike the *normed* method in the generic Assay class this method is optimized for scATAC-Seq data. This method uses the normalization indicated by attribute self.normMethod which by default is set to *norm_tf_idf*. Document frequency is learned from *cell_idx*. The returned matrix contains *feat_idx*, while term frequency uses total ATAC counts unless subset renormalization is requested. Parameters: * **cell_idx** ("ndarray" | "None") – Indices of cells to be included in the normalized matrix (Default value: All those marked True in ‘I’ column of cell attribute table) * **feat_idx** ("ndarray" | "None") – Indices of features to be included in the normalized matrix. Defaults to the complete physical feature axis. * **renormalize_subset** ("bool") – If True, use counts among "feat_idx" as the term-frequency denominator. * **log_transform** ("bool") – Must be False. Return type: "ChunkedArray" Returns: A chunked array (delayed matrix) containing normalized data. class cytearc.assay.ADTassay(z, name, cell_data, *, workspace=None, nthreads=1, **kwargs) This subclass of Assay is designed for normalization of ADT/HTO (feature-barcodes library) data from CITE-Seq experiments. Parameters: * **z** ("Group") – Zarr hierarchy where raw data is located * **name** ("str") – A label/name for assay. * **cell_data** ("MetaData") – Metadata class object for the cell attributes. * ****kwargs** ("Any") normMethod Pointer to the function to be used for normalization of the raw data cytearc.assay.norm_dummy(_, counts) A dummy normalizer. Doesn’t perform any normalization. This is useful when the ‘raw data’ is already normalized. Parameters: * **_** ("Assay") * **counts** ("ChunkedArray") – A chunked array with ‘raw’ counts data Return type: "ChunkedArray" Returns: A chunked array class cytearc.metadata.MetaData(zgrp) Metadata table for cells and features backed by one Zarr group. Changes made through this class are synchronized with the backing store. active_index(key) Return global row indices selected by a boolean column. Return type: "ndarray" property columns: list[str] Return all metadata column names. default_block_rows(column='I') Prefer the Zarr chunk length of "column" for row iteration. Return type: "int" drop(column) Delete an unprotected metadata column. Return type: "None" fetch(column, key='I') Return stored column values for rows selected by "key". Rows flagged by the column’s linked missing mask hold stored placeholders; "to_pandas_dataframe" shows them as missing. Return type: "ndarray" fetch_all(column) Return all stored values from a metadata column. Rows flagged by the column’s linked missing mask hold stored placeholders; "to_pandas_dataframe" shows them as missing. Return type: "ndarray" fetch_all_columns(columns) Return several whole columns; object stores read them concurrently. Return type: "list"["ndarray"] get_dtype(column) Return the dtype of a metadata column. Return type: "dtype"["Any"] get_index_by(value_targets, column, key=None) Return row indices for requested values in a metadata column. Values match as case-insensitive text, as in "CaseInsensitiveIndex", and every row that matches a target is returned, in target order. With "key", indices are positions among the rows that "key" selects. A target that matches no row adds no index and is counted in a warning. Rows flagged by the column’s missing mask never match. Return type: "ndarray" grep(pattern, only_valid=False) Return feature names matching a case-insensitive regex. Return type: "list"["str"] head(n=5) Return the first "n" rows of all columns, with masked rows missing. Return type: "DataFrame" index_to_bool(idx, invert=False) Convert row indices into a table-sized boolean array. Return type: "ndarray" insert(column_name, values, fill_value=None, key='I', overwrite=False, force=False) Insert a column into the table. Raises: **ValueError** – If "column_name" is protected, already exists without "overwrite", or contains "/" or "\", which Zarr reads as path separators. Return type: "None" iter_row_blocks(*, cell_key='I', columns=None, block_rows=None) Yield contiguous row blocks over this table. Each block covers a half-open global index range "[start, stop)". "active_global_indices" lists rows in that range that pass "cell_key". Column arrays are aligned to those active indices only. Return type: "Iterator"["MetaDataRowBlock"] multi_sift(columns, lows, highs, keep_bounds=False) Return a boolean mask where all column filters are satisfied. Return type: "ndarray" reset_key(key) Set every value in a boolean metadata key to true. Return type: "None" sift(column, min_v=-inf, max_v=inf, keep_bounds=False) Return rows whose values fall within the requested bounds. Rows flagged by the column’s linked missing mask never pass. Return type: "ndarray" to_pandas_dataframe(columns, key=None) Return requested columns as a DataFrame, optionally filtered by key. Rows flagged by a column’s linked missing mask are shown as missing: NaN for numeric columns, which become float64, a nullable boolean for boolean columns, and a missing value for other columns. Return type: "DataFrame" update_key(values, key) Restrict a boolean metadata key using the supplied values. Return type: "None"