Read metadata and select cells

CyteArc keeps counts, metadata, graphs, and saved analysis results in a Zarr store. It reads count blocks as needed, so working with a datastore does not require loading the complete expression matrix into memory. Cell and feature metadata describe the rows and columns of that matrix.

This page starts with metadata queries and cell selections, then shows where counts and saved results live. The examples require CyteArc’s extra optional dependencies. Low-level layout details for contributors live in Zarr internals.

Open a prepared pancreas dataset¶

DataStore is the main entry point. Each assay owns feature metadata, normalization, and feature selection. Cell-level columns are shared across assays. The Boolean cell column I marks the active cells used by live-metadata methods. Saved runs keep their own frozen cell selection. Every assay spans all cells of the store. When an assay measured only some of them, as after a merge of sources with different assays, the Boolean column <assay>_I, such as ADT_I, marks the cells it measured; the other cells hold zero counts for that assay. Without that column, the assay measured every cell. Those zero counts are no measurement, so an operation that reads an assay’s values, such as normalization or marker search, refuses its unmeasured cells: select the measured ones first with ds.select_measured_cells("ADT", cell_selection=...). Plots show the values of unmeasured cells as missing.

The default assay supplies method defaults when from_assay is omitted. It does not merge assay-specific feature tables or results.

This page uses the pre-analyzed Bastidas-Ponce pancreas store also used in Core plotting features and Score cell-cycle activity. It contains a completed pipeline run named docs_default. Opening this run gives us its saved cell selection, computed clusters, and UMAP without running the analysis again.

Open the downloaded store and its saved analysis.

The selected clustering remains in its exact artifact. The catalog’s literal clusters column is an imported cell-type annotation, not a copy of this analytical result. Opening the run does not modify either one.

Read cell and feature metadata¶

Cell and feature tables are MetaData objects (ds.cells, ds.RNA.feats), not pandas DataFrames. Use head for a quick look, to_pandas_dataframe to export selected columns, and fetch / fetch_all for single columns.

insert writes a new column and aligns values to the active subset unless you override key. When the values cover only the rows that key selects, the other rows are missing: they are flagged in the column’s missing mask, to_pandas_dataframe and plots show them as missing, and sift never selects them. Pass fill_value to store a real value in those rows instead; it must fit the values, such as False for a Boolean column. None, NaN, and pd.NA among the values are missing too. Re-inserting an existing column requires overwrite=True, and a failed replacement keeps the previous column. In this prepared store, every cell is active and the saved run contains the same cells, so its cluster labels align with the values we insert. For another analysis, check the cell selection before copying results into live metadata.

The new column marks one cluster on the same active cells used for the insert.

fetch returns values for the active subset (default column I). fetch_all returns every row in the store. With every cell active the lengths match. Pass the new Boolean column as key to select one cluster without changing I:

Select cells from metadata¶

sift returns a boolean mask over all metadata rows for one numeric range. Its bounds are exclusive by default: the next call selects cells with more than 1,000 and fewer than 15,000 counts. Pass keep_bounds=True to include the endpoints.

multi_sift keeps rows that satisfy every range, with one lower and one upper bound per column. get_index_by returns row indexes for exact categorical matches:

Combine count and detected-gene limits.

Find rows by their published annotation.

These helpers return masks or indexes aligned with the metadata table. They do not modify I until you explicitly insert or update a cell key. See the MetaData API in Assays and metadata API reference for update and delete helpers.

Optional: inspect counts and saved results¶

The metadata examples above are enough to read annotations and select cells. Continue here when you need to inspect the underlying arrays or understand where analysis results are stored.

Count matrices and normalization¶

Raw counts are a Zarr array (often sharded), exposed as rawData, a chunked array with a NumPy-like interface that streams by row. In this store the array is at RNA/counts. RNA assays also store countsT, a gene-major copy used by HVG and marker stages. Routine analysis does not need to touch either array directly.

Normalized values are computed on demand through the lower-level assay normed() view from raw counts. features.normalize(cell_selection, features) is the public persisted path and requires exact stored cell- and feature-selection references. The direct normed() view follows its explicit or literal metadata indexes; in a newly created store the physical feature I column is all true. Inspect the shapes without loading the complete matrix:

For a worked example that reads count blocks for an external calculation, see Extending CyteArc with custom analyses.

Inspect saved analysis results¶

Analysis methods return lightweight references to results stored in Zarr. The completed docs_default run retained the HVG and normalization references it created. Asking for the same normalization again reuses that result rather than recomputing:

Inspect its status and open the underlying group only when a custom method needs direct access:

Identical inputs and parameters reuse a complete result. Branching, invalidation, and lineage are covered in Reuse results and compare analysis settings.

Look inside the store¶

CyteArc uses Zarr for chunked on-disk arrays. The store is a directory tree: counts, cell and feature attributes, and immutable artifacts live under named groups. Each array has its own shape, dtype, and chunk layout. CyteArc reads and writes those chunks without loading every array in the store.

show_zarr_tree prints the hierarchy. With depth=0 you see the top-level assays and cellData.

Cell statistics computed from an assay are stored under cellData with the assay name as a prefix (RNA_…, ADT_…).

The I column is the default user-owned cell key, tracking which cells are active for live-metadata APIs. Values are boolean. Use snapshot_cell_selection("I") to capture this live column before passing it to an analytical producer. Some metadata, mapping, and export utilities still accept cell_key directly.

This store keeps every barcode active (True). Analytical filtering returns a separate immutable selection artifact and leaves this column unchanged. If you deliberately author a live selection column, its False rows also remain in the table rather than being deleted.

Each assay group holds featureData and its persisted artifacts. Count matrices are Zarr arrays, often sharded. This store keeps RNA counts at RNA/counts.

For feature arrays, pass start="RNA/featureData" to the same method.

Each persisted result is an artifact. Assay-scoped results live under {assay}/artifacts/{kind}/{artifact_id}; datastore-scoped selections and integrated results live under artifacts/{kind}/{artifact_id}. The kind names the operation family and the identifier is random; CyteArc recognises an equivalent result by its recorded operation, revision, parameters, and inputs instead of recomputing it. Nothing here encodes parameters in the path, so a second PCA at different dimensionality becomes a sibling entry rather than a new branch of the tree.

To inspect saved result groups, use ds.show_zarr_tree(start="RNA/artifacts", depth=1).

Zarr internals covers the complete on-disk layout.

Zarr versions and storage profiles¶

Current CyteArc versions write new datasets as Zarr v3. RNA assays also write a gene-major countsT copy next to counts. The catalog dataset used here was rebuilt from its raw source with this codebase, so it already has the current layout. Re-import older RNA stores that use Zarr v2 or lack countsT before analysis.

Count matrices from the writers use sharded arrays (default profile fast_local). Set the profile with CYTEARC_ZARR_PROFILE (fast_local or cloud) or zarrProfile= when opening a DataStore.

Storage profiles and conversion belong to the physical store, while mem_budget and nthreads control execution. See Memory and execution for why RNA stores two orientations, and Explore a remote dataset for object storage and local scratch.

Common mistakes and limitations¶

• Treating on-disk objects as in-memory ones: MetaData is not a pandas DataFrame; fetch returns rows selected by I by default, while fetch_all returns every row, and an analysis result is a stored artifact, not a matrix to index directly.

• Expecting filtering to delete cells or edit saved results: filtering never deletes rows or rewrites live I, and saved artifacts are immutable products of analysis methods, not groups to edit by hand.

• Opening an older RNA store (Zarr v2, or without the countsT copy) in a current CyteArc version; re-import the store before analysis.

Metadata changes and artifacts are written into the Zarr store. Low-level layout details intended for contributors remain in Zarr internals.