Artifacts, lineage, and summaries API reference

Artifacts, lineage, and summaries API reference#

An artifact is an immutable persisted result. Its provenance lets CyteArc verify its inputs, inspect its lineage, and reuse a completed match. Analysis code normally creates and consumes artifacts through DataStore methods.

See Why CyteArc records provenance, Analysis pipeline API reference, and Building neighbourhood graphs step by step.

Types#

`cytearc.ArtifactRef`
`cytearc.ArtifactResolutionError`	A machine-readable failure to resolve or validate an artifact.
`cytearc.ArtifactStatus`
`cytearc.ArtifactLineage`	A snapshot of the upstream lineage of artifact outputs.
`cytearc.DataStoreSummary`
`cytearc.metadata.CellValues`	The per-cell values of one cell-aligned artifact, aligned to cells.
`cytearc.storage.refs.ExternalArtifactRef`	One exact artifact held by another datastore.
`cytearc.storage.operation_revisions.OperationRevision`	One released change to what an operation computes.
`cytearc.storage.ARTIFACT_KINDS`	Build an immutable unordered collection of unique elements.

class cytearc.ArtifactRef(scope, kind, artifact_id, assay=None)#

class cytearc.ArtifactResolutionError(message, *, code, context)#: A machine-readable failure to resolve or validate an artifact.

class cytearc.ArtifactStatus(ref, path, exists, complete, provenance=None, execution_options=None, created_at_ns=None, cytearc_version=None)#

property current_revision: int | None#: The revision this CyteArc release records for the same provenance.

input_ref(name)#

Return the exact artifact reference recorded as input name.

Raises:: ArtifactResolutionError – With code corrupt_payload when the input is missing or is not an exact artifact reference.
Return type:: ArtifactRef

property is_current: bool#

Whether the artifact records the revision this release records.

Only the artifact’s own revision is judged, never its inputs.

property revision: int | None#: The operation revision the artifact records, or None if unknown.

property superseded_by: tuple[OperationRevision, ...]#: Released revisions newer than the recorded one that apply, oldest first.

class cytearc.ArtifactLineage(graph, outputs)#: A snapshot of the upstream lineage of artifact outputs.

class cytearc.DataStoreSummary(zarr_mode, workspace, default_assay, cytearc_version, resources, total_cells, active_cells, cell_columns, assays, artifacts, pipeline_run_counts, labeled_pipeline_runs)#

to_dict()#

Return a deterministic representation containing JSON-safe values.

Return type:: dict[str, Any]

class cytearc.metadata.CellValues(source, value, values, cell_ids, cell_idx, cell_selection, missing, categorical)#

The per-cell values of one cell-aligned artifact, aligned to cells.

source#: The artifact that holds the values.

value#: The name of the array that was read.

values#: One value, or one row of values, per cell.

cell_ids#: The ids of the cells, aligned to values.

cell_idx#: The row of each cell in the cell table, in ascending order.

cell_selection#: The cell selection that the rows follow.

missing#: True for each cell whose value is missing, or None.

categorical#: Whether the values are labels that group cells.

to_pandas()#

Return the values indexed by cell id, showing missing rows as missing.

Return type:: Series | DataFrame

class cytearc.storage.refs.ExternalArtifactRef(dataset_fingerprint, ref, anchor_assay=None)#

One exact artifact held by another datastore.

dataset_fingerprint names the dataset whose axes the artifact is aligned to: the prepared dataset fingerprint of one of its assays, the anchor assay. It names a dataset, not a datastore, so a mount and its source share it, and the caller supplies the datastore that holds the artifact. The anchor is the artifact’s own assay unless anchor_assay names another one, which a datastore-scoped artifact, or an artifact of a different assay, must do.

property fingerprint_assay: str#: The assay of the other datastore that dataset_fingerprint describes.

class cytearc.storage.operation_revisions.OperationRevision(revision, release, change, applies)#

One released change to what an operation computes.

revision#: The revision number; an operation’s first change is 2.

release#: The first CyteArc release that records it, such as "1.0.0".

change#: One line that says what changed.

applies#: A pure predicate that selects the affected artifacts, or None for all.

An ArtifactRef is a location-free name: a scope, a kind, an assay for assay-scoped artifacts, and a random 256-bit artifact_id. Provenance, pipeline records, and lineage store refs, never paths or store locations, so the datastore decides which stores resolve a ref. A datastore resolves its own artifacts; a mounted target resolves its own and then, read only, its source’s (see Mounted targets).

Supported artifact kind names are listed in cytearc.storage.ARTIFACT_KINDS.

cytearc.storage.ARTIFACT_KINDS#: Build an immutable unordered collection of unique elements.

Selections and summaries#

Cell and feature selections are immutable Boolean artifacts aligned to a complete stored axis. Their integrity includes the values and the exact ordered row identities. A live metadata column may be the source of a selection, but changing that column does not change or invalidate the historical artifact. Replacing, reordering, adding, or removing axis IDs fails closed; CyteArc does not remap a stored result by matching IDs.

Feature summaries hold sufficient statistics for one exact cell selection and can be reused by detected-feature, HVG, prevalent-peak, and cell-cycle producers. New code does not create or read mounted summary_stats_* groups. RNA summaries persist normed_tot, normed_n, and sigmas; ATAC summaries persist prevalence and document_frequency.

Producer	Artifact inputs	Scientific identity
assay universe	none	dataset and ordered feature IDs
manual selection	`all_features`	supplied values fingerprint
RNA or ATAC summary	`cell_selection`	normalizer settings
detected features	`feature_summary`	`min_cells`
highly variable genes	`feature_summary`	resolved variability settings
prevalent peaks	`feature_summary`	`top_n`
mapping overlap	mapping reference and query `all_features`	exact inputs

features.hvgs, features.prevalent_peaks, features.detected, and features.snapshot return exact feature-selection refs. They do not create metadata columns. features.resolve accepts an explicit compatible ref.

Graph lineage#

Graph-derived analyses follow named artifact inputs:

connectivity_map -> neighbors -> coordinates and ann_index
ann_index -> the same coordinates
batch_correction -> reduction -> normalized -> feature_selection

Native graphs project one feature selection. Imported-coordinate graphs project none. Integrated graphs follow their ordered source_i refs and can project zero, one, or several distinct selections. Graph and neighbour consumers require their exact refs. Analytical producers return artifacts and leave metadata unchanged.

Inspect and trace results#

Use public datastore methods rather than reading private Zarr paths:

refs = ds.artifacts.list(kind="reduction", complete_only=True)
status = ds.artifacts.inspect(refs[0])

status.operation
status.parameters
status.inputs
status.execution_options
status.created_at_ns
status.cytearc_version
status.revision
status.current_revision
status.is_current
status.superseded_by

artifacts.list uses the default assay unless another assay is supplied. Store-level outputs can be listed with scope="datastore". artifacts.load(ref) opens the payload only after CyteArc confirms that the artifact exists and is complete, and artifacts.load_values(ref) reads the per-cell values of a cell-aligned artifact aligned to their cells (Read per-cell values).

A producer makes its artifact complete with one final write. A write that fails or is interrupted before that write removes its incomplete artifact. Once the final write is issued, CyteArc never removes the artifact, because the write can persist even when it reports an error. A failure or interruption during that write can therefore leave an incomplete artifact that artifacts.list(complete_only=False) shows and that nothing reuses.

Results whose values must be finite are checked while they are written: normalized values, reduced coordinates, loadings, and centers, Harmony corrections and their fitted state, embedding initializations, embeddings, neighbour distances, and connectivity-map and integrated-graph weights. A NaN or infinite value, or a finite value too large for the stored dtype, raises ValueError naming the operation, the array, and the first row that holds it; no artifact is published.

Operation revisions and stale results#

Provenance records an operation revision when it is 2 or more; an artifact without one is revision 1. A CyteArc release that changes what an operation computes adds a revision (Operation revisions), and reuse requires the revision that the running release records, so results of an older revision are computed again. status.revision is the recorded revision, status.current_revision the revision that the running release records for the same provenance, and status.is_current whether they agree. is_current judges only the artifact’s own revision: an artifact built from a superseded input is current itself. status.superseded_by lists the released revisions after the recorded one that apply to the artifact, each with the change it made. The cytearc_version attribute names the release that wrote an artifact for diagnostics and never affects reuse.

A superseded artifact is never reused but stays listable, loadable, and traceable. When planning finds one and no exact match, it logs one INFO line that names it and the changes since its revision. Payload validation still applies: marker-table readers require fold_change_policy metadata and refuse a table that does not record it. DataStore.artifacts.lineage reports show a superseded artifact with the status stale. Results built on it keep their own current revision, but once it is computed again it has a new reference, so the results built on it are computed again too. Passing a superseded artifact explicitly as an input is allowed and records it in lineage as usual.

Find one saved result#

ds.artifacts.find locates a graph, embedding, or clustering without requiring its storage kind or producing-operation name. By default it searches computed results. It reads complete saved results and requires exactly one match. No match or more than one match raises ArtifactResolutionError, a ValueError; it never computes a result or chooses the newest one.

layout = ds.artifacts.find("embedding", graph=graph_ref, method="umap")
clusters = ds.artifacts.find("clusters", graph=graph_ref, method="leiden", resolution=0.5)
coordinates = ds.artifacts.load_values(layout)

Computed result	Methods
`"embedding"`	`"umap"`, `"tsne"`
`"clusters"`	`"leiden"`, `"paris"`
`"graph"`	`"wnn"`, `"snn"`; omit `method` to include native connectivity graphs too

Omitting method includes all methods for the requested computed result. Use from_assay= to restrict an assay, or graph= to match that exact graph and its scope. Without either, the search includes the default assay and datastore-scoped results. Extra keywords such as resolution= match saved parameters. Float parameters use the producer’s numeric conversion, so resolution=1 and resolution=1.0 find the same results. Other parameter types retain exact matching. Superseded results remain candidates.

To find an imported embedding by its original source key, pass imported=:

imported_layout = ds.artifacts.find("embedding", imported="X_umap", from_assay="RNA")

imported must be a nonempty string and is available only for "embedding". It searches imported results separately from computed results, using the same assay scope rules. It cannot be combined with method=, graph=, or dimreduc_key=. Other recorded parameter filters remain available. The existing cytebase.embedding and cytebase.embeddings helpers remain supported. For other kinds or arbitrary provenance filters, use ds.artifacts.list.

Exact provenance filters#

DataStore.artifacts.list can match the operation, parameters, and inputs that provenance records. The recorded revision is not a filter, so a listing includes superseded artifacts:

Filter	Match rule
`operation`	Exact operation name, such as `"run_normalization"`
`parameters`	Exact values for each supplied top-level parameter after provenance serialization
`inputs`	Exact values for each supplied top-level input after provenance serialization

Supplied filters are combined with AND. Mapping key order does not matter, and an ArtifactRef matches its serialized to_dict() form. Each supplied top-level entry must match exactly, while the stored provenance may contain other top-level entries. A supplied nested mapping is compared as a complete value, not as another partial query. Passing {} matches an empty mapping; passing None leaves that field unfiltered.

Any operation, parameters, or inputs filter returns only complete artifacts with valid provenance, even when complete_only=False. To search with every top-level provenance entry available from a known result, inspect it and pass those complete fields back unchanged:

refs = ds.artifacts.list(kind="reduction", complete_only=True)
known = ds.artifacts.inspect(refs[0])

matching_refs = ds.artifacts.list(
    kind=known.ref.kind,
    from_assay=known.ref.assay,
    operation=known.operation,
    parameters=known.parameters,
    inputs=known.inputs,
)

This still uses containment matching. A stored artifact with the same supplied entries plus additional top-level entries also matches.

DataStore.artifacts.lineage follows artifact inputs upstream:

lineage = ds.artifacts.lineage(
    {
        "baselineGraph": baseline_graph,
        "alternativeGraph": alternative_graph,
    }
)

markdown_report = lineage.to_markdown()
mermaid_source = lineage.to_mermaid()

This identifies the exact selections, normalization, coordinates, and graph behind a result and shows where branches diverge. Each artifact in the report has one status: complete; stale when a later operation revision supersedes it; incomplete; missing; or unresolved external for an input in another datastore that was not supplied.

An input stored in another datastore, such as the mapping reference behind a query projection, is recorded as an ExternalArtifactRef. It names the artifact’s dataset by the prepared dataset fingerprint of one of its assays; anchor_assay names that assay when the artifact is datastore-scoped or belongs to another assay. The fingerprint names a dataset, not a store, and a mount shares its source’s. Pass the mapping references to DataStore.artifacts.lineage(target, references=...) to follow such inputs into their datastores; without them, an external input is shown as unresolved. A mount records the source artifacts it uses as plain refs, because it resolves them itself.

ArtifactResolutionError is a ValueError with a machine-readable code and JSON-safe context. Failures distinguish missing or incomplete artifacts, wrong kind/scope/assay, changed row identity or selection values, corrupt payloads, and incompatible artifact contracts. The error explains what failed; it does not choose a replacement result.

Read per-cell values#

artifacts.load_values(ref) reads the per-cell values of a cell-aligned artifact and returns a cytearc.metadata.CellValues aligned to the ids of its cells. A cell-aligned artifact holds a row for each cell of the cell selection that its provenance records as its cell_selection input. The table lists these kinds, the canonical array that artifacts.load_values reads by default, whether that array holds labels that group cells or measurements, and the other per-cell arrays that value= may name. Every other kind is refused, among them cell selections, reference labels, graphs, and per-feature results. Reductions and Harmony corrections hold a row per cell too, but their rows follow the cell selection of their lineage rather than a recorded input; open them with artifacts.load.

Kind	Canonical array	Holds	Other label arrays	Other measurement arrays
`cell_cycle`	`phase`	labels		`s_score`, `g2m_score`
`cluster_cut`	`labels`	labels
`cluster_labels`	`values`	labels
`doublet_score`	`values`	measurements
`embedding`	`values`	measurements
`enrichment_scores`	`scores`	measurements
`fate_map`	`probabilities`	measurements	`valid`
`hto_identity`	`values`	labels
`imported_coordinates`	`data`	measurements
`label_transfer`	`labels`	labels	`abstention_reason`, `candidate_codes`, `vote_class_codes`	`vote_class_fractions`, `vote_fraction`, `top_two_margin`, `vote_entropy`, `nearest_distance`, `reference_distance_percentile`
`membership_strength`	`values`	measurements
`metadata_snapshot`	`values`	measurements
`pseudotime`	`pseudotime`	measurements	`valid`
`quality_metric`	`values`	measurements
`smart_label`	`values`	labels

A metadata snapshot is cell-aligned when it records a cell selection, as the custom source and sink vector of pseudotime scoring does. The metadata snapshot of a pipeline run records none: it holds whole metadata columns, one row for each row of the cell or feature table, and is valid but not cell-aligned. artifacts.load_values, plot groupings and colors, and the other readers refuse it with a ValueError that says its rows are not aligned to a recorded cell selection and points to the run’s frozen fields, such as run.cells.fetch(column). Only a snapshot whose recorded cell selection is malformed raises ArtifactResolutionError with code corrupt_payload.

The canonical array of each kind is part of the identity of the results that read it. Consumers such as select_cells record the artifact that they read but not which of its arrays, so changing a kind’s canonical array in this table would change what they compute from the same recorded inputs. Such a change needs an operation revision for each consumer (Operation revisions).

cell_selection= reads the cells of a selection that the artifact’s own selection contains; any other selection raises ValueError. Values come back in cell table order. A row that the artifact records as missing keeps its stored placeholder and is flagged in missing. to_pandas() returns the values indexed by cell id, as a Series, or as a DataFrame with a column for each position of a row, and shows missing rows as missing. Before it reads anything, the read is charged against the datastore memory budget, and it raises MemoryError when it does not fit: the values and their missing mask, the cell ids, and the int64 cell rows and positions that it builds, each read chunk by chunk.

phase = ds.artifacts.load_values(cell_cycle_ref)  # canonical "phase" labels
s_score = ds.artifacts.load_values(cell_cycle_ref, value="s_score")
phase.to_pandas()  # Series indexed by cell id
coordinates = ds.artifacts.load_values(umap_ref).to_pandas()  # one column per dimension
subset = ds.artifacts.load_values(clusters_ref, cell_selection=t_cells)

select_cells, artifact groupings and colors in plots, and pseudotime source and sink labels read the same canonical arrays, and every reader refuses a kind that the table does not list, whichever array it is asked for. Label consumers, such as snapshot_cluster_labels, compare.compute_bulk, and clusters.relabel_by_overlap, accept the kinds whose canonical array holds labels.

cytearc.metadata.selection.CELL_VALUE_NAMES#

The per-cell arrays of every cell-aligned artifact kind, a read-only mapping from each kind to its CellValueSpec: the canonical array, whether it holds labels, and the kind’s other per-cell arrays. A kind is cell-aligned exactly when it is listed, and readers refuse every other kind, whatever array they are asked for. Reductions and Harmony corrections hold a row per cell too, but their cells follow the cell selection of their lineage rather than a cell_selection input, so they are not listed.

The canonical array names are part of the identity of the results that read them: select_cells, plot groupings, and other consumers record the artifact that they read but not which of its arrays. Changing a kind’s canonical array therefore changes what those consumers compute from the same recorded inputs, and needs an operation revision for each of them.

DataStore summary#

summary() scans literal live I cell and feature columns in blocks and omits store locations and credentials. It reports artifact inventories, pipeline-run counts by status, and completed labeled runs. It never selects a pipeline run. Each assay’s assay_type is the type that the open resolved, Assay.assayType (Assay types), so a read-only open, which cannot record a type, reports the type of a preset-named assay that the store’s assayTypes record lacks. Use summary.to_dict() for a deterministic JSON-safe record.

`cytearc.DataStore.summary`	Return a read-only, metadata-only summary of this datastore.
`cytearc.datastore.artifact_accessor.ArtifactAccessor.list`	List every artifact ref matching the exact provenance predicates.
`cytearc.datastore.artifact_accessor.ArtifactAccessor.inspect`	Inspect a logical artifact without mutating the store.
`cytearc.datastore.artifact_accessor.ArtifactAccessor.load`	Open a complete artifact through a read-only Zarr group.
`cytearc.datastore.artifact_accessor.ArtifactAccessor.load_values`	Read the per-cell values of a cell-aligned artifact.
`cytearc.datastore.artifact_accessor.ArtifactAccessor.lineage`	Build a read-only upstream lineage report for artifact outputs.
`cytearc.datastore.namespaces.FeaturesAccessor.resolve`	Validate and return one exact feature-selection reference.

DataStore.summary()#

Return a read-only, metadata-only summary of this datastore.

Return type:: DataStoreSummary

ArtifactAccessor.list(*, kind=None, from_assay=None, scope='assay', complete_only=False, operation=None, parameters=None, inputs=None)#

List every artifact ref matching the exact provenance predicates.

Return type:: list[ArtifactRef]

ArtifactAccessor.inspect(ref)#

Inspect a logical artifact without mutating the store.

Return type:: ArtifactStatus

ArtifactAccessor.load(ref)#

Open a complete artifact through a read-only Zarr group.

Return type:: Group

ArtifactAccessor.load_values(ref, *, value=None, cell_selection=None)#

Read the per-cell values of a cell-aligned artifact.

Parameters:

ref (ArtifactRef) – A cell-aligned artifact, such as a clustering or an embedding.
value (str | None) – Name of the per-cell array to read, or None for the canonical one.
cell_selection (ArtifactRef | None) – A subset of the artifact’s cell selection, or None for all.

Return type:

CellValues

Returns:

The values with their cell ids and missing mask, as CellValues.

Raises:

TypeError – If an argument has the wrong type.
ValueError – If ref is not a complete cell-aligned artifact.
MemoryError – If the read does not fit the datastore memory budget.

ArtifactAccessor.lineage(target, *, references=None)#

Build a read-only upstream lineage report for artifact outputs.

Return type:: ArtifactLineage

FeaturesAccessor.resolve(assay, features)#

Validate and return one exact feature-selection reference.

Return type:: ArtifactRef

Module-level helpers#

These functions accept a Zarr root group and are useful in tooling. Analysis notebooks should use the datastore methods above. The functions inspect exactly the group they are given: the zw group of a mounted datastore also resolves its source’s artifacts, while a target group opened directly with zarr.open_group holds only the target’s own.

cytearc.storage.list_artifacts(root, *, scope, assay=None, kind=None, complete_only=False, operation=None, parameters=None, inputs=None)#

Return type:: list[ArtifactRef]

cytearc.storage.inspect_artifact(root, ref)#

Return type:: ArtifactStatus