Source: https://docs.nygen.io/CyteArc/api/artifacts.html Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe Artifacts, lineage, and summaries API reference *********************************************** An artifact is an immutable persisted result. Its provenance lets CyteArc verify its inputs, inspect its lineage, and reuse a completed match. Analysis code normally creates and consumes artifacts through "DataStore" methods. See Why CyteArc records provenance, Analysis pipeline API reference, and Building neighbourhood graphs step by step. Types ===== +------------+--------------------------------------------------------------------------------------------+ | "cytearc.A | | | rtifactRef | | | " | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.A | A machine-readable failure to resolve or validate an artifact. | | rtifactRes | | | olutionErr | | | or" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.A | | | rtifactSta | | | tus" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.A | A snapshot of the upstream lineage of artifact outputs. | | rtifactLin | | | eage" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.D | | | ataStoreSu | | | mmary" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.m | The per-cell values of one cell-aligned artifact, aligned to cells. | | etadata.Ce | | | llValues" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.s | One exact artifact held by another datastore. | | torage.ref | | | s.External | | | ArtifactRe | | | f" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.s | One released change to what an operation computes. | | torage.ope | | | ration_rev | | | isions.Ope | | | rationRevi | | | sion" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.s | Build an immutable unordered collection of unique elements. | | torage.ART | | | IFACT_KIND | | | S" | | +------------+--------------------------------------------------------------------------------------------+ class cytearc.ArtifactRef(scope, kind, artifact_id, assay=None) class cytearc.ArtifactResolutionError(message, *, code, context) A machine-readable failure to resolve or validate an artifact. class cytearc.ArtifactStatus(ref, path, exists, complete, provenance=None, execution_options=None, created_at_ns=None, cytearc_version=None) property current_revision: int | None The revision this CyteArc release records for the same provenance. input_ref(name) Return the exact artifact reference recorded as input "name". Raises: **ArtifactResolutionError** – With code "corrupt_payload" when the input is missing or is not an exact artifact reference. Return type: "ArtifactRef" property is_current: bool Whether the artifact records the revision this release records. Only the artifact’s own revision is judged, never its inputs. property revision: int | None The operation revision the artifact records, or None if unknown. property superseded_by: tuple[OperationRevision, ...] Released revisions newer than the recorded one that apply, oldest first. class cytearc.ArtifactLineage(graph, outputs) A snapshot of the upstream lineage of artifact outputs. class cytearc.DataStoreSummary(zarr_mode, workspace, default_assay, cytearc_version, resources, total_cells, active_cells, cell_columns, assays, artifacts, pipeline_run_counts, labeled_pipeline_runs) to_dict() Return a deterministic representation containing JSON-safe values. Return type: "dict"["str", "Any"] class cytearc.metadata.CellValues(source, value, values, cell_ids, cell_idx, cell_selection, missing, categorical) The per-cell values of one cell-aligned artifact, aligned to cells. source The artifact that holds the values. value The name of the array that was read. values One value, or one row of values, per cell. cell_ids The "ids" of the cells, aligned to "values". cell_idx The row of each cell in the cell table, in ascending order. cell_selection The cell selection that the rows follow. missing True for each cell whose value is missing, or None. categorical Whether the values are labels that group cells. to_pandas() Return the values indexed by cell id, showing missing rows as missing. Return type: "Series" | "DataFrame" class cytearc.storage.refs.ExternalArtifactRef(dataset_fingerprint, ref, anchor_assay=None) One exact artifact held by another datastore. "dataset_fingerprint" names the dataset whose axes the artifact is aligned to: the prepared dataset fingerprint of one of its assays, the anchor assay. It names a dataset, not a datastore, so a mount and its source share it, and the caller supplies the datastore that holds the artifact. The anchor is the artifact’s own assay unless "anchor_assay" names another one, which a datastore-scoped artifact, or an artifact of a different assay, must do. property fingerprint_assay: str The assay of the other datastore that "dataset_fingerprint" describes. class cytearc.storage.operation_revisions.OperationRevision(revision, release, change, applies) One released change to what an operation computes. revision The revision number; an operation’s first change is 2. release The first CyteArc release that records it, such as ""1.0.0"". change One line that says what changed. applies A pure predicate that selects the affected artifacts, or None for all. An "ArtifactRef" is a location-free name: a scope, a kind, an assay for assay-scoped artifacts, and a random 256-bit "artifact_id". Provenance, pipeline records, and lineage store refs, never paths or store locations, so the datastore decides which stores resolve a ref. A datastore resolves its own artifacts; a mounted target resolves its own and then, read only, its source’s (see Mounted targets). Supported artifact kind names are listed in "cytearc.storage.ARTIFACT_KINDS". cytearc.storage.ARTIFACT_KINDS Build an immutable unordered collection of unique elements. Selections and summaries ======================== Cell and feature selections are immutable Boolean artifacts aligned to a complete stored axis. Their integrity includes the values and the exact ordered row identities. A live metadata column may be the source of a selection, but changing that column does not change or invalidate the historical artifact. Replacing, reordering, adding, or removing axis IDs fails closed; CyteArc does not remap a stored result by matching IDs. Feature summaries hold sufficient statistics for one exact cell selection and can be reused by detected-feature, HVG, prevalent-peak, and cell-cycle producers. New code does not create or read mounted "summary_stats_*" groups. RNA summaries persist "normed_tot", "normed_n", and "sigmas"; ATAC summaries persist "prevalence" and "document_frequency". +-----------------------------------+-----------------------------------+-----------------------------------+ | Producer | Artifact inputs | Scientific identity | |===================================|===================================|===================================| | assay universe | none | dataset and ordered feature IDs | +-----------------------------------+-----------------------------------+-----------------------------------+ | manual selection | "all_features" | supplied values fingerprint | +-----------------------------------+-----------------------------------+-----------------------------------+ | RNA or ATAC summary | "cell_selection" | normalizer settings | +-----------------------------------+-----------------------------------+-----------------------------------+ | detected features | "feature_summary" | "min_cells" | +-----------------------------------+-----------------------------------+-----------------------------------+ | highly variable genes | "feature_summary" | resolved variability settings | +-----------------------------------+-----------------------------------+-----------------------------------+ | prevalent peaks | "feature_summary" | "top_n" | +-----------------------------------+-----------------------------------+-----------------------------------+ | mapping overlap | mapping reference and query | exact inputs | | | "all_features" | | +-----------------------------------+-----------------------------------+-----------------------------------+ "features.hvgs", "features.prevalent_peaks", "features.detected", and "features.snapshot" return exact feature-selection refs. They do not create metadata columns. "features.resolve" accepts an explicit compatible ref. Graph lineage ============= Graph-derived analyses follow named artifact inputs: connectivity_map -> neighbors -> coordinates and ann_index ann_index -> the same coordinates batch_correction -> reduction -> normalized -> feature_selection Native graphs project one feature selection. Imported-coordinate graphs project none. Integrated graphs follow their ordered "source_i" refs and can project zero, one, or several distinct selections. Graph and neighbour consumers require their exact refs. Analytical producers return artifacts and leave metadata unchanged. Inspect and trace results ========================= Use public datastore methods rather than reading private Zarr paths: refs = ds.artifacts.list(kind="reduction", complete_only=True) status = ds.artifacts.inspect(refs[0]) status.operation status.parameters status.inputs status.execution_options status.created_at_ns status.cytearc_version status.revision status.current_revision status.is_current status.superseded_by "artifacts.list" uses the default assay unless another assay is supplied. Store-level outputs can be listed with "scope="datastore"". "artifacts.load(ref)" opens the payload only after CyteArc confirms that the artifact exists and is complete, and "artifacts.load_values(ref)" reads the per-cell values of a cell- aligned artifact aligned to their cells (Read per-cell values). A producer makes its artifact complete with one final write. A write that fails or is interrupted before that write removes its incomplete artifact. Once the final write is issued, CyteArc never removes the artifact, because the write can persist even when it reports an error. A failure or interruption during that write can therefore leave an incomplete artifact that "artifacts.list(complete_only=False)" shows and that nothing reuses. Results whose values must be finite are checked while they are written: normalized values, reduced coordinates, loadings, and centers, Harmony corrections and their fitted state, embedding initializations, embeddings, neighbour distances, and connectivity-map and integrated-graph weights. A NaN or infinite value, or a finite value too large for the stored dtype, raises "ValueError" naming the operation, the array, and the first row that holds it; no artifact is published. Operation revisions and stale results ------------------------------------- Provenance records an operation revision when it is 2 or more; an artifact without one is revision 1. A CyteArc release that changes what an operation computes adds a revision (Operation revisions), and reuse requires the revision that the running release records, so results of an older revision are computed again. "status.revision" is the recorded revision, "status.current_revision" the revision that the running release records for the same provenance, and "status.is_current" whether they agree. "is_current" judges only the artifact’s own revision: an artifact built from a superseded input is current itself. "status.superseded_by" lists the released revisions after the recorded one that apply to the artifact, each with the "change" it made. The "cytearc_version" attribute names the release that wrote an artifact for diagnostics and never affects reuse. A superseded artifact is never reused but stays listable, loadable, and traceable. When planning finds one and no exact match, it logs one INFO line that names it and the changes since its revision. Payload validation still applies: marker-table readers require "fold_change_policy" metadata and refuse a table that does not record it. "DataStore.artifacts.lineage" reports show a superseded artifact with the status "stale". Results built on it keep their own current revision, but once it is computed again it has a new reference, so the results built on it are computed again too. Passing a superseded artifact explicitly as an input is allowed and records it in lineage as usual. Find one saved result --------------------- "ds.artifacts.find" locates a graph, embedding, or clustering without requiring its storage kind or producing-operation name. By default it searches computed results. It reads complete saved results and requires exactly one match. No match or more than one match raises "ArtifactResolutionError", a "ValueError"; it never computes a result or chooses the newest one. layout = ds.artifacts.find("embedding", graph=graph_ref, method="umap") clusters = ds.artifacts.find("clusters", graph=graph_ref, method="leiden", resolution=0.5) coordinates = ds.artifacts.load_values(layout) +----------------------------------------------------+----------------------------------------------------+ | Computed result | Methods | |====================================================|====================================================| | ""embedding"" | ""umap"", ""tsne"" | +----------------------------------------------------+----------------------------------------------------+ | ""clusters"" | ""leiden"", ""paris"" | +----------------------------------------------------+----------------------------------------------------+ | ""graph"" | ""wnn"", ""snn""; omit "method" to include native | | | connectivity graphs too | +----------------------------------------------------+----------------------------------------------------+ Omitting "method" includes all methods for the requested computed result. Use "from_assay=" to restrict an assay, or "graph=" to match that exact graph and its scope. Without either, the search includes the default assay and datastore-scoped results. Extra keywords such as "resolution=" match saved parameters. Float parameters use the producer’s numeric conversion, so "resolution=1" and "resolution=1.0" find the same results. Other parameter types retain exact matching. Superseded results remain candidates. To find an imported embedding by its original source key, pass "imported=": imported_layout = ds.artifacts.find("embedding", imported="X_umap", from_assay="RNA") "imported" must be a nonempty string and is available only for ""embedding"". It searches imported results separately from computed results, using the same assay scope rules. It cannot be combined with "method=", "graph=", or "dimreduc_key=". Other recorded parameter filters remain available. The existing "cytebase.embedding" and "cytebase.embeddings" helpers remain supported. For other kinds or arbitrary provenance filters, use "ds.artifacts.list". Exact provenance filters ------------------------ "DataStore.artifacts.list" can match the operation, parameters, and inputs that provenance records. The recorded revision is not a filter, so a listing includes superseded artifacts: +----------------------------------------------------+----------------------------------------------------+ | Filter | Match rule | |====================================================|====================================================| | "operation" | Exact operation name, such as | | | ""run_normalization"" | +----------------------------------------------------+----------------------------------------------------+ | "parameters" | Exact values for each supplied top-level parameter | | | after provenance serialization | +----------------------------------------------------+----------------------------------------------------+ | "inputs" | Exact values for each supplied top-level input | | | after provenance serialization | +----------------------------------------------------+----------------------------------------------------+ Supplied filters are combined with AND. Mapping key order does not matter, and an "ArtifactRef" matches its serialized "to_dict()" form. Each supplied top-level entry must match exactly, while the stored provenance may contain other top-level entries. A supplied nested mapping is compared as a complete value, not as another partial query. Passing "{}" matches an empty mapping; passing "None" leaves that field unfiltered. Any "operation", "parameters", or "inputs" filter returns only complete artifacts with valid provenance, even when "complete_only=False". To search with every top-level provenance entry available from a known result, inspect it and pass those complete fields back unchanged: refs = ds.artifacts.list(kind="reduction", complete_only=True) known = ds.artifacts.inspect(refs[0]) matching_refs = ds.artifacts.list( kind=known.ref.kind, from_assay=known.ref.assay, operation=known.operation, parameters=known.parameters, inputs=known.inputs, ) This still uses containment matching. A stored artifact with the same supplied entries plus additional top-level entries also matches. "DataStore.artifacts.lineage" follows artifact inputs upstream: lineage = ds.artifacts.lineage( { "baselineGraph": baseline_graph, "alternativeGraph": alternative_graph, } ) markdown_report = lineage.to_markdown() mermaid_source = lineage.to_mermaid() This identifies the exact selections, normalization, coordinates, and graph behind a result and shows where branches diverge. Each artifact in the report has one status: "complete"; "stale" when a later operation revision supersedes it; "incomplete"; "missing"; or "unresolved external" for an input in another datastore that was not supplied. An input stored in another datastore, such as the mapping reference behind a query projection, is recorded as an "ExternalArtifactRef". It names the artifact’s dataset by the prepared dataset fingerprint of one of its assays; "anchor_assay" names that assay when the artifact is datastore-scoped or belongs to another assay. The fingerprint names a dataset, not a store, and a mount shares its source’s. Pass the mapping references to "DataStore.artifacts.lineage(target, references=...)" to follow such inputs into their datastores; without them, an external input is shown as unresolved. A mount records the source artifacts it uses as plain refs, because it resolves them itself. "ArtifactResolutionError" is a "ValueError" with a machine-readable "code" and JSON-safe "context". Failures distinguish missing or incomplete artifacts, wrong kind/scope/assay, changed row identity or selection values, corrupt payloads, and incompatible artifact contracts. The error explains what failed; it does not choose a replacement result. Read per-cell values -------------------- "artifacts.load_values(ref)" reads the per-cell values of a cell- aligned artifact and returns a "cytearc.metadata.CellValues" aligned to the "ids" of its cells. A cell-aligned artifact holds a row for each cell of the cell selection that its provenance records as its "cell_selection" input. The table lists these kinds, the canonical array that "artifacts.load_values" reads by default, whether that array holds labels that group cells or measurements, and the other per-cell arrays that "value=" may name. Every other kind is refused, among them cell selections, reference labels, graphs, and per-feature results. Reductions and Harmony corrections hold a row per cell too, but their rows follow the cell selection of their lineage rather than a recorded input; open them with "artifacts.load". +----------------------+----------------------+----------------------+----------------------+----------------------+ | Kind | Canonical array | Holds | Other label arrays | Other measurement | | | | | | arrays | |======================|======================|======================|======================|======================| | "cell_cycle" | "phase" | labels | | "s_score", | | | | | | "g2m_score" | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "cluster_cut" | "labels" | labels | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "cluster_labels" | "values" | labels | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "doublet_score" | "values" | measurements | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "embedding" | "values" | measurements | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "enrichment_scores" | "scores" | measurements | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "fate_map" | "probabilities" | measurements | "valid" | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "hto_identity" | "values" | labels | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "imported_coordinate | "data" | measurements | | | | s" | | | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "label_transfer" | "labels" | labels | "abstention_reason", | "vote_class_fraction | | | | | "candidate_codes", | s", "vote_fraction", | | | | | "vote_class_codes" | "top_two_margin", | | | | | | "vote_entropy", | | | | | | "nearest_distance", | | | | | | "reference_distance_ | | | | | | percentile" | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "membership_strength | "values" | measurements | | | | " | | | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "metadata_snapshot" | "values" | measurements | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "pseudotime" | "pseudotime" | measurements | "valid" | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "quality_metric" | "values" | measurements | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ | "smart_label" | "values" | labels | | | +----------------------+----------------------+----------------------+----------------------+----------------------+ A metadata snapshot is cell-aligned when it records a cell selection, as the custom source and sink vector of pseudotime scoring does. The metadata snapshot of a pipeline run records none: it holds whole metadata columns, one row for each row of the cell or feature table, and is valid but not cell-aligned. "artifacts.load_values", plot groupings and colors, and the other readers refuse it with a "ValueError" that says its rows are not aligned to a recorded cell selection and points to the run’s frozen fields, such as "run.cells.fetch(column)". Only a snapshot whose recorded cell selection is malformed raises "ArtifactResolutionError" with code "corrupt_payload". The canonical array of each kind is part of the identity of the results that read it. Consumers such as "select_cells" record the artifact that they read but not which of its arrays, so changing a kind’s canonical array in this table would change what they compute from the same recorded inputs. Such a change needs an operation revision for each consumer (Operation revisions). "cell_selection=" reads the cells of a selection that the artifact’s own selection contains; any other selection raises "ValueError". Values come back in cell table order. A row that the artifact records as missing keeps its stored placeholder and is flagged in "missing". "to_pandas()" returns the values indexed by cell id, as a Series, or as a DataFrame with a column for each position of a row, and shows missing rows as missing. Before it reads anything, the read is charged against the datastore memory budget, and it raises "MemoryError" when it does not fit: the values and their missing mask, the cell ids, and the int64 cell rows and positions that it builds, each read chunk by chunk. phase = ds.artifacts.load_values(cell_cycle_ref) # canonical "phase" labels s_score = ds.artifacts.load_values(cell_cycle_ref, value="s_score") phase.to_pandas() # Series indexed by cell id coordinates = ds.artifacts.load_values(umap_ref).to_pandas() # one column per dimension subset = ds.artifacts.load_values(clusters_ref, cell_selection=t_cells) "select_cells", artifact groupings and colors in plots, and pseudotime source and sink labels read the same canonical arrays, and every reader refuses a kind that the table does not list, whichever array it is asked for. Label consumers, such as "snapshot_cluster_labels", "compare.compute_bulk", and "clusters.relabel_by_overlap", accept the kinds whose canonical array holds labels. cytearc.metadata.selection.CELL_VALUE_NAMES The per-cell arrays of every cell-aligned artifact kind, a read- only mapping from each kind to its "CellValueSpec": the canonical array, whether it holds labels, and the kind’s other per-cell arrays. A kind is cell-aligned exactly when it is listed, and readers refuse every other kind, whatever array they are asked for. Reductions and Harmony corrections hold a row per cell too, but their cells follow the cell selection of their lineage rather than a "cell_selection" input, so they are not listed. The canonical array names are part of the identity of the results that read them: "select_cells", plot groupings, and other consumers record the artifact that they read but not which of its arrays. Changing a kind’s canonical array therefore changes what those consumers compute from the same recorded inputs, and needs an operation revision for each of them. DataStore summary ================= "summary()" scans literal live "I" cell and feature columns in blocks and omits store locations and credentials. It reports artifact inventories, pipeline-run counts by status, and completed labeled runs. It never selects a pipeline run. Each assay’s "assay_type" is the type that the open resolved, "Assay.assayType" (Assay types), so a read-only open, which cannot record a type, reports the type of a preset-named assay that the store’s "assayTypes" record lacks. Use "summary.to_dict()" for a deterministic JSON-safe record. +------------+--------------------------------------------------------------------------------------------+ | "cytearc.D | Return a read-only, metadata-only summary of this datastore. | | ataStore.s | | | ummary" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | List every artifact ref matching the exact provenance predicates. | | atastore.a | | | rtifact_ac | | | cessor.Art | | | ifactAcces | | | sor.list" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Inspect a logical artifact without mutating the store. | | atastore.a | | | rtifact_ac | | | cessor.Art | | | ifactAcces | | | sor.inspec | | | t" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Open a complete artifact through a read-only Zarr group. | | atastore.a | | | rtifact_ac | | | cessor.Art | | | ifactAcces | | | sor.load" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Read the per-cell values of a cell-aligned artifact. | | atastore.a | | | rtifact_ac | | | cessor.Art | | | ifactAcces | | | sor.load_v | | | alues" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Build a read-only upstream lineage report for artifact outputs. | | atastore.a | | | rtifact_ac | | | cessor.Art | | | ifactAcces | | | sor.lineag | | | e" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Validate and return one exact feature-selection reference. | | atastore.n | | | amespaces. | | | FeaturesAc | | | cessor.res | | | olve" | | +------------+--------------------------------------------------------------------------------------------+ DataStore.summary() Return a read-only, metadata-only summary of this datastore. Return type: "DataStoreSummary" ArtifactAccessor.list(*, kind=None, from_assay=None, scope='assay', complete_only=False, operation=None, parameters=None, inputs=None) List every artifact ref matching the exact provenance predicates. Return type: "list"["ArtifactRef"] ArtifactAccessor.inspect(ref) Inspect a logical artifact without mutating the store. Return type: "ArtifactStatus" ArtifactAccessor.load(ref) Open a complete artifact through a read-only Zarr group. Return type: "Group" ArtifactAccessor.load_values(ref, *, value=None, cell_selection=None) Read the per-cell values of a cell-aligned artifact. Parameters: * **ref** ("ArtifactRef") – A cell-aligned artifact, such as a clustering or an embedding. * **value** ("str" | "None") – Name of the per-cell array to read, or None for the canonical one. * **cell_selection** ("ArtifactRef" | "None") – A subset of the artifact’s cell selection, or None for all. Return type: "CellValues" Returns: The values with their cell ids and missing mask, as "CellValues". Raises: * **TypeError** – If an argument has the wrong type. * **ValueError** – If "ref" is not a complete cell-aligned artifact. * **MemoryError** – If the read does not fit the datastore memory budget. ArtifactAccessor.lineage(target, *, references=None) Build a read-only upstream lineage report for artifact outputs. Return type: "ArtifactLineage" FeaturesAccessor.resolve(assay, features) Validate and return one exact feature-selection reference. Return type: "ArtifactRef" Module-level helpers ==================== These functions accept a Zarr root group and are useful in tooling. Analysis notebooks should use the datastore methods above. The functions inspect exactly the group they are given: the "zw" group of a mounted datastore also resolves its source’s artifacts, while a target group opened directly with "zarr.open_group" holds only the target’s own. cytearc.storage.list_artifacts(root, *, scope, assay=None, kind=None, complete_only=False, operation=None, parameters=None, inputs=None) Return type: "list"["ArtifactRef"] cytearc.storage.inspect_artifact(root, ref) Return type: "ArtifactStatus"