Source: https://docs.nygen.io/CyteArc/tutorials/reuse-and-tracing/
Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe
Structured page: https://docs.nygen.io/CyteArc/tutorials.reuse-and-tracing.json

Reuse results and compare analysis settings

Changing the number of neighbours from 21 to 31 does not require rerunning normalization or PCA, or rebuilding the neighbour-search index. Query the saved index, build a new graph, and keep the original graph for comparison.

This example changes one setting at a time: first the neighbour count, then the number of principal components. A lineage diagram shows which results the analyses share. The examples require the extra optional dependencies.

Open a prepared analysis

The prepared 100,000-cell Tabula Sapiens subset has a saved run labelled docs_default, with 30 principal components and 21 neighbours. The saved run uses PCA without batch correction. We use its cells, variable genes, and saved results as the starting point. These settings belong to this example; they can differ from the current pipeline defaults.

# Open count stores and run CyteArc analyses.
import cytearc

# Keep routine logs and progress bars out of the results.
cytearc.configure_output(level="WARNING", progress=False)

# Download the prepared example store.
dataset = cytearc.cytebase.connect("cytearc_docs").download_dataset(
    "tabula_sapiens_100k",
    destination="cytearc_datasets",
    zarr=True,
)

# Open the count store for this analysis.
ds = cytearc.DataStore(f"{dataset}/data.zarr", nthreads=4, min_features_per_cell=-1)

# Open the saved analysis used to check reuse.
baseline_run = ds.pipeline.open(label="docs_default")

# Keep the cells used by the saved analysis.
cell_selection = baseline_run["analysis_cell_selection"]

# Keep the variable genes used by the saved analysis.
hvg_ref = baseline_run["highly_variable_features"]

# Inspect the opened store's cells and features.
ds

DataStore has 100000 (100000) cells with 1 assays: RNA
   Cell metadata:
            'I', 'ids', 'names', '10X_run', 'RNA_nCounts', 
            'RNA_nFeatures', 'RNA_percentMito', 'RNA_percentRibo', '_scvi_batch', '_scvi_labels', 
            'ambient_removal', 'anatomical_position', 'assay', 'assay_ontology_term_id', 'broad_cell_class', 
            'cdna_plate', 'cdna_well', 'cell_type', 'cell_type_ontology_term_id', 'compartment', 
            'development_stage', 'development_stage_ontology_term_id', 'disease', 'disease_ontology_term_id', 'donor_assay', 
            'donor_id', 'donor_method', 'donor_tissue', 'donor_tissue_assay', 'ethnicity_original', 
            'free_annotation', 'is_primary_data', 'library_plate', 'manually_annotated', 'method', 
            'n_genes_by_counts', 'notes', 'observation_joinid', 'pct_counts_ercc', 'pct_counts_mt', 
            'published_2022', 'replicate', 'sample_id', 'sample_number', 'scvi_leiden_donorassay_full', 
            'self_reported_ethnicity', 'self_reported_ethnicity_ontology_term_id', 'sex', 'sex_ontology_term_id', 'suspension_type', 
            'tissue', 'tissue_in_publication', 'tissue_ontology_term_id', 'tissue_type', 'total_counts', 
            'total_counts_ercc', 'total_counts_mt'
   RNA assay has 60606 features and following metadata:
            'I', 'ids', 'names', '_index', 'dropOuts', 
            'ercc', 'feature_biotype', 'feature_length', 'feature_reference', 'feature_type', 
            'genome', 'mean', 'mean_counts', 'mt', 'nCells', 
            'n_cells_by_counts', 'pct_dropout_by_counts', 'std', 'total_counts'

Change the neighbour count from 21 to 31

Increasing k requests more neighbours from the same search index. CyteArc writes a new neighbour list and connectivity graph, keeping the normalization, PCA, and index unchanged.

# Reopen the saved normalized expression.
normalized = baseline_run["normalized"]

# Reopen the saved PCA result.
pca = baseline_run["pca"]

# Keep the saved nearest-neighbor index.
ann = baseline_run["ann_index"]

# Keep the saved 21-neighbor result.
neighbors_k21 = baseline_run["neighbors"]

# Keep the baseline connectivity graph.
graph_k21 = baseline_run["connectivity_map"]

# Query the existing index with 31 neighbors per cell.
neighbors_k31 = ds.graph.neighbors(ann, k=31)

# Build connectivity for the new neighbor count.
graph_k31 = ds.graph.connectivity(neighbors_k31)

# Inspect which saved index the new neighbor query used.
query_status = ds.artifacts.inspect(neighbors_k31)

# Check which results were reused and which changed.
reuse_checks = {
    "normalization reused": ds.features.normalize(cell_selection, hvg_ref) == normalized,
    "PCA reused": ds.reduction.pca(normalized, dims=30) == pca,
    "ANN index reused": query_status.input_ref("ann_index") == ann,
    "neighbors changed": neighbors_k31 != neighbors_k21,
    "graph changed": graph_k31 != graph_k21,
}

# Stop if any result differs from the reuse behavior described here.
assert all(reuse_checks.values()), reuse_checks

# Display the verified comparison.
reuse_checks

{'normalization reused': True,
 'PCA reused': True,
 'ANN index reused': True,
 'neighbors changed': True,
 'graph changed': True}

See Build a neighborhood graph stage by stage <https://docs.nygen.io/CyteArc/tutorials/graph-construction> for how changing k affects an analysis. Here, notice that only neighbours and the graph needed new results. Normalization, PCA, and the search index did not need to run again.

The index check reads the new neighbour result’s recorded input. Matching it to ann confirms that the query used the existing index.

Compare 30 and 40 principal components

A new PCA dimensionality creates a separate analysis branch. The search index, neighbours, and connectivity depend on the reduction, so the new branch needs its own result at each of those steps. The earlier 30-component analysis remains available alongside the new 40-component one.

# Fit a separate reduction with 40 principal components.
pca_dims40 = ds.reduction.pca(normalized, dims=40)

# Build a search index from the 40-component reduction.
ann_dims40 = ds.graph.ann_index(pca_dims40)

# Find 21 neighbors using the new reduction.
neighbors_dims40 = ds.graph.neighbors(ann_dims40, k=21)

# Build the connectivity graph for this reduction.
graph_dims40 = ds.graph.connectivity(neighbors_dims40)

# Check which results were reused and which changed.
branch_checks = {
    "PCA changed": pca_dims40 != pca,
    "ANN index changed": ann_dims40 != ann,
    "neighbors changed": neighbors_dims40 != neighbors_k21,
    "graph changed": graph_dims40 != graph_k21,
}

# Stop if the new branch unexpectedly shares one of these results.
assert all(branch_checks.values()), branch_checks

# Display the verified comparison.
branch_checks

{'PCA changed': True,
 'ANN index changed': True,
 'neighbors changed': True,
 'graph changed': True}

The comparisons above refer to these original 30-component results. Creating the new branch has not replaced them:

# Revisit the original references after building the 40-component branch.
{
    "baseline PCA": pca,
    "baseline neighbors": neighbors_k21,
    "baseline graph": graph_k21,
}

{'baseline PCA': ArtifactRef(assay='RNA', kind='reduction', artifact_id='532aa2d022fe...'),
 'baseline neighbors': ArtifactRef(assay='RNA', kind='neighbors', artifact_id='91937d072cd9...'),
 'baseline graph': ArtifactRef(assay='RNA', kind='connectivity_map', artifact_id='f2ac2849e34c...')}

Compare analysis branches

Put all three graphs in one read-only report: the two neighbour-count branches and the dims=40 branch. Each shared input appears once, so the diagram shows where a parameter change first requires a new result.

# Trace the shared inputs and differences between all three graphs.
lineage = ds.artifacts.lineage(
    {
        "k21 graph": graph_k21,
        "k31 graph": graph_k31,
        "dims40 graph": graph_dims40,
    }
)

# Display the shared inputs and diverging analysis branches.
lineage

Output

```mermaid
flowchart LR
    artifact0["RNA / metadata_snapshot | snapshot_run_metadata | 7340c6fca965"]
    artifact1["datastore / cell_selection | snapshot_pipeline_input_selection | 829e00dc1708"]
    artifact2["RNA / feature_summary | summarize_rna_features | cab031960e6e"]
    artifact3["RNA / feature_selection | select_hvgs | 561ec515123d"]
    artifact4["RNA / normalized | run_normalization | 525af1d08271"]
    artifact5["RNA / feature_scaling | calculate_feature_scaling | 739b82327259"]
    artifact6["RNA / reduction | run_pca | 532aa2d022fe"]
    artifact7["RNA / ann_index | build_ann_index | e7f9b6ebbb42"]
    artifact8["RNA / neighbors | query_neighbors | 91937d072cd9"]
    artifact9["RNA / connectivity_map | build_connectivity_map | f2ac2849e34c | outputs: k21 graph"]
    artifact10["RNA / neighbors | query_neighbors | 977480804cdf"]
    artifact11["RNA / connectivity_map | build_connectivity_map | 6f1ab9b8a1f6 | outputs: k31 graph"]
    artifact12["RNA / reduction | run_pca | 78a3b6d966c1"]
    artifact13["RNA / ann_index | build_ann_index | bacdddea3a64"]
    artifact14["RNA / neighbors | query_neighbors | 9c5871d16ce0"]
    artifact15["RNA / connectivity_map | build_connectivity_map | 893da01dab8e | outputs: dims40 graph"]
    artifact6 -->|"coordinates"| artifact10
    artifact7 -->|"ann_index"| artifact10
    artifact10 -->|"neighbors"| artifact11
    artifact1 -->|"pca_cell_selection"| artifact12
    artifact4 -->|"normalized"| artifact12
    artifact5 -->|"feature_scaling"| artifact12
    artifact12 -->|"coordinates"| artifact13
    artifact12 -->|"coordinates"| artifact14
    artifact13 -->|"ann_index"| artifact14
    artifact14 -->|"neighbors"| artifact15
    artifact1 -->|"cell_selection"| artifact2
    artifact0 -->|"feature_snapshot"| artifact3
    artifact2 -->|"feature_summary"| artifact3
    artifact1 -->|"cell_selection"| artifact4
    artifact3 -->|"feature_selection"| artifact4
    artifact4 -->|"normalized"| artifact5
    artifact1 -->|"pca_cell_selection"| artifact6
    artifact4 -->|"normalized"| artifact6
    artifact5 -->|"feature_scaling"| artifact6
    artifact6 -->|"coordinates"| artifact7
    artifact6 -->|"coordinates"| artifact8
    artifact7 -->|"ann_index"| artifact8
    artifact8 -->|"neighbors"| artifact9
```

### Artifact details

#### RNA / metadata_snapshot / 7340c6fca965
- Status: `complete`
- Path: `RNA/artifacts/metadata_snapshot/7340c6fca965030b215edd1b1facf25e9c1dfeb08e0e39db0700c817575e40d9`
- Operation: `snapshot_run_metadata`
- Parameters: `assay="RNA"; axis="feature"; ordered_columns=["names"]`
- Other inputs: `column_fingerprints={"names":"aec1f6699a0a3c28c281c654aadc04f71a1f02d60a3bc7d4b40de04d0799e623"}; ordered_row_ids_fingerprint="17a23298a6a15eb06fc922060b386f9917f667121c8574b7b86b0dc9ec57dab8"`

#### datastore / cell_selection / 829e00dc1708
- Status: `complete`
- Path: `artifacts/cell_selection/829e00dc1708d12ef12c2fc776cefd4c7b89b80fb9607f7ae7710e64cb21fe29`
- Operation: `snapshot_pipeline_input_selection`
- Parameters: `assay="RNA"`
- Execution options: `source_column="I"`
- Other inputs: `ordered_row_ids_fingerprint="caf85a0156e46e8b6b90e763843befafe8ab7dd8f1e524ebe4eddf5d2ba7b6b9"; values_fingerprint="a2e5d973fe213b1367db7bb885ba86a84db118d73284c330b4600de71439798d"`

#### RNA / feature_summary / cab031960e6e
- Status: `complete`
- Path: `RNA/artifacts/feature_summary/cab031960e6e3baa142847ad652c69a55adff284922cca7b61356cb108640f00`
- Operation: `summarize_rna_features`
- Parameters: `normalization_method={"module":"cytearc.assay","qualname":"norm_lib_size"}; size_factor=1000`
- Execution options: `nthreads=8`
- Other inputs: `dataset_fingerprint="5cb8c7b364157dd18cf642b62137c0f18af8bd12e47be925802272a7c4eda6f3"`

#### RNA / feature_selection / 561ec515123d
- Status: `complete`
- Path: `RNA/artifacts/feature_selection/561ec515123d33564d3846774604d37cccd2ea8104183739b96cf576de2e1b0b`
- Operation: `select_hvgs`
- Parameters: `bin_strategy="adaptive"; blacklist="^MT-|^RP[SL]\\d+[AXY]?\\d*$|^RPLP[0-2]$|^RPSA$|^MRP[SL]\\d+[A-C]?$|^HLA-|^HIST[1-4]H|^H1-[1-6]$|^H1F[1-6]$|^H2AC\\d+$|^H2BC\\d+$|^H3C\\d+$|^H4C\\d+$|^H2AW$|...; blacklist_fingerprint="a3c4883166b5e93e6e11df5850179fcdd0a743a900bff99f8cbbf3f3e0c0b38c"; keep_bounds=false; lowess_frac=0.1; max_cells=99980; max_mean={"special_float":"inf"}; max_var={"special_float":"inf"}; min_cells=20; min_mean={"special_float":"-inf"}; min_var={"special_float":"-inf"}; n_bins=200; ... 3 more`
- Execution options: `invalidate_cache=false; nthreads=8; plot_kwargs={}; show_plot=false`

#### RNA / normalized / 525af1d08271
- Status: `complete`
- Path: `RNA/artifacts/normalized/525af1d08271de95530b7fe2cd1fff9db9045341235b902268ddf704a7b8daeb`
- Operation: `run_normalization`
- Parameters: `log_transform=true; normalization_method={"external_hook":true,"module":"cytearc.assay","qualname":"norm_lib_size"}; renormalize_subset=true; size_factor=1000.0`
- Execution options: `invalidate_cache=false`
- Other inputs: `dataset_fingerprint="5cb8c7b364157dd18cf642b62137c0f18af8bd12e47be925802272a7c4eda6f3"`

#### RNA / feature_scaling / 739b82327259
- Status: `complete`
- Path: `RNA/artifacts/feature_scaling/739b8232725998036ea643b6d155e35f8841c0d3202e23477fbc30cc9e19d6a9`
- Operation: `calculate_feature_scaling`
- Parameters: `enabled=true`
- Execution options: `batch_size=16777; invalidate_cache=false; local_cache="auto"`

#### RNA / reduction / 532aa2d022fe
- Status: `complete`
- Path: `RNA/artifacts/reduction/532aa2d022fe3a0a41d91a65bd3fd1bdd03f934f9e2f1e107d4356ea67c5d165`
- Operation: `run_pca`
- Parameters: `dims=30; feat_scaling=true`
- Execution options: `batch_size=16777; invalidate_cache=false; local_cache="auto"; show_elbow_plot=false`

#### RNA / ann_index / e7f9b6ebbb42
- Status: `complete`
- Path: `RNA/artifacts/ann_index/e7f9b6ebbb42b714f33cf43a4de35d1521b5b1b6a8e27d8568587026e6ffaf78`
- Operation: `build_ann_index`
- Parameters: `ann_ef=50; ann_efc=50; ann_m=48; ann_metric="l2"; ann_parallel=false; rand_state=4466`
- Execution options: `batch_size=100000; invalidate_cache=false; nthreads=1`

#### RNA / neighbors / 91937d072cd9
- Status: `complete`
- Path: `RNA/artifacts/neighbors/91937d072cd999181452a281debaa17cb51adcae39ccda32c7defb6d3271523e`
- Operation: `query_neighbors`
- Parameters: `distance_metric="l2"; k=21`
- Execution options: `batch_size=100000; invalidate_cache=false; nthreads=1`

#### RNA / connectivity_map / f2ac2849e34c
- Status: `complete`
- Path: `RNA/artifacts/connectivity_map/f2ac2849e34c99791cb11061c87dc030a8163888c0554661e5e6c7af956e9a26`
- Operation: `build_connectivity_map`
- Outputs: `k21 graph`
- Parameters: `bandwidth=1.5; local_connectivity=1.0`
- Execution options: `invalidate_cache=false`

#### RNA / neighbors / 977480804cdf
- Status: `complete`
- Path: `RNA/artifacts/neighbors/977480804cdf1731a8c7f5a720b83564b49ceb744e091963280e958a06078701`
- Operation: `query_neighbors`
- Parameters: `distance_metric="l2"; k=31`
- Execution options: `batch_size=100000; invalidate_cache=false; nthreads=1`

#### RNA / connectivity_map / 6f1ab9b8a1f6
- Status: `complete`
- Path: `RNA/artifacts/connectivity_map/6f1ab9b8a1f6577b01413d4b1872a2741b15810a100cdbacd8099c73d2cb336a`
- Operation: `build_connectivity_map`
- Outputs: `k31 graph`
- Parameters: `bandwidth=1.5; local_connectivity=1.0`
- Execution options: `invalidate_cache=false`

#### RNA / reduction / 78a3b6d966c1
- Status: `complete`
- Path: `RNA/artifacts/reduction/78a3b6d966c16fc409124694e87a9c6a57cca58223efd4159d418a73e5b36cb7`
- Operation: `run_pca`
- Parameters: `dims=40; feat_scaling=true`
- Execution options: `batch_size=16777; invalidate_cache=false; local_cache="auto"; show_elbow_plot=false`

#### RNA / ann_index / bacdddea3a64
- Status: `complete`
- Path: `RNA/artifacts/ann_index/bacdddea3a64949508e3e0b55d69d639b8cd9df3877d3a0c98e077179be34480`
- Operation: `build_ann_index`
- Parameters: `ann_ef=50; ann_efc=50; ann_m=48; ann_metric="l2"; ann_parallel=false; rand_state=4466`
- Execution options: `batch_size=100000; invalidate_cache=false; nthreads=1`

#### RNA / neighbors / 9c5871d16ce0
- Status: `complete`
- Path: `RNA/artifacts/neighbors/9c5871d16ce0d98de02164c5f8cad5e0be124c68c777dfbd26ccb6512d63cfb7`
- Operation: `query_neighbors`
- Parameters: `distance_metric="l2"; k=21`
- Execution options: `batch_size=100000; invalidate_cache=false; nthreads=1`

#### RNA / connectivity_map / 893da01dab8e
- Status: `complete`
- Path: `RNA/artifacts/connectivity_map/893da01dab8e08b2b240f9d33ed35b6b0480901972c735c1d64fcf9c8f0e401d`
- Operation: `build_connectivity_map`
- Outputs: `dims40 graph`
- Parameters: `bandwidth=1.5; local_connectivity=1.0`
- Execution options: `invalidate_cache=false`

Expand the output to inspect the full report, including its Mermaid source and exact artifact references. This diagram summarizes the branches verified above:

The k branches diverge after the ANN index. The dims=40 branch forks earlier, at PCA, then carries its own ANN, neighbours, and graph. The full report also includes the shared cell and feature selections and feature-scaling result.

Export the same report when it needs to travel with an analysis. to_markdown() is what notebook display uses. Inspect a short preview before writing or sending the complete string elsewhere:

# Export the displayed lineage report as Markdown.
lineage_markdown = lineage.to_markdown()

# Preview the opening lines of the exported report.
lineage_markdown.splitlines()[:12]

['```mermaid',
 'flowchart LR',
 '    artifact0["RNA / metadata_snapshot | snapshot_run_metadata | 7340c6fca965"]',
 '    artifact1["datastore / cell_selection | snapshot_pipeline_input_selection | 829e00dc1708"]',
 '    artifact2["RNA / feature_summary | summarize_rna_features | cab031960e6e"]',
 '    artifact3["RNA / feature_selection | select_hvgs | 561ec515123d"]',
 '    artifact4["RNA / normalized | run_normalization | 525af1d08271"]',
 '    artifact5["RNA / feature_scaling | calculate_feature_scaling | 739b82327259"]',
 '    artifact6["RNA / reduction | run_pca | 532aa2d022fe"]',
 '    artifact7["RNA / ann_index | build_ann_index | e7f9b6ebbb42"]',
 '    artifact8["RNA / neighbors | query_neighbors | 91937d072cd9"]',
 '    artifact9["RNA / connectivity_map | build_connectivity_map | f2ac2849e34c | outputs: k21 graph"]']

lineage.to_mermaid() returns only the diagram source when a tooling pipeline needs that form alone.

Find and inspect saved results

The completed run retains every immutable reference needed to keep side comparisons separate.

# Inspect the saved chain from normalization to connectivity.
{
    "normalization": normalized,
    "PCA": pca,
    "neighbor index": ann,
    "neighbors": neighbors_k21,
    "graph": graph_k21,
}

{'normalization': ArtifactRef(assay='RNA', kind='normalized', artifact_id='525af1d08271...'),
 'PCA': ArtifactRef(assay='RNA', kind='reduction', artifact_id='532aa2d022fe...'),
 'neighbor index': ArtifactRef(assay='RNA', kind='ann_index', artifact_id='e7f9b6ebbb42...'),
 'neighbors': ArtifactRef(assay='RNA', kind='neighbors', artifact_id='91937d072cd9...'),
 'graph': ArtifactRef(assay='RNA', kind='connectivity_map', artifact_id='f2ac2849e34c...')}

For how results record their inputs, see Why CyteArc records provenance <https://docs.nygen.io/CyteArc/concepts/provenance>. The artifact catalog can also find results by their exact inputs. It returns every match instead of guessing which result you intended. Unpacking into [reopened_graph] checks that the search found exactly one graph; zero or multiple matches raise an error:

# Find the exact saved graph from its neighbor input.
[reopened_graph] = ds.artifacts.list(
    from_assay="RNA",
    kind="connectivity_map",
    operation="build_connectivity_map",
    inputs={"neighbors": neighbors_k21},
    complete_only=True,
)

# Check that the catalog returned the baseline graph.
assert reopened_graph == graph_k21

# Inspect the saved graph's operation, parameters, and inputs.
status = ds.artifacts.inspect(reopened_graph)

# Inspect the graph operation and the exact inputs it records.
{
    "operation": status.operation,
    "parameters": status.parameters,
    "inputs": status.inputs,
    "complete": status.complete,
}

{'operation': 'build_connectivity_map',
 'parameters': {'local_connectivity': 1.0, 'bandwidth': 1.5},
 'inputs': {'neighbors': {'type': 'artifact',
   'scope': 'assay',
   'kind': 'neighbors',
   'artifact_id': '91937d072cd999181452a281debaa17cb51adcae39ccda32c7defb6d3271523e',
   'assay': 'RNA'}},
 'complete': True}

Recompute a matching result

invalidate_cache=True skips reuse <https://docs.nygen.io/CyteArc/reference/glossary#term-reuse> even when the parameters match. Previously completed artifacts remain on disk. The new reference has a different id and path. The operation and parameters stay the same.

CyteArc also stops reusing a result on its own when a release changes what the operation computes. Such a release gives the operation a new revision, and an earlier result of that operation is computed again, with one log line that names the result it replaced and what changed. The earlier result remains on disk, and lineage reports mark it as stale. See Operation revisions <https://docs.nygen.io/CyteArc/developers/operation-revisions>.

For normalization, the call below would write a fresh normalized artifact while retaining the exact immutable cell_selection and feature_selection inputs. It is not executed here because a throwaway duplicate adds no evidence to the lineage figure.

# Force normalization to create a new result with the same inputs.
forced = ds.features.normalize(cell_selection, hvg_ref, invalidate_cache=True)

# Check that forcing recomputation produced a distinct result.
forced != normalized