Source: https://docs.nygen.io/CyteArc/api/graph_construction.html Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe Graph construction API reference ******************************** These "DataStore" methods build a neighbourhood graph as explicit, provenance-backed stages. Each builder returns an "ArtifactRef"; pass that exact ref to the next stage. Prefer "ds.pipeline.run()" for the fixed rich RNA recipe. Use the individual methods for a partial workflow, an alternative branch, or parameters intentionally kept out of the pipeline surface. See Building neighbourhood graphs step by step for an executable walkthrough. Explicit chain ============== Capture the live Boolean cell column once at the workflow boundary. The returned selection is immutable; later live changes do not alter it. cells = ds.snapshot_cell_selection("I") features = ds.features.hvgs(cells, top_n=1000, show_plot=False) normalized = ds.features.normalize(cells, features) pca = ds.reduction.pca(normalized, dims=21) ann = ds.graph.ann_index(pca) neighbors = ds.graph.neighbors(ann, k=11) graph = ds.graph.connectivity(neighbors) initialization = ds.embeddings.initialization(pca) The full stage order is: 1. "snapshot_cell_selection()" and an exact feature-selection ref 2. "normalize()" 3. "pca()", "lsi()", or "custom()", or none 4. Optional "harmony()" 5. "ann_index()" 6. "neighbors()" 7. "connectivity()" Without a reduction, "graph.ann_index(normalized)" builds the graph on the normalized artifact itself: its float32 values of the selected features are the coordinates, one per feature, as in a pipeline run with "pca_dims=0". Every later stage, including the embedding initialization and an SNN integration, accepts such a graph. Harmony corrects reduced coordinates only, and WNN integration, mapping references, and doublet scoring need them too, so each rejects normalized coordinates with an error that names a reduction such as "reduction.pca". Because normalized values can be coordinates, "features.normalize" writes them with a checked writer: a value that is NaN or infinite once rounded to float32, such as one from a custom normalizer or the "log1p" of a negative count, raises "NonFiniteArtifactError", a "ValueError" that names "features.normalize", and no complete artifact is saved. The hnswlib index holds every cell in memory, about "4 * dims + 8 * ann_m + 130" bytes per cell on x86-64 Linux ("cytearc.neighbors.index.ann_index_peak_bytes"), where "dims" is the number of coordinate dimensions: the PCA dimensions, or the selected features of normalized coordinates. "graph.ann_index" holds it while it adds the coordinate blocks and while it saves it, and "graph.neighbors" while it loads and queries it, beside the indices and distances of every cell’s neighbours, 8 bytes per neighbour. Each compares what it holds with the datastore’s memory budget before it creates or loads the index and raises "MemoryError" over it, naming the bytes, the cells, the dimensions, and the limit. The index moves between the store and memory through a temporary file of about "4 * dims + 8 * ann_m + 20" bytes per cell ("cytearc.neighbors.index.ann_index_file_bytes") in the system’s temporary directory, which needs that much free space. "initialization()" depends on explicit reduction, Harmony, imported, or normalized coordinates. It is needed for UMAP unless an initialization array is supplied, but is not part of connectivity construction. "graph.neighbors" reads the coordinate ref named by the ANN artifact. Its optional "coordinates=" argument is only an equality check; it cannot redirect an index to different coordinates. Downstream consumers ==================== UMAP, t-SNE, clustering, diffusion, trajectories, graph metrics, and graph-backed plots require an exact graph or neighbour artifact. Analytical producers return artifacts and do not add live metadata fields: umap = ds.embeddings.umap( graph, initialization, ) leiden = ds.clusters.leiden( graph, resolution=0.5, ) ds.plots.embedding(layout=umap, color_by=leiden) The scientific cell selection comes from graph lineage. Load a payload from its exact ref, or pass the ref to another consumer. Graph consumers do not accept an independent feature selection. Named lineage edges identify the normalized feature selections used to construct a native graph. Imported-coordinate graphs have no such selection; integrated graphs preserve the ordered projections of their explicit sources. Methods ======= +------------+--------------------------------------------------------------------------------------------+ | "cytearc.D | Capture a live boolean cell column as an immutable selection. | | ataStore.s | | | napshot_ce | | | ll_selecti | | | on" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Normalize explicit immutable cell and feature selections. | | atastore.n | | | amespaces. | | | FeaturesAc | | | cessor.nor | | | malize" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Fit or reuse PCA for a normalized artifact. | | atastore.n | | | amespaces. | | | ReductionA | | | ccessor.pc | | | a" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Fit or reuse latent semantic indexing for normalized data. | | atastore.n | | | amespaces. | | | ReductionA | | | ccessor.ls | | | i" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Register custom feature loadings as a reusable reduction. | | atastore.n | | | amespaces. | | | ReductionA | | | ccessor.cu | | | stom" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Snapshot live batch columns, then fit or reuse Harmony correction. | | atastore.n | | | amespaces. | | | ReductionA | | | ccessor.ha | | | rmony" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Build or reuse an approximate nearest-neighbor index. | | atastore.n | | | amespaces. | | | GraphAcces | | | sor.ann_in | | | dex" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Query an ANN artifact and persist compact neighbor matrices. | | atastore.n | | | amespaces. | | | GraphAcces | | | sor.neighb | | | ors" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Convert persisted neighbors into a weighted connectivity graph. | | atastore.n | | | amespaces. | | | GraphAcces | | | sor.connec | | | tivity" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Build or reuse K-means initialization for explicit coordinates. | | atastore.n | | | amespaces. | | | Embeddings | | | Accessor.i | | | nitializat | | | ion" | | +------------+--------------------------------------------------------------------------------------------+ | "cytearc.d | Load the cell neighbourhood as a scipy sparse matrix. | | atastore.n | | | amespaces. | | | GraphAcces | | | sor.load" | | +------------+--------------------------------------------------------------------------------------------+ DataStore.snapshot_cell_selection(cell_key='I') Capture a live boolean cell column as an immutable selection. The returned reference is the explicit starting input for atomic graph construction methods such as "normalize()". A complete matching snapshot is reused when possible. Parameters: **cell_key** ("str") – Boolean cell metadata column to capture. Return type: "ArtifactRef" Returns: A complete datastore-scoped cell-selection artifact. FeaturesAccessor.normalize(cell_selection, features, *, log_transform=None, renormalize_subset=None, invalidate_cache=False) Normalize explicit immutable cell and feature selections. Raises: **UnmeasuredCellsError** – If the assay did not measure a selected cell. Return type: "ArtifactRef" ReductionAccessor.pca(normalized, *, dims=21, pca_cell_selection=None, feat_scaling=True, batch_size=None, local_cache='auto', show_elbow_plot=False, invalidate_cache=False) Fit or reuse PCA for a normalized artifact. Parameters: * **normalized** ("ArtifactRef") – Normalized artifact to reduce. * **dims** ("int") – Requested number of principal components. (Default: 21) * **pca_cell_selection** ("ArtifactRef" | "None") – Optional stored cell-selection artifact used to fit PCA while projecting every cell in "normalized". * **feat_scaling** ("bool") – Whether to standardize features before fitting PCA. * **batch_size** ("int" | "None") – Number of selected cells processed per block. When omitted, whole stored row bands are combined as needed to fit at least "dims + 1" rows. An explicit smaller value is expanded to that aligned minimum with a warning. Several blocks narrower than the selected features, or more than 4096 features, are fitted with IncrementalPCA, whose result depends on the block size, so the block size then joins the artifact identity. * **local_cache** ("bool" | "str") – Local staging policy for normalized data on remote stores. * **show_elbow_plot** ("bool") – Whether to display explained variance after a new PCA fit. * **invalidate_cache** ("bool") – Force a new reduction artifact. Return type: "ArtifactRef" Returns: Reference to the PCA reduction artifact. ReductionAccessor.lsi(normalized, *, dims=11, skip_first=True, rand_state=4466, solver='streaming', n_iter=5, n_oversamples=10, batch_size=None, local_cache='auto', invalidate_cache=False) Fit or reuse latent semantic indexing for normalized data. Parameters: * **normalized** ("ArtifactRef") – Normalized artifact to reduce. * **dims** ("int") – Requested number of retained LSI dimensions. * **skip_first** ("bool") – Whether to omit the first singular component. * **rand_state** ("int") – Non-negative integer seed used by the randomized decomposition. * **solver** ("Literal"["'streaming'", "'materialized'"]) – Memory-bounded streaming solver or materialized compatibility solver. The materialized solver holds the whole matrix and raises MemoryError before reading it when that exceeds the memory budget. * **n_iter** ("int") – Power iterations used by randomized LSI. * **n_oversamples** ("int") – Extra random vectors used to stabilize the fitted singular subspace. * **batch_size** ("int" | "None") – Number of selected cells processed per block. * **local_cache** ("bool" | "str") – Local staging policy for normalized data on remote stores. * **invalidate_cache** ("bool") – Force a new reduction artifact. Return type: "ArtifactRef" Returns: Reference to the LSI reduction artifact. ReductionAccessor.custom(loadings, normalized, *, batch_size=None, local_cache='auto', invalidate_cache=False) Register custom feature loadings as a reusable reduction. Parameters: * **loadings** ("ndarray") – Two-dimensional feature-by- dimension matrix of finite real loadings. Its row count must match the normalized feature selection. * **normalized** ("ArtifactRef") – Normalized artifact associated with the loadings. * **batch_size** ("int" | "None") – Number of selected cells processed per block. * **local_cache** ("bool" | "str") – Local staging policy for normalized data on remote stores. * **invalidate_cache** ("bool") – Force a new reduction artifact. Return type: "ArtifactRef" Returns: Reference to the custom reduction artifact. ReductionAccessor.harmony(reduction, batch_columns, *, harmony_params=None, batch_size=None, invalidate_cache=False) Snapshot live batch columns, then fit or reuse Harmony correction. Raises: **MemoryError** – If the fit exceeds the memory budget. Return type: "ArtifactRef" GraphAccessor.ann_index(coordinates, *, ann_metric='l2', ann_efc=50, ann_ef=50, ann_m=48, ann_parallel=False, rand_state=4466, batch_size=None, invalidate_cache=False) Build or reuse an approximate nearest-neighbor index. Raises: **MemoryError** – If the index exceeds the memory budget. Return type: "ArtifactRef" GraphAccessor.neighbors(ann_index, *, coordinates=None, k=11, batch_size=None, invalidate_cache=False) Query an ANN artifact and persist compact neighbor matrices. Raises: **MemoryError** – If the query exceeds the memory budget. Return type: "ArtifactRef" GraphAccessor.connectivity(neighbors, *, local_connectivity=1.0, bandwidth=1.5, invalidate_cache=False) Convert persisted neighbors into a weighted connectivity graph. Parameters: * **neighbors** ("ArtifactRef") – Neighbors artifact. * **local_connectivity** ("float") – UMAP-style local- connectivity adjustment. * **bandwidth** ("float") – Distance-kernel bandwidth multiplier. * **invalidate_cache** ("bool") – Force a new connectivity artifact. Return type: "ArtifactRef" Returns: Reference to the connectivity-map artifact. EmbeddingsAccessor.initialization(coordinates, *, n_centroids=1000, rand_state=4466, batch_size=None, kmeans_sampling=0.1, kmeans_batch_size=10000, invalidate_cache=False) Build or reuse K-means initialization for explicit coordinates. Pass the returned reference explicitly to an embedding operation. Parameters: * **coordinates** ("ArtifactRef") – Coordinate artifact to cluster, such as a reduction. * **n_centroids** ("int") – Requested number of K-means centroids. * **rand_state** ("int") – K-means random seed. * **batch_size** ("int" | "None") – Number of cells processed per block. * **kmeans_sampling** ("float") – Fraction of cells considered during centroid seeding. * **kmeans_batch_size** ("int") – Number of cells per internal K-means update. * **invalidate_cache** ("bool") – Force a new initialization artifact. Return type: "ArtifactRef" Returns: Reference to the embedding-initialization artifact. Raises: **MemoryError** – If the fit exceeds the memory budget. GraphAccessor.load(graph, *, symmetric=None, upper_only=None, use_k=None) Load the cell neighbourhood as a scipy sparse matrix. Parameters: * **graph** ("ArtifactRef") – Connectivity-map or integrated- graph artifact. * **symmetric** ("bool" | "None") – If True, makes the graph symmetric by adding it to its transpose. * **upper_only** ("bool" | "None") – If True, then only the values from upper triangular of the matrix are returned. This is only used when symmetric is True. * **use_k** ("int" | "None") – Number of top k-nearest neighbours to keep in the graph. It must be an integer from 1 to the graph’s k. By default, all neighbours are used. (Default value: None) Return type: "csr_matrix" Returns: A scipy sparse matrix representing cell neighbourhood graph.