Source: https://docs.nygen.io/CyteArc/tutorials/dataset-merging/ Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe Structured page: https://docs.nygen.io/CyteArc/tutorials.dataset-merging.json Integrating datasets by merging To analyse cells from several datasets together, first place their compatible assays in one datastore. DataStoreMerge aligns features, carries metadata, and records which source each cell came from. The expression values are unchanged by the merge. Batch correction, when appropriate, is a separate step; this page builds and inspects the uncorrected dataset. Load compatible source stores The control and interferon beta stimulated Kang PBMC stores use the same RNA feature space. These prepared stores contain cells with author-provided cell-type labels. Their I columns mark the cells that passed quality control. # Import tools for creating and inspecting a merged count store. from pathlib import Path from tempfile import TemporaryDirectory import pandas as pd import cytearc # Keep routine progress messages out of the teaching output. cytearc.configure_output(level="ERROR", progress=False) # Connect to the repository of prepared documentation datasets. repository = cytearc.cytebase.connect("cytearc_docs") # Download the control PBMC store. ctrl_path = repository.download_dataset( name="kang_15K_pbmc_rnaseq", destination="cytearc_datasets", zarr=True ) Download the stimulated sample from the same repository. # Download the interferon-stimulated PBMC store. stim_path = repository.download_dataset( name="kang_14K_ifnb-pbmc_rnaseq", destination="cytearc_datasets", zarr=True ) Open both source stores before checking their axes. # Open the control sample. ds_ctrl = cytearc.DataStore(f"{ctrl_path}/data.zarr", nthreads=4) # Open the stimulated sample. ds_stim = cytearc.DataStore(f"{stim_path}/data.zarr", nthreads=4) Confirm assay type, cell counts, and feature counts before merging. DataStoreMerge validates the feature axes and requires every source to declare the same assay type, which the merged assay keeps; matching gene symbols alone do not establish compatible genome builds or quantification conventions. # Compare assay types and cell and feature counts before merging. pd.DataFrame( [ { "source": label, "assay": type(store.RNA).__name__, "cells": store.cells.N, "active cells": int(store.cells.fetch_all("I").sum()), "features": store.RNA.feats.N, } for label, store in (("ctrl", ds_ctrl), ("stim", ds_stim)) ] ) source assay cells active cells features 0 ctrl RNAassay 8487 8486 35635 1 stim RNAassay 10111 10111 35635 Merge counts and metadata names supplies the source labels, source_column names their metadata column, and prepend_text keeps imported metadata names distinct from columns authored in the merged store. reset_cell_filter=False copies each source’s I values into the merged store. Cells marked False are still present in the counts and metadata, but remain outside the active selection. A merge never replaces a store that a DataStore has opened, even with overwrite=True, so this example writes into a new temporary folder each time it runs. # Keep the merged example in a temporary folder. merge_directory = TemporaryDirectory() # Choose a new path for the merged counts. merged_path = Path(merge_directory.name) / "kang_dataset_merging.zarr" # Write the prepared counts and metadata to the new store. cytearc.DataStoreMerge( datasets=[ds_ctrl, ds_stim], zarr_path=str(merged_path), names=["ctrl", "stim"], assays=["RNA"], prepend_text="orig", reset_cell_filter=False, source_column="sample_id", ).dump() # Open the completed merge to inspect its cells and features. merged = cytearc.DataStore(str(merged_path), nthreads=4) # Check the merged cell and feature dimensions. merged DataStore has 18597 (18598) cells with 1 assays: RNA Cell metadata: 'I', 'ids', 'names', 'RNA_I', 'RNA_nCounts', 'RNA_nFeatures', 'RNA_percentMito', 'RNA_percentRibo', 'orig_RNA_UMAP1', 'orig_RNA_UMAP2', 'orig_RNA_clusters', 'orig_RNA_leiden_cluster', 'orig_cluster_labels', 'sample_id' RNA assay has 35635 features and following metadata: 'I', 'ids', 'names', 'dropOuts', 'nCells' sample_id records the source label. Columns imported from the sources keep the orig_ prefix so their origin remains explicit. RNA_I marks the cells that the merged RNA assay measured; every cell here comes from an RNA source, so it is True for all of them. When a source lacks an assay, or measured only some of its cells with it, the merged _I column keeps that per-cell membership. Operations that read that assay’s values then refuse the cells it did not measure, whose counts are zero-filled, and merged.select_measured_cells("", cell_selection=...) keeps the measured ones; here it returns the selection unchanged, because RNA measured every cell. The merged active population contains labelled cells from both sources. # Read source labels and imported cell types for active cells. merged_labels = merged.cells.to_pandas_dataframe( ["sample_id", "orig_cluster_labels"], key="I" ) # Count active cells and distinct imported cell types from each source. merged_labels.groupby("sample_id")["orig_cluster_labels"].agg( cells="count", cell_types="nunique" ) cells cell_types sample_id ctrl 8486 13 stim 10111 13 Inspect a prepared joint analysis The merge is complete. To see what these datasets look like together, open the catalog’s prepared merged store. It uses the same merge recipe and already contains PCA, clustering, and UMAP computed without batch correction. This is a separate store from merged, so the following plots do not run an analysis on the store you just created. The saved example run is named docs_default. # Download the separate, pre-analyzed joint example. prepared_path = repository.download_dataset( name="kang_29K_ctrl-ifnb_pbmc_rnaseq", destination="cytearc_datasets", zarr=True ) # Open the datastore for the following analysis. ds = cytearc.DataStore(f"{prepared_path}/data.zarr", nthreads=4) # Open the prepared baseline for the comparisons below. baseline = ds.pipeline.open(label="docs_default") # List the results available in the separate prepared analysis. sorted(baseline) ['analysis_cell_selection', 'ann_index', 'cluster_selection', 'clusters', 'connectivity_map', 'embedding_initialization', 'feature_universe', 'highly_variable_features', 'input_cell_selection', 'leiden_1.0', 'neighbors', 'normalized', 'pca', 'umap'] The saved run links each output name to its stored artifact . Its cell selection and metadata stay fixed even if the datastore’s live metadata changes later. Compare the source labels, the authors’ cell types, and the newly computed clusters on the same UMAP. Agreement between the latter two is useful context when interpreting source separation. # Compare source labels, imported cell types, and computed clusters. ds.plots.embedding( layout=baseline["umap"], color_by=["sample_id", "orig_cluster_labels", baseline["clusters"]], n_columns=3, ) Three UMAP panels show merged control and stimulated samples, imported cell-type labels, and newly computed clusters on the same coordinates. Several cell-type regions separate by sample. Image: https://docs.nygen.io/CyteArc/build/d5803a1ee418a0ad6ccc0c6e490c1c57.png A table of proportions shows whether each Leiden cluster contains cells from both sources. # Compare source proportions within each computed cluster. pd.crosstab( baseline.cells.fetch("clusters"), baseline.cells.fetch("sample_id"), rownames=["cluster"], colnames=["source"], normalize="index", ).round(3) source ctrl stim cluster 1 0.004 0.996 2 0.006 0.994 3 0.008 0.992 4 0.443 0.557 5 0.002 0.998 6 0.002 0.998 7 0.974 0.026 8 0.010 0.990 9 0.429 0.571 10 0.355 0.645 11 0.005 0.995 12 0.512 0.488 13 0.571 0.429 14 0.997 0.003 15 0.993 0.007 16 0.416 0.584 17 0.988 0.012 18 0.998 0.002 19 0.971 0.029 20 0.994 0.006 The stimulated sample received interferon beta, and PBMC cell types do not all respond identically to that treatment. Source-associated structure can therefore include biological response as well as technical variation. The next page, Correcting batch effects with Harmony , uses a different dataset with measured sequencing batches to compare an uncorrected analysis with Harmony. It also introduces metrics for batch mixing. Keep uncorrected counts for condition-level differential expression.