Source: https://docs.nygen.io/CyteArc/api/import_export.html Revision: fccee7b6e0c167e47ce605c8d292fba5b567ebfe Import and export API reference ******************************* Readers ======= class cytearc.readers.CrReader(grp_names) A class to read in CellRanger (Cr) data. Parameters: **grp_names** ("dict"["str", "Any"]) – A dictionary that specifies where to find the matrix, features and barcodes. autoNames Specifies if the data is from RNA or ATAC sequencing. grpNames A dictionary that specifies where to find the matrix, features and barcodes. nFeatures Number of features in dataset. nCells Number of cells in dataset. assayFeats A DataFrame with information about the features in the assay. cell_names() Returns a list of names of the cells in the dataset. Return type: "list"["str"] abstractmethod consume(batch_size, lines_in_mem) Yield CSR matrix chunks of cell rows. Parameters: * **batch_size** ("int") – Number of cells per yielded chunk. * **lines_in_mem** ("int") – MTX lines buffered in memory (Matrix Market readers only). Yields: scipy.sparse.coo_matrix chunks. abstractmethod count_value_ranges(maxBytes, featureGroups=None) Return the range of the canonical values of each group of features. The ranges cover the selected cells, with duplicate coordinates summed as the import stores them. Pass a range to "count_storage_dtype()" to resolve the storage dtype of its group’s counts. Parameters: * **maxBytes** ("int") – Memory available to the scan. * **featureGroups** ("ndarray" | "None") – Group, numbered from zero, of each feature, such as the assay that stores it. None puts every feature in one group. Return type: "list"["CountValueRange"] feature_ids(assay=None) Returns a list of feature IDs in a specified assay. Parameters: **assay** ("str" | "None") – Select which assay to retrieve feature IDs from. Return type: "list"["str"] feature_names(assay=None) Returns a list of features in the dataset. Parameters: **assay** ("str" | "None") – Select which assay to retrieve features from. Return type: "list"["str"] feature_types() Returns a list of feature types in the dataset. Return type: "list"["str"] get_cell_columns() Yield optional cell metadata columns supplied by the reader. Return type: "Iterator"["tuple"["str", "ndarray"]] get_feature_columns() Yield source feature types and optional feature metadata. Return type: "Iterator"["tuple"["str", "ndarray"]] abstract property matrix_dtype: dtype[Any] Return the dtype in which "consume" yields count values. max_window_nnz(window_rows) Bound nnz in any source row window. Return type: "int" producer_staging_bytes(batch_size, lines_in_mem) Return source-reader bytes retained while a matrix batch is yielded. Return type: "int" reclassify_features(indexes, feature_type, *, require_previous='Antibody Capture') Reclassify global feature rows before a writer captures the schema. Return type: "None" rename_assays(name_map) Renames specified assays in the Reader. Parameters: **name_map** ("dict"["str", "str"]) – A Dictionary containing current name as key and new name as value. Return type: "None" class cytearc.readers.CrH5Reader(h5_fn, is_filtered=True, filtering_cutoff=500) A class to read in CellRanger (Cr) data, in the form of an H5 file. Subclass of CrReader. Parameters: **h5_fn** ("str") – File name for the h5 file. autoNames Specifies if the data is from RNA or ATAC sequencing. grpNames A dictionary that specifies where to find the matrix, features and barcodes. nFeatures Number of features in dataset. nCells Number of cells in dataset. assayFeats A DataFrame with information about the features in the assay. h5obj A File object from the h5py package. grp Current active group in the hierarchy. cell_names() Returns a list of names of the cells in the dataset. Return type: "list"["str"] close() Closes file connection. Return type: "None" consume(batch_size, lines_in_mem=None) Yield CSR chunks from the Cell Ranger H5 matrix. Parameters: * **batch_size** ("int") – Number of cells per chunk. * **lines_in_mem** ("int" | "None") – Unused; kept for CrReader API compatibility. Return type: "Generator"["coo_matrix", "None", "None"] count_value_ranges(maxBytes, featureGroups=None) Return the range of the canonical values of each group of features. Each barcode is one compressed vector of the matrix. Selected barcodes whose values are contiguous in the matrix are scanned together, as "consume" reads them; the barcodes between them hold no values. Raises: * **MemoryError** – If one barcode holds more values than fit in "maxBytes". * **ValueError** – If a value is NaN or infinite. Return type: "list"["CountValueRange"] get_feature_columns() Yield source feature types and optional feature metadata. Return type: "Iterator"["tuple"["str", "ndarray"]] property matrix_dtype: dtype[Any] Return the stored dtype in native byte order, with float16 as float32. SciPy sparse matrices hold neither float16 nor another byte order, so "consume" reads the values in this dtype. max_window_nnz(window_rows) Return the largest selected-cell row-window nnz. Return type: "int" producer_staging_bytes(batch_size, lines_in_mem) Count row-pointer arrays created while one H5 batch is produced. Return type: "int" class cytearc.readers.CrDirReader(loc, mtx_separator=' ', index_offset=-1, is_filtered=True, filtering_cutoff=500) Bases: "MtxReader" Read a Cell Ranger output directory in Matrix Market format. The directory must contain exactly one complete matrix, feature, and barcode triplet, such as "matrix.mtx.gz", "features.tsv.gz", and "barcodes.tsv.gz". Use "inspect_mtx()" and "MtxReader" to choose among several triplets. All reading and streaming methods come from "MtxReader". Parameters: * **loc** ("str") – Path to the directory that contains the Cell Ranger output. * **mtx_separator** ("str") – Column delimiter in the MTX file. * **index_offset** ("int") – Value added to each MTX coordinate. The default turns one-based coordinates into zero-based indexes. * **is_filtered** ("bool") – Whether the barcodes are already filtered. When False, only barcodes whose total count exceeds "filtering_cutoff" are kept. * **filtering_cutoff** ("int") – Total count that a barcode must exceed to be kept when "is_filtered" is False. cell_names() Returns a list of names of the cells in the dataset. Return type: "list"["str"] consume(batch_size, lines_in_mem=100000) Yield chunks of cell rows from the Matrix Market file. Parameters: * **batch_size** ("int") – Number of cells per yielded chunk. * **lines_in_mem** ("int") – Number of Matrix Market lines parsed at a time. Yields: scipy.sparse.coo_matrix chunks in "matrix_dtype". count_value_ranges(maxBytes, featureGroups=None) Return the range of the counts of each group of features. Matrix Market counts are non-negative integers, and construction already scanned them with duplicate coordinates summed and kept the largest count of each feature, so this does not use "maxBytes". When the reader dropped cells, several groups need one more read of the file to find the largest counts of the kept cells. Return type: "list"["CountValueRange"] feature_ids(assay=None) Returns a list of feature IDs in a specified assay. Parameters: **assay** ("str" | "None") – Select which assay to retrieve feature IDs from. Return type: "list"["str"] feature_names(assay=None) Returns a list of features in the dataset. Parameters: **assay** ("str" | "None") – Select which assay to retrieve features from. Return type: "list"["str"] feature_types() Returns a list of feature types in the dataset. Return type: "list"["str"] get_cell_columns() Yield optional cell metadata columns supplied by the reader. Return type: "Iterator"["tuple"["str", "ndarray"]] get_feature_columns() Yield source feature types and optional feature metadata. Return type: "Iterator"["tuple"["str", "ndarray"]] property matrix_dtype: dtype[Any] Return the dtype in which "consume" yields count values. max_window_nnz(window_rows) Bound nnz in any source row window. Return type: "int" producer_staging_bytes(batch_size, lines_in_mem) Return source-reader bytes retained while a matrix batch is yielded. Return type: "int" reclassify_features(indexes, feature_type, *, require_previous='Antibody Capture') Reclassify global feature rows before a writer captures the schema. Return type: "None" rename_assays(name_map) Renames specified assays in the Reader. Parameters: **name_map** ("dict"["str", "str"]) – A Dictionary containing current name as key and new name as value. Return type: "None" class cytearc.readers.MtxReader(matrix_path, feature_path=None, cell_path=None, *, feature_reference_path=None, cell_id_key=None, mtx_separator=' ', index_offset=-1, is_filtered=True, filtering_cutoff=500, temp_dir=None) Read a selected Matrix Market triplet as cell-row sparse batches. Construction reads the matrix once to validate it and to find its coordinate order, the entries of each cell, and the largest count of each kept cell and each feature, with duplicate coordinates summed. Batches hold the counts in the narrowest unsigned dtype that holds the largest kept count. consume(batch_size, lines_in_mem=100000) Yield chunks of cell rows from the Matrix Market file. Parameters: * **batch_size** ("int") – Number of cells per yielded chunk. * **lines_in_mem** ("int") – Number of Matrix Market lines parsed at a time. Yields: scipy.sparse.coo_matrix chunks in "matrix_dtype". count_value_ranges(maxBytes, featureGroups=None) Return the range of the counts of each group of features. Matrix Market counts are non-negative integers, and construction already scanned them with duplicate coordinates summed and kept the largest count of each feature, so this does not use "maxBytes". When the reader dropped cells, several groups need one more read of the file to find the largest counts of the kept cells. Return type: "list"["CountValueRange"] get_cell_columns() Yield optional cell metadata columns supplied by the reader. Return type: "Iterator"["tuple"["str", "ndarray"]] get_feature_columns() Yield source feature types and optional feature metadata. Return type: "Iterator"["tuple"["str", "ndarray"]] property matrix_dtype: dtype[Any] Return the dtype in which "consume" yields count values. max_window_nnz(window_rows) Bound nnz in any source row window. Return type: "int" producer_staging_bytes(batch_size, lines_in_mem) Return source-reader bytes retained while a matrix batch is yielded. Return type: "int" class cytearc.readers.H5adReader(h5ad_fn, cell_attrs_key='obs', cell_ids_key='_index', feature_attrs_key='var', feature_ids_key='_index', feature_name_key='gene_short_name', matrix_key='X', obsm_attrs_key='obsm', category_names_key='__categories', embedding_roles=None, cluster_keys=(), *, temp_dir=None) A class to read in data from a H5ad file (h5 file with AnnData information). Parameters: * **h5ad_fn** ("str") – Path to H5AD file * **cell_attrs_key** ("str") – H5 group under which cell attributes are saved.(Default value: ‘obs’) * **feature_attrs_key** ("str") – H5 group under which feature attributes are saved.(Default value: ‘var’) * **cell_ids_key** ("str") – Key in *obs* group that contains unique cell IDs. By default the index will be used. * **feature_ids_key** ("str") – Key in *var* group that contains unique feature IDs. By default the index will be used. * **feature_name_key** ("str") – Key in *var* group that contains feature names. (Default: gene_short_name) * **matrix_key** ("str") – Group where in the sparse matrix resides (default: ‘X’) * **category_names_key** ("str") – Looks up this group and replaces the values in *var* and ‘obs’ child datasets with the corresponding index value within this group. * **temp_dir** ("str" | "Path" | "None") – Parent directory for temporary CSC row storage. None uses the system temporary directory. h5 A File object from the h5py package. matrixKey Group where in the sparse matrix resides (default: ‘X’) cellAttrsKey Group wherein the cell attributes are present featureAttrsKey Group wherein the feature attributes are present groupCodes Used to ensure compatibility with different AnnData versions. nFeatures Number of features in dataset. nCells Number of cells in dataset. cellIdsKey Key in *obs* group that contains unique cell IDs. By default the index will be used. featIdsKey Key in *var* group that contains unique feature IDs. By default the index will be used. featNamesKey Key in *var* group that contains feature names. (Default: gene_short_name) catNamesKey Looks up this group and replaces the values in *var* and ‘obs’ child datasets with the corresponding index value within this group. sourceMatrixDtype dtype in which the source holds the matrix values, in native byte order. float16 is not a count storage dtype, so float16 source values are read as float32. embedding_roles Exact "obsm" keys to import as immutable UMAP or t-SNE artifacts. cluster_keys Exact "obs" keys to import as immutable cluster-label artifacts instead of raw cell metadata. assay_feature_slices(key, name_map=None) Resolve feature ranges and metadata for each assay. Return type: "dict"["str", "_H5adAssayFeatures"] assay_membership() Return the membership column of each assay that the file declares. Return type: "dict"["str", "str"] cell_ids() Returns a list of cell IDs. Return type: "ndarray" consume(batch_size) Returns a generator that yield chunks of data. Return type: "Generator"["coo_matrix", "None", "None"] property consumeDtype: dtype[Any] Return the dtype in which "consume" yields the matrix values. It is "sourceMatrixDtype", except that the converted rows of a CSC integer source yield their duplicate-summed values in int64, or uint64 for unsigned and boolean sources. consume_dataset(batch_size=1000, row_start=0, row_end=None) Returns a generator that yield chunks of data. Return type: "Generator"["coo_matrix", "None", "None"] consume_group(batch_size, row_start=0, row_end=None) Returns a generator that yield chunks of data. Return type: "Generator"["coo_matrix", "None", "None"] consume_row_range(batch_size, row_start, row_end) Yield source batches covering "[row_start, row_end)". Return type: "Generator"["coo_matrix", "None", "None"] count_value_ranges(maxBytes, featureGroups=None) Return the range of the canonical values of each group of features. Duplicate coordinates of a sparse matrix are summed, as the import stores them, so the ranges depend neither on the order of distinct coordinates, nor on orientation or encoding. Pass a range to "count_storage_dtype()" to resolve the storage dtype of its group’s counts. Parameters: * **maxBytes** ("int") – Memory available to the scan. * **featureGroups** ("ndarray" | "None") – Group, numbered from zero, of each feature, such as the assay that stores it. None puts every feature in one group. Raises: * **MemoryError** – If one compressed vector or dense row does not fit in "maxBytes". * **ValueError** – If the matrix holds NaN or an infinite value. Return type: "list"["CountValueRange"] feat_ids() Returns a list of feature IDs. Return type: "ndarray" feat_names() Returns a list of feature names. Return type: "ndarray" feature_types(key) Return decoded feature types from a var column. Return type: "list"["str"] get_cell_columns() Yield raw "obs" metadata, excluding selected cluster artifacts. Missing categorical and text values are "None"; missing numeric values are NaN. Return type: "Generator"["tuple"["str", "ndarray"], "None", "None"] get_feat_columns() Creates a Generator that yields the feature columns. Return type: "Generator"["tuple"["str", "ndarray"], "None", "None"] materialize_csc(maxBytes=67108864) Convert CSC into temporary row storage in bounded blocks. The rows hold the source values with duplicate coordinates summed. Floating-point values keep "sourceMatrixDtype"; integers are held in the 64-bit integer dtype in which their duplicates are summed, so a sum past the range of a narrow source dtype is kept, as a CSR import keeps it. Return type: "None" materialized_csr_bytes() Return bytes retained by the materialized CSC-to-CSR conversion. Return type: "int" max_batch_nnz(batch_size) Return the largest contiguous row-window nnz without loading values. Return type: "int" producer_batch_staging_bytes(batch_size) Bound sparse row pointers retained while one batch is produced. Return type: "int" class cytearc.readers.CSVReader(csv_fn, has_header=True, id_column=None, rows_are_cells=True, sep=',', skip_rows=0, skip_cols=None, cell_data_cols=None, batch_size=10000, pandas_kwargs=None) A class to read in data from a CSV file. Construction checks that every row has as many fields as the header, or as the first row of a file without one, and raises ValueError otherwise: pandas reads in chunks, pads a short row with missing values, and drops the extra fields of a row that starts a chunk. Python’s csv module counts the fields with "sep" and the "quotechar", "quoting", "doublequote", "escapechar", and "skipinitialspace" settings that pandas splits the rows with. Parameters: * **csv_fn** ("str") – Path to the CSV file * **has_header** ("bool") – Does the CSV file has a header. (Default value: True) * **id_column** ("int" | "None") – The column number which contains row name. (Default value: None) * **rows_are_cells** ("bool") – If True then each row represents a cell and hence each column is a feature. If False then each row is feature and each column in a cell * **sep** ("str") – The column separator in the CSV file, one character. (Default value: ‘,’) * **skip_rows** ("int") – Number of rows to skip from the top of the file. (Default value: 0) * **skip_cols** ("list"["str"] | "None") – Names of columns to skip. Must be provided as a list even if just one column. (Default value: None) * **cell_data_cols** ("list"["str"] | "None") – Names of columns to include in cell metadata rather than count matrix. Must be provided as a list even if just one column. (Default value: None) * **batch_size** ("int") – Number of lines to read at a time. Decrease this value if you have too many columns. (Default value: 50,000) * **pandas_kwargs** ("dict"["str", "Any"] | "None") – A dictionary of keyword arguments to be passed to Pandas read_csv function. It cannot set "comment", "dialect", or "lineterminator". nFeatures Number of features in dataset. nCells Number of cells in dataset. countDtype Dtype that holds the count values of every row. countRange Range of the count values of every row, which decides their storage dtype. cellDataDtypes Dtype of each "cell_data_cols" column across every row; "object" marks a text column. cell_ids() Returns a list of cell IDs. Return type: "ndarray" consume() Returns a generator that yield chunks of data. Return type: "Generator"["tuple"["ndarray", "ndarray" | "None"], "None", "None"] feature_ids() Returns a list of feature IDs. Return type: "ndarray" H5AD inspection =============== Use this before "H5adReader" when you do not know which matrix and metadata keys an H5AD file uses. "H5adReader" checks the matrix shape against the lengths of "obs" and "var" when it is constructed, and a mismatch raises "ValueError". Overriding "matrix_key", for example with "raw/X", in the constructor or through "H5adReader.from_inspect", therefore also needs the matching "feature_attrs_key", here "raw/var". "inspect_h5ad(path, matrix_key="raw/X")" returns reader arguments that pair them. "H5adReader" decodes AnnData categorical columns and pandas nullable columns, including the nullable string arrays that AnnData writes for dataframe indexes. By default it reads the index that the dataframe’s "_index" attribute names. Missing categorical or object values become "None"; missing numeric values become "NaN". Unsupported group encodings and multi-dimensional columns are skipped with a warning. "H5adToZarr" keeps missing rows under a linked missing mask and stores nullable booleans as booleans. Pass "embedding_roles" and "cluster_keys" to select analytical H5AD values for artifact import. Selected "obsm" arrays and cluster labels are excluded from live metadata and returned as exact refs by "H5adToZarr.dump()". Other supported "obs" columns remain literal metadata. CSC input is converted to temporary row storage on local disk. Pass "temp_dir" to "H5adReader" or "H5adReader.from_inspect" to choose its parent directory; the default uses the system temporary directory, including "TMPDIR". The temporary files are removed when the reader closes. cytearc.inspect_h5ad(h5ad_fn, *, matrix_key=None) Report the matrix and metadata layout of an H5AD file. Parameters: * **h5ad_fn** ("str") – Path to the H5AD file. * **matrix_key** ("str" | "None") – Optional matrix path to force (for example "X" or "raw/X"). When set, that candidate must exist and match obs/var dimensions. Return type: "H5adInspectResult" Returns: Keys, shape, and column names needed to configure "H5adReader". class cytearc.H5adInspectResult(h5adFn, matrixKey, matrixCandidates, matrixEncoding, integerLike, cellAttrsKey, cellIdsKey, featureAttrsKey, featureIdsKey, featureNameKey, categoryNamesKey, assaySplitKey, suggestedAssays, layers, title, description, nCells, nFeatures) class cytearc.H5adImportResult(assayNames, analysisAssay, cellSelection, embeddingArtifacts, clusterArtifacts) Immutable outputs created while importing one H5AD file. assayNames Assay groups written to the destination store. analysisAssay Assay owning explicitly selected analytical outputs. cellSelection Artifact covering the imported cell axis. embeddingArtifacts Embedding artifacts keyed by exact "obsm" key. clusterArtifacts Cluster artifacts keyed by exact "obs" key. Matrix Market inspection ======================== "inspect_mtx" reports every complete matrix, feature, and cell triplet in a supported source. Pass one returned candidate to "MtxReader". cytearc.inspect_mtx(source) Return complete Matrix Market import candidates found in one source. Return type: "tuple"["MtxCandidate", "..."] class cytearc.readers.MtxCandidate(source, matrixPath, featurePath, cellPath, matrixOrientation, nCells, nFeatures, nEntries, archivePath=None, cellMetadataPath=None, featureReferencePath=None, cellIdKeys=(), relatedFiles=()) One complete Matrix Market matrix, feature, and cell selection. Seurat import ============= Import a serialized Seurat object from an ".rds" file. Inspect with "inspect_seurat", open "SeuratReader", then write with "SeuratToZarr". This path does not attach to a live R session and does not read or write ".h5seurat". Sidecar matrices such as BPCells directories and HDF5 files resolve inside "sidecar_root", which defaults to the directory of the ".rds" file. A reader opened from a stream needs an explicit "sidecar_root" before it reads sidecar-backed layers. See Import and export for the worked contract and CyteArc for Seurat users for workflow mapping. class cytearc.SeuratReader(source, *, limits=None, temp_dir=None, assays=None, assay_layers=None, reductions=None, sidecar_root=None, sidecar_path_remaps=None, matrix_limits=SourceLimits(maxFeatures=2147483647, maxCells=2147483647, maxNnz=9223372036854775807, maxBlockBytes=536870912, maxMetadataBytes=268435456, tileCells=1024, compressedChunkNnz=4194304)) cytearc.inspect_seurat(source, *, limits=None, temp_dir=None, assays=None, assay_layers=None, reductions=None, sidecar_root=None, sidecar_path_remaps=None, matrix_limits=SourceLimits(maxFeatures=2147483647, maxCells=2147483647, maxNnz=9223372036854775807, maxBlockBytes=536870912, maxMetadataBytes=268435456, tileCells=1024, compressedChunkNnz=4194304)) Return type: "SeuratInspectResult" class cytearc.SeuratInspectResult(source, sourceDigest, payloadDigest, compression, activeAssay, nCells, assays, reductions, cellMetadata, activeIdentity, notices=()) class cytearc.SeuratImportResult(assayNames, defaultAssay, cellSelection, activeIdentity, reductionArtifacts, notices=()) Result of writing a Seurat object into a CyteArc Zarr store. assayNames Assay groups written to the destination store. defaultAssay Active assay selected from the Seurat object. cellSelection Artifact for the imported cell filter column. activeIdentity Imported Seurat active identity as immutable cluster labels. reductionArtifacts Imported reductions keyed by result name. notices Non-fatal import notices collected from the reader. class cytearc.SeuratToZarr(reader, zarr_loc, workspace=None, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, *, overwrite=False) Convert a serialized Seurat object into a CyteArc Zarr store. Parameters: * **reader** ("SeuratReader") – Open "SeuratReader" for the source ".rds" file. * **zarr_loc** ("str" | "Store") – Destination Zarr path or store. * **workspace** ("str" | "None") – Workspace name in the destination store. None uses the legacy layout without a workspace group. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). When None, auto-detected. * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly for every assay. When None, every assay uses the default policy. An import that does not fit "mem_budget" raises MemoryError before the destination is created, naming the largest smaller policy that fits. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **overwrite** ("bool") – Replace an existing CyteArc store that no "DataStore" has opened. Construction prepares every selected assay’s counts and reads them once. Each assay stores its counts in the dtype that "count_storage_dtype()" resolves from their canonical values, and a count that the stored dtype cannot hold raises instead of wrapping. Raises: **FileExistsError** – If "zarr_loc" is not empty and may not be replaced. dump(batch_size=None) Write assays, RNA "countsT", and importable reductions. Parameters: **batch_size** ("int" | "None") – Number of source cells per batch, at most one destination row band, which is the default. Return type: "SeuratImportResult" Returns: Imported assay names, cell selection, and reduction artifacts. Storage controls ================ Writers, merge, subset, and "DataStore" accept the same optional storage controls. "profile" chooses the physical encoding. "policy" chooses paired count-matrix geometry, and a writer uses it exactly. Without it, the imports write the default geometry, and an import whose counts write or "countsT" transpose does not fit "mem_budget" raises "MemoryError" naming the largest policy, with the default "unitBytes" and "chunkBytes" halved together, that fits. A smaller geometry makes every later "countsT" read slower. Subset, "repack_zarr", merge, and "add_grouped_assay" instead fit the geometry to "mem_budget": they halve the default "unitBytes" and "chunkBytes" together until their writes fit. Writers that choose their source batches (the sparse and Seurat imports, and merge) admit batches of one destination row band, so a geometry never leaves the write narrower batches than its bands. A write that does not fit, with one- row count shards or with its explicit "policy", fails before it creates the destination: writers raise when they are constructed, and merge when it plans. "MtxToZarr" and "CrToZarr" take "lines_in_mem" in the constructor, so the fit reserves the Matrix Market parse buffer that the write uses. A resumed merge keeps the geometry of its completed counts. The geometry never changes the store’s identity. "io" overrides automatic read, compute, and write widths. Unset values stay under automatic planning from "mem_budget" and "nthreads". class cytearc.storage.io_policy.StorageIoPolicy(readWorkers=None, computeWorkers=None, writeWorkers=None) Requested widths for one store or writer. Unset values mean automatic planning. readWorkers Maximum concurrent read units. None uses automatic memory- admitted read width. computeWorkers Maximum concurrent compute workers. None uses the worker budget. writeWorkers Maximum concurrent destination writers. None uses automatic write width. class cytearc.storage.count_matrix.CountMatrixPolicy(unitBytes, chunkBytes) Byte targets that determine paired "counts" / "countsT" geometry. unitBytes Target bytes for one destination shard and one all-cell feature read group. chunkBytes Target bytes for one inner Zarr chunk. property chunksPerShard: int Inner chunks packed into one shard from the byte targets. Count storage dtype ------------------- Every import stores each assay’s counts in a dtype resolved from the values of that assay, whatever the reader and source encoding, and no import takes a count dtype argument. When every canonical (duplicate- summed) value is a non-negative integer, the counts are stored in the narrowest of "uint8", "uint16", "uint32", and "uint64" that holds them. Other counts keep their source dtype in native byte order, with "float16" read as "float32". Assays split from one source matrix (10x HDF5 and Matrix Market feature types, H5AD "assay_split_key") each resolve their own dtype. Each import reads every count once before it creates the destination: "CrReader" subclasses and "H5adReader" report the range of each group of features over the selected cells through "count_value_ranges", "CSVReader" records it as "countRange" in its first pass, and "SparseToZarr" and "SeuratToZarr" scan their sources. Count matrices hold finite values, so imports reject NaN and infinity before they create the destination. Writers cast through a checked cast, so a count that the stored dtype cannot hold raises instead of wrapping. Subset and repack keep the source dtype. Merge stores the common type of the source count dtypes, widened so that features summed by name cannot overflow it, and rejects integer sources without a common integer dtype. Grouped and melded assays keep their "float64" values. cytearc.storage.count_dtype.count_storage_dtype(sourceDtype, valueRange) Return the dtype that stores counts with "valueRange". Parameters: * **sourceDtype** ("Any") – Dtype in which the source holds the values. * **valueRange** ("CountValueRange") – Range of every canonical value of the matrix. Return type: "dtype"["Any"] Returns: The narrowest unsigned dtype that holds every value when all are non-negative integers; otherwise the source dtype in native byte order, with float16 read as float32. class cytearc.utils.count_values.CountValueRange(integral=True, maximum=0) Running summary of canonical count values. integral Whether every value seen is a non-negative integer. maximum The largest value seen while "integral" holds. update(values) Add a block of canonical values, in any shape. A fractional or negative value settles the range; later blocks are only checked for NaN and infinity. Raises: * **TypeError** – If the values are not real numbers. * **ValueError** – If a value is NaN or infinite. Return type: "None" Writers ======= Every writer owns the reserved metadata columns "ids", "names" and "I" and the "__cytearc_missing__" prefix of missing-value masks. Source cell or feature metadata columns with these names are skipped with a warning, so imported identifiers are never replaced. Zarr reads "/" and "\" in a column name as path separators, so writers store a source column whose name contains either one under the name with "_" in their place and log the rename. A source name that is already valid keeps its name; a renamed column whose name is taken gets the first free "_2", "_3", and so on. "SeuratToZarr" raises for reserved names instead of skipping them. The H5AD, CSV, Matrix Market, and Cell Ranger readers reject missing or repeated cell and feature IDs. The cell column "_I" of every assay that an import writes is reserved for the membership of that assay (Assay membership), because a merge reads a column of that name as which cells the assay measured. "H5adToZarr" restores such a column as the membership of the assay only when the file declares it in "uns["cytearc"]["assayMembership"]", which maps each assay to its column, as CyteArc’s exports write it; the declared column must be boolean without missing values, and a declaration that names another column or a column that "obs" lacks raises "ValueError" before the destination is created. Every other source column named for an imported assay, in an H5AD file without the declaration or from "CSVtoZarr", "CrToZarr", or "MtxToZarr", is skipped with a warning, as "I" is. "SeuratToZarr" writes the membership itself and raises for a metadata column named for any imported assay. A column named for an assay that the import does not write, such as "ADT_I" in an RNA-only import, stays an ordinary column. "SparseToZarr" imports no cell metadata. Import writers take their assay type from the assay name. Pass "assay_type" (or "assay_types" on "CrToZarr") to declare a custom- named assay as a preset such as "RNA". A value that is not a preset raises "ValueError"; use "Assay" for a generic assay. "DataStore(assay_types=...)" and "add_melded_assay(assay_type=...)" apply the same rule. The presets and the persisted "assayTypes" attribute are described in Assay types. Writers create a new store and replace only a store that no "DataStore" has opened. "CrToZarr", "MtxToZarr", "H5adToZarr", "SparseToZarr", "CSVtoZarr", "SeuratToZarr", and "SubsetZarr" accept an absent path, an empty directory, or an empty store, and raise "FileExistsError" for a destination that holds any key, such as an earlier store or a ".DS_Store" file. With "overwrite=True" ("overwrite_existing_file=True" for "SubsetZarr") they replace a store whose root group holds no prepared assay, no "matrixSource", and only members that CyteArc writes ("cellData", "matrices", "artifacts", "pipeline", assays, and workspaces of them). A store that a "DataStore" has opened and content that CyteArc did not write, also inside its groups and arrays, are never replaced, and "SubsetZarr" refuses a destination that overlaps a source. A local destination inside a Zarr store, such as "s.zarr/RNA" or "s.zarr/new.zarr", given as a path or a local store, and a destination whose root group is an assay group raise "ValueError". "CrToZarr", "MtxToZarr", "H5adToZarr", and "SeuratToZarr" check their destination before they read the counts, and "SubsetZarr" when it is constructed. Each checks again as it creates the store with Zarr mode ""w-"", which refuses keys that appeared since the check, or with mode ""w"" for the store that "overwrite=True" replaces. A local path is opened as it is written, also when it holds "#", "?", or ";". "subset_assay_zarr" adds one count array to an existing store and replaces nothing: "out_grp" must not exist and must not be, contain, or lie inside "in_grp", and its parent group must not already record the layout of another count matrix. "SubsetZarr", "DataStoreMerge", "mount_datastore", and "repack_store" refuse a source store that holds a pending derived assay, one that "add_grouped_assay" or "add_melded_assay" has not published, and raise "ValueError" naming it before they write anything. Once no process is writing it, remove it with "discard_interrupted_assay" on a "DataStore" of its workspace. cytearc.storage.destinations.check_destination(location, *, overwrite=False, storage_options=None) Raise unless a writer may create its store at "location"; write nothing. Parameters: * **location** ("str" | "Store") – Destination path, URI, or Zarr store. * **overwrite** ("bool") – Accept a CyteArc store that no "DataStore" has opened. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options used to open a URI. Raises: * **ValueError** – If the destination lies inside a Zarr store or is an assay group. * **FileExistsError** – If the destination is not empty and may not be replaced. Return type: "None" cytearc.storage.destinations.create_destination(location, *, overwrite=False, storage_options=None) Check a destination with "check_destination()" and create its store. Parameters: * **location** ("str" | "Store") – Destination path, URI, or Zarr store. * **overwrite** ("bool") – Replace a CyteArc store that no "DataStore" has opened. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options used to open a URI. Return type: "Group" Returns: The new, empty root group, open for writing. cytearc.storage.destinations.refuse_pending_assays(root, *, operation, subject='The source store') Raise unless a source store holds no derived assay left pending. Parameters: * **root** ("Group") – Root group of the source store, as the writer reads it. * **operation** ("str") – What the store cannot be, such as ""subset"" or ""repacked"". * **subject** ("str") – How the message names the store. Raises: **ValueError** – If the store holds a pending derived assay. Return type: "None" class cytearc.writers.CrToZarr(cr, zarr_loc, workspace=None, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, assay_types=None, lines_in_mem=100000, *, overwrite=False) A class for converting data in the Cellranger format to a Zarr hierarchy. Parameters: * **cr** ("CrReader") – A CrReader object, containing the Cellranger data. * **zarr_loc** ("str" | "Store") – The file name for the Zarr hierarchy or a store * **workspace** ("str" | "None") – Workspace name in the destination store. None uses the legacy layout without a workspace group. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly. When None, the default policy is used. An import that does not fit "mem_budget" raises MemoryError before the destination is created, naming the largest smaller policy that fits. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **assay_types** ("dict"["str", "str"] | "None") – Preset assay types, such as "{"GEX": "RNA"}", for assays whose names are not presets. Other assays take their type from their name. * **lines_in_mem** ("int") – Matrix Market lines parsed at a time (Matrix Market readers only). The layout fit and the write reserve this parse buffer. * **overwrite** ("bool") – If True, replace a CyteArc store that no "DataStore" has opened. Each assay stores its counts in the dtype that "count_storage_dtype()" resolves from the canonical values of its own features over the selected cells. Raises: **FileExistsError** – If "zarr_loc" is not empty, unless "overwrite" replaces it. cr A CrReader object, containing the Cellranger data. z The Zarr hierarchy (array or group). dump(batch_size=None) Writes the count values into the Zarr matrix. Parameters: **batch_size** ("int" | "None") – Number of source cells per batch, at most one destination row band, which is the default. Raises: **ValueError** – If "batch_size" is not positive. Return type: "None" Returns: None "MtxToZarr" is an alias of "CrToZarr" for use with "MtxReader". cytearc.writers.MtxToZarr alias of "CrToZarr" class cytearc.writers.H5adToZarr(h5ad, zarr_loc, assay_name=None, workspace=None, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, assay_split_key=None, assay_name_map=None, analysis_assay=None, assay_type=None, *, overwrite=False) A class for converting data in anndata’s H5ad format to Zarr hierarchy. Parameters: * **h5ad** ("H5adReader") – Reader for the source H5AD file. * **zarr_loc** ("str" | "Store") – The file name for the Zarr hierarchy or a store * **assay_name** ("str" | "None") – the name of the assay (e. g. ‘RNA’) * **assay_split_key** ("str" | "None") – A var column used to split features into assays. * **assay_name_map** ("dict"["str", "str"] | "None") – Feature type to assay name overrides. * **workspace** ("str" | "None") – An optional workspace id. None uses the legacy layout without a workspace group. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly. When None, the default policy is used. An import that does not fit "mem_budget" raises MemoryError before the destination is created, naming the largest smaller policy that fits. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **analysis_assay** ("str" | "None") – Imported assay that owns explicitly selected H5AD embeddings and clusters. Required for multi-assay imports with analytical outputs. * **assay_type** ("str" | "None") – Preset assay type, such as "RNA", for a single imported assay whose name is not a preset. When None, the assay name decides the type. Not allowed with "assay_split_key". * **overwrite** ("bool") – If True, replace a CyteArc store that no "DataStore" has opened. Each assay stores its counts in the dtype that "count_storage_dtype()" resolves from the canonical source values of its own features, so the same counts import identically from every H5AD encoding. Raises: **FileExistsError** – If "zarr_loc" is not empty and may not be replaced. h5ad A h5ad object (h5 file with added AnnData structure). assayName Name of the imported assay; None with "assay_split_key". z The Zarr hierarchy (array or group). storageDtypes Count storage dtype of each imported assay. policy Count-matrix layout policy of every imported assay. dump(batch_size=None) Write h5ad matrix data into Zarr "counts" and RNA "countsT". Parameters: **batch_size** ("int" | "None") – Number of source cells per batch, at most one destination row band, which is the default. Raises: **ValueError** – If "batch_size" is not positive. Return type: "H5adImportResult" Returns: Explicit selection, embedding, and cluster artifact references. class cytearc.writers.SparseToZarr(csr_mat, zarr_loc, cell_ids, feature_ids, assay_name=None, workspace=None, feature_names=None, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, assay_type=None, *, overwrite=False) A class for converting data in a sparse matrix to a Zarr hierarchy. Parameters: * **csr_mat** ("csr_matrix") – A CSR format sparse matrix * **zarr_loc** ("str" | "Store") – Output Zarr filename with path * **cell_ids** ("ndarray" | "list"["str"]) – Cell IDs for the cells in the dataset. * **feature_ids** ("ndarray" | "list"["str"]) – Feature IDs for the features in the dataset. * **assay_name** ("str" | "None") – Name for the output assay. If not provided then automatically set to RNA. * **workspace** ("str" | "None") – Workspace name in the destination store. None uses the legacy layout without a workspace group. * **feature_names** ("ndarray" | "list"["str"] | "None") – Optional display names aligned with "feature_ids". * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly. When None, the default policy is used. An import that does not fit "mem_budget" raises MemoryError before the destination is created, naming the largest smaller policy that fits. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **assay_type** ("str" | "None") – Preset assay type, such as "RNA", for an assay whose name is not a preset. When None, the assay name decides the type. * **overwrite** ("bool") – Replace an existing CyteArc store that no "DataStore" has opened. The counts are stored in the dtype that "count_storage_dtype()" resolves from the canonical (duplicate-summed) values of the matrix. Raises: * **ValueError** – Raised if number of input cell or feature IDs does not match the matrix, or if a count is NaN or infinite. * **FileExistsError** – If "zarr_loc" is not empty and may not be replaced. mat Input CSR matrix assayName The name of the output assay. z The Zarr hierarchy (array or group). dump(batch_size=None) Write out the data matrix into the Zarr hierarchy. Parameters: **batch_size** ("int" | "None") – Number of source cells per batch, at most one destination row band, which is the default. Raises: **ValueError** – If "batch_size" is not positive. Return type: "None" Returns: None class cytearc.writers.CSVtoZarr(cr, zarr_loc, assay_name, workspace=None, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, assay_type=None, *, overwrite=False) A class for converting data from CSV format to a Zarr hierarchy. Parameters: * **cr** ("CSVReader") – A CSVReader object * **zarr_loc** ("str" | "Store") – The file name for the Zarr hierarchy. * **assay_name** ("str") – A label for the assay. Ex. “RNA” or “ATAC” * **workspace** ("str" | "None") – Workspace name in the destination store. None uses the legacy layout without a workspace group. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly. When None, the default policy is used. An import that does not fit "mem_budget" raises MemoryError before the destination is created, naming the largest smaller policy that fits. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **assay_type** ("str" | "None") – Preset assay type, such as "RNA", for an assay whose name is not a preset. When None, the assay name decides the type. * **overwrite** ("bool") – If True, replace a CyteArc store that no "DataStore" has opened. The counts are stored in the dtype that "count_storage_dtype()" resolves from the range the reader found in its first pass over every row. A count that the stored dtype cannot hold raises instead of wrapping. Raises: **FileExistsError** – If "zarr_loc" is not empty and may not be replaced. csvr A CSVReader object z The Zarr hierarchy (array or group). dump() Writes the count values into the Zarr matrix. Raises: * **OverflowError** – If a count no longer fits the stored dtype because the file changed after the reader’s pass. * **ValueError** – If the file no longer holds the rows that the reader’s pass counted. Return type: "None" Returns: None class cytearc.writers.SubsetZarr(zarr_loc, assays, in_workspace=None, out_workspace=None, cell_key=None, cell_idx=None, reset_cell_filter=True, overwrite_existing_file=False, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None) Split Zarr file using a subset of cells. Parameters: * **zarr_loc** ("str" | "Store") – Path for the output (subsetted) Zarr file * **assays** ("list"["Any"]) – Source assays to be subsetted. These assays must be from the same DataStore * **in_workspace** ("str" | "None") – Source workspace name (None for legacy layout). * **out_workspace** ("str" | "None") – Target workspace name in the output Zarr file. * **cell_key** ("str" | "None") – Name of a boolean column in cell metadata. The cells with value True are included in the subset. Only used when cell_idx is None. A column without True values writes a subset without cells. * **cell_idx** ("ndarray" | "None") – Explicit indices of cells to include in the subset: at least one, each a distinct non- negative integer smaller than the number of cells. * **reset_cell_filter** ("bool") – If True, then the cell filtering information is removed, i.e. even the filtered out cells are set as True as in the ‘I’ column. To keep the filtering information set the value for this parameter to False. (Default value: True) * **overwrite_existing_file** ("bool") – If True, replaces an existing CyteArc store that no "DataStore" has opened. (Default value: False) * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly for every assay. When None, each assay uses the default policy with unitBytes and chunkBytes halved together until its subset write and countsT transpose fit "mem_budget". Either way, a subset that does not fit raises MemoryError before the destination is created. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. Each assay keeps the dtype of its source counts, so the subset holds the source values exactly. Raises: **FileExistsError** – If "zarr_loc" is not empty and may not be replaced. dump() Write subsetted cell metadata and count matrices, including RNA "countsT". Return type: "None" Returns: None cytearc.writers.to_h5ad(assay, h5ad_filename, embeddings_cols=None, skip_recalc_nfeats=True, nthreads=4, *, run=None, matrix='raw') Save an assay or a completed pipeline run as an H5AD file. Rows that a nullable metadata column’s linked missing mask flags are written as missing values: NaN in a float64 column for numeric columns, a nullable boolean for boolean columns, and a missing category for other columns. Parameters: * **assay** ("Any") – Assay to save in H5ad format * **h5ad_filename** ("str") – Name for the H5ad file to be created. * **embeddings_cols** ("list"["str"] | "None") – Cell-metadata column prefixes treated as embeddings (for example UMAP, tSNE). When None, uses "["UMAP", "tSNE"]". Pass an empty list to skip embeddings. * **skip_recalc_nfeats** ("bool") – Skip a preliminary nonzero- count pass. (Default value: True) * **nthreads** ("int") – Number of processing threads to use (Default value: 4) * **run** ("object" | "None") – Completed pipeline run opened from the datastore that owns "assay". The frozen run selections and fields are exported. Consecutive frozen UMAP fields are already in "obsm['X_umap']" as in "to_anndata(run=...)"; cluster labels remain in "obs['clusters']". Live embedding columns and feature-count recalculation options do not apply to run export. * **matrix** ("Literal"["'raw'", "'normed'"]) – ""raw"" counts, or ""normed"" values, which require "run". Return type: "None" Returns: None cytearc.writers.to_mtx(assay, mtx_directory, compress=False) Save an assay as a Matrix Market directory. Parameters: * **assay** ("Any") – CyteArc assay. For example: *ds.RNA* * **mtx_directory** ("str") – Out directory where MTX file will be saved along with barcodes and features file * **compress** ("bool") – If True, then the files are compressed and saved with .gz extension, using Cell Ranger 3 names; "features.tsv.gz" then also holds a feature-type column. (Default value: False). Return type: "None" Returns: None Raises: **UnmeasuredCellsError** – If the assay did not measure a cell. cytearc.writers.chunked_to_zarr(df, z, loc, nthreads, msg=None, mirror=None, resources=None) Creates a Zarr hierarchy from a chunked array. Parameters: * **df** ("Any") – ChunkedArray to materialize and write. * **z** ("Group") – Root Zarr group. * **loc** ("str") – Array path within the group. * **nthreads** ("int") – Threads for block compute. * **msg** ("str" | "None") – Progress bar message (default: "Writing data to {loc}"). * **mirror** ("Array" | "None") – Optional second array of the same shape to write each band into during the same pass (local staging cache). * **resources** ("ResourceBudget" | "None") – Optional memory and worker budget for the write. Return type: "None" cytearc.writers.create_zarr_dataset(g, name, chunks, dtype, shape, overwrite=True) Creates and returns a Zarr array. Parameters: * **g** ("Group") – Parent Zarr group. * **name** ("str") – Array name within the group. * **chunks** ("tuple"["int", "..."] | "int") – Chunk shape, or a single integer applied to every axis. * **dtype** ("Any") – NumPy dtype for the array. * **shape** ("tuple"["int", "..."]) – Array shape. * **overwrite** ("bool") – If True, replace an existing array of the same name. Return type: "Array" Returns: A Zarr Array. cytearc.writers.create_zarr_obj_array(g, name, data, dtype=None, overwrite=True, chunk_size=100000, shape=None) Creates and returns a metadata column array. Parameters: * **g** ("Group") – Parent Zarr group. * **name** ("str") – Array name within the group. * **data** ("Any") – Values to write, or None to create an empty array. * **dtype** ("str" | "Any") – Optional dtype. Inferred from "data" when omitted. * **overwrite** ("bool") – If True, replace an existing array of the same name. * **chunk_size** ("int") – Chunk length along the first axis. * **shape** ("int" | "None") – Explicit length when "data" is None. Return type: "Array" Returns: A Zarr Array. cytearc.writers.create_zarr_count_assay(z, assay_name, workspace, n_cells, feat_ids, feat_names, dtype, *, profile=None, policy=None) Creates and returns a Zarr array with name ‘counts’. Parameters: * **z** ("Group") – Root Zarr group. * **assay_name** ("str") – Assay group that will own the counts array. * **workspace** ("str" | "None") – Workspace name. None uses the legacy layout. * **n_cells** ("int") – Number of cells (rows). * **feat_ids** ("ndarray" | "list"["str"]) – Feature identifiers written to feature metadata. * **feat_names** ("ndarray" | "list"["str"]) – Feature display names written to feature metadata. * **dtype** ("Any") – Storage dtype for counts. Import writers store the dtype that "count_storage_dtype()" resolves from the canonical values. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile. When None, chosen from the store. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy. When None, the default plan is used. Return type: "Array" Returns: The created "counts" array. cytearc.writers.subset_assay_zarr(zarr_loc, in_grp, out_grp, cells_idx, feat_idx, storage_options=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None) Selects a subset of the data in an assay in the specified Zarr hierarchy. Parameters: * **zarr_loc** ("str" | "Store") – The file name for the Zarr hierarchy. * **in_grp** ("str") – Array in Zarr hierarchy to subset. * **out_grp** ("str") – Path of a new array in Zarr hierarchy to write subsetted assay to. * **cells_idx** ("ndarray") – Distinct non-negative indices of the cells to keep, in the order of the subset’s rows. * **feat_idx** ("ndarray") – Distinct non-negative indices of the features to keep, at least one, in the order of the subset’s columns. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the Zarr store. * **mem_budget** ("int" | "str" | "None") – Memory available to the conversion. Accepts bytes, a suffixed size (e.g. ‘8G’), or a fraction of total system memory (e.g. ‘0.6’). * **nthreads** ("int" | "None") – Worker count for write-time concurrency. When None, auto-detected. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly. When None, the default policy is used with unitBytes and chunkBytes halved together until the subset write fits "mem_budget". Either way, a subset that does not fit raises MemoryError before "out_grp" is created. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. The subset keeps the dtype of the source counts. Return type: "None" Returns: None Raises: **FileExistsError** – If "out_grp" exists or its parent group holds counts. cytearc.writers.write_renorm_subset_to_zarr(assay, cell_idx, feat_idx, z, loc, nthreads, log_transform=False, msg=None, mirror=None) Write library-size normalized subset data in a single scattered read pass. For HVG subsets the per-cell scale factor is the row sum within each block, so a separate "counts.sum(axis=1)" pass is not needed before writing. Parameters: * **assay** ("Any") – Source assay providing counts and library sizes. * **cell_idx** ("ndarray") – Cell indices to write. * **feat_idx** ("ndarray") – Feature indices to write. * **z** ("Group") – Destination Zarr group. * **loc** ("str") – Array path within the group. * **nthreads** ("int") – Threads for block compute. * **log_transform** ("bool") – If True, write log1p normalized values. * **msg** ("str" | "None") – Optional progress message. * **mirror** ("Array" | "None") – Optional second array of the same shape written in the same pass. Return type: "None" Selection and layout behavior ============================= "cytearc.datastore.datastore.DataStore.to_anndata()" supports cell selection plus either "feature_names" or "feature_indexes". The two feature selectors are mutually exclusive. "to_anndata" needs the optional "anndata" package and raises "ImportError" without it; install it with "pip install 'cytearc[extra]'". "to_h5ad" writes with h5py, which CyteArc always installs, and needs no "anndata", with or without a "run". "SubsetZarr" selects cells of one "DataStore" but retains every feature in each assay it writes. It keeps the membership column "_I" of each assay it writes, for the selected cells, and does not copy the membership column of an assay that "assays=" leaves out (Assay membership). Without a "run", "to_h5ad" and "to_mtx" export a complete assay. For feature-selective disk export outside a pipeline run, call "to_anndata" and use AnnData’s writer. A Matrix Market directory holds no cell metadata, so "to_mtx" cannot say which cells an assay measured, and a cell that it did not measure would be written as a zero row that "MtxToZarr" reads back as a measured cell. "to_mtx" therefore raises "UnmeasuredCellsError" before it writes anything when the assay’s membership column "_I" marks a cell unmeasured (Assay membership). Export the measured cells from a subset that "SubsetZarr(..., cell_key="_I")" writes, or use "to_h5ad" or "to_anndata" without layers, which declare the membership. "to_h5ad" streams the matrix into the file one block of rows at a time, so it never holds the complete matrix, and it converts a dense block to CSR a bounded step of rows at a time, which the block’s stream charges against the memory budget. It writes a temporary file in the same directory and moves it onto the target once complete, so a failed export leaves no partial file and keeps an earlier file of the same name; a replaced file keeps its permissions. Without a "run", text without missing values is a string array and both indexes are stored as "_index". Nullable metadata columns keep a stored placeholder in each row that their linked missing mask flags. "to_anndata" and "to_h5ad" export those rows as missing values with or without "run": numeric columns become float64 with "NaN", boolean columns become nullable booleans, and other columns hold a missing value, which H5AD files store as a missing category. Without "run", "to_anndata" and "to_h5ad" export the membership column "_I" of the exported assay as an ordinary "obs" column and declare it in "uns["cytearc"]["assayMembership"]", for example "{"RNA": "RNA_I"}", so "H5adToZarr" restores it as the membership of an imported assay of that name and a merged store survives export, import, and a second merge. The membership columns of other assays, including the assays of "to_anndata" "layers", describe assays that the file does not hold and are not exported, as a merge or subset that leaves an assay out does not copy its membership column; an import would otherwise keep them as ordinary metadata, which a later merge with "prepend_text=None" that includes those assays would refuse. A run’s frozen fields declare no membership. Because a layer of another assay cannot declare its membership, every exported cell must be one that the layer’s assay measured: "to_anndata" with such a layer, with or without "run", raises "UnmeasuredCellsError" before it reads any value, and a "cell_key" of the cells that the layer’s assay measured exports them. A layer of the exported assay is declared with it. Pass a completed run to write its frozen cells, feature universe, and result fields directly: cytearc.to_h5ad(ds.RNA, "analysis.h5ad", run=run) cytearc.to_h5ad(ds.RNA, "analysis_normalized.h5ad", run=run, matrix="normed") The assay must be the exact assay object owned by the datastore that opened the run. A run export to a file holds exactly what "to_anndata()" returns for the same "run" and "matrix", because one resolution of the run’s rows, columns, and values serves both: the run’s cells, its frozen cell fields in "obs" with the consecutive UMAP fields in "obsm["X_umap"]" and "clusters" in "obs", and its frozen feature fields in "var". "to_anndata" holds that object in memory; "to_h5ad" streams it into the file. The run’s plan decides which text fields are categoricals: text that repeats or has missing values is a categorical whose categories are in natural order ("d2" before "d10", with a superscript or circled digit kept as text), and text missing in every row is a categorical without categories. "to_anndata" returns those fields as pandas categoricals with exactly those categories, and the file stores the same categories and codes, so the object, the file, and a file that "AnnData.write_h5ad" writes from the object agree whatever the AnnData version. Other text is a string array, and the indexes are named "ids" and "gene_ids". The export stores "X" compressed, stores distinct text as a string array, and leaves out the empty "layers", "obsp", "uns", "varm", and "varp" groups. "embeddings_cols", feature-count recalculation, and a writer-specific thread override apply only to ordinary full-assay export and are rejected when "run" is supplied. With "matrix="raw"", the default, "X" holds the raw counts of the run’s cells over its feature universe. With "matrix="normed"", "X" is the run’s "normalized" artifact: the float32 values that its PCA read, as "normalize()" stored them for the run, over the run’s highly variable features in stored feature order. "var" then holds the frozen fields of those features only, and "to_anndata" "layers" align to them. The values are read in the artifact’s stored row bands and never converted to float64. The export checks that the store’s dataset fingerprint still matches the artifact and raises "ValueError" when the run has no normalized output, or when the artifact’s cell selection is not the run’s cell selection or its feature selection is not the run’s highly variable features. Without a "run", "to_anndata(matrix="normed")" normalizes the selected cells and features with the assay’s "normed", and "to_h5ad" rejects "matrix="normed"". "H5adToZarr.dump()" returns an "H5adImportResult" containing the written assays, analysis assay, all-cell selection, and imported embedding and clustering refs. Set "analysis_assay" when a multi-assay import selects analytical values. Use "DataStore.artifacts.load(ref)" for payload access or pass the exact ref to a consumer. Import does not flatten these results into metadata columns. Ordinary "to_h5ad" export writes a complete assay and live metadata. Run-aware export reads only the completed run’s frozen selections and fields, and for normalized values its "normalized" artifact, so export does not require physical result columns. "CrToZarr", "MtxToZarr", "H5adToZarr", "SparseToZarr", and "SeuratToZarr" select source batch rows automatically when "batch_size" is omitted, starting from the smallest destination row- shard height, which the fitted count layout admits. An explicit positive "batch_size" is capped at that height. Merge ===== Use "DataStoreMerge" to merge DataStores. Pass "assays=["RNA"]" when only one assay type is needed. Features are matched by exact feature ID by default. Gene symbols are display labels and do not merge distinct IDs; suffixes such as "_1" are preserved. An assay present in multiple inputs raises an error if none of its IDs overlap. When inputs use different identifier conventions for the same genes, for example Ensembl IDs in one input and gene symbols in another, pass "feature_key="names"" to match features by name. The merged feature IDs are then the names, and features that share a name within one input are summed. Feature annotation columns are merged when the inputs agree. A column whose values differ for a shared feature, such as per-dataset highly variable gene flags, is left out with a warning. The same applies when features that share a name within one input disagree. Merged cell IDs are "{name}__{cell_id}", so source names cannot contain "__". Cell metadata columns keep their attributes when the inputs agree. Differing "levels" of an unordered categorical column become their union in first-seen order; any other differing attribute, including ordered levels, is dropped with a warning. Each merged assay keeps the type that its sources declare in "assayTypes", such as "HTO" or "GeneActivity", rather than the preset of its assay class, so a merged HTO assay can be demultiplexed. Sources that declare different types for one assay are rejected before anything is written; reopen the source whose declaration is wrong with "zarr_mode="r+"" and a matching "assay_types". RNA assays write both "counts" and a gene-major "countsT" copy, which roughly doubles stored counts for those assays. Non-RNA assays never write "countsT". Every merged assay gets the membership column "_I" described in Assay membership. A merged cell keeps its source’s membership: False when the source lacks the assay, True when the source measured every cell with it, and the source’s own value when the source records partial membership. A source’s membership column is not merged again as an ordinary column such as "orig__I", and the membership column of a source assay that "assays=" leaves out is not merged at all, so a merge with "assays=["RNA"]" writes no "ADT_I" or "orig_ADT_I" that a later merge including ADT would trip over. Planning rejects a malformed source membership column, an ordinary column whose merged name equals a membership column, and a source whose cells outside an assay have counts of it. A source column named "_I" for a merged assay that lacks the membership attributes raises "Cell column '_I' is reserved for the membership of assay ...". Import the file again, which restores a declared membership and otherwise skips the column with a warning, or drop the plain column from the source with "source.cells.drop("_I")", after which the source counts every cell of the assay as measured. "MetaData" writes no membership column itself: "insert", "update_key", and "reset_key" refuse "_I" for any assay of the store, and "drop" refuses one that carries the membership role (Assay membership). The destination must be empty or hold a merge written by "DataStoreMerge". A destination that aliases, contains, or lies inside a source store, or that already holds other content, is refused. "overwrite=True" replaces only a merge-owned destination whose assays have not been prepared by opening it as a "DataStore", and it clears the recorded default assay. Interrupted merges resume at whole-component boundaries ("cellData", each assay "counts", and each RNA "countsT") rather than mid-matrix. A resume requires the same configuration, the same source counts, assay types, and membership; a completed "countsT" whose layout differs from the plan is not rewritten in place. The manifest must record "assayTypes" and "sourceMembership"; without those fields, restart with "overwrite=True". Completed counts keep the layout persisted with them, so a resume under another "mem_budget" reuses them, while counts that a resume rewrites get a layout fitted to its budget. class cytearc.merge.DataStoreMerge(datasets, zarr_path, names, *, assays=None, out_workspace=None, overwrite=False, prepend_text='orig', reset_cell_filter=True, seed=42, storage_options=None, source_column=None, mem_budget=None, nthreads=None, profile=None, policy=None, io=None, missing_assay_policy='zero_fill', feature_key='ids') Merge multiple DataStores into one Zarr store. Construction is side-effect free. Call "plan()" to inspect the resolved merge, then "dump()" to write or resume it. Parameters: * **datasets** ("list"["Any"]) – Source DataStores to merge. At least two are required. * **zarr_path** ("str" | "Store") – Destination Zarr path or store. * **names** ("list"["str"]) – Unique name for each source, used in metadata and provenance. Merged cell IDs are "{name}__{cell_id}", so a name cannot contain "__". * **assays** ("list"["str"] | "None") – Optional assay-name filter. None merges every assay present in any source. * **out_workspace** ("str" | "None") – Workspace name in the destination store. None uses the legacy layout without a workspace group. * **overwrite** ("bool") – If True, replace a merge-owned destination whose assays have not been prepared by opening it as a DataStore. * **prepend_text** ("str" | "None") – Prefix added to colliding metadata column names. * **reset_cell_filter** ("bool") – If True, mark every merged cell as selected. * **seed** ("int" | "None") – RNG seed for interleaving source cell blocks. None draws a fresh, unseeded interleaving, so the merged row order differs between runs. * **storage_options** ("dict"["str", "Any"] | "None") – Backend options passed when opening the destination. * **source_column** ("str" | "None") – Optional cell-metadata column storing source names. * **mem_budget** ("int" | "str" | "None") – Memory budget for the merge. Accepts bytes, a size such as "8G", or a fraction of detected system memory. When None, the tightest source budget is used. * **nthreads** ("int" | "None") – Maximum worker budget. When None, the tightest source worker count is used. * **profile** ("Literal"["'fast_local'", "'cloud'"] | "None") – Zarr encoding profile ("fast_local" or "cloud"). When None, chosen from the destination location. * **policy** ("CountMatrixPolicy" | "None") – Count-matrix geometry policy, used exactly for every assay. When None, each assay uses the default policy with unitBytes and chunkBytes halved together until its counts write and countsT transpose fit "mem_budget". A resumed merge keeps the layout of every assay whose counts are complete, so a change of budget between attempts does not block it. * **io** ("StorageIoPolicy" | "None") – Optional explicit read, compute, and write widths. Unset values stay under automatic planning. * **missing_assay_policy** ("Literal"["'zero_fill'", "'error'"]) – "zero_fill" writes zeros for a missing assay in a source. "error" rejects that merge. * **feature_key** ("Literal"["'ids'", "'names'"]) – "ids" matches features across sources by exact feature ID. "names" matches them by feature name, which merges sources that use different ID schemes for the same genes. The merged IDs are then the names, and features that share a name within one source are summed. Merged counts use the common type of the source count dtypes, widened so that features summed by name cannot overflow it. dump() Write or resume the merge and return a component-level result. Return type: "MergeResult" plan() Return a side-effect-free merge plan. Return type: "MergePlan" class cytearc.merge.MergePlan(zarrPath, outWorkspace, sourceNames, nCells, assays, profile, seed, missingAssayPolicy, willResume, canDump, blockedReason, cellDataAction, manifest=) Side-effect-free preview of a DataStoreMerge. class cytearc.merge.AssayMergePlan(assayName, assayType, sourcePresent, missingSources, nFeatures, featureOverlapFraction, dtype, chunks, shards, writeCountsT, estimatedWriteTasks, countsAction, countsTAction) Resolved plan for one assay in a DataStoreMerge. "featureOverlapFraction" is the share of union feature keys (IDs, or names when merging by name) present in at least two sources with that assay. A modality present in only one source reports "1.0" because zero-fill is intentional for the missing sources. class cytearc.merge.MergeResult(zarrPath, nCells, assayNames, components, resumed) Result returned by DataStoreMerge.dump(). class cytearc.merge.ComponentResult(name, action) Outcome for one written or skipped merge component.