Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Why CyteArc shuffles cells

Count files usually list their cells grouped by sample, donor, plate or lane. A store in the file’s order holds the cells of one or two samples in any contiguous block of rows, such as one count shard of about 11,000 cells or a mini-batch that a training loader reads, so the block is a biased sample of the dataset. CyteArc therefore writes the cells of every import in a uniformly random order. Any contiguous block of rows is then a random sample of the store’s cells, and a loader can read large blocks, the fastest way to read a store, and still mix every sample in each batch.

What the order guarantees

The order is a uniform random permutation of the imported cells: every order is equally likely, so every contiguous block of rows, of any size and at any position, is a simple random sample of the cells. shuffle_seed, 0 by default, chooses the permutation, and one seed gives the same order on every platform and NumPy version, so an import of the same file with the same seed writes the same store.

Every per-cell array follows the one order: counts and countsT, the cell IDs and names, the cell metadata columns, assay membership, and the embeddings, cluster labels, reductions and identities that an import brings. Values that you attach to cells later, such as labels from another tool or an scVI latent space, are matched to the cells by cell ID, so the order never misplaces them: ds.cells.insert and the other calls that take one value per cell take a pandas Series or DataFrame indexed by cell ID and refuse a bare array.

Analyses that draw random numbers, such as UMAP and Leiden clustering, draw them over the cells in store order. A store imported in another order therefore gives slightly different layouts and clusters, as another seed would, and its cluster numbers can differ.

Merges and composites order their cells otherwise, in blocks that each hold the cells of one count shard of one source, shuffled within the block, so a contiguous slice of their rows, such as a training mini-batch, can hold the cells of a single source. The import and export reference describes the order.

Where each cell came from

Each cell keeps its row in the imported file, counted from 0, in the int64 cell column sourceRow, and the cell table records the seed in its cellOrder attribute. A reader that drops cells, such as CrH5Reader(is_filtered=False), keeps the row in the file of each cell it keeps, so the column has gaps where the dropped cells were. insert, update_key, reset_key and drop refuse sourceRow.

Exports write the cells back in the file’s order. to_h5ad, to_mtx, DataStore.to_anndata, the clone exports and the AIRR export sort the exported cells by sourceRow, so an import followed by an export gives back the file’s cells in the file’s order, and cell_order="stored" keeps the store’s order instead. Subsets, repacks and mounts keep each cell’s sourceRow, and a merge gives each cell its row in the concatenation of its sources, so their exports follow the files too. The import and export reference describes each export’s order.

What the shuffle costs

These figures come from imports of gzip-compressed samples of a CELLxGENE embryo dataset on 8 CPUs with a 24 GiB memory budget and a local destination.

Keep the file’s order

Pass shuffle_cells=False to the writer to write the cells in the file’s order:

import cytearc

# Import the file with its cells in the file's order.
reader = cytearc.H5adReader.from_inspect(cytearc.inspect_h5ad("counts.h5ad"))
cytearc.H5adToZarr(reader, zarr_loc="counts.zarr", shuffle_cells=False).dump()

Every import writer takes the option: H5adToZarr, CrToZarr and MtxToZarr, TapestriToZarr, CSVtoZarr, SparseToZarr and SeuratToZarr. An import in the file’s order writes neither sourceRow nor cellOrder, and its store is the store that CyteArc 1.4.0 and earlier releases write, except that H5adToZarr stores the file’s categoricals as categorical columns unless given categorical_columns=False. Stores imported by those releases keep their order, open unchanged and export in their stored order, which is their file’s order.

A file imported again with the shuffle gets a store in another order than the store that an earlier release imported from it, so the two stores have different dataset fingerprints, and a composite or a mount that recorded the earlier store refuses the new one in its place. Import with shuffle_cells=False to rebuild the earlier store.