A tool that holds cells, such as an analysis platform or an atlas, can hand its pseudobulk profiles to CytePool as a cube file: the sums of every sample, cell type and gene, the tables that describe them, the number of cells behind each profile, and a record of where they came from. This page builds a cube from sums computed outside CytePool, saves it, and checks it. Cube files and provenance records describes the file and the record in full.
The cells here are those of the simulated PBMC study (Datasets),
standing in for the exporter’s store: 12 donors sampled before and after a
stimulation, six cell types. python docs/datasets.py prepare simulated_pbmc writes
them to the cache, and the code below opens them by name.
import os
from types import SimpleNamespace
import numpy as np
import cytepool
from cytepool.cube import Cube, from_cells
from cytepool.io import read_cube, write_cube
cells = np.load("simulated_pbmc_cells.npz")
n_cells, n_genes = map(int, cells["shape"])
cell_sample, cell_type = cells["cell_sample"], cells["cell_type"]
print(f"CytePool {cytepool.__version__}, numpy {np.__version__}")
print(f"{n_cells:,} cells x {n_genes:,} genes in {len(set(cell_sample))} samples")
print("cell types:", ", ".join(cells["cell_types"]))CytePool 0.1.dev42+g6991e63f8, numpy 2.5.4
19,151 cells x 2,000 genes in 24 samples
cell types: B cells, CD4 T cells, CD8 T cells, NK cells, Monocytes, Dendritic cells
Pool the cells where they are¶
The exporter sums the counts of the cells of every sample and cell type into an array on disk, shaped cell types x samples x genes as a cube’s layers are. Here numpy does it from the cells’ sparse counts. A numpy memmap opened from that file is a lazy layer: the cube reads one cell type at a time from it, so the sums never need to fit in memory together. HDF5 datasets and Zarr arrays work the same way.
samples = np.unique(cell_sample)
groups = list(cells["cell_types"])
s = np.searchsorted(samples, cell_sample)
g = np.array([groups.index(name) for name in cell_type])
row = np.repeat(np.arange(n_cells), np.diff(cells["indptr"]))
entry = (g[row] * len(samples) + s[row]) * n_genes + cells["indices"]
sums = np.bincount(
entry, weights=cells["data"], minlength=len(groups) * len(samples) * n_genes
)
shape = (len(groups), len(samples), n_genes)
on_disk = np.lib.format.open_memmap(
"counts.npy", mode="w+", dtype=np.uint32, shape=shape
)
on_disk[:] = sums.reshape(shape)
on_disk.flush()
counts = np.load("counts.npy", mmap_mode="r")
n_cells_pooled = np.zeros((len(samples), len(groups)), dtype=np.int64)
np.add.at(n_cells_pooled, (s, g), 1)
print(f"counts: {type(counts).__name__} of shape {counts.shape}, {counts.dtype}")
print(f"{int(counts.sum()):,} counts from {int(n_cells_pooled.sum()):,} cells")counts: memmap of shape (6, 24, 2000), uint32
62,621,014 counts from 19,151 cells
Build the cube with a source record¶
Cube takes the layers, the sample, group and feature tables, the cell numbers
and a provenance record. Its source is the exporter’s own account of where the
data came from; CytePool stores it as given and never interprets it. The fields below
are those that make a cube traceable to its cells: the tool and its version, the
dataset, the frozen selection of cells (which stays in the exporter’s files), the
annotations that defined samples and groups, and the settings.
source = {
"tool": "example-exporter",
"version": "0.1",
"dataset": {"id": "simulated_pbmc", "file": "simulated_pbmc_cells.npz"},
"cell_selection": {"artifact": "selections/all-cells", "n_cells": n_cells},
"samples": {"columns": ["donor", "condition"]},
"groups": {"column": "cell_type"},
"settings": {"layer": "raw counts", "filters": None},
}
sample_table = {
"sample_id": samples,
"donor": [name.split("_")[0] for name in samples],
"condition": [name.split("_")[1] for name in samples],
}
cube = Cube(
{"counts": counts},
samples=sample_table,
groups={"group_id": groups},
features={"feature_id": cells["feature_id"]},
n_cells=n_cells_pooled,
provenance={"source": source},
)
print(cube)Cube with 24 samples x 6 groups x 2,000 features
layers: counts (uint32, lazy)
filled: 144 of 144 sample x group entries have cells (100.0%)
samples: sample_id, donor, condition
groups: group_id
features: feature_id
Save it and open it again¶
write_cube reads the lazy layer one cell type at a time, hashes each as it
writes it, and stores the digests with the file. read_cube opens the file
without reading its counts. Cube.history prints the provenance record, with
the source first and the fingerprint last.
write_cube(cube, "pbmc.cube.zarr")
opened = read_cube("pbmc.cube.zarr")
print(sorted(os.listdir("pbmc.cube.zarr")))
print(opened)
print()
print(opened.history())['features', 'groups', 'layers', 'n_cells', 'samples', 'zarr.json']
Cube with 24 samples x 6 groups x 2,000 features
layers: counts (uint32, lazy)
filled: 144 of 144 sample x group entries have cells (100.0%)
samples: sample_id, donor, condition
groups: group_id
features: feature_id
Source, as the caller recorded it (unresolved: CytePool does not follow its references):
tool: "example-exporter"
version: "0.1"
dataset: {"id": "simulated_pbmc", "file": "simulated_pbmc_cells.npz"}
cell_selection: {"artifact": "selections/all-cells", "n_cells": 19151}
samples: {"columns": ["donor", "condition"]}
groups: {"column": "cell_type"}
settings: {"layer": "raw counts", "filters": null}
References: CytePool resolves none on its own. The source record's references need the tool that wrote them. A fingerprint names a cube without saying where it is: Cube.fingerprint() of a cube at hand says whether it is that cube.
Fingerprint: sha256:b4a9d782cf081e10a8e835d66ae7b061874fc377ef838f168cf29911f79e7501
Check the export¶
The fingerprint names the cube’s values, not how it was made. Pooling the same cells
with CytePool’s own from_cells therefore gives the same fingerprint when the
export is right: a check an exporter can run on part of its data. On the opened file,
Cube.verify_fingerprint recomputes the fingerprint from the stored values and
checks every digest.
csr = SimpleNamespace(
data=cells["data"],
indices=cells["indices"],
indptr=cells["indptr"],
shape=(n_cells, n_genes),
)
pooled = from_cells(
csr,
samples=cell_sample,
groups=cell_type,
sample_table=sample_table,
group_table={"group_id": groups},
features=cells["feature_id"],
layers="counts",
)
print("same values as from_cells:", opened.fingerprint() == pooled.fingerprint())
print("stored values verified: ", opened.verify_fingerprint() == opened.fingerprint())same values as from_cells: True
stored values verified: True