Cube files

A cube file is a Zarr v3 group that any Zarr v3 reader can open. write_cube() saves a cube and read_cube() opens one, leaving its layers on disk until they are used; a read then opens only the chunks that hold the selected groups and features. cytepool.io.zarr is the minimal Zarr v3 reader and writer behind them.

Reading and writing cube files; cytepool.io.anndata converts cubes to and from AnnData and MuData.

cytepool.io.read_cube(path)

Open a cube file.

The metadata and the sample, group and feature tables are read now. The layers stay on disk: when a method needs a group of a layer, the cube reads the chunks that hold the samples and features it needs, and checks those values then (see “Lazy layers” in Cube). Opening a cube therefore takes little memory, whatever its size, and a cube derived with select() reads only the chunks that hold the selected features of the selected groups.

The digests stored with the file give the cube’s fingerprint without reading the layers. The tables, cell numbers and normalization are checked against theirs now, and each group of a layer whenever a read covers all of it.

Parameters:

path – The cube file’s directory.

Returns:

A cube whose layers are ZarrArray objects.

Raises:

• FileNotFoundError – If there is no store at path.

• ValueError – If the store is not a cube file, has another format version, is corrupt, uses a Zarr feature this reader does not support, or holds tables, cell numbers or a normalization that do not match its stored digests.

• TypeError – If a stored value has a type the cube does not accept.

• RuntimeError – If the store was replaced while it was being read.

Used in: Export a cube from another tool, Differential expression per cell type, Bulk RNA-seq

Explained in: The cube and its additive layers, Cube files and provenance records

cytepool.io.write_cube(cube, path, *, compression='gzip', level=None, overwrite=False)

Save a cube as a cube file, a Zarr v3 group (see the module documentation).

The store appears at path only once it is complete, so a write that fails leaves no store there; with overwrite=True it leaves the old store as it was. Lazy layers are read, and their values checked, one group at a time. Each group is hashed as it is written, and the digests are stored with the file, so the fingerprint of the saved cube needs no further pass over its values.

Parameters:

• cube – The cube to save.

• path – The directory to create, by convention ending in .cube.zarr. Its parent must exist.

• compression – "gzip" (the default), "zstd" (Python 3.14 or newer) or None for no compression.

• level – The compression level: 0 to 9 for gzip (default 4), -131072 to 22 for zstd (default 3); None without compression.

• overwrite – Replace what is at path, the whole directory, if it is a cube file (a Zarr group whose format is "cytepool.cube") or an empty directory. Nothing else is ever replaced, nor the working directory or one that holds it.

Raises:

• TypeError – If cube is not a Cube, or an argument has the wrong type.

• ValueError – If compression or level is invalid, path is a URL, or a lazy layer gives a value that breaks a constraint.

• OverflowError – If a sum of a derived lazy layer does not fit (see “Sums” in Cube).

• FileExistsError – If path exists and overwrite is False, or it is not something overwrite replaces.

• FileNotFoundError – If the parent directory of path does not exist.

Used in: Export a cube from another tool, Differential expression per cell type

Explained in: The cube and its additive layers, Cube files and provenance records

A minimal reader and writer for Zarr v3 arrays and groups on a local file system.

This module implements, with numpy and the standard library only, the part of the Zarr core specification, version 3.1 (Miles et al. 2025), that cube files need, with the bytes and gzip codecs of the Zarr specifications (Zarr developers 2023) and the zstd codec of its registered extensions (Maitin-Shepard 2025).

Reading supports:

• group and array metadata documents (zarr.json);

• the core data types bool, int8 to int64, uint8 to uint64, float16, float32, float64, complex64 and complex128, the floating-point types in the binary formats of IEEE 754 (IEEE 2019);

• the regular chunk grid, and the default and v2 chunk key encodings with either separator;

• fill values in every form the specification permits; a chunk that is not stored reads as the fill value;

• the bytes codec in either byte order, followed by any sequence of gzip codecs, whose chunks are gzip streams (RFC 1952) of DEFLATE data (RFC 1951; both Deutsch 1996), and zstd codecs, whose chunks are Zstandard frames (RFC 8878; Collet and Kucherawy 2021); zstd needs the compression.zstd module of Python 3.14 or newer.

Anything else, such as another data type, the blosc, crc32c, transpose or sharding_indexed codec, another chunk grid or a storage transformer, raises a ValueError that names it; zarr-python reads such stores.

A read opens only the chunks that hold a selected element and decodes each of them once, one at a time, so it holds the result and about one chunk: compressed chunks are read from disk and decoded in pieces of at most 256 KiB. A zstd decoder also keeps its frame’s window, outside Python’s allocations: up to the size of the chunk, and 2 MiB at zstd’s default level of 3. A chunk compressed twice holds the output of both steps. Selections take integers and slices, and, through ZarrArray.oindex, integer arrays and boolean masks along any axes.

Reading is strict. A metadata document must be a JSON object (Bray 2017) without repeated keys or NaN and Infinity tokens, and every field must be valid; a field the specification does not define is an error unless its value is an object with "must_understand": false. A chunk must decode to exactly the bytes its shape needs, compressed streams must be complete and pass their checksums, and bool chunks may hold only the bytes 0 and 1. Memory and time stay bounded on malformed files: arrays whose chunks would hold more than 2 GiB (MAX_CHUNK_BYTES) are refused, at most the bytes a valid encoding of a chunk can take are read from disk, every decoding step stops just past the most bytes a valid chunk can produce, and a chunk may hold at most 1,024 gzip members or zstd frames (writers use one).

Writing produces arrays with a regular chunk grid, the default chunk key encoding with /, a fill value of zero (false for bool) and the bytes codec, little endian, followed by gzip, zstd (with a checksum) or no compression. A chunk whose bytes are all zero equals the fill value and is not stored.

References

Miles A, Striebel J, Rzepka N, Maitin-Shepard J, Moore J, eds (2025). Zarr core specification, version 3.1. https://zarr-specs.readthedocs.io/en/latest/v3/core/

Zarr developers (2023). Bytes codec, version 1.0, and gzip codec, version 1.0. Zarr specifications, https://zarr-specs.readthedocs.io/en/latest/v3/codecs/

Maitin-Shepard J (2025). Zstd codec. Zarr extensions, zarr-developers/zarr-extensions

Deutsch P (1996). GZIP file format specification version 4.3. RFC 1952, RFC Editor. doi:10.17487/RFC1952

Deutsch P (1996). DEFLATE Compressed Data Format Specification version 1.3. RFC 1951, RFC Editor. doi:10.17487/RFC1951

Collet Y, Kucherawy M (2021). Zstandard Compression and the ‘application/zstd’ Media Type. RFC 8878, RFC Editor. doi:10.17487/RFC8878

Bray T (2017). The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259, RFC Editor. doi:10.17487/RFC8259

IEEE (2019). IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2019. doi:10.1109/IEEESTD.2019.8766229

cytepool.io.zarr.METADATA_FILE = 'zarr.json'

The name of the metadata document of every Zarr v3 group and array.

cytepool.io.zarr.DATA_TYPES = mappingproxy({'bool': dtype('bool'), 'int8': dtype('int8'), 'int16': dtype('int16'), 'int32': dtype('int32'), 'int64': dtype('int64'), 'uint8': dtype('uint8'), 'uint16': dtype('uint16'), 'uint32': dtype('uint32'), 'uint64': dtype('uint64'), 'float16': dtype('float16'), 'float32': dtype('float32'), 'float64': dtype('float64'), 'complex64': dtype('complex64'), 'complex128': dtype('complex128')})

The Zarr v3 core data types this module reads and writes, by name.

cytepool.io.zarr.DEFAULT_LEVELS = mappingproxy({'gzip': 4, 'zstd': 3})

The compression level write_array uses when none is given.

cytepool.io.zarr.MAX_CHUNK_BYTES = 2147483648

The largest chunk, in bytes once decoded, that this module reads or writes.

class cytepool.io.zarr.ZarrArray(path, metadata, stamp)

A Zarr v3 array in a local directory, read chunk by chunk.

Open one with open_array. Indexing with integers and slices, as in array[2], array[1, :, 100:200] or array[..., ::2], reads only the chunks the selection touches and returns a new numpy array in native byte order, which owns its memory (a numpy scalar when every axis gets an integer, as in numpy). array.oindex[...] also takes, along any axis, a 1-D integer array or a boolean mask (as a list or a numpy array), and selects every combination of the positions each axis selects (orthogonal indexing, as in zarr-python): array.oindex[0, [3, 1], [5, 9]] is numpy.asarray(array)[0][numpy.ix_([3, 1], [5, 9])]. Integer arrays may be in any order, repeat positions and count negative positions from the end. numpy.asarray(array) reads the whole array. A chunk that is not stored reads as the fill value.

Every read first checks that the array’s zarr.json is still the file it was when the array was opened, and raises otherwise, so an array replaced on disk is never read through out-of-date metadata: on the same device the inode, size and modification time must match; through another mount, where inode numbers can differ, the size and modification time. A store replaced by an identical copy therefore also counts as changed on the same device.

A ZarrArray holds no open files, so it can be pickled and read from another process that sees the same path.

property path

The array’s directory.

property shape

The array’s shape.

property dtype

The data type of the values a read returns, in native byte order.

property ndim

The number of dimensions.

property chunks

The chunk shape of the regular chunk grid.

property fill_value

The value of elements in chunks that are not stored, as a numpy scalar.

property attributes

A copy of the array’s attributes.

property dimension_names

The name of each dimension (None for an unnamed one), or None.

property oindex

Orthogonal indexing, which selects along each axis independently.

array.oindex[key] reads what key selects. key gives each axis an integer, a slice, a 1-D integer array or a boolean mask with one value per position (arrays as lists or numpy arrays), and may hold one .... The result has an axis for every axis that gets no integer, holding the selected positions in the order given, as in zarr-python’s oindex. It raises the errors of __getitem__, and IndexError for an array that is not 1-D or a mask of the wrong length.

cytepool.io.zarr.open_array(path)

Open a Zarr v3 array for reading.

Parameters:

path – The array’s directory, which holds its zarr.json.

Returns:

A ZarrArray; nothing but the metadata is read.

Raises:

• FileNotFoundError – If there is no array metadata at path.

• ValueError – If the metadata is invalid or uses a feature this module does not support (the message names it).

cytepool.io.zarr.read_group_attributes(path)

Read and check a Zarr v3 group’s metadata and return its attributes.

Consolidated metadata, if present, is ignored: each node is read from its own zarr.json.

Parameters:

path – The group’s directory, which holds its zarr.json.

Raises:

• FileNotFoundError – If there is no group metadata at path.

• ValueError – If the metadata is invalid, describes an array, or belongs to a Zarr v2 store.

cytepool.io.zarr.write_group(path, attributes=None)

Write a Zarr v3 group’s metadata, creating its directory if needed.

Parameters:

• path – The group’s directory; its parent must exist.

• attributes – JSON-serialisable attributes.

Raises:

• FileExistsError – If the directory already has a zarr.json.

• ValueError, TypeError – If an attribute cannot be written as JSON.

cytepool.io.zarr.write_array(path, data, *, chunks, compression='gzip', level=None, attributes=None, dimension_names=None)

Write an array as a new Zarr v3 array directory.

Chunks are written first and zarr.json last, so a write that fails leaves a directory that does not open as an array. A chunk whose bytes are all zero is not stored, since zero is the fill value.

Parameters:

• path – The directory to create; its parent must exist and it must not.

• data – A numpy array, or any object with shape and dtype whose data[start:stop] returns those rows along the first axis, such as a ZarrArray. It is read one row of chunks at a time.

• chunks – The chunk shape: a positive length for each dimension.

• compression – "gzip", "zstd" or None (see check_compression).

• level – The compression level, or None for the default.

• attributes – JSON-serialisable attributes.

• dimension_names – A name or None for each dimension.

Raises:

• TypeError – If data has no shape and dtype, its data type is not a core data type, it is or gives rows as a numpy masked array, or another argument has the wrong type.

• ValueError – If an argument is invalid, a chunk would exceed MAX_CHUNK_BYTES, or data gives rows of the wrong shape or dtype.

• FileExistsError – If path exists.

cytepool.io.zarr.check_compression(compression, level)

Check a compression and level for write_array and return the level used.

Parameters:

• compression – "gzip", "zstd" (Python 3.14 or newer) or None for no compression.

• level – 0 to 9 for gzip, -131072 to 22 for zstd (0 asks zstd for its default), or None for DEFAULT_LEVELS (4 for gzip, 3 for zstd). It must be None without compression.

Returns:

The compression level, or None without compression.

Raises:

• TypeError – If compression is not a string or None, or level is not an integer.

• ValueError – If compression is unknown or unavailable, or level is out of range or given without compression.