Clones API reference

Clones API reference#

A clone is the set of cells that descend from one ancestor cell. Lineage tracing methods mark the ancestor with a heritable barcode, such as LARRY’s, CellTag’s or TREX’s expressed barcodes, and other methods infer clones from the cells’ own mutations, such as copy-number changes or mitochondrial variants. Each tool writes which clone each cell belongs to in its own format. ds.clones reads those files, joins their cells to a store’s cells, and saves the clones as a clones artifact that CyteArc’s clone analyses read whatever defined the clones.

ds.clones.import_clones imports the clones of one file or several, and ds.clones.from_column takes them from a cell metadata column. ds.clones.load_clones reads the artifact as a Clones record, ds.clones.export_clones writes it as a file that other tools read, and ds.plots.clone_abundance draws how many clones have each size. ds.clones.expansion places each cell in an expansion category by its clone’s size within its sample or other group, ds.clones.diversity and ds.clones.rarefaction measure how many clones each group holds and how evenly its cells spread over them, and ds.clones.overlap and ds.clones.track compare groups by the clones they share and follow chosen clones across them.

cells = ds.snapshot_cell_selection("I")
clones = ds.clones.import_clones("trex_run/clones.txt", cell_selection=cells, barcode_suffix="-1")
table = ds.clones.load_clones(clones)
table.clones.head()  # the largest clones and their cells
ds.plots.clone_abundance(clones, group_by="sample")

Import clones from a file#

import_clones recognises each file’s format from its contents: an HDF5 file is AnnData, a file that starts with %%MatrixMarket is a clone matrix, and a text table is recognised by the names of its columns. format= names the format when the contents leave it open, and columns= reads any other table by the columns it names.

`format`	File	Columns read
`"cospar"`	CoSpar’s clone table	`Cell_ID`, `Clone_ID`
`"trex"`	TREX’s `clones.txt`	`cell_id`, and `clone_nr`, `clone#` or, in the 2022 releases’ CSV, `#clone_id`
`"celltag_multi"`	CellTag-multi’s clone tables, such as the iEP table on Zenodo	`cell.bc`, and `clone_id` or `clone.id`; an unnamed index column and annotation columns are skipped
`"celltagr"`	CellTagR’s clone composition	`cell.barcode`, `clone.id`
`"sticr"`	STICR’s cell metadata on the UCSC Cell Browser	`cell_Id`, `VBC`
`"cassiopeia"`	Cassiopeia’s allele table, one row per cell and target site	`cellBC`, `lineageGrp`; group 0 means no clone, and a group written as `4.0` is group `4`
`"numbat"`	Numbat’s `clone_post_{i}.tsv`	`cell`, `clone_opt`, and `p_opt` as each cell’s `clone_probability`
`"table"`	any delimited table	the columns that `columns` names
`"matrix"`	a Matrix Market clone matrix with its cell list beside it, such as LARRY’s on GEO GSE140802	the cell list’s columns that `columns["cell"]` names
`"cospar_h5ad"`	an AnnData file with CoSpar’s `obsm['X_clone']`	the cells by clones matrix
`"mito"`	an AnnData file with MiTo’s `obs['MiTo_clone']`	`"unassigned"` means no clone
`"h5ad"`	an AnnData file	the `obs` column that `columns["clone"]` names

columns maps "cell", "clone" and "probability" to column names. "cell" may list several columns, whose values joined by "__" form the barcode, as a merged store joins a sample and a barcode. A table whose header holds one field fewer than its rows, as R’s write.table writes it, takes each cell from its row names, as does one whose first column has an empty name, as R’s write.csv writes row names, so a wide table with one clone column per labelling round, such as the celltag_data.txt of the Biddy et al. workflow, needs only columns={"clone": "CellTagD13"} to read one round. Clone labels are text as the file writes them; an empty field, NA, NaN or nan means no clone.

Tables may be plain or gzip-compressed, whatever their names, separated by tabs, commas or spaces, quoted as R quotes them, and without a final newline. A header that starts with #, as TREX writes, is a header, and blank lines hold no rows. A delimiter after the last field of every row also gives a row one field more than its header, an empty one, so a table whose every row ends in its delimiter with nothing after it is refused rather than read either way: name the row names in the header, as R’s write.table(..., col.names=NA) writes them, or end the header with the delimiter too. A row with more fields than the table’s first row raises ValueError, and errors name the lines of the file.

A clone matrix holds a nonzero entry where a cell belongs to a clone. Its cell list is a file beside it: <name>metadata.txt.gz beside <name>clone_matrix.mtx.gz, as GEO ships LARRY’s, or cells.tsv, with clones.tsv naming the clones, as export_clones writes. The axis whose length equals the cell list’s rows holds the cells, so LARRY’s in vitro matrix, cells by clones, and its in vivo and cytokine matrices, clones by cells, read alike; a matrix that fits both ways holds its cells in rows. Clones that no list names, and the columns of CoSpar’s X_clone, are labelled by their position from 0, as SciPy and AnnData number them.

# LARRY's cytokine perturbation set from GEO: the metadata names each cell
# by its library and inDrops barcode.
clones = ds.clones.import_clones(
    "GSM4185644_stateFate_cytokinePerturbation_clone_matrix.mtx.gz",
    cell_selection=cells,
    columns={"cell": ["Library", "Cell barcode"]},
)

Join barcodes to cells#

A cell’s ID is barcode_prefix, the barcode and barcode_suffix, matched exactly. barcode_pattern is a regular expression whose named group barcode keeps the part of each barcode that names the cell, found as re.search finds it; a barcode that the pattern does not match names no cell. Each option takes one value for every file or one per file.

# STICR names cells "<sample>_<barcode>-1"; a store of one sample names them by barcode.
clones = ds.clones.import_clones(
    "meta.tsv",
    cell_selection=sample_cells,
    barcode_pattern=r"^GW15_GZ_6wk_Rep1_1_(?P<barcode>[ACGT]{16}-1)$",
)

Barcodes that match no selected cell, such as droplets that cell calling rejected or cells of another sample, are counted by the cell ID they name and saved in the artifact’s n_file_barcodes, n_unmatched_barcodes and unmatched_barcode_examples attributes. A warning names up to five of them, and an import that reuses the saved clones gives the warning again. ds.perturbation.import_guide_calls joins barcodes by the same rules.

A file is identified by the SHA-256 of its bytes, and a clone matrix also by its lists, so a moved file reuses the saved clones, an edited one gives new clones, and files listed in another order give the same clones. A file listed twice with the same prefix and suffix raises ValueError.

Several files#

Clone IDs are unique only within one run of a tool: TREX numbers its clones from 1 in every run. With several files, each file’s labels therefore start with its barcode_prefix, or, without one, with its name before the first . and _, so equal IDs of two files stay two clones. Two files with the same namespace raise ValueError, as two TREX runs that both write clones.txt do without prefixes.

# One TREX run per sample, named by bare 10x barcodes, in a merged store.
clones = ds.clones.import_clones(
    ["d2/clones.txt", "d4/clones.txt"],
    cell_selection=cells,
    barcode_prefix=["d2__", "d4__"],
    barcode_suffix="-1",
)

Cells in several clones#

A cell listed in two different clones, in one file or in two, raises ValueError that names a few such cells and their clones. multiple="drop" leaves such cells without a clone and counts them in the n_cells_in_several_clones attribute. Rows that repeat a cell’s clone, as Cassiopeia lists a cell once per target site, count the cell once. A selected cell that no file lists has no clone.

Clones from a cell metadata column#

from_column saves the clones that a cell metadata column assigns, such as a column imported with an AnnData file. A missing or blank label means no clone. Integer labels read as their text, and floating-point whole numbers, which pandas writes for integer labels beside missing ones, read as integers; other floating-point labels raise TypeError. The artifact records a fingerprint of the column, so editing it gives new clones.

clones = ds.clones.from_column("clone_id", cell_selection=cells)

Read, export and plot clones#

A clones artifact labels each selected cell with its clone, an empty label for a cell without one, which groupings treat as missing. codes numbers the clones by size, largest first, then by label in natural order, -1 for none, and clone_size holds the cells of each cell’s clone, 0 for none. A source that gives probabilities adds clone_probability, NaN where none is given; a cell’s rows that give it no probability leave the one its other rows give. The store keeps each cell’s label at the width of the longest label, so a very long label costs memory for every cell, and an import whose labels would not fit the memory budget raises MemoryError before it saves anything. ds.select_cells(clones, include=["3"]) selects the cells of one clone, and plots colour cells by clone. Read per-cell values lists the arrays.

load_clones returns a Clones record. Its cells table holds one row per cell, indexed by cell ID: clone, a categorical of the clone labels in rank order, clone_size and, where saved, clone_probability. Its clones table holds one row per clone, largest first: clone and n_cells. Its source names the operation that saved the clones, "import_clones" or "clones_from_column".

export_clones(clones, path) writes CoSpar’s clone table, Cell_ID,Clone_ID, one row per cell with a clone, and format="matrix" writes a cells by clones Matrix Market file with cells.tsv and clones.tsv beside it, replacing files of those names. Both hold this store’s cell IDs and the clone labels, and import_clones reads them back to the same clones; clone probabilities are not written. The table refuses labels that a clone table reads as no clone, such as NA, which the matrix keeps, and the matrix refuses labels and cell IDs that hold a line break, which the table keeps. A path ending in .gz is gzip-compressed. Each returns the number of cells written.

ds.clones.export_clones(clones, "clones.csv")  # for CoSpar
ds.clones.export_clones(clones, "clones/matrix.mtx", format="matrix")

ds.plots.clone_abundance(clones) draws two panels: the number of clones of each size, on log scales, and the top largest clones (default 20). group_by names a cell metadata column, such as the sample or time point, and groups a cell-label artifact, such as a clustering; with either, the first panel draws one line per group, counting each clone’s cells within the group, and the second stacks each clone’s cells by group. Cells without a group are left out, and the plot raises ValueError when no cell with a group has a clone. Its tables are sizes (group, clone_size, n_clones) and top (clone, then the cells of each group, or cells).

Measure clonal expansion#

A clone that holds more than one cell has expanded: its founding cell divided. ds.clones.expansion counts each clone’s cells within groups of cells and places each cell in an expansion category by its clone’s size in its group. The groups are the combinations of the labels of the group_by cell metadata columns, such as the sample, and of a groups label artifact, such as a clustering. Without a grouping, a clone’s cells count in every sample together, as scirpy’s clonal_expansion counts them without expanded_in. A cell without a clone or a label is counted in no group, and when no measured cell has both, every summary raises ValueError and saves nothing.

expansion = ds.clones.expansion(clones, group_by="sample")
table = ds.clones.load_expansion(expansion)
table.categories  # cells and clones of each sample in each category
large = ds.select_cells(expansion, include=["21 to 100 cells", "more than 100 cells"])
ds.plots.clonal_expansion(expansion, group_by="cell_type")

One of two kinds of bounds sets the categories:

Option	Categories	As in
`breakpoints`, the default `(1, 2, 5, 20, 100)`	By the clone’s cells in the group: `"1 cell"`, `"2 cells"`, `"3 to 5 cells"`, `"6 to 20 cells"`, `"21 to 100 cells"` and `"more than 100 cells"`	scirpy’s `breakpoints`, whose categories are `"<= 1"`, `"<= 2"`, `"<= 5"` and so on
`frequency_breakpoints`, such as `(0.0001, 0.001, 0.01, 0.1, 1)`	By the clone’s frequency, its share of the group’s cells with a clone, in bins `(lower, upper]` that end at 1: `"0 to 0.0001"`, `"0.0001 to 0.001"`, `"0.001 to 0.01"`, `"0.01 to 0.1"` and `"0.1 to 1"`	scRepertoire’s `clonalHomeostasis` and `cloneSize`, which name these bins Rare, Small, Medium, Large and Hyperexpanded

A frequency depends on the size of its group: in a sample of 1,300 cells a clone of one cell already has the frequency 0.00077, so the first bin of these bounds stays empty, and a clone of two cells is Medium. Compare frequencies between groups of similar sizes, or use bounds in cells. The names of bounds in cells sort in their order, so groupings and legends that sort labels naturally keep them from the smallest clones to the largest. Frequency bounds are named with six significant digits, or with as many more as tell every bound apart, such as "0.1234561 to 0.1234562".

The artifact labels the cells of cell_selection, or the clones’ cells without one; a selected cell that the clones do not hold has no clone. Its canonical values hold each cell’s category, empty for a cell without a clone or a group, so select_cells keeps the cells of large clones and ds.plots.embedding(layout=umap, color_by=expansion) colours cells by category. group_clone_size, the cells of the cell’s clone in its group, group_clone_frequency, their share of the group’s cells with a clone, and group_codes, the cell’s group, are its other per-cell arrays. The result records the clones, the cells, the label artifact, the bounds and, with group_by, each cell’s group, so measuring again with the same values returns the saved result. The measurement holds about 300 bytes per cell and 2,000 bytes per group at its peak, beside the clones and labels that it reads.

load_expansion returns a ClonalExpansion record with three tables, whose groups are keyed by the group_by columns and by the kind of the groups artifact, such as cluster_labels, and named in group by their labels joined by commas, or "all cells":

Table	Rows	Columns
`cells`	One per cell	`clone`, `category`, `group`, the group’s key, `group_clone_size`, `group_clone_frequency`
`categories`	One per group and category, empty categories included	`group`, the group’s key, `category`, `cells`, `clones`, and `fraction`, the share of the group’s cells with a clone that the category holds
`rank_windows`	One per group and window of ranks	`group`, the group’s key, `window`, `cells`, `fraction`

The labels of the cells table are pandas categoricals, coded by the saved codes rather than held as text per cell; category is ordered, smallest clones first, and the labels of each key column follow their natural order, as the groups do. Groups need one name each, so labels that hold commas and give two groups one name are refused when the expansion is measured, and a column of labels may not share the name of a column of the tables.

The rank windows describe a group’s clonal space as scRepertoire’s clonalProportion does: the group’s clones are ranked by their cells in the group, and each window counts the cells of the clones ranked in it. splits sets the windows; the default (10, 100, 1000, 10000, 30000, 100000) gives scRepertoire’s windows "1 to 10" to "30001 to 100000", and "after 100000" holds the cells that scRepertoire leaves out. Tied clones hold the same cells, so ties do not change the counts. A sample whose ten largest clones hold most of its cells is dominated by a few clones.

ds.plots.clonal_expansion draws one stacked bar per group of the share of its cells in each category, or of its clones with count="clones", as scirpy’s summarize_clonal_expansion counts them, and one stacked bar per group of the share of its cells in each rank window, leaving out windows without cells. The groups are the expansion’s own unless group_by names a cell metadata column or groups a cell-label artifact: each cell keeps the category measured within its sample, so grouping by cell type shows which cell types hold the expanded cells of their samples, and the ranks are counted within the plotted groups. normalize=False draws counts. Its tables are categories and rank_windows.

The categories were validated on the clonotypes of two 10x Genomics immune profiling runs, “Human PBMC from a Healthy Donor, 10k cells (v2)” (6,189 cells with a receptor) and “CD8+ T cells of Healthy Donor 1” (46,526 cells), whose expanded CD8+ T cell clones fill every category. Each run holds one donor, so four pseudo samples and six pseudo clusters were assigned from a hash of each barcode. With clonotypes equal to scirpy’s, the category of every cell equals that of scirpy 0.25.1’s clonal_expansion, within the pseudo samples and across all cells, with the default bounds and with (1, 2), and summarize_clonal_expansion by pseudo cluster, counting cells and clones, equals the plot’s tables. On the T cell clonotypes of the PBMC run, which equal scRepertoire’s CTstrict and CTaa keys, the fractions of frequency_breakpoints=(0.0001, 0.001, 0.01, 0.1, 1) in the four pseudo samples equal scRepertoire 2.8.0’s clonalHomeostasis within 4e-16, and the cells of the rank windows equal its clonalProportion. On a million simulated cells in eight samples, 899,879 of them in 314,595 clones of up to 465 cells, measuring takes about 0.7 s and holds 220 MiB of arrays, about 230 bytes per cell, and loading the tables takes about 0.7 s and 290 MiB.

Measure diversity#

The diversity of a population of cells is how many clones it holds and how evenly its cells spread over them. ds.clones.diversity measures it for groups of cells: the combinations of the labels of the group_by cell metadata columns, such as the sample, and of a groups label artifact, such as a clustering. cell_selection keeps only some cells, such as one cell type. Cells without a clone or a label are left out, and a clone’s size in a group is its cells there. Nothing is saved: the result follows from the clones and the labels in about 0.3 s for a million cells.

result = ds.clones.diversity(clones, group_by="sample")
result.indices  # one row per sample: Shannon entropy, Simpson indices, Chao1, ...
result.hill  # Hill numbers: observed, asymptotic, at a common size and coverage
curves = ds.clones.rarefaction(clones, group_by="sample")
ds.plots.clonal_rarefaction(clones, group_by="sample")

The indices table holds one row per group: group, its labels joined by commas ("all cells" without labels), each label in its key column, and, for its n cells and S clones of frequencies p, f1 of one cell and f2 of two:

Column	Definition	As in
`cells`, `clones`, `singletons`, `doubletons`	`n`, `S`, `f1`, `f2`
`coverage`	The sample coverage, the estimated share of the population’s cells whose clone the sample holds: `1 - f1 / n * (n - 1) f1 / ((n - 1) f1 + 2 f2)`, or with `f1 - 1` and `f2 = 1` without doubletons (Chao and Jost 2012)	iNEXT’s `SC`
`shannon`	`-sum(p log p)`, in nats	scRepertoire’s `shannon`
`pielou`	Shannon entropy over its maximum, `log S`; NaN for one clone	scRepertoire’s `pielou` and `norm.entropy`
`simpson`, `inverse_simpson`, `gini_simpson`	`sum(p**2)`, its inverse, and one minus it	scRepertoire’s and immunarch’s `inv.simpson` and `gini.simpson`
`gini`	The Gini coefficient of the clone sizes, 0 when all are equal	scRepertoire’s and immunarch’s `gini`
`d50`	The fewest clones, largest first, that hold at least half the cells	scRepertoire’s and immunarch’s `d50`
`chao1`	`S + (n - 1) / n * f1*2 / (2 f2)`, or `S + (n - 1) / n f1 (f1 - 1) / 2` without doubletons	iNEXT’s and immunarch’s Chao1
`chao1_bias_corrected`	`S + f1 (f1 - 1) / (2 (f2 + 1))`	vegan’s `estimateR` and scRepertoire’s `chao1`
`ace`	The abundance-based coverage estimate with clones of at most 10 cells rare (Chao and Lee 1992); NaN when every rare clone is a singleton	vegan’s `estimateR` and scRepertoire’s `ace`

Every observed index depends on how many cells a group holds: a larger sample finds more clones. Hill numbers (Hill 1973) put the indices on one scale, the effective number of clones: order 0 counts clones, order 1, the exponential of Shannon entropy, weighs them by frequency, and order 2, the inverse Simpson index, counts the dominant ones. The hill table holds, for each group, named in group and the key columns, and order 0, 1 and 2, the observed Hill number; the estimated asymptotic one of the whole population, Chao1 for order 0, the exponential of the entropy estimate of Chao, Wang and Jost (2013) for order 1, and the inverse of Good’s unbiased Simpson estimate for order 2, infinite when no clone holds two cells; the Hill number at_size, the common size, with its coverage_at_size and method_at_size; and the Hill number at_coverage, the common coverage, with its size_at_coverage and method_at_coverage. Below a group’s size its Hill numbers are rarefied, the expected values of a random subsample, and above it extrapolated, as Chao et al. (2014) derive, with the Shannon extrapolation of Chao et al. (2023) that iNEXT 3.0.2 uses: the curve approaches the asymptote geometrically from the observed value and passes through the rarefied value one cell earlier. A size between whole numbers interpolates linearly, as iNEXT does. By default the common size is the smallest of the groups’ doubled sizes and the common coverage the lowest of their coverages at doubled sizes, iNEXT’s estimateD defaults, so no group is extrapolated beyond twice its size; size and coverage set them. Compare groups by the Hill numbers at a common coverage, or at a common size; a group far smaller than the others sets both defaults low, so pass size or coverage, or leave it out. When every group’s coverage is 1, as when no clone of a group holds a single cell, the common coverage is 1, which each group reaches at its observed size, so the Hill numbers at it are the observed ones.

ds.clones.rarefaction computes each group’s curves: one row per group, order and size, with group and the key columns, the size, method ("rarefaction", "observed" or "extrapolation"), diversity and coverage. By default each group gets iNEXT’s 40 sizes, about half evenly from 1 cell to its size and half from there to twice its size; endpoint and knots change them, and sizes gives every group the same sizes. Compiled loops rarefy each group to all its sizes at once, for every order and the coverage. ds.plots.clonal_rarefaction draws one panel per order, the Hill number against the cells (base="size") or against the sample coverage (base="coverage"), solid where rarefied, dashed where extrapolated, with a point at each group’s observed size, and with base="size" a panel of each group’s coverage against its size. A curve that still rises steeply at its observed point has not reached its plateau: more cells would find more clones. iNEXT’s bootstrap confidence intervals are not computed; compare conditions with each sample’s estimate as one observation, in a test that treats samples as replicates, such as those of CytePool.

The implementation follows the papers rather than the code of iNEXT, which is licensed under the GPL, and validates against its outputs. On two small abundance vectors, the clonotypes of the four pseudo samples of both 10x runs above (11,523 to 11,745 cells each in the CD8+ run), all 46,526 CD8+ cells together, and six edge cases (no singletons, no doubletons, every clone a singleton, a lone singleton, one clone, two cells), the Hill numbers of iNEXT 3.0.2’s curves equal CyteArc’s within 1.4e-12 at every size, rarefied, observed and extrapolated, the coverages within 3e-9, the asymptotic estimates within 7e-13, and estimateD by size within 5e-13 and by coverage within 1e-7, the tolerance of iNEXT’s root finding; the default sizes are iNEXT’s. Where the two differ, CyteArc’s rarefied coverage equals exact rational arithmetic within 1e-16, where iNEXT’s carries errors up to 3e-12, because the chance of missing a clone is computed as a product of ratios rather than from log-gamma functions. The indices equal scRepertoire 2.8.0’s clonalDiversity without bootstraps for all twelve of its metrics, immunarch 0.10.3’s repDiversity (chao1, d50, div, gini, gini.simp, inv.simp and Hill numbers of orders 1 to 6) and vegan 2.7.6’s estimateR (S.chao1, S.ACE), within 6e-14, on the same pseudo samples and edge cases. On the same million simulated cells in eight samples, diversity takes about 0.3 s and rarefaction 0.7 s, holding at most 115 MiB of arrays. Split into 2,000 groups of about 450 cells, such as samples by clusters, the cells take 1.9 s for diversity and 1.3 s for rarefaction.

Memory#

Every summary checks the memory budget before each step that reads or allocates, from the sizes and widths that the stored arrays declare, and raises MemoryError before it reads what would not fit or writes anything. The steps hold about:

Step	Bytes
Reading the clones	12 per cell of the clones, and for `track` and the tables of `load_expansion` 100 and three times the stored width of each clone’s label
Matching the clones to other cells, when `cell_selection` of `expansion` differs from the clones’ cells	56 per cell of the clones and 32 per measured cell, and the read before, below
Grouping	100 per cell, 70 per cell for each key column, and three times the stored width of each `group_by` column’s labels; 32 per cell of `cell_selection`, with the read before, and 130 and three times the label width per cell of the `groups` artifact, whatever cells the clones hold; one chunk of the widest column of labels, and that chunk again with the read before when it reads several columns and labels or a column of several chunks; 500 per distinct label and per group
Measuring expansion	300 per measured cell
Loading expansion	500 per cell and the cells’ IDs as text
Rarefaction	40 for each size below a group’s cells and each of its distinct clone sizes, and 100 per row of the curves
Comparing groups	250 per pair of groups and 24 per entry of the groups-by-groups matrices, 40 per clone of each group, and the clones read
Following clones	100 per row of the table and 16 more per row for each key column, a byte per chosen clone and group, 40 per clone of each group, and the clones and labels read
Each summary	2,000 per group

A grouping with a label for each cell therefore needs far more memory than its cells. A step that reads two cell selections one after the other, the measured cells and the clones’ cells, also counts the read before: the thread of Zarr’s pool that decoded a stored chunk lets go of it only when that thread next runs, which on a busy machine can come after the next read has begun, so the largest stored chunk of the selections and the clones’ codes, as stored and decoded, and 64 KiB for the objects of both reads are held beside the next read. Grouping reads its columns a stored chunk at a time, so when it reads more than one chunk, of several columns and labels or of a column stored in several chunks, it counts the largest chunk again for the read before. A selection stores a flag for each cell of the store, so its chunk can outweigh the cells it selects. Reading cell IDs as text holds one chunk of the stored IDs, by default 100,000 of them, both decoded and as stored, and for each cell three times the stored width of an ID, 128 bytes of positions, and a string of 64 bytes and four per character. Peaks that tracemalloc recorded stayed within these budgets on long and emoji labels, long clone labels, a few measured cells of many clone cells, label artifacts and selections of more cells than the clones hold, a label for each cell, many pairs of groups, clones of one cell each, 13 key columns, and long clone labels held beside many rows.

Compare groups#

Two samples of one animal share the clones that both sampled, such as two time points or two tissues, and two cell types share the clones whose cells took both fates. ds.clones.overlap compares every pair of groups by their clones. The groups are the combinations of the labels of group_by cell metadata columns and of a groups label artifact, among the cells of cell_selection, as for diversity. Cells without a clone or a label are left out, and at least two groups must hold a cell with a clone. Nothing is saved.

result = ds.clones.overlap(clones, group_by="sample")
result.pairs  # one row per pair of samples
ds.plots.clonal_overlap(clones, group_by="cell_type")
tracked = ds.clones.track(clones, group_by="time_point", top=10)
ds.plots.clone_tracking(clones, group_by="time_point", order=["day 2", "day 4", "day 6"])

The pairs table holds one row per pair, group_1 and group_2 named by their labels joined by commas, with these columns, for groups of S1 and S2 clones and clone sizes x and y of X and Y cells:

Column	Definition	As in
`shared`	The clones of both groups	scRepertoire’s `raw`, immunarch’s `public`
`jaccard`	`shared / (S1 + S2 - shared)`	scRepertoire’s and immunarch’s `jaccard`
`sorensen`	`2 shared / (S1 + S2)`	immunarch’s `tversky` with its default weights
`overlap`	`shared / min(S1, S2)`	scRepertoire’s and immunarch’s `overlap`
`cosine`	`shared / sqrt(S1 S2)`, the cosine of presence	scRepertoire’s `cosine`
`morisita_horn`	`2 sum(x y) / ((sum(x^2) / X^2 + sum(y^2) / Y^2) X Y)` (Horn 1966)	scRepertoire’s and immunarch’s `morisita`
`abundance_cosine`	`sum(x y) / sqrt(sum(x^2) sum(y^2))`, the cosine of the clone sizes	immunarch’s `cosine`
`shared_fraction_1`, `shared_fraction_2`	The share of each group’s cells whose clone the other group holds

The first five count clones and grow with the cells of both groups, since larger samples find more of the rare clones that two samples share by chance; the Morisita-Horn index and the cosine of clone sizes weigh clones by their cells, depend little on sample size, and come near 1 when the same expanded clones dominate both groups. group_sizes holds each group’s cells with a clone and its clones; cytearc.clones.OVERLAP_INDICES names the indices. A comparison that the memory budget cannot hold, as the memory table above counts it, raises a MemoryError; fewer groups, from fewer group_by columns or a cell_selection, avoid it. ds.plots.clonal_overlap draws one index as a heatmap of the pairs, "morisita_horn" by default, on a figure at most a page wide, with values printed for at most 12 groups and group names for at most 40; its tables are matrix and pairs, which name every group.

ds.clones.track follows clones across groups, such as time points: by default the top (10) with the most cells in each group, together, as scRepertoire’s clonalCompare chooses them, with ties going to the larger clone overall; labels names them instead. Its cells table holds each chosen clone’s cells and share of the cells with a clone of every group, zeros included, so a clone that expands between time points or spreads between cell types shows it. A table that the memory budget cannot hold raises a MemoryError, which a smaller top or fewer groups avoid. ds.plots.clone_tracking draws one line per clone, labelled with its ID, across the groups, in their natural order or in order, which may name time points as numbers; on a figure at most a page wide, its legend lists at most 80 clones, the largest first, and its axis names at most 40 groups; its table is cells.

The indices were validated on the clonotypes of the two 10x runs above. On the four pseudo samples of CD8+ donor 1, which share about 500 clonotypes per pair and whose expanded clones give a Morisita-Horn index of 0.994 to 0.997, and on PBMC 10k with clonotypes equal to scRepertoire’s CTstrict and CTaa, every index equals immunarch 0.10.3’s repOverlap and scRepertoire 2.8.0’s clonalOverlap within 5e-14, for every pair. scRepertoire’s help page gives abundance-weighted formulas for its overlap, raw and cosine, but its outputs follow the presence formulas above, which CyteArc reproduces. On the same million simulated cells in eight samples, overlap takes about 0.5 s and track 0.6 s, each holding at most 130 MiB of arrays. Split into 2,000 groups of samples by clusters, the cells take 1.9 s and 480 MiB for overlap, which compares 2 million pairs, and 1.1 s and 145 MiB for following the largest clone of each group, 546,000 rows. Each ran within the smallest memory budget it accepts: 187 MiB by sample, and 617 MiB and 302 MiB for the 2,000 groups.

Clones#

class cytearc.clones.Clones(cells, clones, source)#

The clones of a set of cells, read from a clones artifact.

cells#: One row per cell, indexed by cell ids: clone (its label, as a categorical of the labels in rank order, NA without one), clone_size (the cells of its clone, 0 without one) and, when the source gave them, clone_probability (the probability of its clone, NaN without one).

clones#: One row per clone, largest first: clone and n_cells.

source#: The operation that saved the clones, such as "import_clones".

Clonal expansion#

class cytearc.clones.ClonalExpansion(cells, categories, rank_windows, clones, labels, breakpoints, frequency_breakpoints, splits, group_by, groups, cell_selection)#

The clonal expansion of a set of cells, from a clonal_expansion artifact.

The groups are the combinations of the labels of the group_by columns and of the groups artifact; these key columns are each group’s key in the tables.

cells#: One row per cell, indexed by cell ids: clone, category (its expansion category, NA without a clone or a group), group (the group’s labels joined by commas), the group’s key, group_clone_size (the cells of its clone in its group, 0 without one) and group_clone_frequency (their share of the group’s cells with a clone, NaN without one). The labels are categoricals, and category is ordered, smallest first.

categories#: One row per group and category, every category of every group: group, the group’s key, category, cells, clones, and fraction, the share of the group’s cells with a clone that the category holds.

rank_windows#: One row per group and rank window: group, the group’s key, window, the cells that the group’s clones ranked in the window hold, and their fraction of the group’s cells.

clones#: The clone artifact, such as a clones artifact.

labels#: The categories, smallest clones first.

breakpoints#: The bounds in cells, or None with frequency bounds.

frequency_breakpoints#: The frequency bounds, or None with bounds in cells.

splits#: The last rank of each window but the last.

group_by#: The cell metadata columns of the groups.

groups#: The label artifact of the groups, or None.

cell_selection#: The cells measured, which the artifact labels: the selection given, or the clones’ cells.

Clonal diversity#

class cytearc.clones.ClonalDiversity(indices, hill, clones, group_by, groups, cell_selection, size, coverage)#

The diversity of the clones of groups of cells.

The groups are the combinations of the group_by columns’ labels and the groups artifact’s labels; each table names a group in group, its labels joined by commas, and holds its labels in the key columns.

indices#: One row per group: group, the key columns and the columns of indices().

hill#: One row per group and order 0, 1 and 2: group, the key columns, order, observed and estimated (asymptotic) Hill numbers, the Hill number at_size of the common size, with its coverage_at_size and method_at_size, and the Hill number at_coverage of the common coverage, with its size_at_coverage and method_at_coverage.

clones#: The clone artifact, such as a clones artifact.

group_by#: The cell metadata columns of the groups.

groups#: The label artifact of the groups, or None.

cell_selection#: The cells kept, or None for every cell.

size#: The common size in cells.

coverage#: The common coverage.

class cytearc.clones.Rarefaction(curves, clones, group_by, groups, cell_selection, orders)#

The rarefaction and extrapolation curves of groups of cells.

curves#: One row per group, order and size: group (its labels joined by commas), the key columns, order, size (cells), method ("rarefaction", "observed" or "extrapolation"), diversity (the Hill number) and coverage.

clones#: The clone artifact, such as a clones artifact.

group_by#: The cell metadata columns of the groups.

groups#: The label artifact of the groups, or None.

cell_selection#: The cells kept, or None for every cell.

orders#: The orders of the curves.

Clone overlap#

class cytearc.clones.CloneOverlap(group_sizes, pairs, clones, group_by, groups, cell_selection)#

The clones that groups of cells share.

The groups are the combinations of the group_by columns’ labels and the groups artifact’s labels; each table names a group by its labels joined by commas.

group_sizes#: One row per group: group, the key columns, cells and clones.

pairs#: One row per pair of groups: group_1 and group_2, the indices shared, jaccard, sorensen, overlap, cosine, morisita_horn and abundance_cosine, and shared_fraction_1 and shared_fraction_2, the share of each group’s cells whose clone the other holds.

clones#: The clone artifact, such as a clones artifact.

group_by#: The cell metadata columns of the groups.

groups#: The label artifact of the groups, or None.

cell_selection#: The cells kept, or None for every cell.

class cytearc.clones.CloneTracking(cells, clones, tracked, group_by, groups, cell_selection)#

The cells of chosen clones in each group of cells.

cells#: One row per chosen clone and group, zeros included: clone, group (its labels joined by commas), the key columns, cells and fraction, the share of the group’s cells with a clone.

clones#: The clone artifact, such as a clones artifact.

tracked#: The chosen clone labels, largest first.

group_by#: The cell metadata columns of the groups.

groups#: The label artifact of the groups, or None.

cell_selection#: The cells kept, or None for every cell.