Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Lineage tracing

Score a fate map computed from expression against the fates that sister cells took, in the LARRY experiment of Weinreb et al. (2020): 130,887 mouse hematopoietic cells profiled on days 2, 4 and 6 of a culture started from barcoded progenitors. The page imports the authors’ clones, measures their sizes, finds which fates share clones and labels each day-2 progenitor by the fates that its clone took. Fate probabilities from expression alone separate the clones that went mostly to neutrophils from those that went mostly to monocytes with an AUROC of 0.78. The AUROC is 0.68 for the quarter of progenitors earliest in pseudotime and 0.87 for the latest quarter, so expression predicts the fates of the earliest progenitors only weakly.

Cells that look alike can be heading for different fates. A single-cell transcriptome is a snapshot, so trajectory methods infer where a cell is heading from its place among other cells, and nothing in the snapshot shows what the cell became. Lineage tracing marks cells with a heritable barcode and profiles their descendants later; cells that share a barcode form a clone.

LARRY barcodes (Lineage And RNA RecoverY) are random 28-base sequences in the 3’ UTR of an enhanced green fluorescent protein gene, delivered by lentivirus and read with each cell’s transcriptome. Weinreb et al. (2020) barcoded mouse hematopoietic progenitors, sorted as Lin⁻Sca1⁺Kit⁺ (LSK) and Lin⁻Sca1⁻Kit⁺ cells, and cultured them for two days, during which they divided three times on average. They profiled half of the cells by single-cell RNA sequencing on day 2, and replated the rest and profiled them on days 4 and 6. A clone’s day-2 cells and its later cells descend from one barcoded cell, so the later cells show which fates the day-2 cells’ sisters took, the closest available measure of where a day-2 cell was heading.

This page is one of four on clonal analysis with ds.clones, each with its own source of clones. Here lentiviral barcodes mark the clones, and three time points follow them through hematopoiesis. The other three take their clones from phylogenetic trees reconstructed from prime-edited target sites in a mouse embryo (Lineage recorders), from somatic mitochondrial DNA mutations in a human tumor (Mitochondrial lineage tracing), and from the receptor chains of T and B cells (Immune repertoires).

Download the experiment

GEO series GSE140802 holds the in vitro experiment as sample GSM4185642: each cell’s metadata, the clone matrix, the gene names and the expression matrix of 130,887 cells and 25,289 genes. The expression matrix, 2.1 GB of normalized values rather than counts, takes most of the download time. GEO’s server can time out when it is busy, so each download resumes where a failed attempt stopped.

The example keeps the downloads and the store, about 7 GB together, in a temporary directory that Python deletes when the kernel stops. To keep them, or where the temporary directory is held in memory, set output_dir to a directory on disk.

Started: 2026-11-13T17:03:59.220432Z | Wall time: 6.151 s

metadata 2.22 clones 0.20 genes 0.07 expression 2065.70 Name: Downloaded MB, dtype: float64

Started: 2026-11-13T17:04:05.380076Z | Wall time: 72.722 s

Build the store

The metadata holds one row per cell, in the same cell order as both matrices: its inDrops library, cell barcode, day, sorted starting population, the authors’ cell type annotation, culture well and position in the authors’ SPRING layout, a two-dimensional drawing of their graph of the cells’ expression. Cell barcodes repeat across libraries, so name each cell by its library and barcode joined by __, as a merged store names cells.

Day246All
Cell type annotation
Baso1873247672110155
Ccr7_DC5710084241
Eos15189351555
Erythroid100153433686
Lymphoid144583130857
Mast4057919652584
Meg9763213672096
Monocyte95075051072719182
Neutrophil106867211444822237
Undifferentiated25575287271789172193
pDC166223101
All282494849854140130887

Started: 2026-11-13T17:05:18.122669Z | Wall time: 0.404 s

On day 2, 25,575 of the 28,249 cells are undifferentiated; by day 6, 17,891 of 54,140 are, and neutrophils (14,448), monocytes (10,727) and basophils (6,721) are the largest mature types.

CyteArc’s Matrix Market reader reads counts, while this matrix holds normalized values, so read its entries with pandas, a chunk at a time, and write the store with SparseToZarr. The file lists its entries gene by gene, so gather them into a matrix with cells as rows before writing it.

Cells 130887 Genes 25289 Nonzero values 176867560 Smallest cell total 2878.75 Largest cell total 2878.75 Name: Expression matrix, dtype: object

Started: 2026-11-13T17:05:18.530063Z | Wall time: 86.301 s

Every cell’s values sum to 2,878.75, so the 177 million nonzero values are each cell’s counts scaled to one common total. values="normalized" records that the assay holds such values, so the store refuses steps that need counts; CyteArc would reach the same decision from the values if the argument were left out.

SparseToZarr holds the whole matrix in memory while it writes the store, shard by shard. When CyteArc’s default count layout does not fit the memory budget, an import stops before it writes and names the largest CountMatrixPolicy that fits. policy="fit" keeps the default layout where it fits and otherwise writes that smaller layout instead of stopping. Smaller shards make later gene-wise reads slower, so on a machine with more memory, give the import a larger mem_budget instead.

Count shards of 1,000 MB within a memory budget of 6.0 GiB; the store takes 5.1 GB on disk
DataStore has 130887 (130887) cells with 1 assays: RNA Cell metadata: 'I', 'ids', 'names', 'RNA_nCounts', 'RNA_nFeatures', 'RNA_percentMito', 'RNA_percentRibo', 'SPRING1', 'SPRING2', 'cell_type', 'day', 'library', 'starting_population', 'well' RNA assay has 25289 features and following metadata: 'I', 'ids', 'names', 'dropOuts', 'nCells'

Started: 2026-11-13T17:06:44.847096Z | Wall time: 72.743 s

With a 6.0 GiB memory budget, the default layout fits this matrix: the import wrote count shards of 1,000 MB, and the store takes 5.1 GB on disk.

Color the SPRING layout by annotation and by day to see where each day’s cells sit among the cell types.

Two SPRING layouts of 130,887 cells. Left, by the authors' annotation: undifferentiated cells fill the center, neutrophils curve along the bottom right, monocytes extend to the right, basophils, eosinophils and mast cells branch at the bottom left, megakaryocytes and erythroid cells on the left, and lymphoid and dendritic cells sit at the top. Right, by day: day-2 cells gather at the top center, and day-4 and day-6 cells fill the branches.

Started: 2026-11-13T17:07:57.594594Z | Wall time: 2.099 s

Day-2 cells sit at the top of the layout, among the undifferentiated progenitors, and day-4 and day-6 cells fill the branches of the mature types: neutrophils and monocytes on the right, basophils, eosinophils and mast cells at the bottom left, megakaryocytes and erythroid cells on the left, and dendritic and lymphoid cells at the top.

Import the clones

The clone matrix holds a 1 where a cell belongs to a clone, with its cells in the order of the metadata, so import_clones reads the metadata beside the matrix as its list of cells, and columns names the two columns that make each cell’s ID. The clones join a frozen cell selection, an immutable record of the cells that an analysis used, so that a later change to the store’s cell columns cannot change what the clones refer to.

Cells in the clone matrix 130887 Cells that name no cell of the store 0 Cells with a clone 49302 Name: Clone import, dtype: int64

Started: 2026-11-13T17:07:59.697424Z | Wall time: 1.214 s

Clones 5864 Smallest clone 2 Median clone 4.0 Largest clone 177 Name: Cells, dtype: object

Started: 2026-11-13T17:08:00.915284Z | Wall time: 0.120 s

All 130,887 cells of the clone matrix joined the store, and 49,302 of them, 38%, belong to one of 5,864 clones of 2 to 177 cells. The GEO deposit lists only clones of two or more cells, so a cell without a clone may be the only sampled cell of its clone.

Measure clone sizes on each day

A clone’s cells on one day are a sample of its descendants on that day, so comparing the days shows how the clones expand.

Left, the number of clones of each size on days 2, 4 and 6, on log scales: the day-2 curve falls steeply and ends at 8 cells, while day-4 and day-6 clones reach 56 and 157 cells. Right, the 20 largest clones, of about 90 to 177 cells, as bars split by day, almost all of their cells from days 4 and 6.

Started: 2026-11-13T17:08:01.038675Z | Wall time: 0.998 s

ClonesOf one or two cells (%)Largest
Day
2267284.68
4410153.556
6395636.6157

Started: 2026-11-13T17:08:02.040843Z | Wall time: 0.152 s

On day 2, 85% of the 2,672 clones hold one or two sampled cells, and the largest holds 8. The share of such small clones falls to 54% on day 4 and 37% on day 6, and the largest holds 56 and then 157 cells; most cells of the 20 largest clones come from day 6.

Find the fates that share clones

Fate coupling measures how often two cell types share clones. For each clone, clones.coupling takes each type’s share of the clone’s cells, after scaling each type by its number of cells with a clone, and measures the cosine between two types’ shares across the clones. This index is CoSpar’s default Wang et al., 2022, and it counts every clone once whatever its size. Each pair also gets a z-score against a null that shuffles the types among the cells with a clone. Measure the coupling among the day-6 cells of the ten mature types, and draw it under a dendrogram that joins first the types that couple most strongly.

Heatmap of the cosine coupling of ten mature cell types on day 6, under an average-linkage dendrogram. Monocytes with neutrophils (0.067), erythroid cells with megakaryocytes (0.062) and basophils with eosinophils (0.042) couple most; the dendrogram joins neutrophils with monocytes, puts basophils, eosinophils, mast cells, erythroid cells and megakaryocytes in a second branch, and Ccr7 dendritic cells, lymphoid cells and plasmacytoid dendritic cells in a third.

Started: 2026-11-13T17:08:02.196401Z | Wall time: 4.545 s

couplingnull_meannull_sdz_score
group_1group_2
MonocyteNeutrophil0.0670.2170.007-20.483
ErythroidMeg0.0620.0540.0071.217
BasoEos0.0420.0240.0028.035
ErythroidMast0.0240.0540.006-5.214
BasoMast0.0220.1170.006-15.993
Neutrophil0.0200.1990.007-24.595

Started: 2026-11-13T17:08:06.745658Z | Wall time: 0.021 s

Pairs of mature types 45 Pairs that couple more than their null 2 Pairs that couple less than their null 43 Clones of two or more day-6 mature cells 2055 Clones of one mature type 1455 Name: Count, dtype: int64

Started: 2026-11-13T17:08:06.769556Z | Wall time: 0.037 s

Neutrophils with monocytes couple most strongly (0.067), then erythroid cells with megakaryocytes (0.062) and basophils with eosinophils (0.042). The dendrogram joins the fates in three branches: neutrophils with monocytes; basophils, eosinophils, mast cells, erythroid cells and megakaryocytes; and the dendritic and lymphoid cells. Weinreb et al. (2020) found that the clones that cell types share in culture follow how close the types are in expression (r = 0.93), and that hierarchies built from either differ only in where mast cells branch.

Only basophils with eosinophils share clones more than chance would give, 8.0 standard deviations above their null mean; erythroid cells with megakaryocytes sit 1.2 standard deviations above theirs, within the null’s spread. The other 43 pairs couple less than their null: by day 6, 1,455 of the 2,055 clones with two or more mature cells hold a single mature type, while shuffling spreads each clone over several types. Neutrophils with monocytes couple at 0.067 against a null mean of 0.217, 20.5 null standard deviations below it. The z-scores here mostly measure how committed the clones are, so this page ranks pairs by the cosine. Where cuts whose clones differ in size are compared, as in Lineage recorders, or where pairs of similar coupling differ in how far they beat chance, as in Mitochondrial lineage tracing, the z-score is the value to compare, and those pages draw it.

Label each progenitor by its clone’s fates

fate_bias gives each early cell the share of its clone’s late cells in each fate and its dominant fate, the fate that holds the most of them. Take the undifferentiated day-2 cells as the early cells, and the day-4 and day-6 cells of the mature types as the late cells, whose types are the fates.

Day-2 progenitors 25575 With a clone 4199 Clone without late cells 2219 Clone's late cells tie between fates 59 With a dominant fate 1921 Name: Cells, dtype: int64

Started: 2026-11-13T17:08:06.809558Z | Wall time: 1.213 s

Dominant fate Neutrophil 730 Monocyte 632 Baso 259 Meg 115 Lymphoid 87 Mast 45 Erythroid 20 Eos 15 Ccr7_DC 11 pDC 7 Name: Day-2 progenitors, dtype: int64

Started: 2026-11-13T17:08:08.026004Z | Wall time: 0.012 s

Stacked bars of the late cells of the 40 of 3,516 clones with the most late cells, colored by mature type and grouped by dominant fate: basophil and mast cell clones, one megakaryocyte clone, then monocyte and neutrophil clones, many of which mix the two types. A panel above counts each clone's day-2 progenitors: none for most of these clones, and at most two.

Started: 2026-11-13T17:08:08.040946Z | Wall time: 0.961 s

Of the 25,575 day-2 progenitors, 4,199 have a clone. The clones of 2,219 of them have no late cells and those of 59 tie between fates, which leaves 1,921 progenitors with a dominant fate. Neutrophils or monocytes dominate the clones of 1,362 of them, 71%, and basophils those of 259. Among the 40 clones with the most late cells, many mix two types, mostly neutrophils with monocytes or basophils with mast cells. Most of these clones have no sampled day-2 progenitor, so the largest clones add few progenitor labels. A clone with few late cells gives a noisy share of each fate.

Compute fate probabilities from expression

A fate map computed from expression predicts the same outcomes without a barcode, and the clones test how well it does. The map walks a neighbor graph of the cells, which the RNA recipe, ds.pipeline.rna, builds: the recipe selects highly variable genes, normalizes each cell to a common total and takes logs, reduces the genes with PCA and connects each cell to its nearest neighbors. Normalizing values already scaled to one total gives the values that the counts would give. Passing the frozen selection of every cell, rather than a cell column, skips the recipe’s quality filter, so the graph keeps every cell that the authors kept. Doublet scoring needs counts, which the deposit lacks, so turn it off, along with the cell-cycle scores, UMAP, clusters and markers, which this page does not use.

Started: 2026-11-13T17:08:09.005775Z | Wall time: 111.263 s

Pseudotime orders the graph from source cells to sink cells by population balance analysis Weinreb et al., 2018, and fate probabilities give each cell the chance that a walk along the graph, moving mostly forward in pseudotime, reaches each sink first, as the fate mapping tutorial explains. Take the day-2 progenitors as the source, sharing a total weight of -1, and the cells of the ten mature types as the sinks, sharing +1.

Graph cells 130887 Scored cells 130887 Name: Cells, dtype: int64

Started: 2026-11-13T17:10:00.272383Z | Wall time: 18.570 s

All 130,887 cells lie in one connected graph and have fate probabilities. The mature cells are the sinks, so their probabilities follow from the choice of sinks; the progenitors, which no sink fixes, are the cells to score.

Score the predictions against the clones

Neutrophils or monocytes dominate the clones of most progenitors with a dominant fate, and Weinreb et al. (2020) benchmarked fate prediction methods on the choice between them. Give each progenitor two neutrophil shares. The predicted share is its probability of reaching neutrophils divided by its probability of reaching neutrophils or monocytes; the traced share is the fraction of neutrophils among its clone’s late neutrophils and monocytes. Draw both for the progenitors whose clones went mostly to neutrophils or to monocytes.

Two SPRING layouts of the 1,362 day-2 progenitors whose clones went mostly to neutrophils or to monocytes, colored from 0 to 1. Left, the traced neutrophil share, mostly 0 or 1, with progenitors of both kinds of clone side by side at the top. Right, the predicted neutrophil share, near one half at the top and rising toward 1 at the bottom, toward the neutrophils.

Started: 2026-11-13T17:10:18.846258Z | Wall time: 14.101 s

Most traced shares are 0 or 1, since most clones went to one of the two types. At the top of the layout, the progenitors of neutrophil clones and of monocyte clones sit side by side, and the predicted shares there stay near one half. The predictions move toward 1 only for progenitors further down, toward the neutrophils, and toward 0 for the few on the right, toward the monocytes.

Score the predictions by clone, since a clone’s progenitors share one traced outcome: average the predicted shares of each clone’s progenitors, and measure how well the averages separate the clones that went mostly to neutrophils from those that went mostly to monocytes by the area under the ROC curve (AUROC), 0.5 for chance and 1 for a perfect separation. Resample the clones for a 95% interval.

Day-2 progenitors 1362 Clones 916 AUROC 0.78 Lower 95% 0.75 Upper 95% 0.81 Name: Neutrophil or monocyte clones, dtype: object

Started: 2026-11-13T17:10:32.950769Z | Wall time: 4.454 s

The mean predicted shares of the 916 clones of these 1,362 progenitors separate the two outcomes with an AUROC of 0.78 (95% interval 0.75 to 0.81): in 78% of the pairs of a neutrophil clone and a monocyte clone, the neutrophil clone has the higher mean predicted share.

Follow the separation through pseudotime

Progenitors early in pseudotime sit closer to the source and further from every fate. Split the scored progenitors into quarters by pseudotime. In each quarter, compare the predicted shares of progenitors from neutrophil clones with those from monocyte clones, and score the quarter’s clones. Then draw the shares of each quarter beside its ROC curve.

PseudotimeMedian share, NeutrophilMedian share, MonocyteDay-2 progenitorsClonesAUROCLower 95%Upper 95%
Quarter
First0.01 to 0.150.510.493412870.680.620.74
Second0.15 to 0.200.550.543402800.670.600.73
Third0.20 to 0.280.670.603402430.790.730.84
Fourth0.28 to 0.640.840.543412360.870.820.92

Started: 2026-11-13T17:10:37.407910Z | Wall time: 17.698 s

Two panels for the 1,362 day-2 progenitors of neutrophil and monocyte clones. Left, split violins of the predicted neutrophil share in each pseudotime quarter, purple for the progenitors of neutrophil clones and pink for those of monocyte clones: in the first and second quarters both halves sit in a narrow band near 0.5, in the third the purple half lies above the pink one, and in the fourth the purple half sits mostly between 0.8 and 1, around a median of 0.84, while the pink half spreads from near 0 to near 1 around a median of 0.54. Right, the ROC curve over the clones of each quarter, from light blue for the first to dark blue for the fourth, above a dashed diagonal for chance: the first and second curves run modestly above the diagonal, with AUROCs of 0.68 and 0.67, and the third and fourth rise more steeply, with 0.79 and 0.87.

Started: 2026-11-13T17:10:55.109444Z | Wall time: 1.347 s

In the earliest quarter, the predicted shares of the two kinds of progenitor barely differ, with medians of 0.51 and 0.49, yet they still rank the clones better than chance, with an AUROC of 0.68 (95% interval 0.62 to 0.74); the second quarter scores about the same, 0.67. The separation grows in the later two quarters: the medians reach 0.67 and 0.60 in the third (AUROC 0.79) and 0.84 and 0.54 in the fourth (AUROC 0.87, 95% interval 0.82 to 0.92). Most of the gain comes from the progenitors of neutrophil clones, whose median share rises from 0.51 in the first quarter to 0.84 in the fourth. The median share of the progenitors of monocyte clones stays between 0.49 and 0.60, and in the fourth quarter their shares spread from near 0 to near 1. The fate map recognizes the progenitors bound for neutrophils once they have progressed in pseudotime, and separates the fates of the earliest progenitors only weakly.

Weinreb et al. (2020) found a similar pattern with population balance analysis, WaddingtonOT and FateID. Their predictions broadly agreed with the clonal fates as cells began to mature, but generally missed the split between neutrophil and monocyte potential among the earliest (Cd34⁺) progenitors, where their correlation with the traced shares stayed below r = 0.26, against r = 0.5 for predictions from held-out clonal data. The two analyses score differently, by an AUROC over clones here and a correlation over cells there, and they choose the earliest progenitors differently, by pseudotime here and by Cd34 expression there, so they agree in direction rather than in their numbers.

The clones API reference describes every clone summary used here, the import of clones from other lineage tracing tools, and clones called from barcode reads.

References
  1. Weinreb, C., Rodriguez-Fraticelli, A., Camargo, F. D., & Klein, A. M. (2020). Lineage tracing on transcriptional landscapes links state to fate during differentiation. Science, 367(6479), eaaw3381. 10.1126/science.aaw3381
  2. Wang, S.-W., Herriges, M. J., Hurley, K., Kotton, D. N., & Klein, A. M. (2022). CoSpar identifies early cell fate biases from single-cell transcriptomic and lineage information. Nature Biotechnology, 40(7), 1066–1074. 10.1038/s41587-022-01209-1
  3. Weinreb, C., Wolock, S., Tusi, B. K., Socolovsky, M., & Klein, A. M. (2018). Fundamental limits on dynamic inference from single-cell snapshots. Proceedings of the National Academy of Sciences, 115(10). 10.1073/pnas.1714723115