Held-out prediction

Held-out prediction#

held_out_prediction() leaves each study of a CubeCollection out in turn, chooses genes up and down in each cell type from the results of the other studies, by SumRank and by a random-effects meta-analysis, scores every case and control of the held-out study by how high its profiles rank those genes, as UCell scores them, and gives the area under the ROC curve of cases against controls, by study and method, as Nakatsuka et al. (2025) compare methods. plot_held_out_auc() draws the AUC of every held-out study by method.

Prediction of case status in held-out studies, from genes chosen in the others.

Genes found by combining studies should separate cases from controls in a study that had no part in finding them. held_out_prediction() measures this as the paper that introduced SumRank does (Nakatsuka et al. 2025): each study of a collection is left out in turn; genes up and genes down in each cell type are chosen from the other studies, by SumRank (sumrank()) and, for comparison, by a random-effects meta-analysis (meta_analysis()); every sample of the held-out study is scored by how high its profiles rank those genes; and the area under the ROC curve (AUC) of the score says how well it separates the study’s cases from its controls. plot_held_out_auc() draws the AUC of every held-out study by method.

Choosing the genes:

In every group (cell type), from the studies other than the held-out one whose results test the group and contrast, and among the genes that every one of them tested, n_genes genes up and n_genes down, the same number for each method, as the paper compares methods by gene sets of one size:

  • "sumrank": the genes with the smallest up_statistic, and those with the smallest down_statistic, below the cap, with the given proportion of the studies summed; ties in the collection’s order of the genes. Every gene has the same numbers of studies and ranks, so this is also the order of the permutation p-values, which therefore need no permutations here.

  • "meta_analysis": the genes with a positive estimate and the smallest p-values, and those with a negative one, from the random-effects meta-analysis of the studies’ log2 fold changes and standard errors with REML and the Knapp-Hartung adjustment (method="reml", test="knapp_hartung"), as the paper ran metagen (method.tau = "REML", hakn = TRUE); ties in the collection’s order.

Scoring a sample:

The genes of a sample’s profile in a group, its counts summed over the cells, are ranked from the highest count, tied genes sharing their average rank, among all the genes the held-out study measured. A set of n of these genes, those of a chosen set that the study measured, has UCell’s score (Andreatta and Carmona 2021), 1 - U / (n G), with U the sum of the set’s ranks minus n (n + 1) / 2 and G the number of genes ranked: the maximum rank is the number of genes, as the paper set it to rank every gene (maxRank = 16000). Then, as the authors’ code does:

  1. In each group, the scores of the genes up, and those of the genes down, are scaled over the held-out study’s samples to [0, 1], minus their smallest value and divided by their range (0 for all when they are equal), which makes the score’s own scale irrelevant.

  2. A sample without a profile in the group, that is not among the samples of the held-out study’s fit of the group, takes the mean of the other samples’ scaled scores, and every sample takes 0 when none has one, as for a set without a gene the study measured.

  3. A sample’s score is the sum over the groups of its scaled scores of the genes up, minus the sum of its scaled scores of the genes down.

The AUC:

The probability that a case scores higher than a control, ties counting one half: the Mann-Whitney statistic divided by the number of pairs of a case and a control (Hanley and McNeil 1982). The samples are those of the held-out study’s design whose value of variable is case or control. 0.5 says the score does not separate them, and below 0.5 the genes predict the wrong direction; the AUC is NaN, with the reason, for a study without a case or without a control.

Deliberate differences from the paper:

  • The paper fits a logistic regression of case status on the score in the other studies and takes the AUC of its predictions in the held-out study from pROC’s auc, whose default direction = "auto" compares the medians of the held-out cases and controls and takes the direction in which the cases score higher (Robin et al. 2011). A logistic regression on one score is monotone in it, so that AUC is the score’s in the direction pROC chooses, which puts a score that ranks controls above cases above 0.5 too. Here the cases are expected to score higher, as genes up minus genes down should make them, so the logistic regression is not needed.

  • The profiles are the summed counts, which rank the genes of a sample as their mean over its cells does; the paper ranked Seurat’s average of the cells’ normalized expression, which weighs each cell’s counts by its depth.

Not done:

The proportion of studies is given, not chosen: the paper chose it as the one whose genes predict best, which a loop over proportions gives. The paper’s prediction of disease severity in cases (a linear regression on the score) is not done.

References

Nakatsuka N, Adler D, Jiang L, Hartman A, Cheng E, Klann E, Satija R (2025). Improving reproducibility of differentially expressed genes in single-cell transcriptomic studies of neurodegenerative diseases through meta-analysis. Nature Communications 16, 7436. doi:10.1038/s41467-025-62579-z

Andreatta M, Carmona SJ (2021). UCell: robust and scalable single-cell gene signature scoring. Computational and Structural Biotechnology Journal 19, 3796-3798. doi:10.1016/j.csbj.2021.06.043

Hanley JA, McNeil BJ (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36. doi:10.1148/radiology.143.1.7063747

Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Mueller M (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12, 77. doi:10.1186/1471-2105-12-77

class cytepool.held_out_prediction.HeldOutPrediction(auc, scores, genes, provenance)#

The prediction of case status in every held-out study, by method.

Built by held_out_prediction(). Three tables, the studies in the collection’s order and the methods in the order "sumrank", "meta_analysis":

  • auc: one row per held-out study and method: study, method, auc (NaN where it cannot be computed), n_cases, n_controls, n_groups (the groups whose genes scored the study), n_genes_up and n_genes_down (the genes chosen, summed over the groups, that the study measured) and reason (why the AUC is NaN; empty otherwise);

  • scores: one row per held-out study, method and sample of its design that is a case or a control: study, method, sample_id, case (whether it is a case) and score;

  • genes: the genes chosen for each held-out study from the other studies: study (held out), method, group_id, direction ("up" or "down"), rank (from 1) and feature_id.

Immutable.

Raises:

  • TypeError – If a table is not a Table, or provenance not a mapping that JSON can write.

  • ValueError – If provenance holds a NaN or an infinity.

Explained in: Meta-analysis

property auc#

The AUC of every held-out study and method.

Explained in: Meta-analysis

property scores#

The score of every case and control of every held-out study, by method.

Explained in: Meta-analysis

property genes#

The genes chosen from the other studies for every held-out study.

Explained in: Meta-analysis

property settings#

The settings, as a new dict (also in the provenance).

property provenance#

A JSON-serialisable record of what produced every number, as a new dict.

It holds the CytePool, numpy, numba and Python versions, the settings, and the provenance of every study’s differential-expression result, by study name, which identifies its cube by its fingerprint and gives its design and the samples of every group’s fit.

cytepool.held_out_prediction.held_out_prediction(collection, results, contrast, variable, case, control, *, groups=None, n_genes=50, proportion_of_studies=1.0)#

Leave each study out, choose genes from the others, and predict its cases.

For every study of the collection in turn, genes up and down in each group are chosen from the results of the other studies, by SumRank and by a random-effects meta-analysis; each case and control of the held-out study is scored by how high its profiles rank those genes, as UCell scores them; and the AUC says how well the scores separate cases from controls. The module description gives each step. The results are those that differential_expression_sumrank() takes: each study’s analysis of its aligned cube.

Parameters:

  • collection – The collection whose aligned cubes the studies’ results analysed.

  • results – The differential-expression result of every study of the collection, by study name, from limma_voom() or limma_trend(), whose tables hold the standard errors of the meta-analysis.

  • contrast – The contrast that the studies tested, cases against controls.

  • variable – The variable of the studies’ designs that tells cases from controls, such as "condition".

  • case – Its value for a case, such as "AD".

  • control – Its value for a control.

  • groups – A group id or a list of them, whose scores are summed; by default every group of the collection. A group that no other study tested gives a held-out study no genes.

  • n_genes – The number of genes chosen up, and down, in each group, by each method; at least 1.

  • proportion_of_studies – The proportion of the studies summed by SumRank, as for sumrank().

Returns:

The AUC of every held-out study and method, the score of every sample, and the genes chosen.

Raises:

  • TypeError – If an argument has the wrong type, or is or holds a numpy masked array.

  • ValueError – If results does not give the result of every study of the collection or a result did not analyse its study’s aligned cube, the contrast is an F-test, the results have no standard errors (dream’s), a study’s design has no variable variable, case equals control, a group is listed twice, or an option is invalid.

  • KeyError – If a group is not a group of the collection, or no study tested contrast.

Used in: Replication across studies

Explained in: Meta-analysis