Replication across studies

Replication across studies#

differential_expression_replication() reads one cell type and contrast of the studies of a CubeCollection, lined up by align_study_results(), and gives for every gene its combined p-value, its partial conjunction p-value of an effect in at least u studies in one direction, the studies that find it and the heterogeneity of its log2 fold changes; for every pair of studies, pi1 of one study’s p-values among the genes the other finds, and their sign agreement; and the number of genes found in at least k studies. plot_study_agreement() and plot_replicated_genes() draw them.

Replication of differential expression across studies.

Most genes that one study finds are not found in the next (Nakatsuka et al. 2025), so a result of several studies says how well they agree next to what they find together. differential_expression_replication() reads the differential expression of one group (cell type) and contrast in every study of a CubeCollection, lined up gene by gene by align_study_results(), and gives for every gene the p-value combined over the studies that tested it, the partial conjunction p-value that the effect is found in at least u studies in the same direction (Benjamini and Heller 2008), how many studies find it and in which direction, and the heterogeneity of its log2 fold changes; for every pair of studies, pi1 of one study’s p-values among the genes the other finds (Storey and Tibshirani 2003) and their sign agreement; and the number of genes found in at least k studies. plot_study_agreement() and plot_replicated_genes() draw them.

The genes:

A study tested a gene when its table of the group and contrast has it; one that did not measure it, measured and left it out, or did not analyse the group is left out of that gene’s numbers, and align_study_results() says which. Each study’s two-sided p-values become one-sided ones by the sign of its log2 fold change (directions of combine_p_values()), so that the combined and partial conjunction p-values are two-sided and effects in opposite directions count against each other. Stouffer’s method, the default, weighs each study by the square root of the number of samples its fit of the group used (Zaykin 2011). A study finds a gene when its adjusted p-value (Benjamini and Hochberg within the study’s group) is below alpha. Both p-values are adjusted by Benjamini and Hochberg’s procedure over the genes that have them.

Agreement between studies:

For an ordered pair of studies, the discovery and the replication, the genes the discovery finds that the replication tested are compared: pi1 = 1 - pi0 of their p-values in the replication study, Storey’s estimate of the proportion of them with an effect there (proportion_of_true_nulls(), with qvalue’s defaults unless null_proportion says otherwise), which counts effects that the replication study is too small to find on its own; the share of them whose log2 fold changes have the same sign; and how many it finds too. pi1 measures replication in the replication study’s p-values, so it is not symmetric.

Heterogeneity:

Cochran’s Q, its p-value, I^2 and tau^2 of the log2 fold changes and their standard errors, by meta_analysis() with meta_method, for the genes that two studies or more tested; NaN for the others.

Not done:

The r-values of Heller, Bogomolov and Benjamini (2014), which claim replication in a follow-up study of the genes a primary study selects. Held-out prediction of case status from the genes the studies find (Nakatsuka et al. 2025) is cytepool.held_out_prediction.held_out_prediction().

References

Nakatsuka N, Adler D, Jiang L, Hartman A, Cheng E, Klann E, Satija R (2025). Improving reproducibility of differentially expressed genes in single-cell transcriptomic studies of neurodegenerative diseases through meta-analysis. Nature Communications 16, 7436. doi:10.1038/s41467-025-62579-z

Benjamini Y, Heller R (2008). Screening for partial conjunction hypotheses. Biometrics 64, 1215-1222. doi:10.1111/j.1541-0420.2007.00984.x

Storey JD, Tibshirani R (2003). Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences 100, 9440-9445. doi:10.1073/pnas.1530509100

Zaykin DV (2011). Optimally weighted Z-test is a powerful method for combining probabilities in meta-analysis. Journal of Evolutionary Biology 24, 1836-1841. doi:10.1111/j.1420-9101.2011.02297.x

Heller R, Bogomolov M, Benjamini Y (2014). Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study. Proceedings of the National Academy of Sciences 111, 16262-16267. doi:10.1073/pnas.1314814111

Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B 57, 289-300. doi:10.1111/j.2517-6161.1995.tb02031.x

class cytepool.replication.ReplicationResult(genes, study_pairs, found_in, studies, provenance)#

The replication of one group and contrast across the studies of a collection.

Built by differential_expression_replication(). Immutable.

genes has a row for every gene of the collection that at least one study tested, in the collection’s order: feature_id and, when the collection’s features have it, symbol; n_studies, the studies that tested it; n_up and n_down, those whose log2 fold change is above and below zero; n_found, those that find it (adjusted p-value below alpha), and n_found_up and n_found_down by direction; direction, 1 or -1, of the combined effect, and sign_agreement, the share of the studies that tested the gene whose log2 fold change has that sign; p_value and adjusted_p_value, the combined p-value of every study that tested the gene, two-sided; and replication_p_value and replication_adjusted_p_value, the partial conjunction p-value that at least at_least studies have an effect in the same direction (NaN where fewer tested the gene); and q_statistic, q_p_value, i_squared and tau_squared, the heterogeneity of its log2 fold changes (NaN where one study tested it).

study_pairs has a row for every ordered pair of the studies that analysed the group and contrast: discovery and replication; n_found, the genes the discovery study finds that the replication study tested; n_found_in_both, those it finds too; sign_agreement, the share of the n_found genes whose log2 fold changes have the same sign in both; and pi1, 1 - pi0 of the replication study’s p-values of those genes (NaN where there is none).

found_in has a row for every k from 1 to the number of studies that analysed the group and contrast: n_studies, k; n_genes, the genes that at least k studies find; and n_genes_same_direction, those that at least k studies find in one direction.

studies has a row for every study of the collection: study; analysed, whether its result has a table of the group and contrast; reason, why not (empty when analysed); n_samples, the samples its fit of the group used (0 when not analysed); weight, its weight in Stouffer’s method (NaN for another method, or when not analysed); n_tested and n_found, the genes it tested and finds.

Raises:

  • TypeError – If a table is not a Table, or provenance is not a mapping that JSON can write.

  • ValueError – If provenance holds a NaN or an infinity.

Explained in: Meta-analysis

property genes#

Every gene that a study tested, with its combined and replication p-values, the studies that find it and its heterogeneity.

property study_pairs#

Every ordered pair of studies, with pi1 and the sign agreement of the genes the first finds.

property found_in#

The genes found in at least k studies, for every k.

property studies#

Every study, whether it analysed the group and contrast, its samples and weight, and the genes it tested and finds.

property settings#

The group, contrast and settings, as a new dict (also in the provenance).

property provenance#

A JSON-serialisable record of what produced every number, as a new dict.

It holds the CytePool, numpy, numba and Python versions; the settings, with the group and contrast; the collection’s provenance, which identifies each study’s cube by its fingerprint and gives the alignment; and the provenance of every study’s differential-expression result, by study name.

cytepool.replication.differential_expression_replication(collection, results, group, contrast, *, alpha=0.05, method='stouffer', at_least=2, conjunction_method='simes', weights=None, meta_method='reml', null_proportion=None)#

How well the studies of a collection replicate each other’s differential expression, gene by gene and study by study.

For one group (cell type) and contrast, the studies’ results, each the analysis of its aligned cube (aligned_cube()), are lined up by align_study_results(), and every gene gets its combined p-value, its partial conjunction p-value of an effect in at least at_least studies in one direction, the studies that find it, and the heterogeneity of its log2 fold changes; every pair of studies its pi1 and sign agreement; and every k the genes found in at least k studies. The module description gives the methods.

Parameters:

  • collection – The collection whose aligned cubes the studies’ results analysed.

  • results – The differential-expression result of every study of the collection, by study name, from limma_voom(), limma_trend() or dream_voom().

  • group – An aligned group id of the collection.

  • contrast – A contrast name that at least two studies tested.

  • alpha – The adjusted p-value below which a study finds a gene, in (0, 1).

  • method – The combination of every study’s p-value: "stouffer" (weighted, by default by the square roots of the studies’ samples), "fisher", "tippett", "pearson", "mudholkar_george" or "simes" (combine_p_values()).

  • at_least – u of the partial conjunction: the fewest studies with an effect in one direction, from 1 to the number of studies that analysed the group and contrast; 2 by default.

  • conjunction_method – "simes", "fisher" or "stouffer" (partial_conjunction()).

  • weights – Stouffer’s weight of each study, by study name, positive and finite; by default the square root of the number of samples its fit of the group used. For method="stouffer" only.

  • meta_method – The estimator of tau^2 for the heterogeneity statistics: "reml", "dl", "pm" or "fixed" (meta_analysis()).

  • null_proportion – Options of proportion_of_true_nulls() for pi1: lambdas, method, smooth_df and smooth_log; qvalue’s defaults by default.

Returns:

The tables of genes, pairs of studies, counts by number of studies and studies, with a provenance record.

Raises:

  • TypeError – If an argument has the wrong type, or is a numpy masked array.

  • ValueError – If results does not give the result of every study of the collection, fewer than two studies analysed the group and contrast, a study’s table lacks a column the analysis reads (as an F-test’s does), at_least exceeds the studies that analysed it, an option is out of range or unknown, or weights are given for a method other than Stouffer’s or do not give every study that analysed it.

  • KeyError – If group is not a group of the collection, or no study tested contrast.

Used in: Replication across studies

Explained in: Meta-analysis