Permutation checks#
Whether the true doses of a dose series pass more genes through its gate than permuted doses do, and how often the gate passes genes with the doses permuted; Could the calls come from noise? explains the check.
Could the genes a dose series calls come from noise? A permutation check of the gate at the series’ own design.
permutation_check() permutes the dose labels of each cell type among
the samples that its design treats alike, runs dose_response() again
on each permuted labelling with the result’s settings, and counts the genes that pass
the result’s gate: how often the gate passes genes with the doses permuted, and a
permutation p-value that tests whether the true doses pass more
(https://docs.nygen.io/CytePool/statistics/dose-response#dose-permutation-check).
- Exchangeable units:
Under the null hypothesis that no gene responds to dose, samples that the design treats alike apart from their dose are exchangeable. Without random effects the units are the samples, in blocks of the same row of the design’s matrix, its terms other than dose: a factor such as sex splits them, and a numeric covariate leaves each sample in a block of its own. With one random intercept, such as
(1 | animal), units that each hold one dose are permuted whole, in blocks of units with the same rows, so that each run keeps the samples and units of every dose group; when units hold several doses, as donors whose cells receive every dose, the doses are permuted within each unit and block of rows.
References
Lee H, Stead JDH, Williams A, Cortés Ramírez SA, Atlas E, Mennigen JA, O’Brien JM, Yauk C (2024). Empirical characterization of false discovery rates of differentially expressed genes and transcriptomic benchmark concentrations in zebrafish embryos. Environmental Science & Technology 58, 6128-6137. doi:10.1021/acs.est.3c10543
Bruns IB, Schultz DR, Demuynck E, et al. (2026). Analytical choices drive toxicogenomic potency estimates: a systematic evaluation of transcriptomic points of departure. Toxicological Sciences 209, kfag115. doi:10.1093/toxsci/kfag115
Bourgon R, Gentleman R, Huber W (2010). Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences 107, 9546-9551. doi:10.1073/pnas.0914005107
Phipson B, Smyth GK (2010). Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology 9, article 39. doi:10.2202/1544-6115.1585
- cytepool.permutation_check(result, *, n_permutations=100, groups=None, rng=None)#
Check whether the true doses of a dose series pass more genes through its gate than permuted doses do.
For each cell type, the dose labels of the samples it uses are permuted among the samples that the design treats alike (the module documentation gives the units),
dose_response()runs again on that cell type with each permuted labelling and the result’s settings and gate, and the genes that pass the gate are counted. The answer is the permutation p-value, (1 + b) / (1 + B) for the b of the B runs that pass at least as many genes as the true doses (Phipson and Smyth 2010). Under the null hypothesis that no gene responds to dose, every assignment of the labels is as likely as the true one, so a small p-value says that the true doses pass more genes than permuted ones, and so that not all of the gate’s calls are noise; it does not say which calls are false. The summary also gives how often the gate passes genes with the doses permuted, given the cell type’s own samples, which varies widely between series of one design. That share estimates how often noise alone passes genes only where no gene responds, as in vehicle-only data: responses scrambled over the permuted dose groups still pass in some runs.A design that allows at most
n_permutationsother assignments runs each of them, which makes the p-value exact; otherwisen_permutationsassignments are drawn at random. Each run keeps the samples of every dose group and the doses of the design, and so the spline’s knots and the genes of the dose-blind expression filter (Bourgon, Gentleman and Huber 2010);batch, which a rerun could find confounded by chance and which no fit models, is left out. Vehicle-only data that the design places in mock dose groups, as Lee et al. (2024) did, runs throughdose_response()and this check like any series (https://docs.nygen.io/CytePool/statistics/dose-response#dose-permutation-check).Each run fits the gate’s test alone, with the result’s seed for a multiple contrast test. Before the runs, each cell type runs once with its true doses and must give the result’s gate: the same genes, p-values equal up to rounding and the same calls, so that the runs are the analysis the result records. Each run takes the time of the cell type’s analysis and of the gate’s test: about 0.7 s for 20,000 genes and 24 samples with the spline test on two threads, twice that with a fold change, and more for a multiple contrast test or under dream.
- Parameters:
result – The result of
dose_response()to check, which keeps its cube and design.n_permutations – The runs per cell type, at least 1: random assignments, or every other assignment when there are at most this many.
groups – A cell type or a list of them, each tested with its gate run; all cell types by default, those not checked with the reason.
rng – A numpy random generator, an integer seed, or None, for the runs. The check takes one seed, the integer given or one drawn from the generator, and records it in
settings["seed"]; each cell type draws its runs from it and the CRC-32 of its name, so that a cell type’s runs depend neither on the other cell types nor on the threads.
- Returns:
The summary of each cell type, its runs and a provenance record.
- Raises:
TypeError – If
resultis not aDoseResponseResult,n_permutationsis not an integer, orgroupsorrnghas the wrong type, or is or holds a numpy masked array.ValueError – If
n_permutationsis below 1,groupsis empty or names a cell type twice,rngis a negative seed, the result keeps no cube and design, its design has more than one random effect or a random slope, a cell type’s true doses do not give the result’s gate again, or a run does not run the gate.KeyError – If a cell type asked for is not in the cube, was not checked by the result, not tested, or did not run the gate.
Used in: Dose response per cell type
Explained in: Choosing a method, Dose response
- class cytepool.PermutationCheck(summary, tables, provenance)#
How often the gate of a dose series passes genes with its doses permuted, and whether the true doses pass more, from
permutation_check().summaryhas one row per cell type asked for:group_id;checked;reason, why a cell type was not checked, empty when checked;n_assignments, the distinct assignments of the dose labels to the cell type’s exchangeable units, as a float (the exact integer is in the provenance);exhaustive, whether the runs were every other assignment;n_runs;n_genes_passing_gate, the genes the gate passes with the true doses (NaN where the gate did not run);share_of_runs_with_passing_genes;mean_passing_genes_per_run;p_value, (1 + b) / (1 + B) for the b of the B runs that pass at least as many genes as the true doses (Phipson and Smyth 2010); andsmallest_possible_p_value, the smallest p-value these runs can give, (1 + t) / (1 + B) for the t runs whose gate gives the true doses’ p-values up to rounding and so passes the same genes whatever the data: runs that only rename dose groups that the gate’s test cannot tell apart, such as dose groups of equal sizes under the dose-group test, and the reflection of doses that sit symmetrically on the axis under the spline. With every other assignment run, B is the number of assignments less one. Dunnett’s test of three or more treated doses, whose p-values come from Monte Carlo integration, gives its renamings p-values apart by that error, so they do not count. Statistics of cell types not checked are NaN, and theirn_runs0.table()gives each run of a cell type, withrunandn_genes_passing_gate. The provenance records each run’s assignment, so that any run can be made again withdose_response()and the result’s seed, which every run takes. Built bypermutation_check(); immutable.- Raises:
TypeError – If
summaryis not aTable,tablesnot a mapping of cell type toTable, orprovenancenot a mapping that JSON can write.ValueError – If
provenanceholds a NaN or an infinity.
Explained in: Dose response
- property summary#
One row per cell type, with whether it was checked, its assignments and runs, the genes the gate passes, how often its runs pass genes, and the p-value.
Explained in: Dose response
- property settings#
The settings
n_permutations,groupsas given, andseed, the seed of the runs, given or drawn, as a new dict, also in the provenance.
- property provenance#
A JSON-serialisable record of what produced every number, as a new dict.
It holds the versions of the software; the settings; the result checked (
result): its cube’s fingerprint, its gate and its seed, which every run takes; and, for every cell type (groups), whether it was checked and why not, and the genes its gate passes with the true doses. A cell type whose units were read also records the kind of units permuted ("samples","whole units"or"doses within units") and the random effect’s grouping (unit), the samples used, in the order of the fit, the doses, each sample’s dose as an index into them (labels), the blocks of units, each unit given by its samples, and the number of assignments as an exact integer; a cell type checked, whether the runs were every other assignment, each run’s assignment (assignments), each sample’s dose index in the order ofsamples, and the runs whose gate gives the true doses’ p-values up to rounding (tied_runs, numbered from 1).
- table(group=None)#
Return the runs of a cell type:
run, from 1, andn_genes_passing_gate.- Parameters:
group – A cell type that was checked; it may be left out when one cell type was asked for.
- Raises:
ValueError – If
groupis left out and several cell types were asked for.TypeError – If
groupis not a string.KeyError – If the cell type was not asked for or not checked; the message says why.
Explained in: Dose response