Add baseline-aware evaluator for perturbation predictions - #1098
Add baseline-aware evaluator for perturbation predictions#1098thantiklermcirony wants to merge 6 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1098 +/- ##
==========================================
+ Coverage 79.94% 80.51% +0.57%
==========================================
Files 55 56 +1
Lines 7533 7759 +226
==========================================
+ Hits 6022 6247 +225
- Misses 1511 1512 +1
🚀 New features to boost your workflow:
|
|
Added 27 test cases in 092dc57 for invalid holdouts and model names, unsupported input forms, and nonfinite arithmetic outcomes; the evaluator implementation is unchanged. The exact published test file passed all 71 cases on Linux/Python 3.12 and 3.14, with zero failures or skips. A paired coverage measurement against the previous 44 tests increased evaluator line coverage from 204/225 (90.67%) to 223/225 (99.11%); this is line coverage, not a claim of exhaustive behavior coverage. Ruff check/format also passed. Reproducible before/after runs, environment records, JUnit and coverage artifacts. The test file SHA-256 is |
PR Checklist
docsis updatedDescription of changes
Relates to #1035. Comparing perturbation predictions currently requires callers to assemble splits, references and baselines themselves. This adds
pt.tl.PerturbationEvaluator, which aligns genes by identifier and returns tidy per-group metrics with explicit unavailable/undefined statuses.Baselines use training controls and single interventions only. Explicit label, combination and context holdouts preserve cell membership; known combination aliases are grouped through an explicit component mapping. Evaluation rejects declared train/test cell overlap and mismatched feature sets while retaining missing prediction groups. MSE and E-distance reuse the existing
DistanceAPI. The guide documents point-baseline, measurement-scale and split limitations.Technical details
This is a proposed first API slice: control-mean and additive baselines, MSE, E-distance, delta Pearson, sign agreement and top-k absolute response overlap. It does not close the entire issue: nearest-neighbour baselines, logFC scoring and DEG discovery are not implemented. Input contracts for the first two would benefit from maintainer agreement. No new dependencies are added.
Validation: 44 contribution tests and 40 independent audit cases passed on Python 3.12.3 and 3.14.7; the existing comparison tests passed. Whole-project mypy checked 101 files; applicable pre-commit hooks and the full Sphinx build passed. An eight-combination Norman example produced 80 score/status rows, agreed with independent numerical calculations, and preserved the case where additive prediction is worse than the control baseline.
Additional context
The clean diff contains only implementation, tests and documentation. The audit workflow, data recipe and independent tests remain in the fork’s audit branch. The five contribution files are byte-identical to the files tested there. No raw data or claim of a new biological prediction method is included.