Skip to content

Add baseline-aware evaluator for perturbation predictions - #1098

Open
thantiklermcirony wants to merge 6 commits into
scverse:mainfrom
thantiklermcirony:feat/perturbation-evaluator
Open

Add baseline-aware evaluator for perturbation predictions#1098
thantiklermcirony wants to merge 6 commits into
scverse:mainfrom
thantiklermcirony:feat/perturbation-evaluator

Conversation

@thantiklermcirony

Copy link
Copy Markdown

PR Checklist

  • Referenced issue is linked
  • Tests added for the new code
  • Documentation in docs is updated

Description of changes

Relates to #1035. Comparing perturbation predictions currently requires callers to assemble splits, references and baselines themselves. This adds pt.tl.PerturbationEvaluator, which aligns genes by identifier and returns tidy per-group metrics with explicit unavailable/undefined statuses.

Baselines use training controls and single interventions only. Explicit label, combination and context holdouts preserve cell membership; known combination aliases are grouped through an explicit component mapping. Evaluation rejects declared train/test cell overlap and mismatched feature sets while retaining missing prediction groups. MSE and E-distance reuse the existing Distance API. The guide documents point-baseline, measurement-scale and split limitations.

Technical details

This is a proposed first API slice: control-mean and additive baselines, MSE, E-distance, delta Pearson, sign agreement and top-k absolute response overlap. It does not close the entire issue: nearest-neighbour baselines, logFC scoring and DEG discovery are not implemented. Input contracts for the first two would benefit from maintainer agreement. No new dependencies are added.

Validation: 44 contribution tests and 40 independent audit cases passed on Python 3.12.3 and 3.14.7; the existing comparison tests passed. Whole-project mypy checked 101 files; applicable pre-commit hooks and the full Sphinx build passed. An eight-combination Norman example produced 80 score/status rows, agreed with independent numerical calculations, and preserved the case where additive prediction is worse than the control baseline.

Additional context

The clean diff contains only implementation, tests and documentation. The audit workflow, data recipe and independent tests remain in the fork’s audit branch. The five contribution files are byte-identical to the files tested there. No raw data or claim of a new biological prediction method is included.

@codecov-commenter

codecov-commenter commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.11504% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.51%. Comparing base (5bf4aca) to head (092dc57).

Files with missing lines Patch % Lines
src/pertpy/tools/_perturbation_evaluator.py 99.11% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1098      +/-   ##
==========================================
+ Coverage   79.94%   80.51%   +0.57%     
==========================================
  Files          55       56       +1     
  Lines        7533     7759     +226     
==========================================
+ Hits         6022     6247     +225     
- Misses       1511     1512       +1     
Files with missing lines Coverage Δ
src/pertpy/tools/__init__.py 91.66% <100.00%> (+0.23%) ⬆️
src/pertpy/tools/_perturbation_evaluator.py 99.11% <99.11%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@thantiklermcirony

Copy link
Copy Markdown
Author

Added 27 test cases in 092dc57 for invalid holdouts and model names, unsupported input forms, and nonfinite arithmetic outcomes; the evaluator implementation is unchanged.

The exact published test file passed all 71 cases on Linux/Python 3.12 and 3.14, with zero failures or skips. A paired coverage measurement against the previous 44 tests increased evaluator line coverage from 204/225 (90.67%) to 223/225 (99.11%); this is line coverage, not a claim of exhaustive behavior coverage. Ruff check/format also passed.

Reproducible before/after runs, environment records, JUnit and coverage artifacts. The test file SHA-256 is 42fb40e002cc004f613f48bd2a35a53880f6203f2c979c62c40acb7622153cdc; published bytes were verified against that run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants