sequential-testing

Runs ~20 options strategies against live market data in shadow mode, records every hypothetical fill under worst/base/optimistic assumptions, and grades each with anytime-valid e-processes. Places no orders.

Sequential Hypothesis Testing with e-Values and p-Values

Sequential testing for efficient and reliable comparison of stochastic algorithms.

Which model is really behind your API relay or agent IDE? Behavioral fingerprinting + anytime-valid sequential tests (FPR<=1%). LLMs can't be random - measured on 9 frontier models.

A/B testing engine: frequentist + Bayesian + CUPED variance reduction + peeking-safe sequential testing

Sequential A/B Testing Framework in Python

A/B test analysis that is correct by default, informative by design, and composable by construction. 88 classes, 60+ paper references.

A/B testing and causal inference scored against known answers: simulations where I set the effect, and a randomised benchmark the observational methods have to recover.

Anytime-valid A/B test analyzer that holds the false-positive rate under 1.5% while you peek at the dashboard continuously, where naive fixed-horizon testing leaks to 23% at 15 looks. mSPRT with confidence sequences, CUPED variance reduction (50% on the demo, SE 0.223 to 0.135), a peeking guard, and a reproducible A/A simulation.

Measure your agent harness, find where it wastes the model, and prove the fix worked. Harness-agnostic, agent-agnostic, zero dependencies. Reference implementation of HTP-1.

Label-free drift attribution (model / noise / world / annotator) with anytime-valid guarantees and a frozen pre-registration — closes the zero-label misattribution gap 0.50→0.00 on the recoverable cause; negatives reported as measured boundaries.

Causal bandit orchestration platform for real-time adaptive experimentation, sequential testing, and interference-aware decisioning.

Automated canary control plane for LLM configs: shadow testing, sequential inference (SPRT/CUSUM), drift monitoring

Small, well-tested Python package for online controlled experiments: sample size, CUPED, mSPRT sequential tests, delta-method ratio metrics.

Your A/B test winner is about half as good as it looked. The winner's curse, peeking and power measured on 32,363 real randomised experiments -- with a live demo that re-splits the traffic in your browser.

Rank-targeted nested sequential design for LLM evaluation: reach the same ranking conclusion for less, and see which comparisons the data never supported.

[Alpha / WIP] Agent-first CLI for Bayesian and sequential A/B/n experiment decisions with JSON, CSV, XLSX, and text inputs.

Typed toolkit for A/B experiment analysis: power analysis, CUPED variance reduction, always-valid sequential testing (mSPRT), Welch/z analysis with CIs - property-based tests (hypothesis) and a worked forecasting-rollout case study.

Progressive delivery for prompts and models: sticky canary routing + an always-valid mSPRT sequential test driving automatic promote/rollback. Peeks every request without inflating the false-rollback rate (measured: 0% vs 11.7% for a naive peeked test).

Full-stack A/B testing engine: frequentist tests, Bayesian inference, sequential O'Brien-Fleming bounds, and multi-metric batch analysis with one-click PDF report export.