Description
The experiment A/B framework is silently broken: none of its 4 active experiments (#30314) accumulate historical data. When push_experiment_state.cjs creates a new orphan branch for the first time, checkoutOrCreateBranch() returns an empty string "" as baseRef. This causes pushSignedCommits to compute git rev-list ""..HEAD, which yields zero commits and returns without error — while the state was never pushed remotely. Every subsequent run starts from empty state and assigns variants randomly instead of using the balanced least-count strategy.
Expected Impact
- All 4 active experiments (
daily-community-attribution, astrostylelite-markdown-spellcheck, smoke-copilot, issue-arborist) will begin accumulating per-variant history
- Statistical inference becomes possible once minimum sample thresholds are reached
- Balanced variant assignment resumes, eliminating sampling bias
Root Cause
In push_experiment_state.cjs, the orphan-branch path passes empty baseRef to pushSignedCommits. Fix: detect "" baseRef and skip the rev-list check (treat orphan first push as always having new content).
Relevant code: push_signed_commits.cjs line 138, push_experiment_state.cjs orphan branch path.
Suggested Agent
Copilot coding agent — targeted JS/Node fix in a single file.
Estimated Effort
Quick (< 1 hour)
Data Source
DeepReport Intelligence Briefing — 2026-05-05 · Daily Experiment Report §25366113777
Generated by DeepReport - Intelligence Gathering Agent · ● 369.8K · ◷
Description
The experiment A/B framework is silently broken: none of its 4 active experiments (#30314) accumulate historical data. When
push_experiment_state.cjscreates a new orphan branch for the first time,checkoutOrCreateBranch()returns an empty string""asbaseRef. This causespushSignedCommitsto computegit rev-list ""..HEAD, which yields zero commits and returns without error — while the state was never pushed remotely. Every subsequent run starts from empty state and assigns variants randomly instead of using the balanced least-count strategy.Expected Impact
daily-community-attribution,astrostylelite-markdown-spellcheck,smoke-copilot,issue-arborist) will begin accumulating per-variant historyRoot Cause
In
push_experiment_state.cjs, the orphan-branch path passes emptybaseReftopushSignedCommits. Fix: detect""baseRef and skip therev-listcheck (treat orphan first push as always having new content).Relevant code:
push_signed_commits.cjsline 138,push_experiment_state.cjsorphan branch path.Suggested Agent
Copilot coding agent — targeted JS/Node fix in a single file.
Estimated Effort
Quick (< 1 hour)
Data Source
DeepReport Intelligence Briefing — 2026-05-05 · Daily Experiment Report §25366113777