[Tracking] Faster local tests with preserved regression detection

[Trecek]

Purpose

Track the next local, single-machine test-performance improvements while preserving regression detection. This is a coordination-only tracker: implement its scoped sub-issues, not this umbrella as one large recipe job.

Six scoped implementation children are now approved and filed: the original three work-elimination slices plus the focused doctor, document-route, and low-priority serialization follow-ups. Operation/scope reductions are distinguished from unmeasured runtime savings. The remaining research lanes are not promoted to proven implementation wins.

Suggested implementation order

Order	Implementation slice	Why this order	Dependency
1	#4957 — Selector bookkeeping	Common cost across local iterations; establishes a cleaner baseline	None
2	#4958 — Combined guard parsing	Improves the guard workload retained by many filtered scopes	Soft sequence, not a technical blocker
3	#4959 — Audit-only headless fixtures	Isolated setup reduction in eleven tests	Independent; recommended third for attribution
4	#4961 — Focused doctor CLI checks	Strongest newly calibrated work reduction: 504 → 9 checker bodies	Independent test-file scope; soft sequence
5	#4962 — Contributing-document route	Removes an unrelated infra execution scope for one demonstrated trigger	Coordinate filter-test ownership with #4957; no hard blocker
6	#4963 — Recipe-grid string reuse	At most 72 avoided serializations; low-priority small cleanup	Independent; lower priority than 1–5

All six numbered issues are linked here and as native GitHub sub-issues. Work may be developed in parallel where files do not overlap, but measure each change independently. No artificial blocking edges are intended. Every part of a multi-part implementation must independently pass task test-check.

Evidence and expectations

The investigation covered task/MCP entrypoints, selector and oracle behavior, fixtures, static guards, recipe loading/serialization, real subprocess timeout paths, installed pytest/xdist/AnyIO behavior, historical profiles, deterministic selector replays, and bytecode headers. It included two independent adversarial reviews per recommendation (12 reviews), followed by direct primary-source checks and corrections.

Evidence snapshot: 5f49d3761; filing checkout: 94e3d18e0. The implementation target files were checked for intervening changes. File and pattern counts below are snapshot observations and must be recomputed if the tree changes.

Fresh timing benchmarks were not run because unrelated PR implementations were concurrently running suites. Operation reductions are supported; elapsed-time ranges are planning estimates, not measured promises.

Slice	Supported work reduction	Expected runtime impact
Selector bookkeeping	Hoist per-item/per-scope filesystem classification; compile an invocation's manifest once. The three-artifact replay performed 309 pattern constructions (3 × 103).	Seconds-scale candidate, potentially larger on heavily filtered scopes; little benefit for unfiltered/root-fallback matching
Combined guard parsing	1,806 immediate second parses plus a 2,027-file duplicate masking pass; one additional 942-file schema pass when cold	Seconds-scale candidate across relevant guard-heavy local runs
Audit-only fixtures	11 factories, 11 audit recoveries, 110 skill-file writes avoided	Small runtime gain; clearer dependencies and isolation
Focused doctor tests (#4961)	504 → 9 checker bodies across nine passing tests; identical named results, 495 unrelated invocations avoided	Proven work elimination, not a 98% runtime estimate; real config/backend/plugin/setup/rendering work remains
Contributing-document route (#4962)	Earlier conservative replay: 476 → 409 candidate modules for that document change, retaining docs and required guards	Specific-trigger execution-scope reduction; no pre-collection or general runtime claim
Recipe-grid string reuse (#4963)	6E → 4E serializations; E ≤ 36 in the inspected snapshot, at most 72 avoided calls	Small source-supported cleanup; no clean timing estimate
Channel-B expiry research	Five requested expiry budgets total 780 worker-seconds; two indivisible 300-second items	Largest minutes-scale potential, but the tested first-observation accelerator lost delayed-fault detection and must not ship as-is

Do not add these estimates together as a promised wall-time improvement. Work overlaps across workers, and a long indivisible test can dominate the run.

Current local-testing architecture

• task test-check is pytest-only after environment/generated-artifact setup. task test-all adds import lint. The configured MCP test gate invokes test-all, conservative filtering, base develop, and a 900-second outer limit. Task commands
• The default is four xdist workers with work stealing. Every worker collects requested tests, and session fixtures/caches are process-local.
• Direct test-filtered and test-local wrappers do not load the MCP base-ref configuration. A conservative run without a resolvable base requests a root full fallback.
• Automatic path filtering occurs after collection. Aggressive mode uses working-tree/untracked changes and excludes large/unmarked cases only when a concrete scope exists; it does not cover committed branch changes like conservative mode.
• RAM-backed unique pytest temp/cache generations, shared UV caches, isolated per-test homes, and several recipe/skill/static-scan caches are already implemented. Do not propose them again as new wins.

Investigation findings relevant to every child

Filtering correctness is separate from bookkeeping speed

The known test-helper gap is real: changing tests/server/_helpers.py selects that non-collectable path without pulling its Fleet consumer. A root full fallback also does not constrain the independent architectural item hook, which can still deselect source-file cases. These are separate coverage-repair candidates, not part of the first selector optimization.

The accepted coverage map is from recorded source commit 5817de1…; admission checks age/schema/success rather than current-tree identity or mapped-target existence. It records only test |run contexts and does not capture arbitrary source readers or all fixture dependencies. An entirely unmapped changed source keeps its directory broad, but a partially mapped/stale source can omit consumers. Selector/refinement

The oracle can widen selection: for a fixed core/_json.py change, aggressive candidate-file scope grew 496 → 754 and added the Channel-B tests. That is a real JSON parser dependency, not a reason to discard the expensive tests. Raw file-count reduction is not runtime saved.

Preserve guards when sharing work

• Keep unpaired import-helper APIs and the distinct traversal/error policies of each guard.
• Return copies or immutable compact data as appropriate; caller mutation and temporary-root calibration must still work.
• Do not introduce a global retained-AST cache. Earlier measured prototypes showed retention cost can exceed reparse cost for few consumers.
• xdist_group affinity is not combined with the installed worksteal scheduler; worker-local cache benefit is conditional.

Counts and old profiles are not current speedups

Earlier classifier, canonical-constant, scan-cache, recipe-discovery and recipe-grid consolidations have already landed. The old two local same-outcome runs at 262.64/344.19 seconds illustrate variance; they are not a current baseline. The current recipe grid has 40 direct recipe/mode points but only E ≤ 36 step-bearing served points (two campaigns have empty step maps): requests are 40+E, and a narrow render reuse would change 6E → 4E, not 240 → 160 unconditionally.

Initial research lanes and current disposition

The numbered list below records the original investigation leads. Subsequent evidence and explicit user approval promoted only the nine doctor cases (#4961), exact contributor-document route (#4962), and local serialization reuse (#4963) into implementation children. Their current scope and acceptance criteria live in those tickets. Timeout redesign, broader caches/fixture sharing, pre-collection filtering, and worker tuning remain research or rejected proposals as detailed below.

1. Channel-B expected-expiry acceleration. Retain all five cases, real children, generous startup allowance, actual outer-timeout/kill paths, and downstream assertions. A child flush or pre-read poll signal is insufficient: deadline movement must follow real heartbeat classification of the relevant input plus Channel-B deposition. Prove invalid confirmation still fails via negative calibration. Do not blindly shorten startup limits; history documents failure of the old 60-second allowance under WSL/xdist. Coordinate with Flaky Channel B drain-race test kills pipeline: merge test gates lack retry path #4307. Timeout cases, heartbeat order
2. Narrow artifact routes from demonstrated consumers. An in-memory contributing.md route change from all infra/ to infra/test_ci_workflow.py reduced candidate files 476 → 409, preserving all 23 docs modules and 13 unconditional infra/hooks modules while excluding the recipe grid. Validate the reader set and exact logical route. The existing unconditional manifest check already detects nonexistent targets; it does not establish the correct consumer. Direct reader
3. Doctor leaf-test isolation. Focused script-version-health/output tests may avoid unrelated checks, but the existing helper also stubs config/backend. Preserve those real inputs and keep aggregate, clean, fleet-gating and read-only tests unrestricted. Coordinate with completed doctor: _is_plugin_installed spawns 'claude plugin list' 3x per run (production latency) #4814 rather than repeating its production probe fix. Helper
4. Recipe-grid work. Reuse only byte-identical final serialized strings across the two non-kitchen labels; retain all per-tool checks and exact aggregate maxima. Registry/card caching and grid partitioning need separate evidence. Direct/served keys differ legitimately in recipe snapshots, known-name lists, and finalized projection. Coordinate with open tests/server: pin project_dir + stop resetting _LOAD_CACHE (BLOCKED on #4816) #4819. Grid
5. Worker-count experiments. Both wrappers currently bake in -n 4; a validated explicit control is needed for a complete serial/1/2/4/8 sweep. Separate in-process serial from one-worker bootstrap, gate workloads, cache layers, actual descendants, and controlled multi-worktree concurrency. Quiet-host measurements do not establish a simultaneous-PR optimum.
6. Static-impact comparison / early collection. pytest-impacted is a beta comparison candidate, not an approved replacement. Its plugin marks skips after collection and its branch semantics differ. Stacking independent filters tends to intersect runnable sets. Preserve our artifact/architectural guards and test helper/rename/non-code/Git-failure cases before adopting any narrower authority.

Shared acceptance / measurement protocol

• Preserve all intended tests and assertions. No duplicate-test deletion, Hypothesis sample reduction, skipped integration coverage, or production timeout changes as a shortcut.
• Use approved task-based checks. Run pre-commit before any commit; honor the normal project commit/worktree rules.
• Freeze the triggering change scenario separately from the candidate implementation. Compare canonical final selected node IDs and outcomes/skip policy, not just collected file counts.
• Record source/diff identity, filter/base/fallback, effective argv, plugin/feature settings, selected scope, worker assignments, phase durations, and resource-accounting boundary.
• Define cold/warm state separately for fresh pytest caches, bytecode, UV, and OS page caches.
• Account for task → controller → workers → test descendants. Do not report summed RSS as physical memory, or controller-only CPU as whole-run work.
• Measure clean repeated pairs when feasible, with implementation checks and speed measurements reported separately. Do not convert ongoing unrelated PR load into a claimed clean speedup.

Relationship to prior work

This tracker is a local-testing follow-up, not a replacement for #4815's prior CI-performance campaign. Closed #4811/#4812/#4814/#4818 already cover specific earlier fixes. Open #4819 remains the owner of its server recipe-cache fixture scope; #4307 contains prior Channel-B reliability context. #4602 is a separate test-file organization audit and is not part of this performance effort.

Prior rejection of indiscriminately shrinking Channel-B timeouts remains valid. The research lane here asks whether controlled expiry after verified consumers can preserve the actual regression checks while avoiding idle waiting on a single local machine.

Completion criteria

• All six approved implementation children are linked below, independently implemented/validated, and their measured or unsupported impact is recorded. Filing is complete; implementation and impact verification are not implied by ticket creation.
• Research lanes have evidence-backed dispositions (ready for a separate scoped change, rejected with a reason, or explicitly still unmeasured), without silently becoming approved implementation work.
• This tracker stays coordination-only; do not dispatch it as one monolithic implementation task.

Provenance

The comprehensive local report is investigation_local_test_performance_2026-09-10.md (dated 2026-09-10), with selector replay, header-census, and per-recommendation review artifacts. These local files are supplemental; the source links, counts, scope, caveats, and acceptance criteria needed by downstream implementers are included in this tracker and its children.

Implementation sub-issues — filed

All six implementation tickets are filed, triaged, and linked as native sub-issues. Implement in the suggested order below; the sequence is soft, not an artificial technical dependency.

• 1. Reduce invariant work in local test selection #4957 — Reduce invariant work in local test selection. Reduce invariant work in local test selection #4957
• 2. Combine repeated parsing in architectural test guards #4958 — Combine repeated parsing in architectural test guards. Combine repeated parsing in architectural test guards #4958
• 3. Remove full server contexts from audit-only headless tests #4959 — Remove full server contexts from audit-only headless tests. Remove full server contexts from audit-only headless tests #4959
• 4. Run only relevant checks in focused doctor CLI tests #4961 — Run only relevant checks in focused doctor CLI tests. Run only relevant checks in focused doctor CLI tests #4961
• 5. Narrow the contributing-document test route to its infra consumer #4962 — Narrow the contributing-document test route to its infra consumer. Narrow the contributing-document test route to its infra consumer #4962
• 6. Reuse identical serialized responses in the recipe test grid #4963 — Reuse identical serialized responses in the recipe test grid. Reuse identical serialized responses in the recipe test grid #4963 — lower-priority small cleanup

All six carry enhancement and recipe:implementation. This parent intentionally has no executable recipe label: it coordinates the children and remaining research. Only the explicitly linked implementation scopes are approved; unrelated research must not be folded into them.

Follow-up investigation — verified dispositions (2026-09-10)

This section supersedes the earlier research-readiness descriptions where they differ. Current approved soft sequence: #4957 → #4958 → #4959 → #4961 → #4962 → #4963. The user explicitly approved the three bounded follow-ups after reviewing these findings. No production/test implementation was made during the investigation or ticket filing. Other research findings do not imply additional approved implementation work.

Approved follow-up #4961: nine focused doctor tests

A diagnostic-only plugin intercepted only the collector's _run_check binding for tests/cli/test_doctor_scripts.py::TestDoctorScriptHealth. It forwarded the selected script_version_health check through the real exception-isolation wrapper and left config loading, backend resolution, the single upfront plugin-installed probe, and CLI output handling real.

Observed through task test-check, with four workers/worksteal and explicit filter mode none:

Quantity	Baseline	Selected check only
Original tests passing	9	9
Offered checker callsites	504	504
Executed checker bodies	504	9
Per-node named check/severity/message	Exactly matched	Exactly matched

This eliminates 495 unrelated checker invocations in that slice. It does not establish a 98% test-runtime improvement: config, plugin discovery, setup, collection, rendering, and selected checks remain. Wall-time differences in the saved logs are not valid benchmarks under current concurrent PR load.

Negative calibration withheld the selected result after its real checker ran. Both baseline and selected versions failed the original len(script_checks) == 1 assertion in test_outdated_scripts_reports_warning_with_count. Thus the selector did not bypass the focused assertion. This is named-result equivalence, not equivalent full doctor JSON.

A narrow follow-up implementation should affect only these nine leaf-output cases. Leave aggregate-presence, clean-doctor, fleet/routing, broken-config, read-only, and recipe-sync-removal tests unrestricted. Do not alter the three already-restricted collector-isolation tests or their intentional config/backend stubs; they are not the measured source of this win. Do not add a production selected-checks API.

Evidence: collector/pre-check setup, target cases, existing differently scoped helper.

Suggested scheduling: #4961 is position 4 after the original three children (or independently if file ownership permits); this is the strongest newly calibrated work-elimination candidate.

Approved follow-up #4962: contributing-document route

Change only the literal docs/developer/contributing.md manifest route from docs/ plus infra/ to docs/ plus infra/test_ci_workflow.py. Direct and source-walker checks found the CI-policy assertion as the relevant infra reader; docs and arch document walkers stay selected.

The earlier deterministic replay at 5f49d3761 showed 476 → 409 candidate modules, retaining all 23 docs modules, arch/contracts, and the 13 unconditional infra/hooks guard files, while excluding the recipe grid. This is a 67-module scope reduction for that trigger, not a collected-node count or wall-time prediction.

Before landing, extend the existing cross-implementation production-manifest assertion with this exact route. Add a resulting-scope assertion retaining docs, the CI file, arch/contracts, and those guards without full infra/. Existing target-existence protection already handles stale file targets; the directory-completeness guard alone does not prove the file is the correct consumer. Dynamic future readers still require maintained dependency declarations.

Evidence: literal route, actual document reader, cross-implementation test, existing target-existence guard.

Suggested scheduling: #4962 is position 5, a separate small routing change after scope-preserving #4957 and the doctor follow-up; it must not expand #4957's equivalence contract.

Timeout research: first-observation accelerator is NOT ready

The deliberate budgets remain 60 + 300 + 60 + 300 + 60 = 780 worker-seconds; the potential remains minutes-scale on affected scopes. A temporary forwarding probe established that the real outer CancelScope.deadline can be shortened after Channel-B completion deposition and actual false heartbeat classification, with no Channel A, an untriggered race, and a live child. It preserved real timeout/kill/reap/adjudication and all original assertions.

The initial bounded checks were encouraging:

• Five healthy accelerated original tests passed.
• Empty-result and empty-stdout acceptance faults failed all five applicable original TIMED_OUT assertions with acceleration off and on.
• Every immediate mutant-on evidence record showed a strictly earlier deadline installed before the injected acceptance. An initial probe defect that bypassed rewriting in fault mode was caught by review and corrected before these runs.

A stronger discriminator rejected the proposed shortcut. Injecting erroneous acceptance on the second post-B empty-stdout classification made the original-deadline case fail, while the accelerated case passed. Baseline evidence recorded two eligible polls; accelerated evidence recorded one, because expiry prevented reaching the delayed fault. This is a synthetic regression discriminator, not an existing production defect.

Therefore: do not implement the first-observation/0.1-second shim as coverage-equivalent; do not replace the missing proof with an arbitrary longer grace. A redesigned split must preserve explicit sustained Channel-B nontermination and demonstrate delayed-fault detection, as well as immediate classification and real timeout/kill behavior. The existing independent two-second timeout test is necessary but does not supply sustained Channel-B coverage or a lower elapsed-time bound. Any reduction of long-duration stress exposure needs an explicit coverage decision.

Evidence: outer timeout, heartbeat loop, Channel-B deposition, original no-stdout assertion, independent timeout test.

Closed or narrowed alternatives

• tests/server: pin project_dir + stop resetting _LOAD_CACHE (BLOCKED on #4816) #4819 must not be implemented as written. Its project-root-only sharing premise is obsolete: the current load key includes both project root and effective-temp identity. Pinning project root while preserving private per-test temps still misses, even after removing resets. Sharing both would also share artifact/staleness state and conflict with project-local overlays. This requires a different isolation design. The old issue was neither closed nor edited by this investigation. Key construction, forwarding, ready-fixture overlay.
• Approved low-priority Reuse identical serialized responses in the recipe test grid #4963: recipe-grid serialization reuse remains a small win: 6E → 4E, E ≤ 36, at most 72 avoided serializations, retaining per-tool checks/maxima. The 183,815-byte baseline is path-normalized, not a raw saved-byte bound. Do not merge direct/served cache keys. Broader registry/card caches are not justified by measured benefit and would need content-safe invalidation/mutation isolation. Render loop/normalization, content hashing.
• Worker experiment needs less enabling work than first thought. Installed xdist 3.8.0 applies min(numprocesses,maxprocesses); PYTEST_ADDOPTS='--maxprocesses=1' task test-check, cap 2, and the default permit a fixed-workload 1/2/4 comparison now. This is version-pinned installed behavior, not a durable wrapper setting. Cap 0 is falsy; values over 4 still yield four. A serial/8-worker sweep still needs a validated Task parameter. No fastest setting is established under uncontrolled concurrent load. xdist 3.8.0 implementation.
• Do not adopt pytest-impacted as another live filter. Its post-collection skip hook would intersect runnable sets with current deselection. In the pinned source, unavailable Git returns None, which reaches skip-all rather than a fail-closed analysis error. Invalid branch refs instead fail configuration; not every Git error is a false green. A reviewer claim that no Python module immediately returns None was rejected: the API continues through explicit non-code strategies. No dependency was installed. Git path, strategy continuation, skip-all hook.

Verification and artifacts

The diagnostic checkout remained clean at 94e3d18e0b5efc891c3bb32c6ae822788f92a273. Environment and generated-artifact checks passed without changes. Only bounded tests ran through the approved Task wrapper; no full suite or clean timing benchmark ran. Intentional mutant failures are evidence, not project regressions (pytest exit 1; Task reports exit 201).

Two final independent validators read the full updated report and verified the new findings. The parent personally re-read their material source/log/evidence citations and preserved rejected reviewer claims in the evidence record.

Local artifacts (not assumed available to other clones):

• Main report: .autoskillit/temp/investigate/investigation_local_test_performance_2026-09-10.md
• Structured run counts, mutation outcomes, and parent adjudications: .autoskillit/temp/investigate/followup-local-performance/run-evidence.json
• Reproduction instructions, temporary plugins, logs, per-node evidence: the expiry-probe/ and doctor-probe/ subdirectories there.

Current implementation order is #4957 → #4958 → #4959 → #4961 → #4962 → #4963; #4963 is deliberately lower priority. All six are filed, not implemented by this investigation. Timeout redesign, pre-collection pruning, cache-sharing, and worker tuning are not promoted to proven speed improvements.

Follow-up ticket filing — user approved (2026-09-10)

The user explicitly approved all three follow-up recommendations, including the lower-priority serialization cleanup. They were filed as native children #4961, #4962, and #4963; the main order and checklist above are the current coordination record. Each child contains pinned source evidence, the relevant diagnostic/counterfactual counts, exclusions, acceptance criteria, and measurement limitations. Existing issue bodies, implementation statuses, and unrelated research were not changed or closed.