You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As a maintainer, I want to measure the cost of preparing a Git-backed workflow
test and the cost of the behavior it proves, so slow tests spend their time on
meaningful Git hazards rather than unnecessary fixture work.
What the fixture does
A workflow Git fixture prepares the repository environment used by one test
scenario. Depending on the suite, that may include:
creating a checkout;
creating and configuring a local remote;
assembling the workflow host;
configuring authentication behavior; and
cleaning up the repository and host state afterward.
Many test files acquire this fixture once for every scenario. A small fixed cost
there is multiplied across the entire Git and pull-request corpus.
The behavior under test is separate. For example, a durability scenario may
interrupt one Git operation and prove that recovery selects the correct
committed state.
Current evidence
A historical comparison found that the Git and pull-request suites became about
1.6 times slower even though the individual git-* test files did not change.
The shared fixture did change. #532 added production authentication delegation
and a credential-helper launcher to the test host. Its contract says that a
local remote requiring no authentication should reach no authentication
mechanism, but the timing increase affected suites using that fixture generally.
This is evidence for a focused measurement, not yet proof that authentication is
the cause.
Measurement
Run one controlled comparison and record the complete method and results on this
issue.
Fixture acquisition
Measure one equivalent repository-fixture acquisition:
and at the current revision if later fixture changes are material.
Use the same machine or runner class, runtime version, fixture shape, and sample
method for every comparison.
Report both the distribution and a representative value rather than one
unexplained elapsed time.
Process and authentication activity
For each acquisition, record:
every Git process started;
whether the local remote opens an authentication session;
whether the credential-helper process starts;
whether any gh process starts; and
the time spent before the test-specific behavior begins.
Use an observable Git shim or equivalent trace rather than inferring process
activity from source code.
Hazard cost
For representative durability scenarios, measure separately:
fixture preparation;
the Git operation;
forced interruption;
recovery and assertion; and
cleanup.
This shows whether the fixture or the behavior being proved dominates the test.
The measurement itself changes no production behavior or test contract.
Decision from the results
Use the evidence to choose one response:
Unnecessary authentication work: If a local remote opens an authentication
session or starts the helper, correct the host so that the command reaches no
mechanism.
Repeated fixture cost: If fixture preparation dominates while the hazards
are inexpensive, evaluate one repository per test file while keeping each
hazard isolated and independently reproducible.
Meaningful hazard cost: If interruption, recovery, or another required
behavior dominates, preserve that cost and record why it is necessary.
Another measured cause: If neither hypothesis explains the increase,
record the observed cause and propose a focused response.
Do not optimize a test by removing the failure or recovery behavior it exists to
prove.
Acceptance
The issue records the revisions, runner, runtime, fixture shape, sample count,
and measurement method.
It reports fixture-acquisition cost for the comparable revisions.
It reports Git process counts and whether authentication, the credential
helper, or gh runs for a local remote.
Representative durability scenarios separate fixture, operation,
interruption, recovery, assertion, and cleanup time.
The evidence identifies a specific cause for the observed increase or rejects
the current hypotheses with enough information for another focused
investigation.
The issue records either a focused follow-up change or why the measured cost
belongs to the behavior contract.
Story
As a maintainer, I want to measure the cost of preparing a Git-backed workflow
test and the cost of the behavior it proves, so slow tests spend their time on
meaningful Git hazards rather than unnecessary fixture work.
What the fixture does
A workflow Git fixture prepares the repository environment used by one test
scenario. Depending on the suite, that may include:
Many test files acquire this fixture once for every scenario. A small fixed cost
there is multiplied across the entire Git and pull-request corpus.
The behavior under test is separate. For example, a durability scenario may
interrupt one Git operation and prove that recovery selects the correct
committed state.
Current evidence
A historical comparison found that the Git and pull-request suites became about
1.6 times slower even though the individual
git-*test files did not change.The shared fixture did change. #532 added production authentication delegation
and a credential-helper launcher to the test host. Its contract says that a
local remote requiring no authentication should reach no authentication
mechanism, but the timing increase affected suites using that fixture generally.
This is evidence for a focused measurement, not yet proof that authentication is
the cause.
Measurement
Run one controlled comparison and record the complete method and results on this
issue.
Fixture acquisition
Measure one equivalent repository-fixture acquisition:
Use the same machine or runner class, runtime version, fixture shape, and sample
method for every comparison.
Report both the distribution and a representative value rather than one
unexplained elapsed time.
Process and authentication activity
For each acquisition, record:
ghprocess starts; andUse an observable Git shim or equivalent trace rather than inferring process
activity from source code.
Hazard cost
For representative durability scenarios, measure separately:
This shows whether the fixture or the behavior being proved dominates the test.
The measurement itself changes no production behavior or test contract.
Decision from the results
Use the evidence to choose one response:
session or starts the helper, correct the host so that the command reaches no
mechanism.
are inexpensive, evaluate one repository per test file while keeping each
hazard isolated and independently reproducible.
behavior dominates, preserve that cost and record why it is necessary.
record the observed cause and propose a focused response.
Do not optimize a test by removing the failure or recovery behavior it exists to
prove.
Acceptance
and measurement method.
helper, or
ghruns for a local remote.interruption, recovery, assertion, and cleanup time.
the current hypotheses with enough information for another focused
investigation.
belongs to the behavior contract.
Out of scope
<WorkflowRun>and<Execution host="workflow">to Markdown tests #547 supplies the missing workflow testingprofile.
polling rather than this repository fixture.
merely to improve timing.