Skip to content

[docs] docs: unbloat trace graders doc #60359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 1 commit into from
Sep 12, 2026
Merged

[docs] docs: unbloat trace graders doc #60359

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/src/content/docs/experimental/trace-graders.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,9 @@ graders:
run: .github/graders/daily-file-diet-operational-value.sh
```

The compiler freezes the evaluator bytes and records their SHA-256 digest. The evaluator returns absolute operational attainment in `[0,1]` for the run's assigned case. A frozen baseline is optional metadata; when present, gh-aw derives `deltaFromBaseline` without changing the primary value.
The compiler freezes the evaluator bytes, records their SHA-256 digest, and expects the evaluator to return absolute operational attainment in `[0,1]` for the run's assigned case. A frozen baseline is optional metadata; when present, gh-aw derives `deltaFromBaseline` without changing the primary value.

Each result records the complete run subject, operational case, evidence time, maturity, and provenance. Operational-value evaluators may query the repositories declared by their frozen evidence contract. They receive the workflow token through `GH_TOKEN` with the agent job's explicitly declared permissions, but do not receive workflow secrets. Enabling the grader does not add evidence permissions to the agent job.
Each result records the run subject, operational case, evidence time, maturity, and provenance. Evaluators may query only the repositories declared by their frozen evidence contract. They receive `GH_TOKEN` with the agent job's explicitly declared permissions, but no workflow secrets, and enabling the grader does not add evidence permissions to the agent job.

Use the `operational value designer` skill (`/operational-value-designer`) to infer operational value from an agentic workflow and design and verify an operational-value evaluator.

Expand All @@ -88,15 +88,15 @@ The command downloads the original grader artifact and reuses its case, run subj
gh aw graders operational-value report daily-file-diet
```

The report command discovers every completed workflow run from the evaluator's declared adoption time through the current time. It applies the current evaluator digest to every run, including runs created before grader artifacts existed, and writes a structured JSON report, an SVG timeline, and a Markdown report under `reports/operational-value`. The versioned JSON is the machine-readable integration contract. Each report observation has the stable identity `(repository, workflowId, runId, runAttempt, evaluatorDigest)`.
The report command discovers completed workflow runs from the evaluator's declared adoption time onward, applies the current evaluator digest to each run, and writes a structured JSON report, an SVG timeline, and a Markdown report under `reports/operational-value`. The versioned JSON is the machine-readable integration contract, and each observation has the stable identity `(repository, workflowId, runId, runAttempt, evaluatorDigest)`.

Pre-grader runs do not have an archived case or event payload. Their evaluator request contains the run ID, attempt, repository, workflow, ref, commit SHA, event name, creation time, and `case: null`. The evaluator must reconstruct the case from that run subject. A result remains explicitly unavailable when the accepted evidence cannot reconstruct it; missing evidence is never scored as zero.
Pre-grader runs do not have an archived case or event payload, so the evaluator request includes run ID, attempt, repository, workflow, ref, commit SHA, event name, creation time, and `case: null`. The evaluator must reconstruct the case from that run subject. If accepted evidence still cannot reconstruct the case, the result remains unavailable rather than being scored as zero.

Mature numeric observations are cached in Monday-based UTC weekly files under the user cache directory. Cache paths are partitioned by repository, workflow ID, evaluator digest, and week. Independent weeks are evaluated concurrently; use `--concurrency` to control the number of evaluator executions (the default is 8). Use `--refresh` to replay every run, `--until` to set an evidence endpoint, `--cache-dir` to relocate the cache, and `--output` to relocate the report artifacts.
Mature numeric observations are cached in Monday-based UTC weekly files under the user cache directory, partitioned by repository, workflow ID, evaluator digest, and week. Independent weeks are evaluated concurrently; use `--concurrency` to control evaluator executions (default `8`). Use `--refresh` to replay every run, `--until` to set an evidence endpoint, `--cache-dir` to relocate the cache, and `--output` to relocate the report artifacts.

Every run remains present in the JSON report and timeline. Weekly primary means retain only the latest observation for each repeated `opportunityKey` within that week. Evaluators may also declare multiple normalized diagnostic metrics with `latest` or `mean` weekly aggregation; the report plots and tabulates each diagnostic independently without combining it into the primary value. The report includes coverage, errors, frozen contract details, baseline and delta when available, and a warning that the observations do not establish causation.
Every run remains present in the JSON report and timeline. Weekly primary means keep only the latest observation for each repeated `opportunityKey` within that week. Evaluators may also declare normalized diagnostic metrics with `latest` or `mean` weekly aggregation; the report plots and tabulates each diagnostic independently without folding it into the primary value. The report also includes coverage, errors, frozen contract details, baseline and delta when available, and a warning that the observations do not establish causation.

The SVG and Markdown files are standalone local exports, not historical storage or source fixtures. Do not commit live report output merely to retain observations. Consumers such as Central Agentic Ops should ingest the JSON contract and own their presentation. The weekly user cache accelerates replay but is not authoritative; deleting it causes gh-aw to rebuild observations from target-repository run metadata and accepted evidence.
The SVG and Markdown files are standalone local exports, not historical storage or source fixtures. Do not commit live report output merely to retain observations. Consumers such as Central Agentic Ops should ingest the JSON contract and own their presentation. The weekly user cache speeds replay but is not authoritative; deleting it causes gh-aw to rebuild observations from target-repository run metadata and accepted evidence.

## Output files

Expand Down
Loading