diff --git a/docs/downstream.md b/docs/downstream.md index d08302d..c92da0e 100644 --- a/docs/downstream.md +++ b/docs/downstream.md @@ -6,8 +6,13 @@ Tool-Semantics Milestone 4 metrics are intended for consumers in the ## myelinmesh — usage-weighted change risk **Tracker:** [myelinmesh#21](https://github.com/askmy-stack/myelinmesh/issues/21) -(v0.4 project integrations). myelinmesh already ingests Tool-Semantics reports -via adapter examples under that repository. +(v0.4 project integrations). Tool-Semantics side of the handoff: +[#67](https://github.com/askmy-stack/tool-semantics/issues/67). + +myelinmesh already ingests Tool-Semantics reports via adapter examples under +that repository. **Close #67** when myelinmesh can ingest the metrics JSON +below without further tool-semantics code changes — or open a dedicated API +issue if a new export field is required (do not expand this tracker). ### What to consume from tool-semantics ≥0.4.0 @@ -19,9 +24,39 @@ via adapter examples under that repository. | Stability | `run_probe_trials(...)` → `StabilityReport` | Separate unstable vs deterministic failures | | Stability JSON | `render_stability_json(report)` | Machine-readable trial aggregates | -`ProbeMetrics` distinguishes missing data from failed evaluations -(`missing_data_count`, `failed_evaluation_count`) and uses `None` rates when a -denominator is empty — consumers must not treat missing rates as `0.0`. +### `ProbeMetrics` field contract + +| Field | Type | Notes | +| --- | --- | --- | +| `probe_count` | int | Probes considered | +| `evaluated_count` | int | Probes with a scored outcome | +| `missing_data_count` | int | Distinct from failures | +| `failed_evaluation_count` | int | Runner / eval errors | +| `tool_selection_accuracy` | float \| null | `null` when denominator empty — **not** `0.0` | +| `argument_validity_rate` | float \| null | same | +| `risk_compliance_rate` | float \| null | same | +| `confirmation_compliance_rate` | float \| null | same | +| `per_probe` | object | Per-probe detail map (opaque to ranking; optional evidence) | + +### `StabilityReport` field contract + +| Field | Type | Notes | +| --- | --- | --- | +| `trial_count` | int | Trials per probe | +| `seed` | int \| null | Repro seed when set | +| `summaries[]` | list | Per-probe trial rollups | +| `summaries[].probe_id` | str | | +| `summaries[].stability_score` | float | | +| `summaries[].unstable` | bool | High variance across trials | +| `summaries[].deterministic_failure` | bool | Fails every trial | +| `summaries[].aggregate_passed` | bool | | +| `summaries[].trials[]` | list | Individual trial rows | +| `metrics` | `ProbeMetrics` | Aggregate across trials | +| `runner` | object \| null | `RunnerMetadata` when model-backed | + +`ProbeMetrics` distinguishes missing data from failed evaluations and uses +`None` rates when a denominator is empty — consumers must not treat missing +rates as `0.0`. ### Suggested fixture flow @@ -32,6 +67,13 @@ denominator is empty — consumers must not treat missing rates as `0.0`. 4. On the myelinmesh side, weight structural change severity by selection / argument failure rates when usage telemetry is available. +### Status checklist (for #67) + +- [x] Export fields documented above match `probes.py` models +- [ ] Link myelinmesh adapter PR here when it lands +- [ ] Confirm ingest of metrics JSON without tool-semantics code changes +- [ ] If a new field is required → file a dedicated API issue (not this tracker) + Open a **new** tool-semantics issue only if myelinmesh needs an export field that is not covered by `ProbeMetrics` / `StabilityReport`.