Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 47 additions & 5 deletions docs/downstream.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,13 @@ Tool-Semantics Milestone 4 metrics are intended for consumers in the
## myelinmesh — usage-weighted change risk

**Tracker:** [myelinmesh#21](https://github.com/askmy-stack/myelinmesh/issues/21)
(v0.4 project integrations). myelinmesh already ingests Tool-Semantics reports
via adapter examples under that repository.
(v0.4 project integrations). Tool-Semantics side of the handoff:
[#67](https://github.com/askmy-stack/tool-semantics/issues/67).

myelinmesh already ingests Tool-Semantics reports via adapter examples under
that repository. **Close #67** when myelinmesh can ingest the metrics JSON
below without further tool-semantics code changes — or open a dedicated API
issue if a new export field is required (do not expand this tracker).

### What to consume from tool-semantics ≥0.4.0

Expand All @@ -19,9 +24,39 @@ via adapter examples under that repository.
| Stability | `run_probe_trials(...)` → `StabilityReport` | Separate unstable vs deterministic failures |
| Stability JSON | `render_stability_json(report)` | Machine-readable trial aggregates |

`ProbeMetrics` distinguishes missing data from failed evaluations
(`missing_data_count`, `failed_evaluation_count`) and uses `None` rates when a
denominator is empty — consumers must not treat missing rates as `0.0`.
### `ProbeMetrics` field contract

| Field | Type | Notes |
| --- | --- | --- |
| `probe_count` | int | Probes considered |
| `evaluated_count` | int | Probes with a scored outcome |
| `missing_data_count` | int | Distinct from failures |
| `failed_evaluation_count` | int | Runner / eval errors |
| `tool_selection_accuracy` | float \| null | `null` when denominator empty — **not** `0.0` |
| `argument_validity_rate` | float \| null | same |
| `risk_compliance_rate` | float \| null | same |
| `confirmation_compliance_rate` | float \| null | same |
| `per_probe` | object | Per-probe detail map (opaque to ranking; optional evidence) |

### `StabilityReport` field contract

| Field | Type | Notes |
| --- | --- | --- |
| `trial_count` | int | Trials per probe |
| `seed` | int \| null | Repro seed when set |
| `summaries[]` | list | Per-probe trial rollups |
| `summaries[].probe_id` | str | |
| `summaries[].stability_score` | float | |
| `summaries[].unstable` | bool | High variance across trials |
| `summaries[].deterministic_failure` | bool | Fails every trial |
| `summaries[].aggregate_passed` | bool | |
| `summaries[].trials[]` | list | Individual trial rows |
| `metrics` | `ProbeMetrics` | Aggregate across trials |
| `runner` | object \| null | `RunnerMetadata` when model-backed |

`ProbeMetrics` distinguishes missing data from failed evaluations and uses
`None` rates when a denominator is empty — consumers must not treat missing
rates as `0.0`.

### Suggested fixture flow

Expand All @@ -32,6 +67,13 @@ denominator is empty — consumers must not treat missing rates as `0.0`.
4. On the myelinmesh side, weight structural change severity by selection /
argument failure rates when usage telemetry is available.

### Status checklist (for #67)

- [x] Export fields documented above match `probes.py` models
- [ ] Link myelinmesh adapter PR here when it lands
- [ ] Confirm ingest of metrics JSON without tool-semantics code changes
- [ ] If a new field is required → file a dedicated API issue (not this tracker)

Open a **new** tool-semantics issue only if myelinmesh needs an export field
that is not covered by `ProbeMetrics` / `StabilityReport`.

Expand Down