chore(drift-sync): needs-human model-family decision (2026-08-07) - #364
Closed
copilotkit-devops-bot[bot] wants to merge 1 commit into
Closed
chore(drift-sync): needs-human model-family decision (2026-08-07)#364copilotkit-devops-bot[bot] wants to merge 1 commit into
copilotkit-devops-bot[bot] wants to merge 1 commit into
Conversation
commit: |
jpr5
added a commit
that referenced
this pull request
Aug 7, 2026
…an (#365) A model family that a provider's own live `/models` listing no longer returns is a **fact**, not a decision. drift-sync was escalating every one of them to a human, reddening the daily cron and emailing the owner to rubber-stamp bookkeeping. Run `31218975992` did exactly that: ten Anthropic families (`claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku`, `claude-3-5-sonnet`, `claude-3-5-haiku`, `claude-3-7-sonnet`, `claude-opus-4`, `claude-sonnet-4`, `claude-haiku-4`, `claude-opus-4-1`), one PR, one red run, one email — with nothing for a human to actually decide. ## Why it escalated Not judgement. A structural dead end. `includeFamilies` membership is checksum-pinned, drift-sync's own gate re-runs that pin, and the allowlist forbids re-pinning it — so removing a retired family always reds its own gate and delivers nothing. With no place to write the fact down, the only reachable outcome was to ask a human. ## What changes A `deprecatedFamilies` ledger in `model-registry.ts` gives the fact somewhere to live. Recording a deprecation is now mechanical, and both classes collapse to the same action: - **Deprecated and unreferenced** — recorded, no alert. - **Deprecated but still referenced** — recorded, and **the mock keeps serving**. Users' tests pin retired models; removing them to satisfy a ledger entry would break those tests. The reference class is an annotation on the recorded line, not a different route. **A genuinely new or unclassified family still pages a human, unchanged.** "Should aimock mock this?" is a real product question. `includeFamilies` is byte-identical — verified by AST extraction (SHA-256 `281ed637a2f5…`, 2300 bytes both sides), and the pin test reds when a family is added. ## Verification Red and green replay the real production input. The replay reproduces changeset key `9c868d12bd208698` — the exact key from run `31218975992` — and its notes diff **empty** against PR #364, so this is the input that paged the owner, not a synthetic stand-in. - **Red** on `main`: ten `needs-human-*`, `reason=needs-human`, exit 1. - **Green**: ten `[deprecation-recorded]`, **zero** needs-human, `reason=ok-applied`, exit 0. All three gates ran and passed. A second run reports `ok-no-churn` and stays silent. **The nine still-referenced mocks still serve**: built `dist/`, ran the real server, POSTed `/v1/messages` per family — 9/9 HTTP 200 with real mocked bodies. **Negative control**: a synthetic new family `claude-zephyr-9` still yields `reason=needs-human`, exit 1, and a `Decision: pending` note. **The ledger cannot reach the serving path**: zero hits in a fresh `dist/` (`createServer` scores 42, proving the search works), no non-test `src/` file references the registry, and the boundary test reds when the symbol is added to `src/constants.ts`. Ten mutations, all red — including a stray ledger entry, unclassify-on-record, a ledger that swallows every fixture, and detect-but-never-record. Along the way this surfaced four fixtures hard-coding `claude-3-opus` as "genuinely retired", which would have reddened CI on the first post-merge drift PR. They now derive their stand-ins and throw loudly rather than pass vacuously. ## Unproven Listings are replayed rather than fetched live; Anthropic fidelity is established by the changeset-key and note equality above, while openai/gemini inputs are synthesized. The ledger is append-only by choice: a resurrected family would keep a stale marker, traded against record/un-record PR churn.
Contributor
|
Superseded. These ten Anthropic deprecations are now recorded mechanically in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Needs a human decision (drift-sync)
The deterministic, zero-LLM drift-sync found a model-family change it must
NOT auto-apply (a genuinely new/unclassified family, a deprecation —
still-referenced or zero-reference — or a registry structural mismatch). It
wrote the note file(s) below and opened this PR so the decision is REACHABLE
in the repo.
Note file(s) in this PR
drift-proposals/anthropic-claude-3-5-haiku-deprecated-referenced.mddrift-proposals/anthropic-claude-3-5-sonnet-deprecated-referenced.mddrift-proposals/anthropic-claude-3-7-sonnet-deprecated-referenced.mddrift-proposals/anthropic-claude-3-haiku-deprecated-referenced.mddrift-proposals/anthropic-claude-3-opus-deprecated-referenced.mddrift-proposals/anthropic-claude-3-sonnet-deprecated-referenced.mddrift-proposals/anthropic-claude-haiku-4-deprecated-referenced.mddrift-proposals/anthropic-claude-opus-4-1-deprecated-unreferenced.mddrift-proposals/anthropic-claude-opus-4-deprecated-referenced.mddrift-proposals/anthropic-claude-sonnet-4-deprecated-referenced.mddrift-sync outcome