DGM: score metadata completeness for fleet promotion (fixes #168) - #170
Merged
Conversation
All six gaps from #168, client side (the coordinated backend half is live on harness-telemetry as of 2026-07-11, so v2 scores verify in prod): - Gap 1: scoreVariants' prov string now carries niches[i] — workflow tournament scores land in a real MAP-Elites cell instead of the anonymous one that promotion can never match. - Gap 2: the score envelope is v2 — niche and provider_class join the HMAC-signed message, so a score for one cell/tier cannot be replayed into another by mutating unsigned transport fields. All signing sites (scoreVariants, runEval, /score) emit v2; user-controlled envelope fields are sanitized (tab/CR/LF) and fingerprints hex-validated BEFORE signing so signed bytes always equal ingested bytes; run_id included. SDK signers/verifiers regenerated to match (v2-first with v1 legacy fallback), pinned by a cross-language golden vector. - Gap 3: fleetEvent asserts (debug builds) that a propose's prompt_text hashes to its claimed prompt_sha; oversized genomes (>64KiB, cap raised from 8KiB) are skipped rather than sent truncated, since a truncated genome can neither validate nor serve. - Gap 4: the wire scale is canonically [0,1]. runEval and scoreVariants divide their 0-100 local scores at the emission boundary (display, eval_best, and targets keep the 0-100 UX); out-of-range evals skip the score AND the paired fleet events; /score accepts an explicit scale ("unit"|"percent") and rejects unknown scale values loudly, with the documented [0,1]/(1,100] heuristic when absent. - Gap 5: scoreVariants proposes the genome before submitting, so a judged variant's cell always has the genome text promotion needs. - Gap 6: run telemetry now carries the same session run_id as score telemetry (and the run branch of the OTLP serializer actually emits it), giving operational and fitness rows a stable join key. zig build test 162/162; SDK regen byte-idempotent; backend suite 23/23 against the same golden vectors. Fixes #168 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Client half of #168 — all six gaps. The coordinated backend (harness-telemetry v2-first verification, legacy v1 window with ingest-time scale normalization, per-row-normalized aggregations, propose fingerprint validation, incumbent-refresh evidence floor) is already deployed to prod (version 726d9395), and the 177 historical 0-100-scale rows were normalized with a pre-migration backup — so this client's v2-only scores verify live from the moment it ships.
See the commit message for the per-gap breakdown. Built by a multi-agent workflow: 2 implementers + cross-repo verifier + 3 adversarial reviewers, a 10-fix review round, then a re-review round (7 findings, all closed). Signature parity is pinned by a golden v2 vector reproduced byte-for-byte in Zig (
src/scoring.zigtest), the worker's actualscoreSignature()(index.test.ts), and the regenerated Python SDK.Verification:
zig build test162/162 · SDK regen byte-idempotent (drift gate green) · backendnode --test23/23 +tsc --noEmitclean · prod post-migration: 0 rows above 1.0, fitness view serves exactly the legitimate reviewer champion (mean 0.96),/v1/statsand/v1/eliteshealthy.Fixes #168