Add changelog registry state discovery and reconciliation commands#3717
Open
cotti wants to merge 1 commit into
Open
Add changelog registry state discovery and reconciliation commands#3717cotti wants to merge 1 commit into
cotti wants to merge 1 commit into
Conversation
Registries merge additively under optimistic concurrency and are not authoritative for removals or discovery, so they can drift from the objects actually in the bucket and nothing could detect or repair that. Backfill planning needs a trustworthy current-state snapshot, so this adds a `changelog registry` command group: - `inspect` (read-only): classifies divergence between a scope's private registry.json and the actual private objects — missing, stale, corrupt, and object-divergent — and can emit a machine-readable RegistryStateSnapshot for the backfill planner. - `repair` (explicit, idempotent): converges the private registry to the actual objects through the same conditional-PUT optimistic concurrency as the live upload path, re-inspecting on 412 so concurrent uploads are never clobbered. Guarded against empty results (--allow-empty) and newer-schema downgrades; fully audited. - `verify-public` (strictly read-only): waits under a bounded retry policy for scrubber propagation and diagnoses public divergence. The public bucket is a hard write boundary enforced structurally: the comparison operates on a reader interface with no write surface. - `republish` (explicit): re-emits the private ObjectCreated event via a metadata-preserving S3 self-copy to recover a lost or DLQ'd scrub event; the scrubber remains the sole public-side writer. Part of elastic/docs-eng-team#670
|
Label error. Requires exactly 1 of: automation, breaking, bug, changelog:skip, chore, ci, dependencies, documentation, enhancement, feature, fix, redesign. Found: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
registry.jsonmanifests (per scope:bundle/{product}/andchangelog/{org}/{repo}/{branch}/) merge additively under optimistic concurrency and are not authoritative for removals or discovery, so they can silently drift from the objects actually in the bucket. Nothing could detect or repair that drift, and backfill planning needs a trustworthy current-state view. This PR adds achangelog registrycommand group backed by a newElastic.Changelog/Reconciliation/service layer:changelog registry inspect(read-only) — compares a scope's private registry against the actual private-bucket objects, prints human-readable diagnostics, and optionally writes a machine-readableRegistryStateSnapshot(--out) that the backfill planner (elastic/docs-eng-team#676) will consume.changelog registry repair(explicit, never implicit; idempotent) — converges the private registry to the actual objects.changelog registry verify-public(strictly read-only) — waits under a bounded retry policy (--max-attempts×--poll-interval-seconds) for scrubber propagation and diagnoses public-side divergence.changelog registry republish(explicit) — re-emits the privateObjectCreatedevent for selected objects to recover a lost/DLQ'd scrub event.Divergence taxonomy
missingstalecorruptobject-divergenttargetis null while the parent bundle resolves one.A manifest with a newer
schema_versionis reported asUnsupportedSchema(not corrupt): its entries can't be judged by an older tool, and repair refuses to rewrite it because that would silently downgrade the schema. For bundle scopes the expectedtargetis re-derived from the bundle YAML in S3 (including the parent-bundle fallback for legacy amends, mirroringRegistryBuilder); changelog scopes enumerate files only.Repair semantics
If-Matchon update,If-None-Match: *on create). A 412 triggers a full re-inspection — fresh registry read and fresh object listing — before the bounded retry, and the registry is always read before listing, so a concurrent upload either appears in the re-listing or invalidates the precondition; it can never be dropped.--dry-runprints the plan without writing; an empty result requires--allow-empty.[RequiresAuth]+[CommandIntent(Intent.Idempotent)]+[MutationScope(MutationScope.Global)](same forrepublish);inspect/verify-publicare[RequiresAuth]only.Public-boundary design
The public bucket is scrubber-owned and is a hard write boundary enforced structurally, not by convention: all inspection and public-side comparison code operates exclusively on a read-only reader interface (
IS3ScopeReader: list + get only), so no code path in the checker can mutate any bucket. Tests additionally assert zeroPutObject/CopyObject/DeleteObjectcalls on the public-side fake.Recovery from a lost scrub event is the explicit private-side
republish: a metadata-preserving S3 self-copy (CopyObjectonto the same key withMetadataDirective: REPLACE, re-supplying the original content type and user metadata) that leaves content and ETag untouched while emitting theObjectCreated:Copynotification the scrubber Lambda reacts to (src/infra/docs-lambda-changelog-scrubber/Program.csmatchesObjectCreated). The scrubber remains the sole public-side writer.Docs
docs/cli/changelog/registry/(namespace + 4 commands);docs/cli-schema.jsonregenerated (verified: diff empty ignoringversion).docs/development/changelog-bundle-registry.mdgains a "State discovery and reconciliation" section, and theregistry.jsonexample is corrected to the actual serialized key (e_tag, notetag).Test plan
45 new tests in
tests/Elastic.Changelog.Tests/Reconciliation/using a stateful in-memory S3 fake (FakeItEasy overIAmazonS3, MD5 ETags matchingS3EtagCalculator, real conditional-PUT 412 semantics):missingIf-None-Match: *, update pathIf-Match--allow-empty; newer schema refused--max-attempts; converges early when the scrubber catches up mid-poll--allincludesregistry.json, partial failures reportedElastic.Changelog.Tests849/849 passed;./build.sh unit-testgreendotnet format+./build.sh lintclean;./build.sh build --skip-dirty-checksucceeds;dotnet publish -c Release(Native AOT) succeeds with zero trim/AOT warnings; repo docs build validates the new supplemental pagesCloses elastic/docs-eng-team#670