Skip to content

engine(content-lane): add the missing documentation_url snake_case alias and make the urlFields pairing assertion derived #9992

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

AWESOME_CLAUDE_CONTENT_SPEC declares that every camelCase URL-bearing frontmatter key also has a
snake_case alias, because the curated list legitimately accepts both conventions. The comment on
sourceUrlFields states the invariant explicitly.

packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:94:

  urlFields: new Set([
    "documentationUrl",
    "docsUrl",
    "downloadUrl",
    "githubUrl",
    "packageUrl",
    "repoUrl",
    "repositoryUrl",
    "sourceUrl",
    "websiteUrl",
    "docs_url",
    "download_url",
    "github_url",
    "package_url",
    "repo_url",
    "repository_url",
    "source_url",
    "website_url",
  ]),

Nine camelCase keys, eight snake_case aliases. documentation_url is missing.

The same gap is in sourceUrlFields at packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:123,
whose own comment says:

    // snake_case aliases, matching urlFields one-for-one (#7250): source-evidence.ts read only the camelCase
    // names, so an entry using a legitimately-listed snake_case key (e.g. the canonical `source_url`) was visible
    // to duplicates.ts but invisible to the source-evidence gate.

"one-for-one" is not true: documentationUrl has no partner in either set.

Two live consumers read these sets by literal parsed key, so a submitted entry that spells the field
documentation_url is invisible to both:

  • src/review/content-lane/duplicates.ts:288.filter(([key]) => spec.urlFields.has(key)). The entry's
    documentation URL never enters ContentDuplicateSignals.urls, so a duplicate submission whose only
    distinguishing URL is documentation_url is not compared against the corpus.
  • src/review/content-lane/source-evidence.ts:273for (const field of spec.sourceUrlFields). The
    documentation URL is never extracted, so it is never fetch-verified as source evidence.

The regression test that is supposed to pin this invariant cannot catch it, because it enumerates the
aliases by hand instead of deriving them. test/unit/content-lane-source-evidence.test.ts:65:

    // Every snake_case alias urlFields carries is now recognized here too.
    for (const field of ["docs_url", "download_url", "github_url", "package_url", "repo_url", "repository_url", "source_url", "website_url"]) {
      expect(AWESOME_CLAUDE_CONTENT_SPEC.sourceUrlFields).toContain(field);
      expect(AWESOME_CLAUDE_CONTENT_SPEC.urlFields.has(field)).toBe(true);
    }

The sibling assertion for protectedFrontmatterFields at test/unit/content-lane-duplicates.test.ts:157
already does this correctly — it walks the whole set and derives the snake_case name — which is why the
protected-fields set has no such gap.

documentation_url appears nowhere in the repository.

Requirements

  • AWESOME_CLAUDE_CONTENT_SPEC.urlFields must contain "documentation_url".
  • AWESOME_CLAUDE_CONTENT_SPEC.sourceUrlFields must contain "documentation_url", placed in the
    snake_case alias block alongside the existing eight, preserving the file's existing ordering
    convention (camelCase block first, then the snake_case block).
  • The enumerated alias loop at test/unit/content-lane-source-evidence.test.ts:65 must be replaced by a
    DERIVED assertion over the whole set — for every member of urlFields that is camelCase (contains an
    uppercase letter), its field.replace(/[A-Z]/g, (l) => "_" + l.toLowerCase()) form must also be present in
    urlFields AND in sourceUrlFields. A hard-coded list with documentation_url appended does NOT satisfy
    this bullet: the point is that the next added URL field cannot reopen the same gap.
  • sourceUrlFields ordering must stay meaningful: its own doc comment says it is read "in extraction ORDER"
    by the source-evidence gate, so the new alias is appended to the snake_case block, not interleaved into the
    camelCase block.
  • What must NOT change: distributionSourceFields, primaryCanonicalSourceFields,
    protectedFrontmatterFields, domainOnlyExclusions, multiEntryCatalogUrls, and
    sourceUrlListFields are all correct as shipped and must be left byte-identical.
    documentationUrl must remain a scalar-only field (it must NOT be added to sourceUrlListFields) —
    test/unit/content-lane-source-evidence.test.ts:93 depends on that.

⚠️ Required pattern: mirror the derived pairing assertion at
test/unit/content-lane-duplicates.test.ts:157 (toSnakeCase walked over the entire set), which is
already the working precedent in this repo for exactly this invariant. What does NOT satisfy this issue:
(a) adding "documentation_url" to the two sets and leaving the hand-enumerated test loop in place;
(b) introducing a generic runtime "normalize any key to camelCase before lookup" layer in
duplicates.ts / source-evidence.ts — that changes matching behaviour for every field in every spec and
is far wider than this defect; (c) a test-only PR that asserts the gap exists without closing it.

Deliverables

  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts"documentation_url" added
    to both urlFields and sourceUrlFields.
  • test/unit/content-lane-source-evidence.test.ts — the enumerated alias loop replaced with a derived
    loop over AWESOME_CLAUDE_CONTENT_SPEC.urlFields, asserting for every camelCase member that its
    snake_case form is in urlFields and in sourceUrlFields. This test must fail against the current
    content-repo-spec.ts and pass after the fix.
  • test/unit/content-lane-source-evidence.test.ts — a regression test named for this bug asserting
    extractSubmittedSourceUrls on frontmatter containing only
    documentation_url: https://docs.acme.example/guide returns exactly one entry
    { field: "documentation_url", url: "https://docs.acme.example/guide" }.
  • test/unit/content-lane-duplicates.test.ts — a regression test asserting
    extractContentDuplicateSignals on an entry whose only URL frontmatter key is documentation_url
    produces a non-empty urls array containing the normalized URL (today it is []).
  • A new engine-package test at packages/loopover-engine/test/content-repo-spec.test.ts (this file does
    not exist yet; create it, importing AWESOME_CLAUDE_CONTENT_SPEC from
    ../dist/review/content-lane/content-repo-spec.js per the convention in
    packages/loopover-engine/test/content-lane-flag.test.ts) asserting the SAME derived camelCase ↔
    snake_case pairing invariant over urlFields and sourceUrlFields.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
one that adds "documentation_url" to the two sets and adds the root test/unit assertions but skips the
new packages/loopover-engine/test/content-repo-spec.test.ts file — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's
coverage.include covers src/**/*.ts and packages/loopover-engine/src/**/*.ts — the touched path
packages/loopover-engine/src/review/content-lane/content-repo-spec.ts IS measured. The change adds set
members rather than branches, but the two consumers it re-enables (duplicates.ts:288's
spec.urlFields.has(key) filter and source-evidence.ts:273's field loop) have both arms exercised only
if the new tests assert BOTH a documentation_url entry being picked up AND an unrelated non-URL key still
being excluded; assert both.

Engine lines are credited by two uploads whose hits are unioned — add the test to
packages/loopover-engine/test/** as well as any root test/** coverage, or the patch gate can still fail.
That is precisely why the new packages/loopover-engine/test/content-repo-spec.test.ts file is a required
Deliverable and not optional.

Expected Outcome

A curated-list entry that spells its documentation link documentation_url — a spelling the spec already
accepts for eight sibling URL fields — is seen by duplicate detection and fetch-verified by the
source-evidence gate, exactly like documentationUrl. The pairing invariant is enforced by a derived
assertion in both the root and engine test suites, so the next URL field added to the spec cannot silently
reintroduce a half-registered alias.

Links & Resources

  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:94urlFields, missing documentation_url
  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:123sourceUrlFields and its "matching urlFields one-for-one" comment
  • src/review/content-lane/duplicates.ts:288urlFields consumer
  • src/review/content-lane/source-evidence.ts:273sourceUrlFields consumer
  • test/unit/content-lane-source-evidence.test.ts:65 — the hand-enumerated alias assertion that cannot catch this
  • test/unit/content-lane-duplicates.test.ts:157 — the derived pairing assertion to mirror

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions