Skip to content

fix(review): contentSignalSourceFromDirectoryEntry ignores ContentRepoSpec, breaking URL duplicate-detection for custom specs #5941

Description

@JSONbored

Context

src/review/content-lane/duplicates.ts synthesizes frontmatter for directory-index entries so they can be run through the same duplicate-detection signal extraction as real content files. Two code paths disagree on which URL fields are considered:

extractContentDuplicateSignals (around line 270) correctly honors the caller-supplied spec: ContentRepoSpec:

const urls = [
  ...new Set(
    Object.entries(fields)
      .filter(([key]) => spec.urlFields.has(key))
      .map(([, value]) => normalizeUrl(value, spec))
      .filter(Boolean),
  ),
];

But contentSignalSourceFromDirectoryEntry (around line 549), which builds the synthetic frontmatter that later gets parsed and fed into extractContentDuplicateSignals, does not take a spec parameter at all — it always emits only a hardcoded field list:

const DIRECTORY_ENTRY_URL_SIGNAL_FIELDS = [/* ...9 hardcoded field names... */, "sourceUrl", "websiteUrl"] as const;

function contentSignalSourceFromDirectoryEntry(entry: DirectoryIndexEntry): string {
  const lines = ["---", `title: ${yamlScalar(entry.title)}`, ...];
  for (const field of DIRECTORY_ENTRY_URL_SIGNAL_FIELDS) {
    const value = entry[field];
    if (value) lines.push(`${field}: ${yamlScalar(value)}`);
  }
  lines.push("---", "");
  return lines.join("\n");
}

directoryIndexToSignals (around line 591) does thread spec through to extractContentDuplicateSignals, but passes contentSignalSourceFromDirectoryEntry(entry) (no spec) as the content argument:

extractContentDuplicateSignals(
  { filePath, content: contentSignalSourceFromDirectoryEntry(entry), label: ..., url: ... },
  spec,
)

For a self-hosted content repo whose ContentRepoSpec.urlFields uses field names other than the 9 hardcoded ones in DIRECTORY_ENTRY_URL_SIGNAL_FIELDS, none of that repo's URL fields ever make it into the synthesized frontmatter — urls in the resulting ContentDuplicateSignals is silently empty for every directory-index entry, breaking URL/domain-based duplicate detection against the accepted-entries corpus for any non-default ContentRepoSpec.

test/unit/content-lane-duplicates.test.ts only tests removing a URL field from the default spec, never a differently-named custom field, so this drift is untested.

Requirements

  • contentSignalSourceFromDirectoryEntry must accept a spec: ContentRepoSpec parameter (defaulting to AWESOME_CLAUDE_CONTENT_SPEC, matching the existing default used elsewhere in this file) and emit frontmatter lines for the fields named in spec.urlFields instead of (or in addition to, if a safe superset approach is simpler) the hardcoded DIRECTORY_ENTRY_URL_SIGNAL_FIELDS list.
  • directoryIndexToSignals must pass its own spec argument through to contentSignalSourceFromDirectoryEntry, so the synthesized frontmatter and the downstream extractContentDuplicateSignals filtering agree on which fields are URL signals.
  • Preserve behavior for the default AWESOME_CLAUDE_CONTENT_SPEC case exactly (same synthesized frontmatter output as today) — this is a fix for the custom-spec case, not a behavior change for the default lane.
  • Keep DirectoryIndexEntry's field access safe for entries missing a given custom field (same if (value) ... guard pattern already in use).

Deliverables

  • contentSignalSourceFromDirectoryEntry(entry, spec) synthesizes URL frontmatter lines using spec.urlFields instead of a fixed hardcoded list.
  • directoryIndexToSignals passes its spec argument through to contentSignalSourceFromDirectoryEntry.
  • A regression test in test/unit/content-lane-duplicates.test.ts using a ContentRepoSpec with a custom (non-default) urlFields set, asserting the resulting ContentDuplicateSignals.urls for a directory-index entry actually contains the custom field's URL.
  • A test confirming the default-spec output is unchanged from before this fix (no regression for AWESOME_CLAUDE_CONTENT_SPEC callers).

Test Coverage Requirements

Aim for 99%+ Codecov patch coverage (100% including the new branch/invariant) on the touched lines in src/review/content-lane/duplicates.ts. This is a fix for silently-empty duplicate-detection signals on a real self-host configuration path, so the custom-spec regression test is required, not just incidental line coverage.

Expected Outcome

A self-hosted content repo with a custom ContentRepoSpec.urlFields gets working URL/domain-based duplicate detection against accepted directory-index entries, instead of urls being silently empty because the synthesized frontmatter only ever emitted the default spec's hardcoded field names.

Links & Resources

  • src/review/content-lane/duplicates.ts (extractContentDuplicateSignals, ~line 260-280; DIRECTORY_ENTRY_URL_SIGNAL_FIELDS + contentSignalSourceFromDirectoryEntry, ~line 535-561; directoryIndexToSignals, ~line 566-599)
  • src/review/content-lane/content-repo-spec.ts (ContentRepoSpec, AWESOME_CLAUDE_CONTENT_SPEC)
  • test/unit/content-lane-duplicates.test.ts (existing test suite)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions