diff --git a/packages/gittensory-miner/README.md b/packages/gittensory-miner/README.md index 8307bb47f7..e144139d98 100644 --- a/packages/gittensory-miner/README.md +++ b/packages/gittensory-miner/README.md @@ -27,6 +27,14 @@ The package also includes a metadata-only ranker: `rankCandidateIssues` composes (potential, feasibility, lane fit, freshness, dup risk) and returns fan-out candidates sorted by `rankScore`. It never clones source and never writes to GitHub. +Discovery is per-tenant, not github.com-specific (#4784): `lib/forge-config.js` (`resolveForgeConfig`) holds the +forge base URL, API version, request headers, repo path, search endpoint/qualifiers, user-agent, and credential env +var behind one resolver with gittensory's github.com values as the only defaults, so the fan-out targets another +forge unchanged. `gittensory-miner discover` surfaces `--api-base-url ` and `--token-env ` and forwards a +tenant goal spec to the ranker, printing `usedDefaultGoalSpec` so a fall-back to the built-in rubric is explicit +rather than silent. See [`docs/repo-agnostic-capability-audit.md`](docs/repo-agnostic-capability-audit.md) for the +#4780 audit this executes. + The package also includes an append-only governor decision ledger: `initGovernorLedger` / `appendGovernorEvent` persist structured allow/deny/throttle/kill-switch outcomes in local SQLite for contributor audit. Insert-only — no enforcement wiring yet. (#2328) diff --git a/packages/gittensory-miner/docs/repo-agnostic-capability-audit.md b/packages/gittensory-miner/docs/repo-agnostic-capability-audit.md index 489b472f54..1f108ac8c9 100644 --- a/packages/gittensory-miner/docs/repo-agnostic-capability-audit.md +++ b/packages/gittensory-miner/docs/repo-agnostic-capability-audit.md @@ -85,15 +85,45 @@ redo them: ## Prioritized checklist for #4784 -- [ ] **High — forge abstraction (`opportunity-fanout.js`):** move the GitHub API version (7), +- [x] **High — forge abstraction (`opportunity-fanout.js`):** move the GitHub API version (7), headers (69–71), repo path (83), search endpoint (241), and search-qualifier dialect (202) behind a per-tenant forge adapter; keep GitHub as the default. -- [ ] **High — thread `apiBaseUrl` to the CLI (`discover-cli.js`):** surface the already-supported +- [x] **High — thread `apiBaseUrl` to the CLI (`discover-cli.js`):** surface the already-supported `opportunity-fanout` `apiBaseUrl` override via config / a flag so a non-`api.github.com` host is reachable. -- [ ] **Medium — credential env var (`discover-cli.js:102`):** make the token env var name +- [x] **Medium — credential env var (`discover-cli.js:102`):** make the token env var name configurable (default `GITHUB_TOKEN`). -- [ ] **Medium — pass tenant label taxonomy + goal spec through (`opportunity-ranker.js`, engine +- [x] **Medium — pass tenant label taxonomy + goal spec through (`opportunity-ranker.js`, engine `DEFAULT_TYPE_LABELS`):** ensure the miner supplies the tenant's labels / goal spec instead of silently falling back to the gittensory defaults. -- [ ] **Low — configurable user-agent (`opportunity-fanout.js:70`).** +- [x] **Low — configurable user-agent (`opportunity-fanout.js:70`).** + +## Resolution (#4784) + +All five checklist items are resolved (or, where noted, explicitly deferred with a reason); gittensory's +own github.com conventions survive only as defaults, and the existing gittensory discovery path is +unchanged (`resolveForgeConfig()` with no overrides is byte-identical to the pre-#4784 hardcoded +behavior). + +- **Forge abstraction — resolved.** [`lib/forge-config.js`](../lib/forge-config.js) is the per-tenant + forge adapter: `DEFAULT_FORGE_CONFIG` holds every github.com value (base URL, API version + version + header name, `accept` header, user-agent, repo path prefix, search endpoint, search qualifiers, + token env var) and `resolveForgeConfig(overrides)` fills any missing field from that default. + `opportunity-fanout.js` now reads these from the resolved forge instead of module constants, so the + API version, request headers, repo path, search endpoint, and search-qualifier dialect are all + per-tenant. +- **`apiBaseUrl` reachable from the CLI — resolved.** `discover` accepts `--api-base-url ` (and + `runDiscover({ apiBaseUrl })`), threading the forge host that the fan-out already supported but that + the CLI never surfaced. A programmatic caller can also pass the rest of the forge knobs via + `runDiscover({ forge })`. +- **Credential env var — resolved.** `discover --token-env ` (and `runDiscover({ tokenEnv })`) + reads a non-`GITHUB_TOKEN` variable, defaulting to `GITHUB_TOKEN`. +- **Tenant goal spec through — resolved.** `runDiscover` forwards `goalSpecsByRepo` / + `goalSpecContentByRepo` to the ranker and surfaces `usedDefaultGoalSpec` in both the JSON and the + human-readable summary, so the fall-back to gittensory's built-in rubric is explicit rather than + silent. The **discovery** label taxonomy is the goal spec's generic `preferredLabels` / + `blockedLabels` (already per-tenant). The engine's `DEFAULT_TYPE_LABELS` (`gittensor:*`) is a + **review-stack** default (overridable per repo via the focus manifest) that the miner discovery + ranker never consults, so it is **deferred**: changing it belongs to the review path, not #4784's + discovery/claim scope. +- **Configurable user-agent — resolved.** `forge.userAgent` (default `loopover-miner`). diff --git a/packages/gittensory-miner/lib/discover-cli.d.ts b/packages/gittensory-miner/lib/discover-cli.d.ts index 370b143935..9f2c662216 100644 --- a/packages/gittensory-miner/lib/discover-cli.d.ts +++ b/packages/gittensory-miner/lib/discover-cli.d.ts @@ -1,9 +1,15 @@ +import type { ForgeConfig } from "./forge-config.js"; import type { CandidateIssueWarning, + FanoutOptions, FanoutTarget, RawCandidateIssue, } from "./opportunity-fanout.js"; -import type { RankedCandidateIssue, RankedCandidateSummary } from "./opportunity-ranker.js"; +import type { + RankCandidateIssuesOptions, + RankedCandidateIssue, + RankedCandidateSummary, +} from "./opportunity-ranker.js"; import type { EnqueueRankedDiscoverySummary } from "./portfolio-discovery.js"; import type { PortfolioQueueStore } from "./portfolio-queue.js"; @@ -12,6 +18,10 @@ export type ParsedDiscoverArgs = targets: FanoutTarget[]; search: string | null; json: boolean; + /** Present only when `--api-base-url` is supplied (#4784); threads the tenant's forge host to the fan-out. */ + apiBaseUrl?: string; + /** Present only when `--token-env` is supplied (#4784); names the credential env var to read. */ + tokenEnv?: string; } | { error: string }; @@ -33,27 +43,36 @@ export type DiscoverResult = { rateLimitRemaining: number | null; rateLimitResetAt: string | null; ranked: DiscoverRankedEntry[]; + /** True when ranking fell back to the built-in default goal spec because no per-tenant spec was supplied (#4784). */ + usedDefaultGoalSpec?: boolean; enqueueSummary: EnqueueRankedDiscoverySummary; }; export type RunDiscoverOptions = { githubToken?: string; apiBaseUrl?: string; + /** Per-tenant credential env var name (#4784); defaults to GITHUB_TOKEN. Overridden by a `--token-env` flag. */ + tokenEnv?: string; + /** Per-tenant forge knobs beyond the host (#4784), forwarded to the fan-out. */ + forge?: Partial; nowMs?: number; + /** Per-tenant goal specs threaded to the ranker so lane fit uses the tenant's conventions, not the defaults (#4784). */ + goalSpecsByRepo?: RankCandidateIssuesOptions["goalSpecsByRepo"]; + goalSpecContentByRepo?: RankCandidateIssuesOptions["goalSpecContentByRepo"]; initPortfolioQueue?: () => PortfolioQueueStore; fetchCandidateIssuesWithSummary?: ( targets: FanoutTarget[], githubToken: string, - options?: { apiBaseUrl?: string }, + options?: FanoutOptions, ) => Promise; searchCandidateIssuesWithSummary?: ( searchQuery: string, githubToken: string, - options?: { apiBaseUrl?: string }, + options?: FanoutOptions, ) => Promise; rankCandidateIssuesWithSummary?: ( candidates: RawCandidateIssue[], - options?: { nowMs?: number }, + options?: RankCandidateIssuesOptions, ) => RankedCandidateSummary; enqueueRankedDiscovery?: ( rankedIssues: RankedCandidateIssue[], diff --git a/packages/gittensory-miner/lib/discover-cli.js b/packages/gittensory-miner/lib/discover-cli.js index d392afb58b..19882e625f 100644 --- a/packages/gittensory-miner/lib/discover-cli.js +++ b/packages/gittensory-miner/lib/discover-cli.js @@ -1,5 +1,6 @@ /** `discover` CLI command (#4247): wires the existing fanout -> rank -> enqueue pipeline together so a miner * can actually run it. Every piece already exists and is independently tested; this module only composes them. */ +import { resolveForgeConfig } from "./forge-config.js"; import { fetchCandidateIssuesWithSummary, searchCandidateIssuesWithSummary, @@ -9,7 +10,7 @@ import { enqueueRankedDiscovery } from "./portfolio-discovery.js"; import { initPortfolioQueueStore } from "./portfolio-queue.js"; const DISCOVER_USAGE = - "Usage: gittensory-miner discover [...] | --search [--json]"; + "Usage: gittensory-miner discover [...] | --search [--json] [--api-base-url ] [--token-env ]"; const MAX_DISCOVER_TITLE_DISPLAY_LENGTH = 240; const OSC_SEQUENCE_PATTERN = /\u001b\][\s\S]*?(?:\u0007|\u001b\\)/g; @@ -36,7 +37,10 @@ function parseRepoTarget(value) { } export function parseDiscoverArgs(args) { - const options = { json: false, search: null }; + // `--api-base-url` and `--token-env` (#4784) thread the tenant's forge host and credential env var into the + // fan-out; they are kept off the parsed result unless supplied, so callers that pass neither see the exact + // pre-#4784 `{ targets, search, json }` shape. + const options = { json: false, search: null, apiBaseUrl: null, tokenEnv: null }; const targets = []; for (let index = 0; index < args.length; index += 1) { @@ -52,6 +56,20 @@ export function parseDiscoverArgs(args) { index += 1; continue; } + if (token === "--api-base-url") { + const value = args[index + 1]; + if (!value || value.startsWith("-")) return { error: DISCOVER_USAGE }; + options.apiBaseUrl = value; + index += 1; + continue; + } + if (token === "--token-env") { + const value = args[index + 1]; + if (!value || value.startsWith("-")) return { error: DISCOVER_USAGE }; + options.tokenEnv = value; + index += 1; + continue; + } if (token.startsWith("-")) { return { error: `Unknown option: ${token}` }; } @@ -67,7 +85,13 @@ export function parseDiscoverArgs(args) { return { error: "Pass either repository targets or --search, not both." }; } - return { targets, search: options.search, json: options.json }; + return { + targets, + search: options.search, + json: options.json, + ...(options.apiBaseUrl !== null ? { apiBaseUrl: options.apiBaseUrl } : {}), + ...(options.tokenEnv !== null ? { tokenEnv: options.tokenEnv } : {}), + }; } // The rate-limit line surfaces the telemetry the fanout already records (#4837) so an operator sees how close a @@ -90,6 +114,13 @@ export function renderDiscoverSummary(result) { if (result.enqueueSummary.skippedBelowMinRank > 0) { lines.push(`skipped (below min rank): ${result.enqueueSummary.skippedBelowMinRank}`); } + // Make the fall-back to gittensory's built-in rubric explicit instead of silent (#4784): when no per-tenant goal + // spec is supplied, lane fit reflects gittensory's defaults, not the target repo's own conventions. + if (result.usedDefaultGoalSpec) { + lines.push( + "note: ranked with the built-in default goal spec (no per-tenant .gittensory-miner.yml supplied)", + ); + } if (result.ranked.length === 0) { lines.push("", "no candidates found."); return lines.join("\n"); @@ -109,7 +140,16 @@ export async function runDiscover(args, options = {}) { return 2; } - const githubToken = options.githubToken ?? process.env.GITHUB_TOKEN ?? ""; + // Credential env var is per-tenant (#4784): a `--token-env FORGE_PAT` flag (or `options.tokenEnv`) reads a + // non-`GITHUB_TOKEN` variable so a non-github.com forge's token is reachable. The default falls through to the + // forge adapter's own `tokenEnvVar` (github.com's `GITHUB_TOKEN`), so there's a single source of truth for the + // default credential env instead of a second hardcoded literal that could drift from `DEFAULT_FORGE_CONFIG`. + const tokenEnv = parsed.tokenEnv ?? options.tokenEnv ?? resolveForgeConfig(options.forge).tokenEnvVar; + const githubToken = options.githubToken ?? process.env[tokenEnv] ?? ""; + // A `--api-base-url` flag (or `options.apiBaseUrl`) surfaces the fan-out's existing forge-host override at the CLI + // (#4784); `options.forge` carries any remaining per-tenant forge knobs for a programmatic caller. + const apiBaseUrl = parsed.apiBaseUrl ?? options.apiBaseUrl; + const fanOutOptions = { apiBaseUrl, forge: options.forge }; const fetchTargets = options.fetchCandidateIssuesWithSummary ?? fetchCandidateIssuesWithSummary; const searchTargets = options.searchCandidateIssuesWithSummary ?? searchCandidateIssuesWithSummary; const rankIssues = options.rankCandidateIssuesWithSummary ?? rankCandidateIssuesWithSummary; @@ -121,10 +161,17 @@ export async function runDiscover(args, options = {}) { try { const fanOut = parsed.search !== null - ? await searchTargets(parsed.search, githubToken, { apiBaseUrl: options.apiBaseUrl }) - : await fetchTargets(parsed.targets, githubToken, { apiBaseUrl: options.apiBaseUrl }); - - const rankedSummary = rankIssues(fanOut.issues, { nowMs: options.nowMs }); + ? await searchTargets(parsed.search, githubToken, fanOutOptions) + : await fetchTargets(parsed.targets, githubToken, fanOutOptions); + + // Pass any caller-supplied per-tenant goal specs through to the ranker so lane fit uses the tenant's + // conventions instead of silently falling back to gittensory's defaults (#4784); the fallback is surfaced via + // `usedDefaultGoalSpec` below rather than hidden. + const rankedSummary = rankIssues(fanOut.issues, { + nowMs: options.nowMs, + goalSpecsByRepo: options.goalSpecsByRepo, + goalSpecContentByRepo: options.goalSpecContentByRepo, + }); const enqueueSummary = enqueue(rankedSummary.issues, { queueStore: portfolioQueue }); const result = { @@ -133,6 +180,7 @@ export async function runDiscover(args, options = {}) { rateLimitRemaining: fanOut.rateLimitRemaining, rateLimitResetAt: fanOut.rateLimitResetAt, ranked: rankedSummary.issues, + usedDefaultGoalSpec: rankedSummary.usedDefaultGoalSpec, enqueueSummary, }; diff --git a/packages/gittensory-miner/lib/forge-config.d.ts b/packages/gittensory-miner/lib/forge-config.d.ts new file mode 100644 index 0000000000..d8774d18bb --- /dev/null +++ b/packages/gittensory-miner/lib/forge-config.d.ts @@ -0,0 +1,17 @@ +/** Per-tenant forge configuration (#4784). Every field is a string knob defaulting to the github.com value in + * `DEFAULT_FORGE_CONFIG`; a tenant overrides only what differs for their forge. */ +export type ForgeConfig = { + apiBaseUrl: string; + apiVersion: string; + apiVersionHeader: string; + acceptHeader: string; + userAgent: string; + repoPathPrefix: string; + searchEndpoint: string; + searchQualifiers: string; + tokenEnvVar: string; +}; + +export const DEFAULT_FORGE_CONFIG: Readonly; + +export function resolveForgeConfig(overrides?: Partial): ForgeConfig; diff --git a/packages/gittensory-miner/lib/forge-config.js b/packages/gittensory-miner/lib/forge-config.js new file mode 100644 index 0000000000..88b67eddc6 --- /dev/null +++ b/packages/gittensory-miner/lib/forge-config.js @@ -0,0 +1,37 @@ +/** Per-tenant forge configuration (#4784): the GitHub-specific protocol details that discovery used to hardcode, + * gathered behind one resolver so a non-github.com tenant (GitHub Enterprise, or another GitHub-compatible forge) + * can override them. gittensory's own github.com conventions survive only as `DEFAULT_FORGE_CONFIG` — calling + * `resolveForgeConfig()` with no overrides is byte-identical to the pre-#4784 hardcoded fan-out behavior, which is + * what keeps the existing gittensory discovery path unchanged. Executes the #4780 repo-agnostic-capability-audit + * checklist (forge abstraction, configurable credential env var, configurable user-agent). */ + +/** The github.com defaults every forge field falls back to. Frozen so a caller can't mutate the shared baseline. */ +export const DEFAULT_FORGE_CONFIG = Object.freeze({ + apiBaseUrl: "https://api.github.com", + apiVersion: "2022-11-28", + apiVersionHeader: "x-github-api-version", + acceptHeader: "application/vnd.github+json", + userAgent: "loopover-miner", + repoPathPrefix: "/repos", + searchEndpoint: "/search/issues", + searchQualifiers: "state:open type:issue", + tokenEnvVar: "GITHUB_TOKEN", +}); + +function trimmedStringOr(value, fallback) { + return typeof value === "string" && value.trim() ? value.trim() : fallback; +} + +/** + * Resolve a full forge config from partial per-tenant overrides. Every field is an independent string knob that + * falls back to its github.com default when the override is missing, non-string, or blank — so a partial override + * (say, only `apiBaseUrl` for a GitHub Enterprise host) still yields a complete, usable config. + */ +export function resolveForgeConfig(overrides = {}) { + const source = overrides && typeof overrides === "object" ? overrides : {}; + const resolved = {}; + for (const [key, fallback] of Object.entries(DEFAULT_FORGE_CONFIG)) { + resolved[key] = trimmedStringOr(source[key], fallback); + } + return resolved; +} diff --git a/packages/gittensory-miner/lib/opportunity-fanout.d.ts b/packages/gittensory-miner/lib/opportunity-fanout.d.ts index d179b9da14..c288f279da 100644 --- a/packages/gittensory-miner/lib/opportunity-fanout.d.ts +++ b/packages/gittensory-miner/lib/opportunity-fanout.d.ts @@ -1,8 +1,23 @@ +import type { ForgeConfig } from "./forge-config.js"; + export type FanoutTarget = { owner: string; repo: string; }; +/** Options shared by every fan-out entry point. `apiBaseUrl` is the legacy top-level forge-host override (it still + * wins over `forge.apiBaseUrl`); `forge` (#4784) carries the rest of the per-tenant forge knobs. */ +export type FanoutOptions = { + apiBaseUrl?: string; + forge?: Partial; + concurrency?: number; + rateLimitLowWaterMark?: number; + rateLimitHighWaterMark?: number; + perPage?: number; + maxPages?: number; + sleepFn?: (ms: number) => Promise; +}; + export type RawCandidateIssue = { owner: string; repo: string; @@ -42,51 +57,23 @@ export function mapWithConcurrency( export function fetchCandidateIssuesWithSummary( targets: FanoutTarget[], githubToken: string, - options?: { - apiBaseUrl?: string; - concurrency?: number; - rateLimitLowWaterMark?: number; - rateLimitHighWaterMark?: number; - perPage?: number; - sleepFn?: (ms: number) => Promise; - }, + options?: FanoutOptions, ): Promise; export function fetchCandidateIssues( targets: FanoutTarget[], githubToken: string, - options?: { - apiBaseUrl?: string; - concurrency?: number; - rateLimitLowWaterMark?: number; - rateLimitHighWaterMark?: number; - perPage?: number; - sleepFn?: (ms: number) => Promise; - }, + options?: FanoutOptions, ): Promise; export function searchCandidateIssuesWithSummary( searchQuery: string, githubToken: string, - options?: { - apiBaseUrl?: string; - concurrency?: number; - rateLimitLowWaterMark?: number; - rateLimitHighWaterMark?: number; - perPage?: number; - sleepFn?: (ms: number) => Promise; - }, + options?: FanoutOptions, ): Promise; export function searchCandidateIssues( searchQuery: string, githubToken: string, - options?: { - apiBaseUrl?: string; - concurrency?: number; - rateLimitLowWaterMark?: number; - rateLimitHighWaterMark?: number; - perPage?: number; - sleepFn?: (ms: number) => Promise; - }, + options?: FanoutOptions, ): Promise; diff --git a/packages/gittensory-miner/lib/opportunity-fanout.js b/packages/gittensory-miner/lib/opportunity-fanout.js index 348aa87ded..aed5c8f964 100644 --- a/packages/gittensory-miner/lib/opportunity-fanout.js +++ b/packages/gittensory-miner/lib/opportunity-fanout.js @@ -1,5 +1,6 @@ import { Buffer } from "node:buffer"; import { resolveAiPolicyVerdict } from "@jsonbored/gittensory-engine"; +import { resolveForgeConfig } from "./forge-config.js"; import { DEFAULT_RATE_LIMIT_HIGH_WATER_MARK, DEFAULT_RATE_LIMIT_LOW_WATER_MARK, @@ -7,7 +8,6 @@ import { } from "./discovery-throttle.js"; import { fetchWithRetry } from "./http-retry.js"; -const defaultApiBaseUrl = "https://api.github.com"; const defaultConcurrency = 5; // How long a parked worker waits before re-checking the live rate-limit-derived concurrency limit (#4844). const throttleParkMs = 25; @@ -15,7 +15,6 @@ const defaultPerPage = 100; // Follow the GitHub Link header past the first page so a repo/search with >100 open issues isn't silently // truncated (#4831); cap the follow loop so a pathological Link chain can't run away. const defaultMaxPages = 10; -const githubApiVersion = "2022-11-28"; function normalizeLimit(value, fallback, min, max) { if (!Number.isFinite(value)) return fallback; @@ -48,13 +47,21 @@ function targetFromFullName(fullName) { return { owner, repo, repoFullName: `${owner}/${repo}` }; } -function targetFromSearchIssue(issue) { +function escapeRegExp(value) { + return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +// Derive owner/repo from a search hit when `repository.full_name` is absent, using the tenant forge's own +// `repoPathPrefix` for the API `repository_url` and a forge-agnostic host for the web `html_url` (#4784). Hardcoding +// `/repos/` and `github.com` here dropped every custom-forge search result whose payload omitted `full_name`. +function targetFromSearchIssue(issue, forge) { const repositoryFullName = targetFromFullName(issue?.repository?.full_name); if (repositoryFullName) return repositoryFullName; + const repoPathPrefix = escapeRegExp(forge.repoPathPrefix.replace(/\/+$/, "")); const repositoryUrl = typeof issue?.repository_url === "string" - ? issue.repository_url.match(/\/repos\/([^/?#]+)\/([^/?#]+)(?:[?#].*)?$/) + ? issue.repository_url.match(new RegExp(`${repoPathPrefix}/([^/?#]+)/([^/?#]+)(?:[?#].*)?$`)) : null; if (repositoryUrl) { const owner = decodeURIComponent(repositoryUrl[1]); @@ -64,7 +71,7 @@ function targetFromSearchIssue(issue) { const htmlUrl = typeof issue?.html_url === "string" - ? issue.html_url.match(/^https:\/\/github\.com\/([^/]+)\/([^/]+)\/issues\/\d+(?:[?#].*)?$/) + ? issue.html_url.match(/^https:\/\/[^/]+\/([^/]+)\/([^/]+)\/issues\/\d+(?:[?#].*)?$/) : null; if (htmlUrl) { const owner = decodeURIComponent(htmlUrl[1]); @@ -75,11 +82,11 @@ function targetFromSearchIssue(issue) { return null; } -function githubHeaders(githubToken) { +function githubHeaders(githubToken, forge) { const headers = { - accept: "application/vnd.github+json", - "user-agent": "loopover-miner", - "x-github-api-version": githubApiVersion, + accept: forge.acceptHeader, + "user-agent": forge.userAgent, + [forge.apiVersionHeader]: forge.apiVersion, }; const token = typeof githubToken === "string" ? githubToken.trim() : ""; if (token) headers.authorization = `Bearer ${token}`; @@ -90,8 +97,8 @@ function apiUrl(apiBaseUrl, path, query = "") { return `${apiBaseUrl.replace(/\/+$/, "")}${path}${query}`; } -function repoPath(target, suffix) { - return `/repos/${encodeURIComponent(target.owner)}/${encodeURIComponent(target.repo)}${suffix}`; +function repoPath(forge, target, suffix) { + return `${forge.repoPathPrefix}/${encodeURIComponent(target.owner)}/${encodeURIComponent(target.repo)}${suffix}`; } function recordRateLimit(summary, response) { @@ -118,7 +125,7 @@ async function githubGetJson(url, githubToken, summary, options) { const response = await fetchWithRetry( fetch, url, - { method: "GET", headers: githubHeaders(githubToken) }, + { method: "GET", headers: githubHeaders(githubToken, options.forge) }, { sleepFn: options?.sleepFn }, ); recordRateLimit(summary, response); @@ -142,7 +149,7 @@ function warning(target, stage, message) { async function fetchRepoDoc(target, path, githubToken, options, summary, warnings) { const url = apiUrl( options.apiBaseUrl, - repoPath(target, `/contents/${encodeURIComponent(path)}`), + repoPath(options.forge, target, `/contents/${encodeURIComponent(path)}`), ); try { const { response, payload } = await githubGetJson(url, githubToken, summary, options); @@ -211,10 +218,10 @@ function normalizeIssue(target, issue, policySource) { }; } -function searchQueryWithIssueQualifiers(searchQuery) { +function searchQueryWithIssueQualifiers(searchQuery, forge) { const trimmed = typeof searchQuery === "string" ? searchQuery.trim() : ""; if (!trimmed) return ""; - return `${trimmed} state:open type:issue`; + return `${trimmed} ${forge.searchQualifiers}`; } // The URL of the next page from a GitHub Link header (`; rel="next"`), or null when this is the last page. @@ -230,7 +237,7 @@ async function fetchTargetIssues(target, githubToken, options, summary, warnings let url = apiUrl( options.apiBaseUrl, - repoPath(target, "/issues"), + repoPath(options.forge, target, "/issues"), `?state=open&per_page=${options.perPage}`, ); const issues = []; @@ -261,12 +268,12 @@ async function fetchTargetIssues(target, githubToken, options, summary, warnings } async function fetchSearchIssues(searchQuery, githubToken, options, summary, warnings) { - const qualifiedQuery = searchQueryWithIssueQualifiers(searchQuery); + const qualifiedQuery = searchQueryWithIssueQualifiers(searchQuery, options.forge); if (!qualifiedQuery) return []; let url = apiUrl( options.apiBaseUrl, - "/search/issues", + options.forge.searchEndpoint, `?q=${encodeURIComponent(qualifiedQuery)}&per_page=${options.perPage}`, ); const items = []; @@ -352,11 +359,16 @@ function liveConcurrencyResolver(normalizedOptions, summary) { } function normalizeOptions(options = {}) { + // A legacy top-level `apiBaseUrl` (the pre-#4784 GitHub-Enterprise override every existing caller uses) still wins + // over `forge.apiBaseUrl`, so nothing that already passes `apiBaseUrl` changes behavior. + const apiBaseUrlOverride = + typeof options.apiBaseUrl === "string" && options.apiBaseUrl.trim() + ? { apiBaseUrl: options.apiBaseUrl } + : {}; + const forge = resolveForgeConfig({ ...(options.forge ?? {}), ...apiBaseUrlOverride }); return { - apiBaseUrl: - typeof options.apiBaseUrl === "string" && options.apiBaseUrl.trim() - ? options.apiBaseUrl.trim() - : defaultApiBaseUrl, + forge, + apiBaseUrl: forge.apiBaseUrl, concurrency: normalizeLimit(options.concurrency, defaultConcurrency, 1, 10), // Below/above these recorded-rate-limit-remaining marks the fanout serializes / runs at full concurrency; in // between it scales down linearly (#4844). @@ -423,7 +435,7 @@ export async function searchCandidateIssuesWithSummary(searchQuery, githubToken, const targetsByKey = new Map(); for (const item of searchItems) { if (!item || typeof item !== "object" || item.pull_request) continue; - const target = targetFromSearchIssue(item); + const target = targetFromSearchIssue(item, normalizedOptions.forge); if (target && !targetsByKey.has(targetKey(target))) targetsByKey.set(targetKey(target), target); } @@ -440,7 +452,7 @@ export async function searchCandidateIssuesWithSummary(searchQuery, githubToken, const policiesByKey = new Map(policyEntries); const issues = []; for (const item of searchItems) { - const target = targetFromSearchIssue(item); + const target = targetFromSearchIssue(item, normalizedOptions.forge); if (!target) continue; const policy = policiesByKey.get(targetKey(target)); if (!policy?.allowed) continue; diff --git a/test/unit/miner-discover-cli.test.ts b/test/unit/miner-discover-cli.test.ts index 5b35ac8aa1..2553c6dcc7 100644 --- a/test/unit/miner-discover-cli.test.ts +++ b/test/unit/miner-discover-cli.test.ts @@ -101,6 +101,47 @@ describe("parseDiscoverArgs (#4247)", () => { error: "Unknown option: --verbose", }); }); + + it("parses the per-tenant --api-base-url and --token-env flags (#4784)", () => { + expect( + parseDiscoverArgs([ + "acme/widgets", + "--api-base-url", + "https://ghe.example.com/api/v3", + "--token-env", + "FORGE_PAT", + ]), + ).toEqual({ + targets: [{ owner: "acme", repo: "widgets" }], + search: null, + json: false, + apiBaseUrl: "https://ghe.example.com/api/v3", + tokenEnv: "FORGE_PAT", + }); + }); + + it("omits --api-base-url / --token-env keys entirely when not supplied (#4784)", () => { + expect(parseDiscoverArgs(["acme/widgets"])).toEqual({ + targets: [{ owner: "acme", repo: "widgets" }], + search: null, + json: false, + }); + }); + + it("rejects --api-base-url / --token-env missing their value (#4784)", () => { + expect(parseDiscoverArgs(["acme/widgets", "--api-base-url"])).toEqual({ + error: expect.stringContaining("Usage: gittensory-miner discover"), + }); + expect(parseDiscoverArgs(["acme/widgets", "--api-base-url", "--json"])).toEqual({ + error: expect.stringContaining("Usage: gittensory-miner discover"), + }); + expect(parseDiscoverArgs(["acme/widgets", "--token-env"])).toEqual({ + error: expect.stringContaining("Usage: gittensory-miner discover"), + }); + expect(parseDiscoverArgs(["acme/widgets", "--token-env", "--json"])).toEqual({ + error: expect.stringContaining("Usage: gittensory-miner discover"), + }); + }); }); describe("renderDiscoverSummary (#4247)", () => { @@ -175,6 +216,21 @@ describe("renderDiscoverSummary (#4247)", () => { enqueueSummary: { enqueued: 0, skippedBelowMinRank: 0, skippedInvalid: 0, eventsAppended: 0 }, }); expect(empty).toContain("no candidates found."); + // Without the flag the fall-back note is absent (the default-goal-spec branch is opt-in on the result). + expect(empty).not.toContain("built-in default goal spec"); + }); + + it("surfaces the default-goal-spec fall-back note when no per-tenant spec was supplied (#4784)", () => { + const text = renderDiscoverSummary({ + fanOutCount: 1, + warnings: [], + rateLimitRemaining: null, + rateLimitResetAt: null, + ranked: [{ repoFullName: "acme/widgets", issueNumber: 1, title: "x", rankScore: 0.8 }], + usedDefaultGoalSpec: true, + enqueueSummary: { enqueued: 1, skippedBelowMinRank: 0, skippedInvalid: 0, eventsAppended: 0 }, + }); + expect(text).toContain("ranked with the built-in default goal spec"); }); it("surfaces rate-limit telemetry, and reports 'unknown' when the fanout captured none (#4837)", () => { @@ -361,6 +417,139 @@ describe("runDiscover (#4247)", () => { else process.env.GITTENSORY_MINER_PORTFOLIO_QUEUE_DB = previousDbPath; } }); + + it("threads --token-env and --api-base-url into the fan-out (#4784)", async () => { + const portfolioQueue = tempQueueStore(); + const previous = process.env.FORGE_PAT; + process.env.FORGE_PAT = "tenant-secret"; + try { + const fetchCandidateIssuesWithSummary = vi.fn(async () => ({ + issues: [fanOutIssue()], + warnings: [], + rateLimitRemaining: null, + rateLimitResetAt: null, + })); + vi.spyOn(console, "log").mockImplementation(() => undefined); + + const exitCode = await runDiscover( + ["acme/widgets", "--api-base-url", "https://ghe.example.com/api/v3", "--token-env", "FORGE_PAT"], + { nowMs: NOW, initPortfolioQueue: () => portfolioQueue, fetchCandidateIssuesWithSummary }, + ); + + expect(exitCode).toBe(0); + expect(fetchCandidateIssuesWithSummary).toHaveBeenCalledWith( + [{ owner: "acme", repo: "widgets" }], + "tenant-secret", + expect.objectContaining({ apiBaseUrl: "https://ghe.example.com/api/v3" }), + ); + } finally { + if (previous === undefined) delete process.env.FORGE_PAT; + else process.env.FORGE_PAT = previous; + } + }); + + it("defaults the credential env var to the forge adapter's tokenEnvVar when no --token-env is given (#4784)", async () => { + const portfolioQueue = tempQueueStore(); + const previous = process.env.FORGE_PAT; + process.env.FORGE_PAT = "tenant-secret"; + try { + const fetchCandidateIssuesWithSummary = vi.fn(async () => ({ + issues: [fanOutIssue()], + warnings: [], + rateLimitRemaining: null, + rateLimitResetAt: null, + })); + vi.spyOn(console, "log").mockImplementation(() => undefined); + + const exitCode = await runDiscover(["acme/widgets"], { + nowMs: NOW, + initPortfolioQueue: () => portfolioQueue, + fetchCandidateIssuesWithSummary, + forge: { tokenEnvVar: "FORGE_PAT" }, + }); + + expect(exitCode).toBe(0); + expect(fetchCandidateIssuesWithSummary).toHaveBeenCalledWith( + [{ owner: "acme", repo: "widgets" }], + "tenant-secret", + expect.any(Object), + ); + } finally { + if (previous === undefined) delete process.env.FORGE_PAT; + else process.env.FORGE_PAT = previous; + } + }); + + it("prefers an explicit githubToken option and a programmatic apiBaseUrl / tokenEnv (#4784)", async () => { + const portfolioQueue = tempQueueStore(); + const fetchCandidateIssuesWithSummary = vi.fn(async () => ({ + issues: [fanOutIssue()], + warnings: [], + rateLimitRemaining: null, + rateLimitResetAt: null, + })); + vi.spyOn(console, "log").mockImplementation(() => undefined); + + const exitCode = await runDiscover(["acme/widgets"], { + nowMs: NOW, + initPortfolioQueue: () => portfolioQueue, + fetchCandidateIssuesWithSummary, + githubToken: "explicit-token", + apiBaseUrl: "https://programmatic.example.com", + tokenEnv: "IGNORED_BECAUSE_TOKEN_IS_EXPLICIT", + }); + + expect(exitCode).toBe(0); + expect(fetchCandidateIssuesWithSummary).toHaveBeenCalledWith( + [{ owner: "acme", repo: "widgets" }], + "explicit-token", + expect.objectContaining({ apiBaseUrl: "https://programmatic.example.com" }), + ); + }); + + it("forwards a per-tenant goal spec to the ranker and surfaces usedDefaultGoalSpec (#4784)", async () => { + const portfolioQueue = tempQueueStore(); + const fetchCandidateIssuesWithSummary = vi.fn(async () => ({ + issues: [fanOutIssue()], + warnings: [], + rateLimitRemaining: null, + rateLimitResetAt: null, + })); + const goalSpecContentByRepo = { "acme/widgets": "minerEnabled: true\n" }; + const rankCandidateIssuesWithSummary = vi.fn(() => ({ + issues: [ + { + ...fanOutIssue(), + potential: 0.5, + feasibility: 0.5, + laneFit: 0.5, + freshness: 0.5, + dupRisk: 0, + rankScore: 0.5, + }, + ], + skippedInvalid: 0, + usedDefaultGoalSpec: false, + defaultGoalSpec: {} as never, + })); + + const log = vi.spyOn(console, "log").mockImplementation(() => undefined); + const exitCode = await runDiscover(["acme/widgets", "--json"], { + nowMs: NOW, + initPortfolioQueue: () => portfolioQueue, + fetchCandidateIssuesWithSummary, + rankCandidateIssuesWithSummary, + goalSpecContentByRepo, + }); + + expect(exitCode).toBe(0); + expect(rankCandidateIssuesWithSummary).toHaveBeenCalledWith( + expect.any(Array), + expect.objectContaining({ goalSpecContentByRepo }), + ); + const payload = JSON.parse(String(log.mock.calls[0]?.[0])); + expect(payload.usedDefaultGoalSpec).toBe(false); + }); }); describe("gittensory-miner discover CLI entrypoint (#4247)", () => { diff --git a/test/unit/miner-forge-config.test.ts b/test/unit/miner-forge-config.test.ts new file mode 100644 index 0000000000..13b0244702 --- /dev/null +++ b/test/unit/miner-forge-config.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from "vitest"; +import { + DEFAULT_FORGE_CONFIG, + resolveForgeConfig, +} from "../../packages/gittensory-miner/lib/forge-config.js"; + +describe("resolveForgeConfig (#4784)", () => { + it("returns gittensory's github.com defaults when no overrides are supplied", () => { + expect(resolveForgeConfig()).toEqual({ + apiBaseUrl: "https://api.github.com", + apiVersion: "2022-11-28", + apiVersionHeader: "x-github-api-version", + acceptHeader: "application/vnd.github+json", + userAgent: "loopover-miner", + repoPathPrefix: "/repos", + searchEndpoint: "/search/issues", + searchQualifiers: "state:open type:issue", + tokenEnvVar: "GITHUB_TOKEN", + }); + // A no-override resolve must exactly equal the shared default baseline (the "unchanged gittensory path" contract). + expect(resolveForgeConfig()).toEqual({ ...DEFAULT_FORGE_CONFIG }); + }); + + it("applies only the supplied per-tenant overrides and keeps defaults for the rest", () => { + const resolved = resolveForgeConfig({ + apiBaseUrl: "https://ghe.example.com/api/v3", + apiVersionHeader: "x-forge-version", + apiVersion: "v9", + tokenEnvVar: "FORGE_PAT", + }); + expect(resolved.apiBaseUrl).toBe("https://ghe.example.com/api/v3"); + expect(resolved.apiVersionHeader).toBe("x-forge-version"); + expect(resolved.apiVersion).toBe("v9"); + expect(resolved.tokenEnvVar).toBe("FORGE_PAT"); + // Untouched fields still fall back to the github.com defaults. + expect(resolved.acceptHeader).toBe("application/vnd.github+json"); + expect(resolved.repoPathPrefix).toBe("/repos"); + expect(resolved.searchEndpoint).toBe("/search/issues"); + expect(resolved.searchQualifiers).toBe("state:open type:issue"); + expect(resolved.userAgent).toBe("loopover-miner"); + }); + + it("trims string overrides and falls back to the default for blank or non-string values", () => { + expect(resolveForgeConfig({ userAgent: " my-tenant-bot " }).userAgent).toBe("my-tenant-bot"); + // Blank/whitespace override -> default (a tenant can't accidentally clear a field to ""). + expect(resolveForgeConfig({ userAgent: " " }).userAgent).toBe("loopover-miner"); + // Non-string override -> default. + expect(resolveForgeConfig({ apiVersion: 123 as never }).apiVersion).toBe("2022-11-28"); + }); + + it("treats a non-object overrides argument as no overrides", () => { + expect(resolveForgeConfig(null as never)).toEqual({ ...DEFAULT_FORGE_CONFIG }); + }); + + it("exposes a frozen default baseline that resolve never mutates", () => { + expect(Object.isFrozen(DEFAULT_FORGE_CONFIG)).toBe(true); + resolveForgeConfig({ apiBaseUrl: "https://other.example" }); + expect(DEFAULT_FORGE_CONFIG.apiBaseUrl).toBe("https://api.github.com"); + }); +}); diff --git a/test/unit/miner-opportunity-fanout-forge.test.ts b/test/unit/miner-opportunity-fanout-forge.test.ts new file mode 100644 index 0000000000..5f71880997 --- /dev/null +++ b/test/unit/miner-opportunity-fanout-forge.test.ts @@ -0,0 +1,179 @@ +import { Buffer } from "node:buffer"; +import { afterEach, describe, expect, it, vi } from "vitest"; + +vi.mock("@jsonbored/gittensory-engine", async () => { + return import("../../packages/gittensory-engine/src/index"); +}); + +import { + fetchCandidateIssues, + searchCandidateIssuesWithSummary, +} from "../../packages/gittensory-miner/lib/opportunity-fanout.js"; + +type Call = { url: string; headers: Record }; + +function jsonResponse(body: unknown, init: ResponseInit = {}) { + return Response.json(body, init); +} + +function contentResponse(content: string) { + return jsonResponse({ + type: "file", + encoding: "base64", + content: Buffer.from(content, "utf8").toString("base64"), + }); +} + +const issue = (number: number) => ({ + number, + title: `Issue ${number}`, + labels: ["help wanted"], + comments: 1, + created_at: "2026-07-01T00:00:00Z", + updated_at: "2026-07-01T01:00:00Z", + html_url: `https://forge.example.com/acme/widgets/issues/${number}`, +}); + +function stubFetch(handler: (url: string) => Response) { + const calls: Call[] = []; + vi.stubGlobal("fetch", async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + calls.push({ url, headers: (init?.headers as Record) ?? {} }); + return handler(url); + }); + return calls; +} + +const CUSTOM_FORGE = { + apiBaseUrl: "https://ghe.example.com/api/v3", + apiVersion: "v9", + apiVersionHeader: "x-forge-version", + acceptHeader: "application/vnd.forge+json", + userAgent: "acme-tenant-bot", + repoPathPrefix: "/repositories", + searchEndpoint: "/search/tickets", + searchQualifiers: "is:open kind:issue", + tokenEnvVar: "FORGE_PAT", +}; + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +describe("opportunity fan-out per-tenant forge config (#4784)", () => { + it("routes repo fetches through the tenant forge host, path prefix, and headers", async () => { + const calls = stubFetch((url) => { + if (url.endsWith("/contents/AI-USAGE.md")) return jsonResponse({}, { status: 404 }); + if (url.endsWith("/contents/CONTRIBUTING.md")) return contentResponse("Contributions welcome."); + if (url.includes("/issues?")) return jsonResponse([issue(7)]); + return jsonResponse({}, { status: 404 }); + }); + + const result = await fetchCandidateIssues([{ owner: "acme", repo: "widgets" }], "tenant-token", { + forge: CUSTOM_FORGE, + }); + + expect(result.map((entry) => entry.issueNumber)).toEqual([7]); + // Every request went to the tenant forge base URL + custom repo path prefix, not api.github.com/repos. + expect(calls.every((call) => call.url.startsWith("https://ghe.example.com/api/v3/repositories/acme/widgets"))).toBe( + true, + ); + expect(calls.some((call) => call.url.includes("/repos/"))).toBe(false); + // Headers carry the tenant's accept/user-agent and a custom API-version header name+value (no github header). + const headers = calls[0]?.headers ?? {}; + expect(headers.accept).toBe("application/vnd.forge+json"); + expect(headers["user-agent"]).toBe("acme-tenant-bot"); + expect(headers["x-forge-version"]).toBe("v9"); + expect(headers["x-github-api-version"]).toBeUndefined(); + expect(headers.authorization).toBe("Bearer tenant-token"); + }); + + it("routes search through the tenant search endpoint and search-qualifier dialect", async () => { + const calls = stubFetch((url) => { + if (url.includes("/search/tickets?")) { + return jsonResponse({ + items: [ + { + ...issue(21), + repository: { full_name: "acme/widgets" }, + html_url: "https://ghe.example.com/acme/widgets/issues/21", + }, + ], + }); + } + if (url.endsWith("/contents/AI-USAGE.md")) return jsonResponse({}, { status: 404 }); + if (url.endsWith("/contents/CONTRIBUTING.md")) return contentResponse("Contributions welcome."); + return jsonResponse({}, { status: 404 }); + }); + + const result = await searchCandidateIssuesWithSummary("label:bug", "tenant-token", { forge: CUSTOM_FORGE }); + + expect(result.issues.map((entry) => entry.issueNumber)).toEqual([21]); + const searchCall = calls.find((call) => call.url.includes("/search/tickets?")); + expect(searchCall).toBeDefined(); + expect(searchCall?.url).toContain( + `q=${encodeURIComponent("label:bug is:open kind:issue")}`, + ); + expect(calls.some((call) => call.url.includes("/search/issues?"))).toBe(false); + }); + + it("resolves search hits from repository_url via the tenant repoPathPrefix when full_name is absent (#4784)", async () => { + stubFetch((url) => { + if (url.includes("/search/tickets?")) { + return jsonResponse({ + items: [ + { + ...issue(33), + // No repository.full_name: the custom forge only returns the API repository_url, which uses the + // tenant's repoPathPrefix ("/repositories"), not GitHub's hardcoded "/repos". + repository_url: "https://ghe.example.com/api/v3/repositories/acme/gadgets", + }, + ], + }); + } + if (url.endsWith("/contents/AI-USAGE.md")) return jsonResponse({}, { status: 404 }); + if (url.endsWith("/contents/CONTRIBUTING.md")) return contentResponse("Contributions welcome."); + return jsonResponse({}, { status: 404 }); + }); + + const result = await searchCandidateIssuesWithSummary("label:bug", "tenant-token", { forge: CUSTOM_FORGE }); + + expect(result.issues.map((entry) => entry.repoFullName)).toEqual(["acme/gadgets"]); + expect(result.issues.map((entry) => entry.issueNumber)).toEqual([33]); + }); + + it("resolves search hits from a non-github html_url when full_name and repository_url are absent (#4784)", async () => { + stubFetch((url) => { + if (url.includes("/search/tickets?")) { + return jsonResponse({ + items: [ + { + ...issue(44), + html_url: "https://ghe.example.com/acme/tools/issues/44", + }, + ], + }); + } + if (url.endsWith("/contents/AI-USAGE.md")) return jsonResponse({}, { status: 404 }); + if (url.endsWith("/contents/CONTRIBUTING.md")) return contentResponse("Contributions welcome."); + return jsonResponse({}, { status: 404 }); + }); + + const result = await searchCandidateIssuesWithSummary("label:bug", "tenant-token", { forge: CUSTOM_FORGE }); + + expect(result.issues.map((entry) => entry.repoFullName)).toEqual(["acme/tools"]); + expect(result.issues.map((entry) => entry.issueNumber)).toEqual([44]); + }); + + it("lets a legacy top-level apiBaseUrl win over forge.apiBaseUrl (back-compat)", async () => { + const calls = stubFetch(() => contentResponse("AI-generated PRs are rejected.")); + + await fetchCandidateIssues([{ owner: "acme", repo: "widgets" }], "", { + apiBaseUrl: "https://legacy.example.com", + forge: { apiBaseUrl: "https://ignored.example.com", repoPathPrefix: "/repositories" }, + }); + + // The top-level override supplies the host; the rest of the forge config (path prefix) still applies. + expect(calls[0]?.url).toBe("https://legacy.example.com/repositories/acme/widgets/contents/AI-USAGE.md"); + }); +});