Let a maintainer prove boot config without a setup PR; infer it for root repos by bishopBethel · Pull Request #256 · devasignhq/agent

bishopBethel · 2026-09-16T12:05:38Z

The two gaps dogfooding Phase 4 turned up.

1. The probe was unreachable for every onboarded repo

It could only fire on an open setup PR — and a repo that's already set up never gets one again, because Phase 3 correctly makes regeneration a no-op when nothing differs. I verified that by running the real generator against devasignhq/agent and bishopBethel/fundsflow: yml identical, workflow identical, both.

So every repo already onboarded was locked out of the one feature that proves its boot config. fundsflow is the case in point — its panel has said "Configured" since onboarding, with config that has never been proven and, as built, never could be.

A maintainer can now ask for a check directly. DevAsign resolves the default-branch head itself, mints a probe, and dispatches; the run reports back into the panel. No PR comment — there's no PR to comment on, and inventing one is how Phase 4's comment-scoping bug happened.

2. Single-package repos got no inference

inferBootCandidates only looked at nested package dirs, so a root-manifest repo fell through to the pre-Phase-3 guess: hardcoded --port 5173, no --strictPort, and no reading of the app's own vite server.port. devasignhq/website declares server.port: 3001; fundsflow is running that legacy guess today. Without --strictPort an occupied port silently moves the app while the probe waits on a dead URL — the exact failure a Phase 3 reviewer flagged, fixed then on the nested path only. Root packages now go through the same inference, templates and re-derivation.

Review found 18 issues; all fixed

The worst one was mine. I specified binding the re-check to "a dispatch run on the default branch" — forgetting that DevAsign fires exactly that shape to re-run a contributor's PR, and those runs execute the contributor's code. Untrusted code could have claimed the probe and filed its report. It's now bound to an unguessable nonce that only the App and the run it started ever see.

Others worth naming:

A second dispatch could hijack an already-claimed probe (now pinned to run_id + run_attempt).
Root candidacy made ordinary workspace monorepos ambiguous, silently downgrading them to a root-level 5173 guess.
Vite's port was read from the first server: anywhere in the file — a vitest test.server block shadowed the real one and dropped the whole server pairing.
A re-check's verdict was labelled "checked on PR #<setup PR>", a PR it never ran on.
A re-check that proved the app boots could never clear a "failing" row, so the panel contradicted itself.

This needs CLI 1.8.0 — the release story changed

I'd aimed for backend-only so it would work with the 1.7.0 you just published. The nonce fix makes that impossible: the runner has to echo the token back, and 1.7.0 doesn't. Security wins, so the CLI is bumped to 1.8.0 here.

A 1.7.0 runner therefore can't claim a re-check. Rather than let the request sit until the 2h expiry, the backend now records why and the panel says to update @devasign/verify. (That last piece is mine, added after the workflow — with a test verified to fail without it.)

Order: merge → deploy → publish 1.8.0 from a clean origin/main checkout. Run npm ci in verify/ first; the main checkout's node_modules drifts and that's what broke the 1.7.0 publish.

Verification

Backend 1715 pass, CLI 123 pass, frontend 267 pass, three clean typechecks. Every behaviour fix was mutation-verified. I re-ran all gates independently and spot-mutated the probe-key and runner-outdated fixes myself.

The integrator also pinned the full dispatch chain end to end with a test that dynamically imports the real CLI's context.ts rather than restating its rules — three links in that chain had no coverage on either side.

🤖 Generated with Claude Code

…oot repos Two gaps dogfooding found in Phase 4. The probe could only ever fire on an open setup PR, and an onboarded repo never gets one again — Phase 3 correctly makes regeneration a no-op when nothing differs. So every repo already set up was locked out of the one feature that proves its boot config, bishopBethel/fundsflow among them: "Configured" since onboarding, never proven, no way to prove it. A maintainer can now ask for a check directly; DevAsign resolves the default-branch head itself, mints a probe and dispatches, and the run reports back into the panel. No PR comment, because there is no PR to comment on. Inference only ever looked at nested package directories, so a single-package repo fell through to the pre-Phase-3 guess that hardcodes port 5173, omits --strictPort, and never reads the app's own vite server.port. That is the common new-customer shape, and without --strictPort an occupied port moves the app while the probe waits on a dead URL. Root packages now go through the same inference, templates and re-derivation as nested ones. Adversarial review found 18 issues, all fixed. The worst was mine: I had bound the re-check to "a dispatch run on the default branch", forgetting DevAsign fires exactly that shape to re-run a contributor's PR — and those runs execute the contributor's code. It is now bound to an unguessable nonce only the App and the run it started ever see. Also: a root candidate made ordinary workspace monorepos ambiguous; vite's port was read from the first `server:` in the file, which a vitest `test.server` shadows; and a re-check's verdict was labelled with the setup PR's number, which it never ran on. Because the nonce must be echoed back, this needs CLI 1.8.0. A 1.7.0 runner cannot claim a re-check, so it now records that and the panel says to update, rather than leaving the request to time out silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The re-check only works with a runner that echoes the App's dispatch token, which 1.7.0 does not send. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

vercel · 2026-09-16T12:05:44Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
contributor	Ready	Preview	Sep 16, 2026 12:13pm UTC
sponsor	Ready	Preview	Sep 16, 2026 12:13pm UTC

devasign-agent

DevAsign Code Review

🐞 Bugs (1) · 🔒 Security (1) · ❌ Tests failing (3)

🟡 Merge score: 58/100

14 of 14 acceptance criteria met.
This PR adds a maintainer-requested boot re-check for already-onboarded repos (dispatched by the App, claimed by a nonce pinned to run_id+run_attempt, reported into the panel with no PR comment) and extends boot inference to root/single-package repos.

⚠️ 3 of 11 verified tests are failing · 2 UI criteria checked without a browser (set up) — check the "Tests by DevAsign" comment before merging.

Prompt to fix all issues

You are helping fix PR "Let a maintainer prove boot config without a setup PR; infer it for root repos" in devasignhq/agent. Automated review surfaced the items below — failed acceptance criteria and review findings. Each item states what was required, what's wrong with the current diff, and how to fix it; the embedded fix blocks include the expected behavior and the relevant diff hunk. Apply each fix so the item is resolved. Items tagged **Blocker** gate approval; the rest are advisory but worth addressing. Don't introduce changes beyond what's listed.

## End goal
A maintainer can trigger a boot-config probe on an already-onboarded repo without an open setup PR, single-package (root-manifest) repos get proper boot inference, and the re-check is securely bound to the run that claimed it via CLI 1.8.0.

## Review findings

### 1. [Critical error · Warn] `backend/src/routes/api.ts` — The `record` closure calls `patchRepoVerify(repo.id, (cur) => ({ ...cur, onboarding: { ...cur.onboarding, bootCheck } }))`, spreading `cur.onboarding`. If `patchRepoVerify`'s reducer receives a verify state whose `onboarding` is undefined (e.g. a concurrent reset between the availability check and this write), `...cur.onboarding` spreads undefined which yields an object missing required onboarding fields (`prNumber`, etc.), and the resulting `RepoVerifyState.onboarding` would be corrupt. The availability guard reads `repo.verify?.onboarding` on a snapshot taken earlier; `patchRepoVerify` operates on the live row inside its own reducer, so the invariant is not guaranteed to hold at write time. The definition of `patchRepoVerify` is not in the provided context, so whether it re-guards or requires a fully-formed onboarding cannot be confirmed here; flagging for human verification.

Fix: Guard against undefined onboarding in the boot-check state writer

File: backend/src/routes/api.ts
Symbol: makeBootCheckHandler / record closure

Issue:
The `record` closure spreads `...cur.onboarding` inside the patchRepoVerify reducer. If the live verify row's onboarding is undefined or has been reset between the availability check and this write, the spread produces a malformed onboarding object missing required fields, corrupting repo verify state.

Expected behavior:
The bootCheck field should only be written when a well-formed onboarding object exists on the live row; otherwise the write should be a no-op or preserve invariants.

Suggested approach:
Inside the reducer, bail out (return cur unchanged) when `cur.onboarding?.prNumber` is absent, or construct onboarding defensively. Confirm patchRepoVerify's contract for missing onboarding.

Relevant diff:
```diff
+    const record = (bootCheck: RepoVerifyState["onboarding"]["bootCheck"]) =>
+      patchRepoVerify(repo.id, (cur) => ({ ...cur, onboarding: { ...cur.onboarding, bootCheck } }));
```

### 2. [Security · Warn] `backend/src/routes/api.ts` — The boot-check endpoint only applies `expensiveLimiter` (per-IP, shared across routes) plus per-repo throttling inside `bootCheckAvailability` (cooldown + daily cap). Each successful call triggers `branchTipSha` and `repositoryDispatch` against the customer's GitHub with their installation token. The per-repo daily cap (MAX_RECHECKS_PER_DAY=10) counts `bootProbes` rows with `kind === 'recheck'` created in the window. Since authorization is `ownedRepo(req, res)` (definition not shown), the throttle relies entirely on the cap being enforced before the GitHub round trip. The `branchTip` call at line 1490 happens BEFORE the second availability re-check at line 1500, so a burst of concurrent requests can each perform a `branchTipSha` GitHub call before any probe row exists to trip the cap — the daily cap only gates dispatch, not the branch-tip read. This is a partial amplification vector against the customer's GitHub API quota. Severity capped at medium because ownedRepo auth and expensiveLimiter (both outside the shown context) constrain the blast radius, and the assumption about ownedRepo scoping cannot be verified from the provided diff.

Fix: Rate-limit the branch-tip GitHub read, not just the dispatch

File: backend/src/routes/api.ts
Symbol: makeBootCheckHandler

Issue:
branchTip (branchTipSha) is called before the second availability re-check and before any bootProbe row is inserted. Concurrent clicks each perform a GitHub branch-tip read before the daily cap can count them, since the cap counts inserted probe rows. This lets a burst amplify calls against the customer's GitHub quota.

Expected behavior:
The daily/cooldown throttle should bound the number of GitHub API calls (including branch-tip reads), not only the dispatches.

Suggested approach:
Insert or reserve a throttle marker before the branch-tip read, or move the branch-tip read after a reservation that the cap counts. Alternatively add a short per-repo in-flight lock around the whole handler so concurrent clicks serialize.

Relevant diff:
```diff
+    const branchTip = deps.branchTip ?? ((i, r, branch) => branchTipSha(i.installationId, r.owner, r.name, branch));
+    const sha = await branchTip(install, repo, repo.defaultBranch).catch((err) => {
```

## Your task
Work through every item above — the failed acceptance criteria and each review finding. For each one: understand the gap from "What's wrong now", implement the change so the Required behavior holds (each fix block's `Expected behavior` describes the target state), and use the `Relevant diff` hunks as the anchor for where to edit. After each change, re-verify it resolves the item. Treat **Blocker**-tagged items as required (they block approval); address the rest too.

Security: 1 pre-existing security finding touches files in this PR (not introduced by it) — view on the Security page.

devasign-agent · 2026-09-16T12:08:43Z

+  };
+}
+export const bootCheckHandler = makeBootCheckHandler();
+api.post("/repositories/:id/verify/boot-check", expensiveLimiter, bootCheckHandler);


🐞 Critical error (nit) — In `record()`, the `patchRepoVerify` guard checks `cur.onboarding?.prNumber` before writing bootChe…

In record(), the patchRepoVerify guard checks cur.onboarding?.prNumber before writing bootCheck, but the callback that writes the failed-dispatch bootCheck references probe.prNumber (captured from current.verify!.onboarding.prNumber! at insert time). The probe was inserted using current.verify!.onboarding.prNumber! (line 1517) with a non-null assertion; if current.verify or current.verify.onboarding is undefined at that point the insert would throw before dispatch. bootCheckAvailability(current, now) at line 1507 only guarantees ob.prNumber is truthy via the no_setup_pr check on repo.verify?.onboarding, so current.verify.onboarding is expected to exist. This is likely safe, but the non-null assertions current.verify!.onboarding.prNumber! depend on the availability check having validated current (not repo) — verify bootCheckAvailability reads repo.verify?.onboarding?.prNumber and that current was passed. It was (line 1507 passes current), so the assertion holds. No defect confirmed here.

Prompt to fix with AI

Fix: Harden non-null assertions on current.verify.onboarding in boot-check probe insert File: backend/src/routes/api.ts Symbol: bootCheckHandler Issue: The probe insert uses `current.verify!.onboarding.prNumber!` with non-null assertions that rely on bootCheckAvailability(current) having validated onboarding.prNumber. This coupling is implicit and fragile if the availability logic changes. Expected behavior: The insert should either re-derive prNumber from the validated availability result or guard explicitly so a future change to bootCheckAvailability cannot cause an unguarded throw. Suggested approach: Extract prNumber into a local after the `still.available` check with an explicit guard, and use that local for both the probe insert and dispatch payload. Relevant diff: ```diff + const still = bootCheckAvailability(current, now); + if (!still.available) return void res.json({ ok: true, dispatched: false, reason: still.reason }); + const probe = db.insert("bootProbes", { + prNumber: current.verify!.onboarding.prNumber!, ```

devasign-agent · 2026-09-16T12:11:50Z

Tests by DevAsign

✅ Passed (8) · ❌ Failed (3) · ⚠️ Unverifiable (3)

8 of 14 criteria verified by tests, 3 failed, 3 unverifiable. Each verdict below links to its evidence.
This PR adds or changes 10 test files of its own; they were not used as evidence.

2 UI criteria were checked without a browser because their browser tests could not run — see setup

1 — A maintainer can request a boot-config check on a repo that has no open setup PR; DevAsign resolves the default-branch head, dispatches a probe run, and the run's result reports back into the repo's panel. (unverifiable)

Verdict: unverifiable

Generated test imported a file that does not exist in the repository, so the criterion was never exercised.

Conversation

bishopBethel commented Sep 16, 2026

1. The probe was unreachable for every onboarded repo

2. Single-package repos got no inference

Review found 18 issues; all fixed

This needs CLI 1.8.0 — the release story changed

Verification

Uh oh!

vercel Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devasign-agent Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

DevAsign Code Review

🟡 Merge score: 58/100

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devasign-agent Bot Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devasign-agent Bot commented Sep 16, 2026

Tests by DevAsign

Uh oh!

devasign-agent Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

DevAsign Code Review

🟡 Merge score: 76/100

Uh oh!

Uh oh!

Uh oh!

devasign-agent Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

devasign-agent Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

devasign-agent Bot commented Sep 16, 2026

Tests by DevAsign

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 16, 2026 •

edited

Loading

devasign-agent Bot left a comment •

edited

Loading

devasign-agent Bot Sep 16, 2026 •

edited

Loading

devasign-agent Bot left a comment •

edited

Loading