feat(profile): run an unattended scenario file in web profiling

[kalvinnchau]

Problem

just web profile records only a session that a person drives and stops with Ctrl-C. An unattended run can capture page load and nothing else, and it never exits on its own.

Change

just web profile --scenario <file> runs a scenario file against the page and ends the capture itself. No scenario lives in the repository.

• Scenario file: any JavaScript module, inside or outside the repository, that default-exports async (page, { signal }) => {}. It gets the Playwright page. signal aborts on Ctrl-C, at the timeout, and when the capture ends for any reason (including Vite exiting), so abort-aware work a scenario leaves behind cannot keep the command alive. Because the file can live outside the tree, the same scenario runs unchanged against any commit that has this flag.

• Harness: loads the file before starting Vite or Chrome, runs it, saves the app's existing client metrics export as client-metrics.json, stops the capture and exits 0.

• Failure: a scenario that throws, or does not finish within five minutes, exits 1. The capture is still finalized, no client-metrics.json is written, and the command prints Scenario failed; artifacts remain at <directory>. followed by the scenario's stack. A missing file or a module without a default function fails before anything starts.

• Ctrl-C during a scenario behaves as before: it exits 0 like any interrupted capture and writes no client-metrics.json.

• Manifest: gains scenario (the resolved file path) and relay. relay is the https:// origin of the BUZZ_RELAY_URL that Vite resolves for its mode from the process environment or the .env files (process environment wins), validated by the app's own relayOrigin. A value the dev server would reject (credentials, path, query, fragment, wrong scheme, malformed) is recorded as null, never verbatim. This applies to every web capture, with or without --scenario. coverage lists client-metrics for scenario runs.

• Mode: the profiler takes the Vite mode itself (--mode <mode>, --mode=, -m, once; default development), resolves the relay for it, and launches Vite with the same mode. Spellings Vite would read as a mode without the profiler seeing one (a short-flag group such as -dm, or --m) are rejected before the capture.

The CLI now prints the stack of a wrapped failure's cause, which also covers the existing Profiling stopped during startup error.

--scenario is rejected for desktop. There is no new metrics code: the numbers are the __buzzClientMetrics export that dev builds already produce. The justfile is unchanged because just web profile already forwards its arguments.

A scenario runs as the developer's real account, so keeping it read-only is the scenario author's job. The harness adds no writes of its own.

Evidence

• node --test tests/integration/profile-dev.test.mjs: 47 pass, 0 fail at 13fc3929. Twelve tests cover the change:

  • flag parsing;
  • the file must default-export a function;
  • a scenario that never finishes fails at the timeout, and its abort-aware wait is aborted;
  • mode parsing: four accepted spellings re-emitted as one --mode, missing value, duplicates, and the rejected -dm / --m spellings;
  • the recorded relay: unset, .env.local only, process environment over .env.local, a mode's own .env.<mode> over .env.local, and six rejected forms from either source;
  • a wrapped failure reports its cause's stack;
  • a scenario ends its own capture and saves the client metrics; the manifest relay comes from .env.local through Vite's real loadEnv;
  • a failed step finalizes the capture without client metrics and names the artifact directory;
  • Ctrl-C during a scenario saves the capture without client metrics;
  • a timed-out scenario holding an abort-aware 60-second timer is aborted and the process exits;
  • a scenario is aborted when Vite exits during the capture, and the process exits;
  • -m staging records the .env.staging relay and launches Vite with --mode staging.

• Mutation checks, each restored afterwards:

  • the timeout wait ignores Ctrl-C: the Ctrl-C test fails;
  • the harness never calls the scenario: three scenario tests fail;
  • runScenario does not abort its signal, or passes the scenario the parent signal: the timeout test fails;
  • the capture does not abort the signal at its end: the Vite-exit test fails;
  • neither aborts (the reviewed behaviour): the timeout and Vite-exit process tests both fail;
  • the relay is returned unvalidated, or read from process.env: the relay and manifest tests fail;
  • the scenario failure is not wrapped: the failed-step and timed-out tests fail;
  • the relay is resolved for development regardless of mode, the mode is not passed on to Vite, or the -dm / --m rejection is removed: the mode tests fail.

• bin/lefthook run check-staged: pass.

• Live, macOS, headed Chrome, Vite dev server, staging relay. The scenario file lives outside the repository (listed below) and the same command ran three times in a row:

  BUZZ_DEV_OPEN_RELAY=1 BUZZ_RELAY_URL=wss://<staging> bin/just web profile --network --scenario /abs/path/channels.mjs

  Run	Exit	Opens	p50	p90	max	Long tasks
  1	0	11	208 ms	303 ms	314 ms	16 (1431 ms)
  2	0	11	187 ms	308 ms	341 ms	16 (1324 ms)
  3	0	11	185 ms	282 ms	332 ms	15 (1239 ms)

  The --network log of run 1 has no publish, read-state-publish or sign requests.

• --scenario nope.mjs failed before creating a profile directory (at e76a8fae).

• Real Vite 8.3.0 given the profiler's normalised arguments reports the same mode the profiler resolved, for no mode and each accepted spelling.

• Live at 13fc3929, relay set only in .env.staging: --mode staging ran the channels scenario to exit 0 with "relay": "https://<staging>"; without --mode the manifest has "relay": null; -dm staging was refused before Vite started.

• Live at 0adccc90, same setup:

  • A scenario awaiting an abort-aware 20-minute timer: its signal aborted 300.0 s after it started waiting, the command exited 1 after 307 s in total, printed the artifact directory and Scenario did not finish within 300 seconds., and left no profiler or Vite process.
  • A throwing scenario: exit 1, Scenario failed; artifacts remain at …, then the stack pointing at the scenario's own line.
  • Relay set only in .env.local, none in the process environment: the channels scenario exited 0 (11 opens, p50 196 ms) and the manifest has "relay": "https://<staging>".
  • BUZZ_RELAY_URL='wss://user:<canary>@<staging>/?token=<canary>': Vite refused to start, the manifest has "relay": null, and the canary appears in neither the profile directory nor the command output.

channels.mjs used for the live runs (not part of this PR)

// Scenario for `just web profile --scenario <this file>` in block/buzz-app.
// Opens the first sidebar channels twice: a cold pass, then a warm pass.
// Read-only: focus stays on the sidebar, so the reading policy marks nothing read.
import { setTimeout as pause } from "node:timers/promises";

const CHANNELS = 5;
const PAUSE_MS = 1_000;
const ROWS = 'nav[aria-label="Subscribed channels"] button[data-channel-id]';

export default async function channels(page, { signal }) {
  // Opens during startup catch-up would measure live setup, not the open.
  await page.waitForFunction(
    () =>
      globalThis.__buzzClientMetrics?.summary().phases[0].coverageMs !==
      undefined,
  );
  const rows = page.locator(ROWS);
  await rows.first().waitFor();
  const ids = await rows.evaluateAll(
    (buttons, limit) =>
      buttons.slice(0, limit).map((button) => button.dataset.channelId),
    CHANNELS,
  );
  // The app finishes an open by timing it or counting it as not timed.
  const finishedOpens = () =>
    page.evaluate(() => {
      const { n, skipped } = globalThis.__buzzClientMetrics.summary().opens;
      return n + skipped;
    });
  for (const id of [...ids, ...ids]) {
    const row = page
      .locator(`${ROWS}[data-channel-id=${JSON.stringify(id)}]`)
      .first();
    // Reselecting the channel on screen opens nothing.
    if ((await row.getAttribute("aria-current")) === "page") continue;
    const before = await finishedOpens();
    await row.click();
    await page.waitForFunction((count) => {
      const { n, skipped } = globalThis.__buzzClientMetrics.summary().opens;
      return n + skipped > count;
    }, before);
    // Let trailing work finish so it is not charged to the next open.
    await pause(PAUSE_MS, undefined, { signal });
  }
}

Limitations

• relay is resolved from the repository's own Vite configuration. A capture started with a different --config is not modelled.
• A scenario outside the repository resolves bare imports from its own location, not from the repository's node_modules.
• The five-minute limit is fixed.
• Each capture starts from a fresh browser profile, so no community is selected unless BUZZ_DEV_OPEN_RELAY=1 is set or the scenario selects one.
• client-metrics.json needs the dev-only __buzzClientMetrics, and the capture runs against the Vite dev server. Differences between runs are meaningful; absolute timings are not production numbers.
• Still macOS only, with headed Chrome.

Not covered

• Full tests/integration/*.test.mjs locally at the first commit: 147 of 174 pass. The 27 failures are in the agent-runtime, launcher and worktree-icon tests (spawn …/bin/rustc EACCES, just install exit 126), files this change does not touch. Hosted CI passed on the first commit. The full suite was not rerun locally for 0adccc90 or 13fc3929. On e76a8fae, Browser journeys (webkit, 3/6) failed once in tests/browser/todos.spec.mjs:9 and passed on a rerun of that job with no change.
• No repeat-run or base-versus-candidate comparison tooling. Repeating a run means invoking the command again.

To test

Save a scenario anywhere, for example /tmp/wait.mjs:

export default async (page) => {
  await page.waitForFunction(
    () => globalThis.__buzzClientMetrics?.summary().phases[0].coverageMs !== undefined,
  );
};

BUZZ_DEV_OPEN_RELAY=1 BUZZ_RELAY_URL=wss://<relay> bin/just web profile --scenario /tmp/wait.mjs

Chrome opens, the app loads its community, and the command exits 0 without input after printing the .profiles/...-web directory. That directory contains client-metrics.json, and manifest.json has "scenario": "/tmp/wait.mjs" and the relay as an https:// origin.

[wesbillman]

One change needed: abort the scenario when its deadline expires so outstanding abort-aware work cannot keep an unattended run alive (details inline).

Star Lord automated source review via Wes’s account (wesbillman). Head e76a8faed0592f4ba5190ffdb67b79380bfad659; base 73064e34dfa5f14c472c3661589d05f509c17884. Source-only: no tests, scenarios, or app execution. The hosted snapshot shows required CI and DCO passing; that does not validate the missing timeout-cleanup case or native behavior.

[wesbillman]

No changes requested. The follow-up addresses the prior timeout finding: scenario-scoped cancellation now aborts outstanding work on timeout/finalization, and capture shutdown also aborts it when Vite exits; regression tests cover cancellation and process exit.

Star Lord automated source review via Wes’s account (wesbillman), head 0adccc90060ac973c5d92c85d200be05c1605aa3, base 73064e34dfa5f14c472c3661589d05f509c17884. Source-only follow-up: no tests, scenarios, or app execution; hosted CI was still running at the snapshot, and the author’s live-run evidence was not independently reproduced.

[wesbillman]

No changes requested in this follow-up: the manifest and launched Vite now use the same normalized mode, with relay validation and cancellation cleanup preserved. I also inspected the updated public description and changed files; no attached media was present.

Star Lord automated source review via Wes’s account — head 13fc39293a014f8549ae8f30a5dbd9027985e774, base 73064e34dfa5f14c472c3661589d05f509c17884. Source-only: no tests or live profiling run; the current-head CI snapshot still has jobs running.