diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c339ac14..c9cb72f0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -98,6 +98,7 @@ jobs: --exclude-unused-npm \ --allow-all \ --include packages/code-review-agent \ + --include packages/cli/src/documents/prompt-command.md \ --target ${{ matrix.target }} \ --output dist/${{ matrix.artifact }} \ packages/cli/src/compiled.ts diff --git a/architecture.md b/architecture.md index 3ee295f3..e060a9dd 100644 --- a/architecture.md +++ b/architecture.md @@ -33,6 +33,10 @@ Existing documents and code get aligned to this section retroactively. | Repository base | the optional Git revision from which one named Workspace Repository initializes its primary checkout | | pinned commit | the commit obtained by resolving a base once; it remains the workflow run's starting repository state even as the run creates descendant commits | | document target | an addressable static heading in a root document's own Markdown flow, named by the canonical path of heading labels that reaches it; selecting one executes the preamble, each ancestor's own content, and that heading's complete subtree | +| Prompt | a person's original request, in ordinary natural language. `xmd prompt` takes exactly one | +| Plan | the executable program produced from a Prompt: an Executable Markdown document combining readable prose that expresses the Prompt's intent with the components that carry it out, each placed beside the prose describing what it does. It begins with one descriptive level-one heading. A Plan is what `xmd prompt` approves and then delivers: printed to stdout by default, written to an `--output` file, and run only under `--run`. It is not a synonym for a workflow, a policy document or any executable Markdown file | +| prompt command document | the one exact checked-in first-party Markdown value root `xmd prompt` executes to convert a Prompt into a Plan. It implements that conversion and its review workflow — the Prompt wording, the draft and repair loops, the `` branches, human review, revision, approval, stopping, exhaustion and the final explanation turn — and returns the exact approved Plan source. Every Plan-producing turn in it states the complete Plan requirements for itself, so a replacement may add or correct a title rather than only carry one forward. It is not itself a Plan. Internal: no command-line option selects another one, and no repository component search can answer for it | +| prompt profile | the trusted-host assembly used only for the prompt command document: its fixed inputs, a constrained Agent provider, Elicitation, the fixed first-party components and the host-declared ``. It uses no repository component search and exposes no custom root, and the ceiling it establishes is not readable from the command line. Its working directory is one host-owned directory dedicated to the logical prompt session, keyed by the digest of that name, created empty and required to be empty on the way in. An explicitly named session's directory is durable, because continuation derives the same session identity from it; an invocation-unique default session's is scope-owned, claimed before it is created, and exactly one cleanup is attempted after profile teardown and before admission on every ending — the leaf removed non-recursively when it is still the empty directory that was handed over, and left as found with the command failing terminally when it has gained content or vanished. Where those directories live is a host dependency no caller or document selects | | expansion | one logical evaluation of an authored executable element within a document execution | | expansion ID | a deterministic identifier for one logical expansion; restoring or retrying that expansion preserves the ID, while a distinct evaluation requested by the document receives another | | Git capability | the contextual interface through which workflow infrastructure queries the Git repository associated with the current working directory | @@ -3541,6 +3545,7 @@ Status is measured against main. | --- | --- | --- | | `xmd syntax` | describes every structural construct and every selected component the production `run` profile would let a document write in the contextual working directory, as deterministic Markdown or as version-1 JSON, from one catalog. Inspection only: it registers the run profile's declarations in a bounded scope and reads the filesystem for which files exist and, for a selected Markdown component, that file's frontmatter. It runs no body, imports no repository TypeScript module, installs no provider, mints no authority and writes no journal. An include it cannot enumerate — a selection-relevant symbolic link to a directory beneath it included — fails the whole request rather than printing a healthy subset | built on the #632 stack | | document validation | validates one supplied root projection and the recursive Markdown source closure normal component selection discovers, returning deterministic version-1 document diagnostics and `valid`, `invalid` or `not-statically-checkable` invocation outcomes without evaluating document code or installing operational host behavior | built on the #654 stack | +| `xmd prompt` | turns one Prompt into a Plan and delivers it, by executing one root document — and, only under `--run`, a second — with a complete scope boundary between them. First the packaged prompt command document, under the internal `` identity, inside the prompt profile: one enclosing Session, a host ceiling of one host-owned directory dedicated to that logical session — keyed by the digest of the name, never the name, created empty and required to be empty before the provider exists or a session is materialized, refused rather than cleaned when it is not, durable when the caller named the session and handed back non-recursively after teardown when it did not — with no additional directories, no MCP servers, no native tools and a private strict denial no permission flag widens, no Files, command, service or network capability for that document, no repository component search, and one host-declared `` whose closed assessment answers `valid: false` for a defect the draft authored and raises for a defect the command line authored. Its instructions require every Plan to begin with one descriptive level-one title and to keep the Prompt's outcomes as readable steps with each component beside the step it performs, through repairs and revisions alike; that is an authorship and human-review requirement, and `` never enforces it. A tenth draft that still has problems may be stopped or explained: the explanation is one more ordinary turn in the same Session carrying only the final diagnostics, is inert text, reopens no draft limit, and ends the command. The host's instruction layer states only that an answer belongs to the message that asked for it, so which shape a turn wants stays in the document. Authorship sits outside durability: it runs on an invocation-owned in-memory stream that is never journaled, persisted, reused or replayed. Then, only after that execution and every provider, Prompt task and Elicitation resource inside it has torn down, the host validates the returned Plan again, resolves props for exactly those bytes, and then delivers the approved Plan where the caller asked: to stdout byte for byte by default, to an exclusively created `--output` path, and — only under `--run` — through the ordinary supplied-source path under the `` identity, one ordinary document with its own Agent provider, its own journal and ordinary run output, result and failure behavior. A journal exists only when `--run` begins, and the flags that configure only a run are refused in preflight without it | built on the #659 stack | | `` / `printErrors(fn)` | prints failures | built on main | | `` | stops authored work with the sentence its author wrote, raised where it is written. An ordinary overridable core default — never structural, never reserved, so a repository `Fail.md` is chosen ahead of it — with a closed schema of one required non-empty `message` and **self-closing only**: a paired spelling never enters its body, and `as` is refused by the body itself because there is nothing to bind. Every refusal reports the invocation and happens before the authored message, so a document that never reached its decision is never reported as having made one. A valid invocation is the ordinary failure of a function component: an `Error` carrying the exact authored message, positioned at the opening tag, rendering nothing and binding nothing. It carries no `printErrors()` declaration, which is what leaves recovery to an authored `` region under ordinary text-root modes; a value body's `throw` is not replaced there, so the authored failure settles the body ahead of missing-`` settlement. No authority, context, provider, resource, module state or durable operation of its own: replay of a completed root restores the recorded outcome without re-expanding the body | built on the #659 stack | | `` | binds one name in the current environment from exactly one source: the content it renders, or the exact value `value` names, bound by reference and never through the JSON boundary component props cross — the scanner resolves no JSON for that one prop, and expansion projects none. Which source it has is read from what the author wrote, before either one runs, so a construct naming both expands no child and evaluates no expression. It opens no scope, owns no resource, adds no middleware boundary and writes no journal record — replay reconstructs both sources through ordinary expansion | built on the #527 stack | diff --git a/deno.json b/deno.json index 5b4d4107..322b3d94 100644 --- a/deno.json +++ b/deno.json @@ -57,7 +57,7 @@ "verify:clean": "deno run --allow-all --node-modules-dir=none --cached-only --frozen scripts/verify-clean.ts", "deps": "deno run --allow-all scripts/deps.ts", "deps:target": "deno run --allow-all scripts/deps-target.ts", - "build": "deno task build:web && deno compile --node-modules-dir=none --cached-only --frozen --exclude-unused-npm --allow-all --include packages/code-review-agent --output dist/xmd packages/cli/src/compiled.ts", + "build": "deno task build:web && deno compile --node-modules-dir=none --cached-only --frozen --exclude-unused-npm --allow-all --include packages/code-review-agent --include packages/cli/src/documents/prompt-command.md --output dist/xmd packages/cli/src/compiled.ts", "build:web": "deno run --allow-all --node-modules-dir=none --cached-only --frozen scripts/preflight.ts scripts/build-web-client.ts", "gen:publish-workflow": "deno run --allow-all packages/cli/src/deno.ts run scripts/gen-publish-workflow.md", "bump": "deno run -A scripts/bump-version.ts", diff --git a/packages/cli/src/agent-stack.ts b/packages/cli/src/agent-stack.ts new file mode 100644 index 00000000..3aca46d7 --- /dev/null +++ b/packages/cli/src/agent-stack.ts @@ -0,0 +1,124 @@ +/** + * The one Agent configuration an invocation resolves, and the two things a + * host does with it (specs/acp-client-spec.md §Command-line configuration). + * + * `xmd run` and `xmd prompt` take the same Agent, permission and provider + * options, so they resolve them once, here, rather than each reading the flags + * again. What they do with the result differs, and deliberately: a run installs + * the registered provider into the Agent Api so a document may reach it, while + * `xmd prompt` hands the same answer to two consumers — the prompt profile, which + * takes the provider name and the default agent and nothing else, and the + * approved document, which runs the ordinary run stack. + * + * `agent-config.ts` stays the pure flag-to-permission mapping. This module is + * where the environment, the provider registry and the host's own machine + * session assembly enter. + */ + +import { + installAgentComponents, + installPermissionMode, + registerAgentProvider, +} from "@executablemd/core"; +import type { AgentProviderFactory, PermissionMode } from "@executablemd/core"; +import { installForegroundLauncher, env as readEnv } from "@executablemd/runtime"; +import { createAcpxProvider, DEFAULT_AGENT_NAME } from "@executablemd/acp"; +import type { AcpxProviderDependencies } from "@executablemd/acp"; +import { Err, Ok } from "effection"; +import type { Operation, Result } from "effection"; + +import { resolveAgentConfig } from "./agent-config.ts"; +import type { AgentFlags } from "./agent-config.ts"; +import type { MachineSessionAssembly } from "./session-coordinator.ts"; + +/** Everything one invocation settled about agents, resolved exactly once. */ +export interface AgentStack { + /** The provider name the caller selected, already known to be registered. */ + provider: string; + /** The agent every consumer defaults to, environment fallback applied. */ + defaultAgent: string; + permissionMode: PermissionMode; + /** What this host states about machine-wide agent sessions, if anything. */ + sessions?: MachineSessionAssembly; +} + +/** + * Read the command line, the environment and the host's assembly into one + * configuration. + * + * A failure comes back as a `Result` rather than as a printed line and an exit, + * so the same resolution serves a command that runs a document and one that + * generates one first. + */ +export function* resolveAgentStack( + flags: AgentFlags, + sessions: MachineSessionAssembly | undefined, +): Operation> { + const config = resolveAgentConfig(flags); + if ("error" in config) { + return Err(new Error(config.error)); + } + if (flags.agentProvider !== "acpx") { + return Err(new Error(`Unknown agent provider "${flags.agentProvider}"`)); + } + const defaultAgent = + config.defaultAgent ?? (yield* readEnv("DEFAULT_AGENT_NAME")) ?? DEFAULT_AGENT_NAME; + return Ok({ + provider: flags.agentProvider, + defaultAgent, + permissionMode: config.permissionMode, + ...(sessions === undefined ? {} : { sessions }), + }); +} + +/** + * What this host built, if it built anything. + * + * Each piece reaches the provider directly rather than through a context: who + * owns a session and which build it belongs to are security decisions, and ones + * a document could replace are not ones. The two advertised sets are stated by + * the host, not inherited. + */ +export function hostAcpDependencies( + sessions: MachineSessionAssembly | undefined, +): AcpxProviderDependencies { + if (sessions === undefined) { + return {}; + } + return { + ...(sessions.coordinator ? { coordinator: sessions.coordinator } : {}), + ...(sessions.routeStore ? { routeStore: sessions.routeStore } : {}), + ...(sessions.executableObserver ? { executableObserver: sessions.executableObserver } : {}), + advertiseNativeLaunch: sessions.advertiseNativeLaunch, + advertiseClientNativeAttachment: sessions.advertiseClientNativeAttachment, + }; +} + +/** + * Install the agent stack a document runs under: the registration, the + * components with the resolved root provider, the permission mode, and the + * terminal this command has to give away. + * + * Nothing starts an agent — the provider validates availability on first use. + */ +export function* installRunAgentStack(stack: AgentStack): Operation { + const acpx = createAcpxProvider(hostAcpDependencies(stack.sessions)); + yield* registerAgentProvider("acpx", acpx); + + // The trusted host selects its own root provider by name. Document-level + // selection goes through the installation protocol; this is the host saying + // what it configured, which no document is composing around. + const providers: Record = { acpx }; + const factory = providers[stack.provider]; + const { defaultAgent, permissionMode } = stack; + yield* installAgentComponents({ + defaultAgent, + permissionMode, + rootProvider: { factory, options: { defaultAgent, permissionMode } }, + }); + yield* installPermissionMode(permissionMode); + // `xmd run` is the one command that has a terminal to give away. Help, + // document inspection and `xmd test` install no launcher, so a document that + // reaches under any of them refuses instead of spawning. + yield* installForegroundLauncher(); +} diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index ca4c45fe..ddb2e613 100755 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -4,6 +4,7 @@ * Usage: * xmd run [options] * xmd [options] (run is the default command) + * xmd prompt "" [options] * xmd workflow start [options] * xmd workflow resume * xmd workflow status|history [--json] @@ -16,6 +17,7 @@ * * Examples: * xmd run packages/core/examples/hello-world.md + * xmd prompt "ask me for my age and write the result to a file" * xmd packages/core/examples/hello-world.md --verbose * xmd run packages/core/examples/hello-world.md --journal events.jsonl * xmd run README.md#Release/Publish @@ -61,8 +63,6 @@ import { inspectDocument, agentIdentityComponents, installAgentComponents, - installPermissionMode, - registerAgentProvider, retainedSource, rootSourcePath, useNormalizedOutput, @@ -71,18 +71,15 @@ import { import { executeInstalled } from "@executablemd/core/host"; import type { ExecutionInstallation } from "@executablemd/core/host"; import type { - AgentProviderFactory, DocumentTargetInfo, FileRootDocument, PropsSchema, RootDocumentSource, } from "@executablemd/core"; -import { command as hostCommand, env as readEnv } from "@executablemd/runtime"; -import { createAcpxProvider, DEFAULT_AGENT_NAME } from "@executablemd/acp"; +import { command as hostCommand } from "@executablemd/runtime"; import type { MachineSessionAssembly } from "./session-coordinator.ts"; import { installTestingComponents, - TestFailureError, testHarnessInstallation, useTesting, } from "@executablemd/testing"; @@ -94,8 +91,9 @@ import { import { installWebComponents, installWebElicitation } from "@executablemd/web"; import { timebox } from "@effectionx/timebox"; import { timeout as runTimeout } from "@executablemd/runtime"; -import { installForegroundLauncher } from "@executablemd/runtime"; -import { resolveAgentConfig } from "./agent-config.ts"; +import { installRunAgentStack, resolveAgentStack } from "./agent-stack.ts"; +import type { AgentStack } from "./agent-stack.ts"; +import { reportFailure } from "./report.ts"; import { TIMEOUT_FLAGS, resolveRunTimeouts } from "./timeouts.ts"; import type { RunTimeouts } from "./timeouts.ts"; import type { AgentFlags } from "./agent-config.ts"; @@ -108,9 +106,19 @@ import { describeError, extractPropsArgs, formatProperties, - resolveProps, + resolvePropsFromSources, } from "./props.ts"; import type { Binding, Extraction } from "./props.ts"; +import { + namesPrompt, + OUTPUT_OPTION, + RUN_OPTION, + scanPromptArgs, + SESSION_OPTION, +} from "./prompt-args.ts"; +import type { PromptScan } from "./prompt-args.ts"; +import { runPrompt } from "./prompt.ts"; +import type { PromptExecution } from "./prompt.ts"; import { componentSearchPath, resolveTestTarget } from "./test-target.ts"; import { renderSyntaxJson, renderSyntaxMarkdown, syntaxCatalog } from "./syntax.ts"; import { testingExecutionHost } from "./testing-host.ts"; @@ -156,19 +164,17 @@ const SECRET_DETECTION_FIELD = { ...field(z.boolean(), field.default(true)), }; -const runConfig = object({ - path: { - description: "markdown document to execute, optionally `#` and one target selector", - ...field(z.string().optional(), cli.argument()), - }, - // Declared so `xmd run --help` lists it with every other option. The value is - // lifted out of argv by readEvalFlags before parsing — see eval-source.ts — - // so this field is never the source of the document. - eval: { - description: "inline markdown document to execute, in place of a path", - aliases: ["-e"], - ...field(z.string().optional()), - }, +/** + * Everything a command that ends in a document execution configures. + * + * Declared once because `xmd run` and `xmd prompt` configure the same + * execution: the includes it resolves components through, what it writes and + * where, the agent stack it runs under, and the three deadlines. `run` adds the + * document, `prompt` adds the request and where to keep the source; nothing + * else differs, and a second copy of this list would be the two commands + * drifting apart one option at a time. + */ +const executionFields = { include: { description: "component search directory", ...field(z.array(z.string()), field.default(["components", "."]), field.array()), @@ -220,6 +226,50 @@ const runConfig = object({ ...field(z.boolean(), field.default(false)), }, secretDetection: SECRET_DETECTION_FIELD, +}; + +const runConfig = object({ + path: { + description: "markdown document to execute, optionally `#` and one target selector", + ...field(z.string().optional(), cli.argument()), + }, + // Declared so `xmd run --help` lists it with every other option. The value is + // lifted out of argv by readEvalFlags before parsing — see eval-source.ts — + // so this field is never the source of the document. + eval: { + description: "inline markdown document to execute, in place of a path", + aliases: ["-e"], + ...field(z.string().optional()), + }, + ...executionFields, +}); + +/** + * `xmd prompt` — the Prompt, where the approved Plan goes, and whether it runs. + * + * The individual `--props-*` options a candidate declares are deliberately + * absent: they exist only once a document does, and the props phase binds them + * per candidate. The aggregate `--props` is absent for the reason `run`'s is — + * the parser coerces a separated value before any schema could judge it. + */ +const promptConfig = object({ + request: { + description: "the steps the coding agent should turn into a Plan", + ...field(z.string().optional(), cli.argument()), + }, + output: { + description: "write the approved Plan here instead of to stdout (path must not exist)", + ...field(z.string().optional()), + }, + run: { + description: "run the approved Plan instead of writing it", + ...field(z.boolean(), field.default(false)), + }, + session: { + description: "logical name for the assistant session (default: unique to this invocation)", + ...field(z.string().optional()), + }, + ...executionFields, }); const testConfig = object({ @@ -284,6 +334,7 @@ const xmd = program({ config: commands( { run: runConfig, + prompt: promptConfig, test: testConfig, syntax: syntaxConfig, "test-agent": testAgentConfig, @@ -480,73 +531,25 @@ function* underRunDeadline(timeouts: RunTimeouts, body: () => Operation): } /** - * Install the agent stack for `xmd run`: permission mode, the ACPX - * registration, and the components with the resolved root provider. Invalid flags and an unknown --agent-provider fail here - * — before any document executes. Nothing starts an agent: the provider - * validates availability on first use. + * Settle the Agent configuration for one invocation, or report why it cannot + * be settled. + * + * Called once per command, before anything a document or an agent could + * observe: incompatible permission flags and an unknown `--agent-provider` are + * command-line failures, and a command line is wrong before any of it runs. + * `undefined` means the caller has already reported and should stop. */ -function* installAgentStack( +function* settleAgentStack( flags: AgentFlags, sessions: MachineSessionAssembly | undefined, -): Operation { - const config = resolveAgentConfig(flags); - if ("error" in config) { - console.error(config.error); - yield* exit(1); - return; - } - - // What this host built, if it built anything. Each piece reaches the provider - // directly rather than through a context: who owns a session and which build - // it belongs to are security decisions, and ones a document could replace are - // not ones. A host that answers who owns a session also answers how it was - // constructed and which build accepted its identity, all from the same - // trusted root — an agent that names its own sessions needs all three, and - // refuses unless it has all three. - // - // The two advertised sets are stated by the host, not inherited: this is the - // ordinary `xmd run` profile saying what it has proven. - const acpx = createAcpxProvider( - sessions === undefined - ? undefined - : { - ...(sessions.coordinator ? { coordinator: sessions.coordinator } : {}), - ...(sessions.routeStore ? { routeStore: sessions.routeStore } : {}), - ...(sessions.executableObserver - ? { executableObserver: sessions.executableObserver } - : {}), - advertiseNativeLaunch: sessions.advertiseNativeLaunch, - advertiseClientNativeAttachment: sessions.advertiseClientNativeAttachment, - }, - ); - yield* registerAgentProvider("acpx", acpx); - const defaultAgent = - config.defaultAgent ?? (yield* readEnv("DEFAULT_AGENT_NAME")) ?? DEFAULT_AGENT_NAME; - - // The trusted host selects its own root provider by name. Document-level - // selection goes through the installation protocol; this is the host saying - // what it configured, which no document is composing around. - const providers: Record = { acpx }; - const factory = Object.hasOwn(providers, flags.agentProvider) - ? providers[flags.agentProvider] - : undefined; - if (!factory) { - console.error(`Unknown agent provider "${flags.agentProvider}"`); +): Operation { + const stack = yield* resolveAgentStack(flags, sessions); + if (!stack.ok) { + console.error(stack.error.message); yield* exit(1); - return; + return undefined; } - - const permissionMode = config.permissionMode; - yield* installAgentComponents({ - defaultAgent, - permissionMode, - rootProvider: { factory, options: { defaultAgent, permissionMode } }, - }); - yield* installPermissionMode(permissionMode); - // `xmd run` is the one command that has a terminal to give away. Help, - // document inspection and `xmd test` install no launcher, so a document that - // reaches under any of them refuses instead of spawning. - yield* installForegroundLauncher(); + return stack.value; } /** @@ -615,7 +618,16 @@ interface DocumentConfig { export interface DocumentMode { testing: boolean; - agent?: AgentFlags; + /** + * The Agent configuration this invocation already settled. + * + * Resolved, not the flags that produced it: an invocation reads + * `DEFAULT_AGENT_NAME` and decides its permission mode once, and a command + * that generates a document before running one has two consumers for that one + * answer. Passing the flags instead would let the second consumer reach a + * different conclusion than the first from the same command line. + */ + agent?: AgentStack; /** * What this host states about machine-wide agent sessions: who owns one, * how it was constructed, which build it belongs to, and which adapters this @@ -684,10 +696,10 @@ export function* installDocumentComponents(mode: DocumentMode, verbose: boolean) // run — and where the workflow's provider, installed further out, would lose // to it at the same `{ at: "min" }`. - // Agent flags are exclusive to `xmd run` — `xmd test` drives agents - // through the deterministic TestAgent stack instead. + // Agent flags belong to the two commands that end in a document execution — + // `xmd test` drives agents through the deterministic TestAgent stack instead. if (mode.agent) { - yield* installAgentStack(mode.agent, mode.machineSessions); + yield* installRunAgentStack(mode.agent); } } @@ -895,6 +907,60 @@ function* runScopedDocument( } } +/** What an approved prompt document runs with, beyond the source and its props. */ +export interface PromptExecutionConfig { + include: string[]; + verbose: boolean; + journal: string | undefined; + raw: boolean; + secretDetection: boolean; +} + +/** + * How `xmd prompt` runs the document a person approved: exactly as `xmd run` + * runs a supplied one. + * + * The prompt profile's scope is already gone by the time this is called, so the + * executed program gets a fresh ordinary Agent provider and inherits neither + * the assistant session nor its instruction layer. The browser form is composed + * around the document for the same reason a run composes one: `xmd prompt` is a + * command a person is sitting in front of. + */ +export function promptExecutor( + config: PromptExecutionConfig, + stack: AgentStack, + sessions: MachineSessionAssembly | undefined, + installService: HostServiceInstaller, +): (approved: PromptExecution) => Operation> { + return (approved) => + scoped(function* (): Operation> { + announceSecretDetection(config.secretDetection); + yield* installWebElicitation(); + return yield* runScopedDocument( + { + root: approved.root, + include: config.include, + verbose: config.verbose, + journal: config.journal, + raw: config.raw, + secretDetection: config.secretDetection, + retainProcessOutput: keepsProcessOutput(config.journal), + }, + { + testing: false, + props: approved.props, + ...(sessions === undefined ? {} : { machineSessions: sessions }), + // The same object generation was configured from. The provider it + // installs is a fresh ordinary one — the profile's scope is already + // gone — but which agent it defaults to and what it may do were + // decided once, for the whole invocation. + agent: stack, + }, + installService, + ); + }); +} + /** * A document-target failure as the command line reports it, or `undefined` * when this failure is not one. @@ -924,16 +990,6 @@ function targetFailureReport(root: RootDocumentSource, error: unknown): string | return [outcome, heading, ...references].join("\n"); } -/** Print a completed document's failure the way `xmd` has always printed it. */ -function reportFailure(error: Error, prefix?: string): void { - const label = prefix === undefined ? "" : `${prefix}: `; - if (error instanceof TestFailureError) { - console.error(`\n${label}tests failed: ${error.message}`); - return; - } - console.error(`${label}${error.message}`); -} - interface TestConfig extends Omit { /** * Optional because `field` types a schema by what it accepts, and @@ -1167,6 +1223,15 @@ interface PropsPhase { * would lose exactly the token the separator was there to protect. */ workflow?: { action?: string; target?: string; argument?: string; value?: string }; + /** + * What fixed grammar established about an `xmd prompt` command line. + * + * Carried rather than re-derived downstream, for the same reason the workflow + * positionals are: the request may have been written after `--`, where the + * parser never sees it, and the generated property occurrences are the ones + * this scan classified. + */ + prompt?: PromptScan; root?: RootDocumentSource; bindings: Binding[]; extraction?: Extraction; @@ -1201,6 +1266,28 @@ interface PropsPhase { * of argv, so it needs no parse at all. */ function* preparePropsPhase(args: string[], evalFlags: EvalFlags): Operation { + // `xmd prompt` declares its own grammar and has no document to inspect: the + // schema its generated options come from is written by an agent that has not + // been asked anything yet. Everything below that reads a document, and every + // refusal that assumes one, is therefore skipped. + if (namesPrompt(args)) { + // Refused here rather than with the other commands' inline refusal below, + // because that one is reached through the parse this branch exists to skip. + // An inline document is what `xmd prompt` sets out to write, so a caller who + // supplied one would otherwise watch it generate a different one instead. + if (evalFlags.values[0] !== undefined) { + return { + args, + bindings: [], + error: + `unrecognized option for xmd prompt: ${EVAL_OPTION} — inline documents are ` + + "exclusive to xmd run", + }; + } + const scan = scanPromptArgs(args); + return { args: scan.fixed, bindings: [], prompt: scan }; + } + // `xmd workflow` reads its options from the head and its remaining positionals // from the tail. The parser only ever sees the head, so a dash-leading token // after `--` is never offered to it as an option; the grammar check below @@ -1264,7 +1351,9 @@ function* preparePropsPhase(args: string[], evalFlags: EvalFlags): Operation" --props-name Ada', + "", + ` ${AGGREGATE_OPTION} `, + " Set document properties as a JSON object", + ` Environment: ${AGGREGATE_ENV}`, + "", + ` ${OUTPUT_OPTION} `, + " Write the approved Plan there instead of to stdout. The path must not", + " exist; an existing one is left alone and the command stops.", + "", + ` ${RUN_OPTION}`, + " Run the approved Plan instead of writing it. With --output the file is", + " written first, and only a successful write is followed by the run.", + "", + ` ${SESSION_OPTION} `, + " Use this logical assistant session instead of one unique to this run.", + "", + `Options that only configure running a Plan — --journal, --raw, --verbose, the`, + "exec and fetch timeouts, the permission flags and secret detection — are", + `refused without ${RUN_OPTION}, because without it nothing runs for them to`, + "configure.", + "", + "Permission flags configure the approved Plan. Writing the Plan is a", + "conversation about text and gives the coding agent nothing: an empty directory", + "of its own, no tools, no MCP servers, and every native permission request", + "denied.", +].join("\n"); + /** * Help for whichever command the arguments name. A command renders its * own help when `--help` is its first argument, so the flag removed @@ -1586,7 +1727,8 @@ function renderHelp(phase: PropsPhase): string { const help = xmd.parse({ args: [command, "--help"] }); const base = help.ok && help.value.config.help ? help.value.config.text : xmd.help({ args: [] }); - const epilogue = command === "run" ? RUN_SOURCE_HELP : ""; + const epilogue = + command === "run" ? RUN_SOURCE_HELP : command === "prompt" ? PROMPT_REQUEST_HELP : ""; const withSource = epilogue === "" ? base : `${base}\n\n${epilogue}`; if (!phase.root) { @@ -1628,23 +1770,11 @@ function* resolveRunProps( } try { - const individualEnv: { binding: Binding; value: string }[] = []; - for (const binding of phase.bindings) { - const value = yield* readEnv(binding.env); - if (value !== undefined) { - individualEnv.push({ binding, value }); - } - } - const aggregateEnv = yield* readEnv(AGGREGATE_ENV); - return { - value: resolveProps({ + value: yield* resolvePropsFromSources({ propsSchema: phase.propsSchema, bindings: phase.bindings, - individual: phase.extraction.individual, - aggregateCli: phase.extraction.aggregate, - aggregateEnv, - individualEnv, + extraction: phase.extraction, }), }; } catch (error) { @@ -1747,6 +1877,19 @@ function* dispatch( break; } const root = propsPhase.root; + const runStack = yield* settleAgentStack( + { + agentProvider: config.agentProvider, + defaultAgent: config.defaultAgent, + approveAll: config.approveAll, + approveReads: config.approveReads, + denyAll: config.denyAll, + }, + sessions, + ); + if (runStack === undefined) { + break; + } announceSecretDetection(config.secretDetection); const result = yield* scoped(function* (): Operation> { // `` reaches a person through the browser form, and `xmd run` @@ -1765,13 +1908,7 @@ function* dispatch( // it, which is what keeps a machine session from being acted on by // a command that never said it could own one. ...(sessions === undefined ? {} : { machineSessions: sessions }), - agent: { - agentProvider: config.agentProvider, - defaultAgent: config.defaultAgent, - approveAll: config.approveAll, - approveReads: config.approveReads, - denyAll: config.denyAll, - }, + agent: runStack, }, installService, ); @@ -1789,11 +1926,64 @@ function* dispatch( } break; } + case "prompt": { + const config = command.config; + const scan = propsPhase.prompt; + if (scan === undefined) { + console.error( + 'xmd prompt names the command first — write `xmd prompt "" [options]`', + ); + yield* exit(1); + break; + } + // Once, here, and handed to both consumers below. Authorship and the run + // that may follow it are one invocation, so they answer to one + // `--default-agent`, one `DEFAULT_AGENT_NAME` and one permission mode. + const promptStack = yield* settleAgentStack( + { + agentProvider: config.agentProvider, + defaultAgent: config.defaultAgent, + approveAll: config.approveAll, + approveReads: config.approveReads, + denyAll: config.denyAll, + }, + sessions, + ); + if (promptStack === undefined) { + break; + } + const exitCode = yield* runPrompt( + { + argv: helpRequest.args, + scan, + include: config.include, + ...(config.output === undefined ? {} : { output: config.output }), + run: config.run, + ...(config.session === undefined ? {} : { session: config.session }), + stack: promptStack, + }, + { + ...(sessions === undefined ? {} : { sessions }), + catalog: syntaxCatalog, + // `` reaches a person through the browser form, and the + // review question is asked by the command rather than by a document. + // A host that answers installs a provider; one that does not installs + // none, and nothing downstream reads a profile to find out which. + installElicitation: installWebElicitation, + execute: promptExecutor(config, promptStack, sessions, installService), + }, + ); + if (exitCode !== 0) { + yield* exit(exitCode); + } + break; + } case "test": { const strayTimeout = findTimeoutFlag(evalFlags.rest); if (strayTimeout) { console.error( - `unrecognized option for xmd test: ${strayTimeout} — timeout options are exclusive to xmd run`, + `unrecognized option for xmd test: ${strayTimeout} — timeout options are exclusive to ` + + "xmd run and xmd prompt", ); yield* exit(1); break; @@ -1801,7 +1991,8 @@ function* dispatch( const agentFlag = findAgentOnlyFlag(evalFlags.rest); if (agentFlag) { console.error( - `unrecognized option for xmd test: ${agentFlag} — agent options are exclusive to xmd run`, + `unrecognized option for xmd test: ${agentFlag} — agent options are exclusive to ` + + "xmd run and xmd prompt", ); yield* exit(1); break; @@ -1809,7 +2000,8 @@ function* dispatch( const propsFlag = findPropsFlag(evalFlags.rest); if (propsFlag) { console.error( - `unrecognized option for xmd test: ${propsFlag} — document properties are exclusive to xmd run`, + `unrecognized option for xmd test: ${propsFlag} — document properties are exclusive to ` + + "xmd run and xmd prompt", ); yield* exit(1); break; @@ -1848,7 +2040,8 @@ function* dispatch( const agentFlag = findAgentOnlyFlag(evalFlags.rest); if (agentFlag) { console.error( - `unrecognized option for xmd workflow: ${agentFlag} — agent options are exclusive to xmd run`, + `unrecognized option for xmd workflow: ${agentFlag} — agent options are exclusive to ` + + "xmd run and xmd prompt", ); yield* exit(1); break; @@ -1976,14 +2169,22 @@ export function* runXmd( } // Recognized before anything reads a document: a malformed duration is a // grammar failure, and a grammar failure never depends on what is on disk. - // Help, `--version`, and the other commands stay outside a run lifecycle, - // which is why the timeout options are read only for a run. + // Help, `--version`, and the commands that execute nothing stay outside a run + // lifecycle, which is why the timeout options are read only for the two that + // end in one. const provisional = xmd.parse({ args: helpRequest.args }); const selected = provisional.ok ? provisional.value.config : undefined; - const isRun = - !helpRequest.requested && selected !== undefined && !selected.help && selected.name === "run"; - - if (!isRun) { + // The two commands that end in a document execution. `xmd prompt`'s deadline + // encloses more than a run's — the catalog, the assistant session, every + // repair, the human review, provider teardown, the output file and the run — + // because all of it is what the caller asked to be bounded. + const executes = + !helpRequest.requested && + selected !== undefined && + !selected.help && + (selected.name === "run" || selected.name === "prompt"); + + if (!executes) { return yield* dispatch(evalFlags, helpRequest, installService, workflowHost, sessions); } diff --git a/packages/cli/src/documents/prompt-command.md b/packages/cli/src/documents/prompt-command.md new file mode 100644 index 00000000..a107a6d0 --- /dev/null +++ b/packages/cli/src/documents/prompt-command.md @@ -0,0 +1,257 @@ +--- +props: + type: object + properties: + request: { type: string } + syntax: { type: string } + session: { type: string } + required: [request, syntax, session] + additionalProperties: false +returns: + type: string +--- + +# `xmd prompt` turns steps into a program + +This document is a workflow that generates an executable Plan from a sequence of +steps. It combines the original Prompt, which describes those steps, with the XMD +components available to carry them out. A coding agent turns both into one +document that explains and executes the sequence. + +The result is the XMD version of a coding agent’s plan. A conventional Markdown +plan must be interpreted again before its steps can happen. An XMD Plan already +contains those executable steps, so running it simply executes them. + +A draft remains text while this workflow reviews it. Nothing in it runs before +you approve it. After approval, `xmd prompt` validates the exact source again. By +default it prints the approved XMD source. `--output` writes that source to a +file instead, and `--run` executes the Plan. With both options, the command +writes the source before running it. + + + + + + +## Create the first draft + + +Create one complete XMD Plan from this Prompt: + +{props.request} + +Every Plan is complete on its own: + +- optional frontmatter, and then one descriptive level-one Markdown heading as + the first body content; +- the Prompt's complete sequence, written as readable steps; +- every outcome the Prompt asked for; +- those steps in an order that makes sense; and +- each XMD component beside the prose describing the action it performs. + +The prose is part of the program and appears when the Plan runs, so its source +and its execution tell the same story. + +For the Prompt "ask me for my age and write it to a file", the shape is: + +```markdown +# Ask for and save your age + +Ask me for my age. + + + +Write it to a file. + +{answer.age} +``` + +You do not have to repeat the Prompt word for word. Divide it, clarify it and +rewrite it into natural prose — but keep every outcome it asked for, keep them in +an order that makes sense, and keep each one beside the component that carries it +out. + +Everything you may use is described below. Use nothing that is not here. + +{props.syntax} + +Reply with the Plan source and nothing else. No enclosing code fence, no +explanation before or after it. + + + + + +## Check and repair the draft + +Each new draft is checked before it is shown to you. If the check finds problems, +the coding agent gets up to three repair attempts to replace it with a corrected +Plan. A revision you request later is a new draft and receives three repair +attempts of its own. + + + + + + + + + +That Plan has problems. These are the exact ones: + + + + +Send one complete replacement Plan that resolves every problem above. + +Every Plan is complete on its own: + +- optional frontmatter, and then one descriptive level-one Markdown heading as + the first body content; +- the Prompt's complete sequence, written as readable steps; +- every outcome the Prompt asked for; +- those steps in an order that makes sense; and +- each XMD component beside the prose describing the action it performs. + +Write the title the Plan needs rather than the one the last draft had: add it if +it was missing, move it if it was not the first body content, and replace it if +it did not describe the Plan. + +Reply with the Plan source and nothing else. No enclosing code fence, no +explanation before or after it. + + + + + +## Review the draft + +The workflow shows you the complete draft after its repair attempts. + +- Choose **Approve** to accept a draft that passed its check. +- Choose **Request changes** to send feedback to the coding agent and create a + new draft. +- Choose **Stop** to end without outputting or running anything. + +A draft with remaining problems cannot be approved. You may review at most ten +drafts, and the tenth cannot be revised. If the tenth draft still has problems, +you may ask the coding agent to explain what went wrong or stop. + + +### Original Prompt + +{props.request} + +### Draft Plan + + + + +### Problems that remain + +The coding agent used all three repair attempts, but the draft still has these +problems: + + + + + + +## Continue from your decision + +Approve keeps this exact draft and leaves the review. Request changes sends your +feedback to the coding agent and starts a new draft. Stop ends here. On a tenth +draft that still has problems there is nothing left to revise into, so the two +remaining choices are to ask the coding agent what went wrong, or to stop. + + + + + + + + + + + + + + + + + +The final Plan still has these problems: + + + + +Explain briefly why the attempts did not resolve them and what the person should +clarify in their next Prompt. Do not create another Plan. + + + + + + +You read that Plan and asked for this to change: + +{review.feedback} + +Send one complete replacement Plan. + +Every Plan is complete on its own: + +- optional frontmatter, and then one descriptive level-one Markdown heading as + the first body content; +- the Prompt's complete sequence, written as readable steps; +- every outcome the Prompt asked for; +- those steps in an order that makes sense; and +- each XMD component beside the prose describing the action it performs. + +Write the title the Plan needs rather than the one the last draft had: add it if +it was missing, move it if it was not the first body content, and replace it if +it did not describe the Plan. + +Reply with the Plan source and nothing else. No enclosing code fence, no +explanation before or after it. + + + + + +## Return the approved Plan + +Only an approved Plan can leave this workflow. Its source is returned exactly as +the coding agent wrote it. + + + + + + + diff --git a/packages/cli/src/packaged-document.ts b/packages/cli/src/packaged-document.ts new file mode 100644 index 00000000..4a99111c --- /dev/null +++ b/packages/cli/src/packaged-document.ts @@ -0,0 +1,55 @@ +/** + * The Markdown documents this CLI ships and executes itself. + * + * `xmd prompt` runs a first-party document rather than a TypeScript policy, so + * that document has to be present wherever the command is: a source checkout, a + * published npm package, and a compiled binary with no checkout at all. It is + * located from this module's own URL, never from the working directory and never + * through the component search path — the command must find the same program + * whatever directory a person happens to be standing in, and a repository file + * must not be able to answer for it. + * + * Each build keeps `src/documents/` beside its module: `deno compile --include` + * embeds each file at the same relative path, and the npm build copies the + * directory into the emitted tree. So the one lookup below is correct in all + * four places, and a build that forgets the asset fails loudly on first use + * rather than silently choosing different behavior. + */ + +import { readFile } from "node:fs/promises"; +import { until } from "effection"; +import type { Operation } from "effection"; + +/** The document `xmd prompt` executes to turn a request into a Plan. */ +export const PROMPT_COMMAND_DOCUMENT = "prompt-command.md"; + +/** + * Where a packaged document lives, as a URL beside this module. + * + * One directory rather than "any Markdown under `src/`": packages keep test + * documents and scenario fixtures beside their modules too, and a build that + * swept those up would publish them and grow the binary for no reason. Being in + * here is what declares a document shipped. + */ +export function packagedDocumentUrl(name: string): URL { + return new URL(`./documents/${name}`, import.meta.url); +} + +/** + * Read one packaged document. + * + * `node:fs` rather than a host File Api: this is the CLI reading its own + * program, not a document reaching the caller's filesystem, and it must not be + * answerable by whatever the running document installed. + */ +export function* readPackagedDocument(name: string): Operation { + const url = packagedDocumentUrl(name); + try { + return yield* until(readFile(url, "utf8")); + } catch (error) { + throw new Error( + `the packaged document ${name} is missing from this build (looked in ${url.href})`, + { cause: error }, + ); + } +} diff --git a/packages/cli/src/prompt-args.ts b/packages/cli/src/prompt-args.ts new file mode 100644 index 00000000..d370a201 --- /dev/null +++ b/packages/cli/src/prompt-args.ts @@ -0,0 +1,430 @@ +/** + * `xmd prompt` argument grammar — everything decidable before a document exists + * (specs/prompt-command-spec.md). + * + * A pure function over argv. It reads no Context Apis, contacts no agent, and + * inspects nothing on disk, which is what lets the command refuse a malformed + * command line before it builds a catalog, opens a session, asks a person, or + * writes a file. + * + * The command line is read in two stages because only the first one is fixed. + * The request, the built-in options and the end-of-options separator are known + * from the grammar alone; whether `--props-name` takes a following token depends + * on what the *generated* document declares, and no document exists yet. So this + * stage classifies what it can, records each generated occurrence with the token + * it provisionally read, and leaves the rest to the candidate that supplies the + * schema. + */ + +import { AGGREGATE_OPTION } from "./props.ts"; +import type { Binding } from "./props.ts"; + +export const PROMPT_COMMAND = "prompt"; +export const OUTPUT_OPTION = "--output"; +export const SESSION_OPTION = "--session"; +export const RUN_OPTION = "--run"; + +/** The built-in options that take a separated value. */ +const VALUE_OPTIONS: readonly string[] = [ + "--include", + "--journal", + "-j", + "--agent-provider", + "--default-agent", + "--timeout", + "--timeout-exec", + "--timeout-fetch", + OUTPUT_OPTION, + SESSION_OPTION, +]; + +/** The built-in options that take none. */ +const SWITCH_OPTIONS: readonly string[] = [ + "--verbose", + "-V", + "--raw", + "--approve-all", + "--approve-reads", + "--deny-all", + "--secret-detection", + "--no-secret-detection", + RUN_OPTION, + "--help", + "-h", + "--version", +]; + +/** + * The options that configure the Plan's execution and nothing else. + * + * `xmd prompt` prints an approved Plan unless `--run` asks for it to be run, so + * each of these describes work that would not happen. Accepting one silently + * would mean answering a caller who asked for a journal, a permission mode or an + * exec deadline with a command that creates none of them. + * + * Everything absent from this list is here for a reason the command always has: + * `--include` builds the catalog and admits properties, `--agent-provider` and + * `--default-agent` settle who writes the Plan, `--session` names the + * conversation, `--timeout` bounds the whole command, and `--output` is where an + * approved Plan goes. + */ +const RUN_ONLY_OPTIONS: readonly string[] = [ + "--journal", + "-j", + "--raw", + "--verbose", + "-V", + "--timeout-exec", + "--timeout-fetch", + "--approve-all", + "--approve-reads", + "--deny-all", + "--secret-detection", + "--no-secret-detection", +]; + +const RUN_ONLY = new Set(RUN_ONLY_OPTIONS); + +const VALUE = new Set(VALUE_OPTIONS); +const KNOWN = new Set([...VALUE_OPTIONS, ...SWITCH_OPTIONS, AGGREGATE_OPTION]); + +function optionName(token: string): string { + const equals = token.indexOf("="); + return equals === -1 ? token : token.slice(0, equals); +} + +function generatesProperty(name: string): boolean { + return name.startsWith("--props-") || name.startsWith("--no-props"); +} + +/** + * Whether this token is an option `xmd` itself defines. + * + * Used to decide what a generated property option may provisionally read as its + * value. A caller writing `--props-name --raw` means the switch, not a value of + * `--raw`; a value that really begins with `-` is written `--props-name=-value`. + */ +function isKnownOption(token: string): boolean { + if (!token.startsWith("-") || token === "-") { + return false; + } + const name = optionName(token); + return KNOWN.has(name) || generatesProperty(name); +} + +/** One `--props-*` token, as written. */ +export interface PropertyOccurrence { + /** The option name, without any `=value`. */ + option: string; + /** The value written with `=`, when it was written that way. */ + inline?: string; + /** + * The following token this scan read as the option's value. + * + * Provisional: whether the option takes one is the candidate's answer, and a + * candidate that declares it a switch turns this token into a second request. + */ + provisional?: string; +} + +/** What fixed grammar establishes about one `xmd prompt` command line. */ +export interface PromptScan { + /** The request, byte for byte, when exactly one was written. */ + request?: string; + /** + * The argv the built-in option parser sees. + * + * Generated property tokens and the aggregate `--props` are removed: the + * parser defines neither, and it coerces a separated value through `Number()` + * before any schema could judge it. Tokens after `--` are left out for the + * same reason the workflow command leaves them out — a dash-leading positional + * handed back to a parser is read as an option again. + */ + fixed: string[]; + /** Every generated property occurrence, in the order it was written. */ + occurrences: PropertyOccurrence[]; + /** Why fixed grammar refuses this command line. */ + error?: string; +} + +/** Whether these arguments select the `prompt` command. */ +export function namesPrompt(args: readonly string[]): boolean { + return args[0] === PROMPT_COMMAND; +} + +const ORDER_HELP = + "document properties follow the request, as in " + + '`xmd prompt "" --props-name `'; + +export function scanPromptArgs(args: readonly string[]): PromptScan { + const fixed: string[] = [PROMPT_COMMAND]; + const occurrences: PropertyOccurrence[] = []; + let request: string | undefined; + let extra: string | undefined; + let runs = false; + let runOnly: string | undefined; + let parsingOptions = true; + let index = 1; + + while (index < args.length) { + const token = args[index]; + + if (parsingOptions && token === "--") { + parsingOptions = false; + index += 1; + continue; + } + + if (parsingOptions && token.startsWith("-") && token !== "-") { + const equals = token.indexOf("="); + const name = optionName(token); + + if (name === AGGREGATE_OPTION) { + index += equals === -1 && args[index + 1] !== undefined ? 2 : 1; + continue; + } + + if (generatesProperty(name)) { + if (request === undefined) { + return { + fixed, + occurrences, + error: `unrecognized option: ${name} — ${ORDER_HELP}`, + }; + } + if (equals !== -1) { + occurrences.push({ option: name, inline: token.slice(equals + 1) }); + index += 1; + continue; + } + const next = args[index + 1]; + if (next === undefined || next === "--" || isKnownOption(next)) { + occurrences.push({ option: name }); + index += 1; + continue; + } + occurrences.push({ option: name, provisional: next }); + index += 2; + continue; + } + + if (!KNOWN.has(name)) { + // The parser stops at the first option it does not define and drops the + // rest, so an option nobody defines would otherwise be accepted in + // silence — and a caller who asked for something the command never did + // has not been answered. `--save` is named because it is the one + // spelling somebody may remember; it was replaced before release, so + // there is no alias to keep. + return { + ...(request === undefined ? {} : { request }), + fixed, + occurrences, + error: + name === "--save" + ? `unrecognized option for xmd prompt: --save — the approved Plan goes to stdout, ` + + `and ${OUTPUT_OPTION} writes it to a file` + : `unrecognized option for xmd prompt: ${name}`, + }; + } + + if (name === RUN_OPTION) { + // `optionName` stops at the first `=`, so every `--run=…` spelling + // arrives here under the name of the switch. Reading one as the switch + // would let a token satisfy the gate below while meaning the opposite, + // and the ordinary parser reads `--run=true` as the default rather than + // as true — so a caller who spelled it that way is answered by neither + // half of this command. Refused instead, before it establishes anything. + if (equals !== -1) { + return { + ...(request === undefined ? {} : { request }), + fixed, + occurrences, + error: + `${RUN_OPTION} does not take a value — write ${RUN_OPTION} to execute the Plan ` + + "or leave it out to write the Plan", + }; + } + runs = true; + } + if (runOnly === undefined && RUN_ONLY.has(name)) { + runOnly = name; + } + const separated = + equals === -1 && VALUE.has(name) && args[index + 1] !== undefined + ? args[index + 1] + : undefined; + // Read here rather than after parsing, because an empty value is exactly + // what the parser cannot report: an option it reads as absent falls back + // to the default, so a caller who asked for a session and named none would + // silently get the generated one instead. + if (name === SESSION_OPTION) { + const value = equals === -1 ? separated : token.slice(equals + 1); + if (value === undefined || value.length === 0) { + return { + ...(request === undefined ? {} : { request }), + fixed, + occurrences, + error: + `${SESSION_OPTION} needs a name — write \`${SESSION_OPTION} \` or leave ` + + "it out for a session unique to this invocation", + }; + } + } + fixed.push(token); + if (separated !== undefined) { + fixed.push(separated); + index += 2; + continue; + } + index += 1; + continue; + } + + if (request === undefined) { + request = token; + // Kept out of the parser's argv when the separator carried it: a + // dash-leading request written after `--` is positional because of where + // it is, and handing it back would make it an option again. + if (parsingOptions) { + fixed.push(token); + } + } else if (extra === undefined) { + extra = token; + } + index += 1; + } + + if (extra !== undefined) { + return { + ...(request === undefined ? {} : { request }), + fixed, + occurrences, + error: + `unrecognized argument for xmd prompt: ${extra} — the command takes exactly one ` + + "request, and " + + ORDER_HELP, + }; + } + + if (request === undefined) { + return { + fixed, + occurrences, + error: 'xmd prompt requires one request — `xmd prompt ""`', + }; + } + + if (request.trim().length === 0) { + return { + request, + fixed, + occurrences, + error: "xmd prompt requires a request with at least one non-whitespace character", + }; + } + + if (runOnly !== undefined && !runs) { + return { + request, + fixed, + occurrences, + error: + `${runOnly} configures running the Plan, and without ${RUN_OPTION} this command ` + + `writes the Plan instead of running it — add ${RUN_OPTION}, or drop ${runOnly}`, + }; + } + + return { request, fixed, occurrences }; +} + +/** + * How a supplied individual option is written: whether it takes a token, and + * whether repeating it accumulates. + * + * This is what a later candidate may not change. The comparison happens before + * any token is extracted, so a switch that became a value option cannot reach + * forward and consume the `--raw` written after it. + */ +export interface OptionSignature { + boolean: boolean; + array: boolean; +} + +export function signatureOf(binding: Binding): OptionSignature { + return { boolean: binding.boolean, array: binding.array }; +} + +function describeSignature(signature: OptionSignature): string { + if (signature.boolean) { + return "a bare switch"; + } + return signature.array ? "a repeated value option" : "a single-value option"; +} + +/** + * Whether the candidate still declares every supplied option the way the + * candidate that first bound it did. + * + * A removed option or a changed shape is the caller's command line meaning + * something else than it did, which no revision may do silently. + */ +export function signatureFailure( + frozen: ReadonlyMap, + bindings: readonly Binding[], +): string | undefined { + const current = new Map(bindings.map((binding) => [binding.option, signatureOf(binding)])); + for (const [option, signature] of frozen) { + const now = current.get(option); + if (now === undefined) { + return ( + `${option} was accepted by an earlier draft and this one declares no such property — ` + + "the command line no longer describes the document under review" + ); + } + if (now.boolean !== signature.boolean || now.array !== signature.array) { + return ( + `${option} was ${describeSignature(signature)} in an earlier draft and is ` + + `${describeSignature(now)} in this one — the command line no longer describes the ` + + "document under review" + ); + } + } + return undefined; +} + +/** + * The token a candidate's own arity turns into a second request. + * + * `--props-loud true` reads as an option and a value until a candidate declares + * `loud` a boolean; from then on `true` is a positional, and the command takes + * exactly one. + */ +export function strayPropertyValue( + occurrences: readonly PropertyOccurrence[], + bindings: readonly Binding[], +): string | undefined { + const byOption = new Map(bindings.map((binding) => [binding.option, binding])); + for (const occurrence of occurrences) { + const { provisional } = occurrence; + if (provisional === undefined) { + continue; + } + const binding = byOption.get(occurrence.option); + if (binding?.boolean === true) { + return ( + `unrecognized argument for xmd prompt: ${provisional} — ${occurrence.option} is a ` + + `switch, so this is a second request; write \`${occurrence.option}=${provisional}\` ` + + "to give it a value" + ); + } + } + return undefined; +} + +/** + * Whether this token is an option the invocation owns, and therefore one a + * generated property may not read as its value. + */ +export function isReservedOption(token: string): boolean { + return isKnownOption(token); +} diff --git a/packages/cli/src/prompt-profile.ts b/packages/cli/src/prompt-profile.ts new file mode 100644 index 00000000..822a9eed --- /dev/null +++ b/packages/cli/src/prompt-profile.ts @@ -0,0 +1,485 @@ +/** + * The prompt profile — the trusted-host assembly the prompt command document + * runs under, and the only thing that ever runs under it + * (specs/prompt-command-spec.md). + * + * `xmd prompt` executes this root on every invocation, and a second one — the + * Plan it returns — only under `--run`, behind a complete scope boundary. This + * module is the one that always happens. It supplies that document's inputs, a + * constrained Agent provider, Elicitation, the fixed first-party components and + * the host-declared draft checker, and it exposes no custom root and no + * repository component search: the document it runs is the one the CLI ships. + * + * The Agent ceiling is assembled here rather than read from the command line, + * because it is not the caller's to choose. Writing a Plan is a conversation + * about text; it never lets an agent touch anything. So the provider gets a + * host-owned directory dedicated to this logical session — created empty, and + * required to be empty before anything is built — no additional directories, no + * MCP servers, an empty native-tool allowlist and a private strict denial of + * every native + * permission request — one that answers inside the provider and consults no + * authored approval scope, so nothing composed around it can widen a ceiling + * with nothing in it. `--approve-all`, `--approve-reads` and `--deny-all` + * configure the approved document, later, and reach none of this. + * + * The document itself is given no Files, command, service or XMD-mediated + * network capability either. It decides what to write; it writes nothing. + */ + +import { ensure, Err, Ok, scoped, until } from "effection"; +import type { Operation, Result } from "effection"; +import { createHash } from "node:crypto"; +import { mkdir, readdir, rmdir } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +import { + agentIdentityComponents, + collect, + installAgentComponents, + installPermissionMode, + installPromptFailurePolicy, + registerAgentProvider, + retainedSource, +} from "@executablemd/core"; +import type { Json } from "@executablemd/core"; +import type { IdentityComponent } from "@executablemd/core/host"; +import { executeInstalled } from "@executablemd/core/host"; +import { createAcpxProvider } from "@executablemd/acp"; +import type { AcpxProviderDependencies } from "@executablemd/acp"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import { API } from "@executablemd/runtime"; + +import { hostAcpDependencies } from "./agent-stack.ts"; +import type { AgentStack } from "./agent-stack.ts"; +import { PROMPT_COMMAND_DOCUMENT, readPackagedDocument } from "./packaged-document.ts"; + +/** + * The identity the prompt command document runs under. + * + * Stable and internal: no path selects it, no include resolves it, and a + * position reading `(:12:1)` says the source is the CLI's own. + */ +export const PROMPT_COMMAND_IDENTITY = ""; + +/** + * The permission mode the prompt command document runs under. + * + * Fixed rather than configured. The provider denies native requests privately, + * so this is what an authored approval scope inside that document would + * compose around if one existed — and the honest answer for a profile that + * grants no native authority is the one that grants none. + */ +const PROFILE_PERMISSION_MODE = "deny-all"; + +/** The closed answer the host gives about one candidate. */ +export interface CandidateAssessment { + valid: boolean; + /** Empty when valid; the complete structured findings when not. */ + diagnostics: Json; +} + +/** What the host supplies to one prompt command document execution. */ +export interface PromptProfile { + /** The request as the person typed it. */ + request: string; + /** The rendered syntax catalog for this run profile and these includes. */ + syntax: string; + /** The logical name every turn in this invocation belongs to. */ + session: string; + /** + * Whether the caller named that session. + * + * A trusted host value rather than something read back out of the name: only + * the host knows whether `--session` was written, and the difference decides + * whether this conversation's directory outlives the invocation. It reaches + * the command document nowhere. + */ + explicitSession: boolean; + /** + * Where this host keeps its profile session directories. + * + * A host dependency, not a caller's: no flag, environment variable, document + * prop or replaceable context reaches it. Production leaves it at the default + * below; a harness that owns a temporary tree supplies that tree instead, so a + * test never reads, creates or removes anything under a real one. + */ + root: string; + /** The one Agent configuration this invocation settled. */ + stack: AgentStack; + /** What the constrained provider is built on, beyond the host's assembly. */ + acp?: AcpxProviderDependencies; + /** Who answers the review question. */ + installElicitation(): Operation; + /** + * The host's assessment of one candidate. + * + * A candidate-authored failure comes back as `valid: false` and is repairable. + * A caller-source failure raises, which ends that execution: the document + * has no way to catch it and no way to recategorize it as feedback + * for an agent that could not have caused it. + */ + assess(source: string): Operation; +} + +/** + * Run the packaged prompt command document and answer with the Plan it approved. + * + * Every resource this builds lives inside one scope, so leaving it is what tears + * the Prompt tasks, the provider and the Elicitation provider down. A teardown + * failure raises out of here rather than being folded into the result, because + * a failure to release is not an outcome the source that was selected survives. + */ +export function* runPromptCommandDocument(profile: PromptProfile): Operation> { + // Before a directory exists, before a provider exists, and therefore before + // any session could be placed or any turn started. A host that cannot + // establish this ceiling refuses rather than writing a Plan under a weaker one. + if (profile.stack.provider !== "acpx") { + return Err( + new Error( + `the ${profile.stack.provider} provider cannot establish the prompt profile's ` + + "ceiling — nothing was written or run", + ), + ); + } + + return yield* scoped(function* (): Operation> { + // First, and before anything is built: this session's directory is claimed, + // established and proven empty, or the command stops here. Nothing has been + // installed yet, so a refusal reaches no provider, no session and no turn — + // and because the claim is taken before the directory is made, an ending of + // any kind, including a cancellation, still hands it back. + const established = yield* useSessionDirectory(profile); + if (!established.ok) { + return established; + } + const workdir = established.value; + + yield* refuseDocumentCapabilities(); + yield* profile.installElicitation(); + + const acpx = createAcpxProvider(profileCeiling(profile, workdir)); + yield* registerAgentProvider("acpx", acpx); + const options = { + defaultAgent: profile.stack.defaultAgent, + permissionMode: PROFILE_PERMISSION_MODE, + } as const; + yield* installAgentComponents({ ...options, rootProvider: { factory: acpx, options } }); + yield* installPermissionMode(PROFILE_PERMISSION_MODE); + // A candidate comes from a turn's complete successful close value or from + // nowhere. `` ordinarily renders whatever a failed turn managed to + // emit and carries on, which for a policy that reviews source would mean + // showing a person half a program; the host decides otherwise here, so a + // failed, cancelled or protocol-invalid turn ends the command before anything + // is presented. The document cannot opt out of it. + yield* installPromptFailurePolicy(function* () { + return true; + }); + + const source = yield* readPackagedDocument(PROMPT_COMMAND_DOCUMENT); + try { + const approved = yield* collect( + yield* executeInstalled( + { + ...retainedSource(PROMPT_COMMAND_IDENTITY, source), + // Invocation-owned and thrown away with the scope. Ordinary document + // and Prompt semantics need a durable stream; nothing about writing a + // Plan needs a durable one, and `--journal` belongs to the Plan you + // approved rather than to the conversation that wrote it. + stream: new InMemoryStream(), + // No repository component search. What the document may name is + // what this profile declares, so a file in the caller's tree cannot + // answer for ``, `` or anything else. + includes: [], + props: { + request: profile.request, + syntax: profile.syntax, + session: profile.session, + }, + }, + [{ components: [...agentIdentityComponents(), validator(profile)] }], + ), + ); + if (typeof approved !== "string") { + return Err(new Error("the prompt command document returned something that is not a Plan")); + } + return Ok(approved); + } catch (error) { + return Err(error instanceof Error ? error : new Error(String(error))); + } + }); +} + +/** + * The host-declared draft checker, as an internal value component. + * + * Declared to the execution, so canonical execution supplies its invocation + * identity and repository resolution cannot replace it. It executes nothing it + * is given: a draft is a string here, and stays one until a person has approved + * it and the host has validated it again. + */ +function validator(profile: PromptProfile): IdentityComponent { + return { + name: "CheckDraft", + origin: "xmd prompt", + forms: ["self-closing"] as const, + props: { + type: "object", + properties: { source: { type: "string" } }, + required: ["source"], + additionalProperties: false, + }, + returns: { + type: "object", + properties: { valid: { type: "boolean" }, diagnostics: { type: "object" } }, + required: ["valid", "diagnostics"], + additionalProperties: false, + }, + factory: () => + function* checkDraft(props: Record) { + const assessment = yield* profile.assess(String(props.source)); + return { valid: assessment.valid, diagnostics: assessment.diagnostics }; + }, + }; +} + +/** + * The Agent ceiling, stated as the dependencies the provider is built from. + * + * Each entry is the whole of one clause: the directory the agent runs in — this + * session's own, proven empty above — the MCP servers it configures, the native + * tools a fresh session may use, + * and who answers a native permission request. `mcpServers: []` and + * `allowedTools: []` are statements rather than omissions — leaving either off + * is the backend's default, which is not this host's. + * + * The host's own assembly is passed through, then overridden: a coordinator or + * a route store says who owns a session, which this profile still has to respect, + * while nothing a caller wrote may reach the four fields below. + */ +function profileCeiling(profile: PromptProfile, workdir: string): AcpxProviderDependencies { + return { + ...hostAcpDependencies(profile.stack.sessions), + ...profile.acp, + // deno-lint-ignore require-yield + *agentCwd() { + return workdir; + }, + mcpServers: [], + permissions: "strict", + newSessionOptions: { systemPrompt: PROMPT_INSTRUCTIONS, allowedTools: [] }, + }; +} + +/** + * What the assistant session is told once, before it is asked anything. + * + * The host owns this layer, and it owns only this: that an answer belongs to the + * message that asked for it. Which shape any particular message wants — a Plan, + * or an explanation of why there is not one — is that message's own business, and + * every message is the prompt command document's text. Hiding a shape here would + * be hiding a policy decision in a place nobody reviewing the workflow can read. + */ +export const PROMPT_INSTRUCTIONS = [ + "You are the coding agent behind `xmd prompt`. A workflow asks you for one thing", + "at a time, on behalf of one person, and every message states what its answer has", + "to be.", + "", + "Answer the message you were sent, in the shape it asked for, and nothing else. A", + "message asking for a Plan is answered with Plan source; a message asking for an", + "explanation is answered with an explanation. Never answer one in the shape the", + "other asked for.", +].join("\n"); + +/** + * Where this host keeps its profile session directories by default. + * + * Under its own state directory rather than the caller's tree: an agent writing + * a document has no reason to read the checkout it will run in, and a ceiling + * that starts there is not a ceiling. + */ +export const DEFAULT_PROFILE_ROOT: string = join(homedir(), ".xmd", "prompt", "sessions"); + +/** + * The directory one logical session's conversation runs in. + * + * Dedicated to that session rather than shared by every invocation, so two + * conversations never see one ambient directory. The leaf is the digest of the + * name and never the name itself: a logical session name is a caller's string, + * and a caller's string that becomes a path is a caller's string that can escape + * one. + * + * A digest also gives the identity `--session` needs. The generated + * invocation-unique name digests to a location nothing else reaches, while the + * same explicit name digests to the same one — which is what lets ACPX find the + * session record it established last time, since a session's key includes the + * directory it lives in. + */ +export function profileDirectoryFor(root: string, session: string): string { + return join(root, createHash("sha256").update(session).digest("hex")); +} + +/** + * This conversation's directory, for as long as the conversation lasts. + * + * An explicitly named session's directory is durable — a later `--session` finds + * the same location and therefore the same ACPX session — so nothing is + * registered against it and nothing removes it. + * + * An invocation-unique one belongs to this scope, and the release is registered + * *before* the first filesystem call that could create it. A leaf this call + * made and then failed on is still a leaf this call made; registering after the + * `mkdir` would leave one behind exactly in the case nobody is watching. Being + * the first thing registered in the scope is also what puts it last in teardown, + * after every provider, Prompt task and Elicitation resource has gone. + */ +function* useSessionDirectory(profile: PromptProfile): Operation> { + const directory = profileDirectoryFor(profile.root, profile.session); + if (profile.explicitSession) { + return yield* establishDirectory(directory); + } + const claim: DirectoryClaim = { established: false }; + yield* ensure(() => releaseSessionDirectory(directory, claim)); + const established = yield* establishDirectory(directory); + claim.established = established.ok; + return established; +} + +/** Whether the directory was ever handed to this conversation to use. */ +interface DirectoryClaim { + established: boolean; +} + +/** + * Establish this session's directory, or refuse. + * + * Created empty, and required to be empty every time — not cleaned. Whatever is + * in there was put there by something this host did not authorize, and deleting + * a stranger's files to get on with the work is the opposite of what a ceiling + * is for. So the command says what it found and where, and stops. + */ +function* establishDirectory(directory: string): Operation> { + try { + yield* until(mkdir(directory, { recursive: true })); + const entries = yield* until(readdir(directory)); + if (entries.length > 0) { + return Err( + new Error( + `${directory} is not empty, and xmd prompt writes a Plan in a directory of its own ` + + "with nothing in it. Move or remove what is in there, or name a different " + + "--session; nothing was written or run", + ), + ); + } + return Ok(directory); + } catch (error) { + return Err( + new Error( + `could not establish ${directory}: ` + + (error instanceof Error ? error.message : String(error)), + ), + ); + } +} + +/** + * Give an invocation-unique conversation's directory back when it is over. + * + * One attempt, non-recursive, always. A directory this conversation was given + * empty and is handing back empty is removed; anything else is reported and + * nothing is deleted, because a leaf that changed underneath a conversation + * nobody authorized to write there is interference, not a tidying job. The + * failure raises out of the profile's scope, so no final admission follows it, + * and the approved Plan reaches no stdout, no file and no run. + * + * A directory the conversation never got — establishment refused it, or never + * made it — is a different question, and one already answered: whatever + * establishment reported is the honest account, and this leaves both it and the + * directory's contents alone. An empty leaf this call did create is still handed + * back, which is the whole reason the release is registered before the `mkdir`. + */ +function* releaseSessionDirectory(directory: string, claim: DirectoryClaim): Operation { + try { + yield* until(rmdir(directory)); + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : undefined; + if (!claim.established) { + return; + } + if (code === "ENOENT") { + throw new Error( + `${directory} was made for this conversation and is already gone. Something removed ` + + "it while the conversation was still running, which nothing here is allowed to do; " + + "nothing was output or run", + ); + } + if (code === "ENOTEMPTY" || code === "EEXIST") { + throw new Error( + `${directory} was empty when this conversation started and is not now. It belongs to ` + + "one invocation, so nothing should have written there; its contents were left alone " + + "and nothing was output or run", + ); + } + throw error instanceof Error ? error : new Error(String(error)); + } +} + +/** + * The capabilities the prompt command document does not get. + * + * Installed above whatever the entrypoint provided, so the document is + * refused rather than served. It decides what to write; writing a file, + * running a command, starting a service and reaching the network are all the + * approved document's business, under the caller's own configuration. + */ +function* refuseDocumentCapabilities(): Operation { + const refuse = (capability: string) => () => { + throw new Error( + `xmd prompt asked for ${capability}, which the prompt profile grants to nothing`, + ); + }; + yield* API.Files.around({ + // deno-lint-ignore require-yield + *checkFilePath() { + return refuse("a file")(); + }, + // deno-lint-ignore require-yield + *readTextFile() { + return refuse("a file")(); + }, + // deno-lint-ignore require-yield + *writeTextFile() { + return refuse("a file")(); + }, + // deno-lint-ignore require-yield + *deleteFile() { + return refuse("a file")(); + }, + // deno-lint-ignore require-yield + *globFiles() { + return refuse("a file")(); + }, + // deno-lint-ignore require-yield + *temporaryDirectory() { + return refuse("a directory")(); + }, + }); + yield* API.Process.around({ + // deno-lint-ignore require-yield + *exec() { + return refuse("a command")(); + }, + }); + yield* API.Fetch.around({ + // deno-lint-ignore require-yield + *fetch() { + return refuse("the network")(); + }, + }); + yield* API.Service.around({ + // deno-lint-ignore require-yield + *start() { + return refuse("a service")(); + }, + }); +} diff --git a/packages/cli/src/prompt.ts b/packages/cli/src/prompt.ts new file mode 100644 index 00000000..5ec57732 --- /dev/null +++ b/packages/cli/src/prompt.ts @@ -0,0 +1,463 @@ +/** + * `xmd prompt` — the trusted host around the prompt command document + * (specs/prompt-command-spec.md). + * + * Every invocation executes one root document — the packaged prompt command + * document — and a second one only when `--run` asks for it: + * + * ```text + * fixed command preflight + * -> build the run-profile syntax catalog + * -> execute the exact packaged prompt command document + * -> await that execution and provider teardown + * -> validate the returned source again + * -> deliver those exact bytes, in exactly one of four ways: + * (default) write the source to stdout + * --output exclusively create the file + * --run execute retainedSource("", source) + * through the ordinary run path + * --output --run create the file, then execute it + * ``` + * + * The complete scope boundary sits before that optional second execution: the + * command document and everything it built are gone before a Plan is admitted, + * so whichever result follows, it follows an invocation that has already let go + * of the conversation that wrote it. + * + * What a person is asked, how many drafts may be repaired, how many may be + * reviewed and what happens when nobody approves anything are not here. They are + * in `src/documents/prompt-command.md`, written in the open, where they can be + * read and argued with. This module is what a policy cannot be trusted to do for + * itself: settle the command line, build the ceiling the assistant runs under, + * answer honestly about a draft, and hold the boundary between text an agent + * wrote and a Plan this host will hand over or run. + * + * Two kinds of failure are told apart throughout, because they have different + * remedies. A *draft* failure is something the agent wrote, so the prompt + * command document is told the facts and may ask for another draft. A *caller* + * failure is something the command line or the environment said, so it raises + * out of the validator and ends that execution: no draft the agent could write + * would fix it, and a policy that could catch it could call it feedback. + */ + +import { Err, Ok, scoped, until } from "effection"; +import type { Operation, Result } from "effection"; +import { open } from "node:fs/promises"; +import type { FileHandle } from "node:fs/promises"; +import { randomUUID } from "node:crypto"; +import { resolve } from "node:path"; +import process from "node:process"; + +import { + agentIdentityComponents, + inspectDocument, + retainedSource, + validateDocument, +} from "@executablemd/core"; +import type { + DocumentValidation, + DocumentValidationCode, + Json, + PropsSchema, + RootDocumentSource, + SyntaxCatalog, +} from "@executablemd/core"; +import type { AcpxProviderDependencies } from "@executablemd/acp"; +import { cwd } from "@executablemd/runtime"; + +import type { AgentStack } from "./agent-stack.ts"; +import { DEFAULT_PROFILE_ROOT, runPromptCommandDocument } from "./prompt-profile.ts"; +import type { CandidateAssessment } from "./prompt-profile.ts"; +import type { MachineSessionAssembly } from "./session-coordinator.ts"; +import { + buildBindings, + describeError, + extractPropsArgs, + resolvePropsFromSources, +} from "./props.ts"; +import type { Binding, Extraction } from "./props.ts"; +import { reportFailure } from "./report.ts"; +import { renderSyntaxMarkdown, useRunProfileRegistry } from "./syntax.ts"; +import { + isReservedOption, + signatureFailure, + signatureOf, + strayPropertyValue, +} from "./prompt-args.ts"; +import type { OptionSignature, PromptScan } from "./prompt-args.ts"; + +/** + * The identity approved text runs under. + * + * Deliberate and fixed, the way `` is for an inline document: the bytes + * came from an agent, not from a file, and a source position reading + * `(:5:1)` says so. It affects positions and diagnostics only — + * components, includes and every relative filesystem operation still resolve + * from the contextual working directory. + */ +export const PROMPT_IDENTITY = ""; + +/** The approved bytes, and the props resolved under exactly those bytes. */ +export interface PromptExecution { + root: RootDocumentSource; + props: Record; +} + +/** What one `xmd prompt` invocation was asked to do. */ +export interface PromptCommand { + /** The argv this invocation holds, and the props source for every candidate. */ + argv: string[]; + /** What fixed grammar established about that argv. */ + scan: PromptScan; + include: string[]; + /** Where the approved Plan is written, when the caller asked for a file. */ + output?: string; + /** Whether the caller asked for the approved Plan to be run. */ + run: boolean; + /** The logical assistant-session name, when the caller chose one. */ + session?: string; + /** + * The Agent configuration this invocation settled, before the command ran. + * + * Settled by the caller rather than here, because the run that may follow + * approval is configured from the same answer: two resolutions of one command + * line is two chances to read `DEFAULT_AGENT_NAME` differently. + */ + stack: AgentStack; +} + +/** What the host supplies. Every entry is a decision only a host can make. */ +export interface PromptDependencies { + /** What this host states about machine-wide agent sessions, if anything. */ + sessions?: MachineSessionAssembly; + /** What the profile's provider is built on, beyond the host's own assembly. */ + acp?: AcpxProviderDependencies; + /** The run profile's complete structured vocabulary. */ + catalog(includes: readonly string[]): Operation; + /** Who answers the review question. */ + installElicitation(): Operation; + /** + * Where this host keeps its profile session directories. + * + * Absent is the ordinary host default. A harness that owns a temporary tree + * names that tree here, which is the only way anything but production selects + * one — there is no flag, no environment variable and no contextual Api to + * reach, so a document cannot move where the ceiling lives. + */ + profileRoot?: string; + /** Run the approved document the way this host runs any supplied one. */ + execute(approved: PromptExecution): Operation>; +} + +/** The prompt-owned findings that are not core's to report. */ +interface PromptDiagnostic { + code: "generated-binding-collision" | "root-props-unreadable"; + message: string; +} + +/** Everything definite that is wrong with one candidate. */ +interface CandidateDefects { + /** Core's complete versioned answer, whenever core produced one. */ + validation?: DocumentValidation; + /** What this command found about the options the candidate generates. */ + prompt?: PromptDiagnostic; +} + +type CandidateOutcome = + | { kind: "valid"; props: Record } + | { kind: "repairable"; defects: CandidateDefects } + | { kind: "terminal"; error: Error }; + +/** The codes that say the root's own declaration could not be read. */ +const DECLARATION_CODES: ReadonlySet = new Set([ + "source-unreadable", + "source-invalid", + "target-invalid", + "frontmatter-invalid", + "props-declaration-invalid", + "returns-declaration-invalid", +]); + +/** + * Run the command, and report the process status it earned. + * + * Every phase is behind a returned value rather than behind a flag another + * phase reads, so a refusal cannot be followed by the work it refused. + */ +export function* runPrompt(command: PromptCommand, deps: PromptDependencies): Operation { + const { scan } = command; + if (scan.error !== undefined || scan.request === undefined) { + console.error(scan.error ?? 'xmd prompt requires one request — `xmd prompt ""`'); + return 1; + } + const request = scan.request; + + let syntax: string; + try { + syntax = renderSyntaxMarkdown(yield* deps.catalog(command.include)); + } catch (error) { + console.error(describeError(error)); + return 1; + } + + // Every supplied individual option's shape, as the first draft that bound it + // declared it. Frozen while the Plan is being written and carried into the + // final gate, so the bytes that are delivered are checked against the command + // line that was written rather than against whichever draft happened to be + // last. + const frozen = new Map(); + + // The command document lives and dies inside that call's scope. Leaving it + // closes the Prompt tasks, the provider and the Elicitation provider, so a + // teardown failure raises out here — before the admission, the output file + // and the run that would otherwise already have happened. + let authored: Result; + try { + authored = yield* runPromptCommandDocument({ + request, + syntax, + session: command.session ?? invocationSessionName(), + // Read from what the caller wrote, not from the shape of the name. Only a + // session somebody can ask for again needs its directory to outlive the + // invocation, and only the host knows whether somebody named one. + explicitSession: command.session !== undefined, + root: deps.profileRoot ?? DEFAULT_PROFILE_ROOT, + stack: command.stack, + ...(deps.acp === undefined ? {} : { acp: deps.acp }), + installElicitation: deps.installElicitation, + assess: (source) => assess(command, frozen, source), + }); + } catch (error) { + console.error(describeError(error)); + return 1; + } + + if (!authored.ok) { + console.error(authored.error.message); + return 1; + } + + // The returned Plan is untrusted again. Whatever the command document concluded + // about a candidate, these are the bytes that would run, and they are checked + // as though nothing had ever validated them. + const admitted = yield* assessCandidate(command, frozen, authored.value); + if (admitted.kind === "terminal") { + console.error(admitted.error.message); + return 1; + } + if (admitted.kind === "repairable") { + console.error( + `the approved document does not validate:\n${JSON.stringify(admitted.defects, null, 2)}`, + ); + return 1; + } + + const source = authored.value; + if (command.output !== undefined) { + // Before the run, so a Plan that fails at run time is still on disk to read + // and hand-edit. An existing path is refused and nothing after it happens. + const written = yield* writeOutput(command.output, source); + if (!written.ok) { + console.error(written.error.message); + return 1; + } + } + + if (!command.run) { + // The approved Plan is the result. It goes to stdout exactly as the agent + // wrote it — no fence, no heading, no trailing newline of this command's — + // so a caller can pipe it into a file, a diff or another program. A caller + // who named `--output` already has it, and gets a quiet command instead. + if (command.output === undefined) { + process.stdout.write(source); + } + return 0; + } + + const executed = yield* deps.execute({ + root: retainedSource(PROMPT_IDENTITY, source), + props: admitted.props, + }); + if (!executed.ok) { + // Reported exactly as `xmd run` reports the same failure: what ran is one + // ordinary document, and how it failed is not this command's news to + // rephrase. A runtime failure ends the command here — there is nothing for + // the command document to reconsider about a Plan you already approved. + reportFailure(executed.error); + return 1; + } + return 0; +} + +/** + * A logical session name nothing else can name. + * + * The conversation belongs to this invocation: a second `xmd prompt` places a + * different session rather than continuing this one, and a request nobody meant + * to repeat never arrives in a history it did not create. `--session` replaces + * it when a caller wants the provider's ordinary continuation instead. + */ +function invocationSessionName(): string { + return `xmd-prompt:${randomUUID()}`; +} + +/** + * The host's answer about one draft, in the shape the command document reads. + * + * A caller-source failure raises rather than answering. That is the whole of + * the classification the command document can observe: it sees facts about drafts, + * and it never sees an argument the command line got wrong. + */ +function* assess( + command: PromptCommand, + frozen: Map, + source: string, +): Operation { + const outcome = yield* assessCandidate(command, frozen, source); + if (outcome.kind === "terminal") { + throw outcome.error; + } + if (outcome.kind === "valid") { + return { valid: true, diagnostics: {} }; + } + return { valid: false, diagnostics: outcome.defects as unknown as Json }; +} + +/** + * Everything decidable about one candidate, in the order that keeps a caller's + * mistake from being taught to the agent. + * + * The registry is installed here, around the two questions that need it, rather + * than around the command document: validation and the catalog have to agree + * about what `` is, and that document has no business reaching a + * vocabulary it only describes. + */ +function* assessCandidate( + command: PromptCommand, + frozen: Map, + candidate: string, +): Operation { + return yield* scoped(function* (): Operation { + yield* useRunProfileRegistry(); + const root = retainedSource(PROMPT_IDENTITY, candidate); + const includes = command.include; + const components = agentIdentityComponents(); + + // Whether the root declares itself readably. Inspection would raise on a + // malformed declaration, and recovering a code from an exception's prose is + // exactly what the structured answer exists to replace. + const declaration = yield* validateDocument({ ...root, includes, components }); + if (declaration.diagnostics.some((entry) => DECLARATION_CODES.has(entry.code))) { + return { kind: "repairable", defects: { validation: declaration } }; + } + + let propsSchema: PropsSchema; + try { + propsSchema = (yield* inspectDocument(root)).props; + } catch (error) { + return { + kind: "repairable", + defects: { prompt: { code: "root-props-unreadable", message: describeError(error) } }, + }; + } + + let bindings: Binding[]; + try { + bindings = buildBindings(propsSchema); + } catch (error) { + return { + kind: "repairable", + defects: { + prompt: { code: "generated-binding-collision", message: describeError(error) }, + }, + }; + } + + // Before a single token is extracted: an option that changed shape would + // otherwise reach forward and read the `--raw` written after it as its value. + const drift = signatureFailure(frozen, bindings); + if (drift !== undefined) { + return { kind: "terminal", error: new Error(drift) }; + } + const stray = strayPropertyValue(command.scan.occurrences, bindings); + if (stray !== undefined) { + return { kind: "terminal", error: new Error(stray) }; + } + + let extraction: Extraction; + let props: Record; + try { + extraction = extractPropsArgs(command.argv, bindings, { reserved: isReservedOption }); + props = yield* resolvePropsFromSources({ propsSchema, bindings, extraction }); + } catch (error) { + return { kind: "terminal", error: toError(error) }; + } + + for (const supplied of extraction.individual) { + frozen.set(supplied.binding.option, signatureOf(supplied.binding)); + } + + const validation = yield* validateDocument({ ...root, props, includes, components }); + if (validation.outcome === "invalid") { + return { kind: "repairable", defects: { validation } }; + } + return { kind: "valid", props }; + }); +} + +/** + * Create the destination and write the approved Plan, or refuse. + * + * Exclusive creation, so an existing path is left exactly as it is and the + * command stops rather than replacing work somebody kept. There is no + * check-then-write: the open is the check. + */ +function* writeOutput(path: string, source: string): Operation> { + const target = resolve(yield* cwd(), path); + let handle: FileHandle; + try { + handle = yield* until(open(target, "wx")); + } catch (error) { + const existing = + error instanceof Error && + (("code" in error && error.code === "EEXIST") || error.message.startsWith("EEXIST:")); + if (existing) { + return Err( + new Error( + `${target} already exists — choose another --output path; the approved Plan was ` + + "not written and nothing ran", + ), + ); + } + return Err(new Error(`could not create ${target}: ${describeError(error)}`)); + } + + const written = yield* writeAll(handle, source); + const closed = yield* closeHandle(handle, target); + if (!written.ok) { + return written; + } + return closed; +} + +function* writeAll(handle: FileHandle, source: string): Operation> { + try { + yield* until(handle.writeFile(source, "utf8")); + return Ok(undefined); + } catch (error) { + return Err(new Error(`could not write the approved Plan: ${describeError(error)}`)); + } +} + +function* closeHandle(handle: FileHandle, target: string): Operation> { + try { + yield* until(handle.close()); + return Ok(undefined); + } catch (error) { + return Err(new Error(`could not close ${target}: ${describeError(error)}`)); + } +} + +function toError(error: unknown): Error { + return error instanceof Error ? error : new Error(String(error)); +} diff --git a/packages/cli/src/props.ts b/packages/cli/src/props.ts index 8cfb456d..8afb5e32 100644 --- a/packages/cli/src/props.ts +++ b/packages/cli/src/props.ts @@ -10,6 +10,8 @@ import type { StandardSchemaV1 } from "@standard-schema/spec"; import { createContext, field, object } from "configliere"; import type { Parser } from "configliere"; import type { Json } from "@executablemd/durable-streams"; +import { env as readEnv } from "@executablemd/runtime"; +import type { Operation } from "effection"; import { z } from "zod"; export interface Binding { @@ -523,6 +525,21 @@ export interface Extraction { rest: string[]; } +export interface ExtractOptions { + /** + * Whether a token is an option the invocation itself owns, and therefore one + * a separated property value may never be. + * + * `xmd run` supplies none: the document is already named, so the schema that + * decides a property's arity cannot change under the same command line. + * `xmd prompt` re-binds the same argv against every candidate, so a value + * option that swallowed `--raw` in one draft would take the flag away from + * the invocation itself. A value that really begins with `-` is written + * `--props-name=-value`. + */ + reserved?: (token: string) => boolean; +} + /** * Remove `--props` and `--props-*` tokens from argv, keeping their * original text. Configliere's own option matching coerces every value @@ -530,7 +547,12 @@ export interface Extraction { * Only the generated bindings are recognized, so this stays a source * adapter rather than a second argument parser. */ -export function extractPropsArgs(args: string[], bindings: Binding[]): Extraction { +export function extractPropsArgs( + args: string[], + bindings: Binding[], + options?: ExtractOptions, +): Extraction { + const reserved = options?.reserved ?? (() => false); const byOption = new Map(bindings.map((binding) => [binding.option, binding])); const collected = new Map(); const individual: { binding: Binding; value: string | string[] }[] = []; @@ -608,6 +630,12 @@ export function extractPropsArgs(args: string[], bindings: Binding[]): Extractio if (next === undefined) { throw new PropsError(`${binding.option} requires a value`); } + if (reserved(next)) { + throw new PropsError( + `${binding.option} requires a value — \`${next}\` is an xmd option, so write ` + + `\`${binding.option}=${next}\` for a value that begins with "-"`, + ); + } record(binding, next); index += 2; } @@ -819,6 +847,39 @@ export function resolveProps(options: ResolveOptions): Record { return props; } +/** + * Resolve one document's properties from every source this host offers. + * + * The command line arrives already extracted; the environment is read here, one + * lookup per declared binding plus the aggregate. Both commands that supply root + * props go through this, so precedence and decoding exist once — a second copy + * would be a second set of rules the specification does not describe. + */ +export function* resolvePropsFromSources(options: { + propsSchema: unknown; + bindings: Binding[]; + extraction: Extraction; +}): Operation> { + const { propsSchema, bindings, extraction } = options; + const individualEnv: { binding: Binding; value: string }[] = []; + for (const binding of bindings) { + const value = yield* readEnv(binding.env); + if (value !== undefined) { + individualEnv.push({ binding, value }); + } + } + const aggregateEnv = yield* readEnv(AGGREGATE_ENV); + + return resolveProps({ + propsSchema, + bindings, + individual: extraction.individual, + ...(extraction.aggregate === undefined ? {} : { aggregateCli: extraction.aggregate }), + ...(aggregateEnv === undefined ? {} : { aggregateEnv }), + individualEnv, + }); +} + /** * Render the document-property section of `xmd run --help`. * Everything shown is declarative — accepted bindings, descriptions, diff --git a/packages/cli/src/report.ts b/packages/cli/src/report.ts new file mode 100644 index 00000000..73be4b0d --- /dev/null +++ b/packages/cli/src/report.ts @@ -0,0 +1,22 @@ +/** + * How a completed document's failure reaches the terminal. + * + * Shared because `xmd run` and `xmd prompt` end in the same thing — one + * ordinary document execution — and a person reading the output of either has + * no reason to see the same failure worded two ways. A test failure is the case + * that makes the difference visible: it earns a heading and a blank line above + * it, and printing only its message would report a failed suite as if it were + * an ordinary error. + */ + +import { TestFailureError } from "@executablemd/testing"; + +/** Print a completed document's failure the way `xmd` has always printed it. */ +export function reportFailure(error: Error, prefix?: string): void { + const label = prefix === undefined ? "" : `${prefix}: `; + if (error instanceof TestFailureError) { + console.error(`\n${label}tests failed: ${error.message}`); + return; + } + console.error(`${label}${error.message}`); +} diff --git a/packages/cli/src/syntax.ts b/packages/cli/src/syntax.ts index bd91e5f8..873c3a4b 100644 --- a/packages/cli/src/syntax.ts +++ b/packages/cli/src/syntax.ts @@ -52,15 +52,28 @@ import { WEB_REGISTRATIONS } from "@executablemd/web"; */ export function* syntaxCatalog(includes: readonly string[]): Operation { return yield* scoped(function* () { - yield* registerComponents([ - ...AGENT_REGISTRATIONS, - ...TESTING_REGISTRATIONS, - ...WEB_REGISTRATIONS, - ]); + yield* useRunProfileRegistry(); return yield* inspectSyntax({ includes, components: agentIdentityComponents() }); }); } +/** + * The registrations the `run` profile installs, as registry state and nothing + * else. + * + * Shared with `xmd prompt`, which both describes this vocabulary to a generator + * and validates what comes back. Registering only here would make the catalog + * advertise `` while validation reported it unresolved — a document told + * to use a component nobody would accept. + */ +export function* useRunProfileRegistry(): Operation { + yield* registerComponents([ + ...AGENT_REGISTRATIONS, + ...TESTING_REGISTRATIONS, + ...WEB_REGISTRATIONS, + ]); +} + /** * The catalog as JSON: two-space indent, one trailing newline. * diff --git a/packages/cli/tests/packaged-document.test.ts b/packages/cli/tests/packaged-document.test.ts new file mode 100644 index 00000000..8c8bb46b --- /dev/null +++ b/packages/cli/tests/packaged-document.test.ts @@ -0,0 +1,148 @@ +/** + * The documents the CLI ships and executes itself. + * + * `xmd prompt` runs a first-party Markdown program, so that program has to be + * present and identical wherever the command is. This suite runs under Deno, + * Node and Bun, which is what makes it evidence: the lookup is package-relative, + * and a resolution that only works under the runtime the author happened to use + * is the failure this catches. + * + * The compiled binary and the published npm package are proven where those + * artifacts exist — `scripts/tests/cli-npm-bin.test.ts` for the package, and the + * `dist/xmd` suites for the binary. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { readTextFile } from "@effectionx/fs"; +import { fileURLToPath } from "node:url"; + +import { + packagedDocumentUrl, + PROMPT_COMMAND_DOCUMENT, + readPackagedDocument, +} from "../src/packaged-document.ts"; +import { useWorkingDirectory } from "./support/prompt-harness.ts"; + +/** + * The approved introduction, exactly. + * + * Pinned rather than sampled: this is the passage the product contract settled, + * down to the typographic apostrophe in "coding agent’s plan". A paraphrase that + * still contained the chosen sentences would pass a looser check while saying + * something else, and an ASCII apostrophe here would quietly redefine the + * approved wording around whatever the file happens to hold. + */ +const INTRODUCTION = [ + "# `xmd prompt` turns steps into a program", + "", + "This document is a workflow that generates an executable Plan from a sequence of", + "steps. It combines the original Prompt, which describes those steps, with the XMD", + "components available to carry them out. A coding agent turns both into one", + "document that explains and executes the sequence.", + "", + "The result is the XMD version of a coding agent’s plan. A conventional Markdown", + "plan must be interpreted again before its steps can happen. An XMD Plan already", + "contains those executable steps, so running it simply executes them.", + "", + "A draft remains text while this workflow reviews it. Nothing in it runs before", + "you approve it. After approval, `xmd prompt` validates the exact source again. By", + "default it prints the approved XMD source. `--output` writes that source to a", + "file instead, and `--run` executes the Plan. With both options, the command", + "writes the source before running it.", +].join("\n"); + +/** What every turn that asks for a Plan has to say, on its own. */ +const PLAN_REQUIREMENTS = [ + "Every Plan is complete on its own:", + "", + "- optional frontmatter, and then one descriptive level-one Markdown heading as", + " the first body content;", + "- the Prompt's complete sequence, written as readable steps;", + "- every outcome the Prompt asked for;", + "- those steps in an order that makes sense; and", + "- each XMD component beside the prose describing the action it performs.", +].join("\n"); + +describe("packaged documents", () => { + it("reads the prompt command document from beside its module, whatever the cwd is", function* () { + // A temporary contextual cwd with no document in it. A lookup that reached + // for the working directory would find nothing here, and one that reached + // through the component search path would be answerable by a repository + // file — neither may decide which program this command runs. + const source = yield* useWorkingDirectory(function* () { + return yield* readPackagedDocument(PROMPT_COMMAND_DOCUMENT); + }); + + const committed = yield* readTextFile( + fileURLToPath(packagedDocumentUrl(PROMPT_COMMAND_DOCUMENT)), + ); + expect(source).toBe(committed); + // It is the prompt command document, not merely some file that exists. + expect(source).toContain("returns:"); + expect(source).toContain("` directly. The + // single-child `` wrapper existed only because the component refused a + // literal `as`, and main supplies that now. + expect(source).toContain(''); + // The closing branch is an unexpected-no-decision fallback, not a second + // copy of exhaustion: exhaustion is decided inside review, and saying it + // twice would make the two endings indistinguishable to a reader. + expect(source).toContain( + '', + ); + expect(source.split("reviewed ten drafts without an approved Plan").length - 1).toBe(2); + // The choices are the words a person reads, with no internal spelling + // behind them. + expect(source).toContain('["Approve", "Request changes", "Stop"]'); + expect(source).toContain('["Explain what went wrong", "Stop"]'); + expect(source).not.toContain('"revise"'); + expect(source).not.toContain('"abort"'); + expect(source).not.toContain(''); + }); + + it("says which build is missing a document rather than behaving differently", function* () { + // The failure a forgotten `deno compile --include` or npm asset copy + // produces. It names the path it looked at, because that is the difference + // between the four builds and the only thing worth reporting. + let failure: Error | undefined; + try { + yield* readPackagedDocument("no-such-document.md"); + } catch (error) { + failure = error instanceof Error ? error : new Error(String(error)); + } + + expect(failure?.message).toContain("no-such-document.md is missing from this build"); + expect(failure?.message).toContain("looked in"); + }); +}); diff --git a/packages/cli/tests/prompt-args.test.ts b/packages/cli/tests/prompt-args.test.ts new file mode 100644 index 00000000..97fa101c --- /dev/null +++ b/packages/cli/tests/prompt-args.test.ts @@ -0,0 +1,340 @@ +/** + * Tier PR — `xmd prompt` fixed grammar (specs/prompt-command-spec.md). + * + * Rows P1–P6, in the half that is decidable without a document. `scanPromptArgs` + * is a pure function over argv, so what the command line means — and every + * refusal it earns — is asserted directly rather than inferred from a process + * that printed nothing. + * + * The rest of P5 and P6 live in `prompt.test.ts`, where a candidate schema + * exists to bind against. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; + +import { + isReservedOption, + namesPrompt, + scanPromptArgs, + signatureFailure, + signatureOf, + strayPropertyValue, +} from "../src/prompt-args.ts"; +import type { OptionSignature } from "../src/prompt-args.ts"; +import { buildBindings, extractPropsArgs, PropsError } from "../src/props.ts"; +import type { Binding } from "../src/props.ts"; + +const REQUEST = "ask me for my age and write the result to a file"; + +function bindingsFor(properties: Record): Binding[] { + return buildBindings({ type: "object", properties, additionalProperties: false }); +} + +function frozen(entries: Record): Map { + return new Map(Object.entries(entries)); +} + +describe("Tier PR — xmd prompt fixed grammar", () => { + it("C1: exactly one request, kept byte for byte", function* () { + expect(namesPrompt(["prompt", REQUEST])).toBe(true); + expect(namesPrompt(["run", "doc.md"])).toBe(false); + + const one = scanPromptArgs(["prompt", REQUEST]); + expect(one.error).toBe(undefined); + expect(one.request).toBe(REQUEST); + + // Preserved, not trimmed: trimming is only how emptiness is tested. + const padded = scanPromptArgs(["prompt", ` ${REQUEST}\n`]); + expect(padded.error).toBe(undefined); + expect(padded.request).toBe(` ${REQUEST}\n`); + + expect(scanPromptArgs(["prompt"]).error).toContain("requires one request"); + expect(scanPromptArgs(["prompt", "", "--raw"]).error).toContain("non-whitespace"); + expect(scanPromptArgs(["prompt", " \t\n "]).error).toContain("non-whitespace"); + expect(scanPromptArgs(["prompt", REQUEST, "second"]).error).toContain( + "unrecognized argument for xmd prompt: second", + ); + + // A request that begins with a dash is written after the separator, and is + // still exactly one request. + const separated = scanPromptArgs(["prompt", "--", "--not-an-option"]); + expect(separated.error).toBe(undefined); + expect(separated.request).toBe("--not-an-option"); + // And it is kept out of the parser's argv, so nothing reads it as an option. + expect(separated.fixed).toEqual(["prompt"]); + }); + + it("C1: individual options follow the request, aggregate props may precede it", function* () { + const early = scanPromptArgs(["prompt", "--props-name", "Ada", REQUEST]); + expect(early.error).toContain("unrecognized option: --props-name"); + expect(early.error).toContain("follow the request"); + // Refused before anything is classified: no request was adopted from the + // tokens that followed, and no occurrence was recorded. + expect(early.request).toBe(undefined); + expect(early.occurrences).toEqual([]); + + const aggregate = scanPromptArgs([ + "prompt", + "--props", + '{"name":"Ada"}', + REQUEST, + "--raw", + "--run", + ]); + expect(aggregate.error).toBe(undefined); + expect(aggregate.request).toBe(REQUEST); + // The aggregate never reaches the parser: it coerces a separated value + // through Number() before any schema could judge it. + expect(aggregate.fixed).toEqual(["prompt", REQUEST, "--raw", "--run"]); + + const inline = scanPromptArgs(["prompt", '--props={"name":"Ada"}', REQUEST]); + expect(inline.error).toBe(undefined); + expect(inline.fixed).toEqual(["prompt", REQUEST]); + }); + + it("C1: built-in options after generated props stay with the invocation", function* () { + const scan = scanPromptArgs([ + "prompt", + REQUEST, + "--props-name", + "Ada", + "--raw", + "--include", + "lib", + "--props-loud", + "--journal", + "trace.jsonl", + "--output", + "out.md", + "--session", + "ada", + "--run", + ]); + expect(scan.error).toBe(undefined); + expect(scan.request).toBe(REQUEST); + expect(scan.fixed).toEqual([ + "prompt", + REQUEST, + "--raw", + "--include", + "lib", + "--journal", + "trace.jsonl", + "--output", + "out.md", + "--session", + "ada", + "--run", + ]); + // `--props-loud` did not swallow `--journal`: a known option is never read + // as a generated property's value. + expect(scan.occurrences).toEqual([ + { option: "--props-name", provisional: "Ada" }, + { option: "--props-loud" }, + ]); + + // Nor at extraction, once a candidate declares `loud` a value option. + const bindings = bindingsFor({ loud: { type: "string" } }); + let failure: unknown; + try { + extractPropsArgs(["prompt", REQUEST, "--props-loud", "--raw"], bindings, { + reserved: isReservedOption, + }); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(PropsError); + expect(String(failure)).toContain("--props-loud requires a value"); + expect(String(failure)).toContain("--props-loud=--raw"); + + // `xmd run` supplies no reserved list, so its behavior is unchanged. + const asRun = extractPropsArgs(["--props-loud", "--raw"], bindings); + expect(asRun.individual).toEqual([{ binding: bindings[0], value: "--raw" }]); + + expect(isReservedOption("--raw")).toBe(true); + expect(isReservedOption("--include=lib")).toBe(true); + expect(isReservedOption("--props-other")).toBe(true); + expect(isReservedOption("Ada")).toBe(false); + expect(isReservedOption("-5")).toBe(false); + }); + + it("C1: scalar, boolean and aggregate sources are all recorded", function* () { + const scan = scanPromptArgs([ + "prompt", + REQUEST, + "--props-name", + "Ada", + "--props-loud", + "--props-tag=alpha", + "--props-tag=beta", + "--props", + '{"count":2}', + ]); + expect(scan.error).toBe(undefined); + expect(scan.occurrences).toEqual([ + { option: "--props-name", provisional: "Ada" }, + { option: "--props-loud" }, + { option: "--props-tag", inline: "alpha" }, + { option: "--props-tag", inline: "beta" }, + ]); + + const bindings = bindingsFor({ + name: { type: "string" }, + loud: { type: "boolean" }, + tag: { type: "array", items: { type: "string" } }, + count: { type: "number" }, + }); + const extraction = extractPropsArgs( + [ + "prompt", + REQUEST, + "--props-name", + "Ada", + "--props-loud", + "--props-tag=alpha", + "--props-tag=beta", + "--props", + '{"count":2}', + ], + bindings, + { reserved: isReservedOption }, + ); + expect(extraction.aggregate).toBe('{"count":2}'); + expect(extraction.individual.map((entry) => [entry.binding.option, entry.value])).toEqual([ + ["--props-name", "Ada"], + ["--props-loud", "true"], + ["--props-tag", ["alpha", "beta"]], + ]); + }); + + it("C1: a boolean binding turns its provisional value into a second request", function* () { + const scan = scanPromptArgs(["prompt", REQUEST, "--props-loud", "true"]); + expect(scan.error).toBe(undefined); + expect(scan.occurrences).toEqual([{ option: "--props-loud", provisional: "true" }]); + + // A candidate that declares `loud` a value option accepts it. + expect(strayPropertyValue(scan.occurrences, bindingsFor({ loud: { type: "string" } }))).toBe( + undefined, + ); + + // One that declares it a switch does not: `true` is then a positional. + const stray = strayPropertyValue(scan.occurrences, bindingsFor({ loud: { type: "boolean" } })); + expect(stray).toContain("unrecognized argument for xmd prompt: true"); + expect(stray).toContain("--props-loud=true"); + }); + + it("C1: options that only configure a run need --run to mean anything", function* () { + // Each of them describes work that a command writing a Plan never does. A + // caller who asked for a journal, a permission mode or an exec deadline and + // got a command that creates none of them was not answered. + // Every spelling, including the short forms and both secret-detection + // switches: a table that covered only the long names would leave `-V` and + // `--secret-detection` accepted and ignored. + for (const flag of [ + ["--journal", "trace.jsonl"], + ["-j", "trace.jsonl"], + ["--raw"], + ["--verbose"], + ["-V"], + ["--timeout-exec", "5s"], + ["--timeout-fetch", "5s"], + ["--approve-all"], + ["--approve-reads"], + ["--deny-all"], + ["--secret-detection"], + ["--no-secret-detection"], + ]) { + const refused = scanPromptArgs(["prompt", REQUEST, ...flag]); + expect(refused.error).toContain(`${flag[0]} configures running the Plan`); + expect(refused.error).toContain("add --run"); + + // With `--run` the same command line is ordinary, wherever the two are + // written relative to each other. + expect(scanPromptArgs(["prompt", REQUEST, ...flag, "--run"]).error).toBe(undefined); + expect(scanPromptArgs(["prompt", REQUEST, "--run", ...flag]).error).toBe(undefined); + } + + // The options the command always uses are never refused: they build the + // catalog, settle the agent, name the session, bound the command and say + // where the Plan goes. + for (const flag of [ + ["--include", "lib"], + ["--agent-provider", "acpx"], + ["--default-agent", "codex"], + ["--session", "ada"], + ["--timeout", "5s"], + ["--output", "plan.md"], + ]) { + expect(scanPromptArgs(["prompt", REQUEST, ...flag]).error).toBe(undefined); + } + }); + + it("C1: --run is a switch, and every valued spelling of it is refused", function* () { + // An option name is read up to its first `=`, so `--run=false` arrives under + // the name of the switch. Taken as the switch it would establish the + // opposite of what was written, and satisfy the run-only gate on the way. + const REFUSAL = + "--run does not take a value — write --run to execute the Plan " + + "or leave it out to write the Plan"; + + for (const spelling of ["--run=false", "--run=true", "--run="]) { + const scan = scanPromptArgs(["prompt", REQUEST, spelling]); + expect(scan.error).toBe(REFUSAL); + // It established nothing: the token reached neither the parser's argv nor + // the record of what this invocation asked for. + expect(scan.fixed).toEqual(["prompt", REQUEST]); + + // And it does not answer for `--run` where a run is what makes an option + // meaningful. Without the fix this command line is accepted, and then + // nothing runs — so the journal the caller asked for is never created. + const gated = scanPromptArgs(["prompt", REQUEST, spelling, "--journal", "trace.jsonl"]); + expect(gated.error).toBe(REFUSAL); + expect(gated.fixed).toEqual(["prompt", REQUEST]); + } + + // The switch itself is unaffected, wherever it is written. + expect(scanPromptArgs(["prompt", REQUEST, "--run"]).error).toBe(undefined); + expect(scanPromptArgs(["prompt", REQUEST, "--run", "--journal", "t.jsonl"]).error).toBe( + undefined, + ); + }); + + it("C1: an option this command does not define is refused, not dropped", function* () { + // The parser stops at the first option it does not define and drops the + // rest, so silence here would mean accepting a command line nobody honoured. + const unknown = scanPromptArgs(["prompt", REQUEST, "--not-a-thing", "value"]); + expect(unknown.error).toBe("unrecognized option for xmd prompt: --not-a-thing"); + + // `--save` was replaced before release, so there is no alias — and the + // refusal says where the Plan goes now rather than only that the option is + // unknown. + const retired = scanPromptArgs(["prompt", REQUEST, "--save", "out.md"]); + expect(retired.error).toContain("unrecognized option for xmd prompt: --save"); + expect(retired.error).toContain("goes to stdout"); + expect(retired.error).toContain("--output writes it to a file"); + // It is not quietly read as the option that replaced it. + expect(retired.occurrences).toEqual([]); + }); + + it("C7: a frozen option's shape is what a later candidate may not change", function* () { + const scalar = bindingsFor({ name: { type: "string" } }); + const boolean = bindingsFor({ name: { type: "boolean" } }); + const array = bindingsFor({ name: { type: "array", items: { type: "string" } } }); + const absent = bindingsFor({ other: { type: "string" } }); + + expect(signatureOf(scalar[0])).toEqual({ boolean: false, array: false }); + const stable = frozen({ "--props-name": signatureOf(scalar[0]) }); + + // Unchanged: nothing is refused, so the same sources resolve again. + expect(signatureFailure(stable, scalar)).toBe(undefined); + + expect(signatureFailure(stable, absent)).toContain("declares no such property"); + expect(signatureFailure(stable, boolean)).toContain("single-value option"); + expect(signatureFailure(stable, boolean)).toContain("bare switch"); + expect(signatureFailure(stable, array)).toContain("repeated value option"); + + // An option nobody supplied is never frozen, so a candidate may add, + // remove or reshape it freely. + expect(signatureFailure(new Map(), absent)).toBe(undefined); + }); +}); diff --git a/packages/cli/tests/prompt-cli.test.ts b/packages/cli/tests/prompt-cli.test.ts new file mode 100644 index 00000000..2386b82b --- /dev/null +++ b/packages/cli/tests/prompt-cli.test.ts @@ -0,0 +1,855 @@ +/** + * Tier PR — the `xmd prompt` command lifecycle + * (specs/prompt-command-spec.md). + * + * The command's grammar, filesystem, journal, lifetime and execution rows: what + * `xmd prompt` does to the disk, to the process status and to the document it + * ends in. The approved document runs through the production executor, so what + * these prove about execution is what `xmd run` does with a supplied root. + * + * The grammar rows shell out, because exit status and help text are what an + * operator sees. Everything with a phase to observe runs in process, where a + * refusal is proven by the tripwires that stayed at zero. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { runCli } from "@executablemd/test-support/launch"; +import { readTextFile, writeTextFile } from "@effectionx/fs"; +import { stat } from "@executablemd/runtime"; +import { ensure, Ok, scoped, spawn, until } from "effection"; +import type { Operation, Result } from "effection"; +import { join } from "node:path"; +import { readdir } from "node:fs/promises"; +import process from "node:process"; + +import { promptExecutor } from "../src/cli.ts"; +import { resolveAgentStack } from "../src/agent-stack.ts"; +import type { AgentStack } from "../src/agent-stack.ts"; +import { runPrompt } from "../src/prompt.ts"; +import type { PromptCommand, PromptExecution } from "../src/prompt.ts"; +import { scanPromptArgs } from "../src/prompt-args.ts"; +import { + AGENT, + createPromptHarness, + timesRead, + useEnvironment, + useRecordedEnvironment, + useWorkingDirectory, +} from "./support/prompt-harness.ts"; +import type { PromptHarness } from "./support/prompt-harness.ts"; + +const REQUEST = "write a greeting"; + +/** A document that declares props and writes what it resolved. */ +const GREETER = [ + "---", + "props:", + " type: object", + " properties:", + " name: { type: string }", + " loud: { type: boolean, default: false }", + " count: { type: number }", + " required: [name]", + " additionalProperties: false", + "---", + "", + 'name={props.name} loud={props.loud} count={props.count}', + "", +].join("\n"); + +/** + * A document that validates and then fails at run time. + * + * Validation runs no command, so nothing before execution can know this one + * exits nonzero — which is the point: a runtime failure is not a candidate + * defect and never returns the command to generation or review. + */ +const FAILS_AT_RUN = ["```bash exec", "exit 3", "```", ""].join("\n"); + +const PLAIN = "Nothing but prose.\n"; + +/** + * A document that validates, runs, and fails its own tests. + * + * A `` boundary is what makes an assertion decide the outcome of an + * ordinary run, so this is the approved document that ends in the one failure + * `xmd run` reports differently from every other: with a heading of its own. + */ +const FAILING_TEST = [ + "", + '', + "", + "", + "", + "", +].join("\n"); + +/** The Agent configuration a dispatch settles once and hands to both consumers. */ +const STACK: AgentStack = { + provider: "acpx", + defaultAgent: AGENT, + permissionMode: "deny-all", +}; + +/** + * One invocation, asking for the approved Plan to be run. + * + * `--run` is the default here because these cases are about the run: what the + * journal holds, what the document writes, how a runtime failure reports. The + * modes that write the Plan instead are exercised by their own cases below, + * which pass `run: false` and say which destination they mean. + */ +function command(dir: string, args: string[], output?: string): PromptCommand { + const argv = ["prompt", ...args, "--run"]; + return { + argv, + scan: scanPromptArgs(argv), + include: [dir], + ...(output === undefined ? {} : { output }), + run: true, + stack: STACK, + }; +} + +/** The same invocation, writing the Plan rather than running it. */ +function writing(dir: string, args: string[], output?: string): PromptCommand { + const argv = ["prompt", ...args]; + return { + argv, + scan: scanPromptArgs(argv), + include: [dir], + ...(output === undefined ? {} : { output }), + run: false, + stack: STACK, + }; +} + +/** The production executor, configured the way the dispatch configures it. */ +function executor( + dir: string, + journal?: string, + stack?: AgentStack, +): (approved: PromptExecution) => Operation> { + return promptExecutor( + { + include: [dir], + verbose: false, + journal, + raw: true, + secretDetection: true, + }, + stack ?? STACK, + undefined, + function* () {}, + ); +} + +function* exists(path: string): Operation { + return (yield* stat(path)).exists; +} + +/** Every phase after the refusal, at zero. */ +function untouched(harness: PromptHarness): Record { + return { + catalogs: harness.catalogCalls.length, + runtimes: harness.fake.created.length, + started: harness.fake.started, + turns: harness.fake.prompts.length, + reviews: harness.reviews.length, + executions: harness.executions.length, + }; +} + +const NOTHING = { + catalogs: 0, + runtimes: 0, + started: false, + turns: 0, + reviews: 0, + executions: 0, +}; + +describe( + "Tier PR — the xmd prompt command", + { sanitizeOps: false, sanitizeResources: false }, + () => { + it("C1: a misplaced individual option refuses before any phase begins", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + const code = yield* runPrompt( + command(dir, ["--props-name", "Ada", REQUEST], "out.md"), + harness.deps, + ); + + expect(code).toBe(1); + expect(untouched(harness)).toEqual(NOTHING); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + }); + + // The agent configuration is settled before the command begins, so an + // incompatible pair of permission flags costs no inspection at all. Read + // at the boundary an operator uses, because that is where the resolution + // the whole invocation shares now happens. + yield* useWorkingDirectory(function* (dir) { + const { code, stderr } = yield* runCli( + ["prompt", REQUEST, "--approve-all", "--deny-all", "--run"], + { cwd: dir }, + ).join(); + + expect(code).toBe(1); + expect(stderr).toContain( + "--approve-all, --approve-reads, and --deny-all are mutually exclusive", + ); + // No provider was built: reaching one is what reports an agent as + // unavailable, and this command line never got that far. + expect(stderr).not.toContain("unavailable"); + }); + }); + + it("C1: --run=false reaches no authorship and no durable effect", function* () { + // `--run=false` used to be read as the switch, which satisfied the gate + // that only exists because a run is what makes `--journal` mean anything. + // The command line was then accepted, nothing ran, and the journal the + // caller asked for was never created — so the gate answered a request it + // had not honoured. Nonzero alone would not catch that: the invocation + // below would have failed anyway, on the agent it cannot reach. + // + // This case names the phases that stayed at zero, which a subprocess + // cannot see. The one after it drives the real parser and dispatch, which + // this one does not reach — the defect lived exactly between those two + // layers, so both are needed to pin it. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const journal = join(dir, "trace.jsonl"); + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, journal); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const argv = ["prompt", REQUEST, "--run=false", "--journal", journal]; + const written = console.error; + const lines: string[] = []; + const value = yield* scoped(function* (): Operation { + yield* ensure(() => { + console.error = written; + }); + console.error = (...parts: unknown[]) => { + lines.push(parts.map((part) => String(part)).join(" ")); + }; + return yield* runPrompt( + { argv, scan: scanPromptArgs(argv), include: [dir], run: false, stack: STACK }, + harness.deps, + ); + }); + + expect(value).toBe(1); + // The refusal is the one the fixed grammar owes this command line, not + // an incidental failure further along. + expect(lines).toEqual([ + "--run does not take a value — write --run to execute the Plan " + + "or leave it out to write the Plan", + ]); + // Every phase after preflight stayed at zero: no catalog, no provider, + // no turn, no review, no execution. + expect(untouched(harness)).toEqual(NOTHING); + // No prompt-profile session was opened either — neither established + // with the provider nor given a directory to run in. + expect(harness.fake.ensured).toEqual([]); + expect(yield* until(readdir(profileRoot))).toEqual([]); + // And nothing durable exists — neither the journal it named, nor an + // output file, nor anything else. + expect(yield* exists(journal)).toBe(false); + expect(yield* until(readdir(dir))).toEqual([]); + }); + }); + + it("C1: a valued --run is refused by the real parser, not just the scanner", function* () { + // The defect was a disagreement between the scanner and the parser: the + // scanner read `--run=false` as the switch while the parser read it as + // false. A case that hands `runPrompt` an already-scanned command skips + // the boundary the bug lived on, so this one goes through the command line + // an operator actually types. + const REFUSAL = + "--run does not take a value — write --run to execute the Plan " + + "or leave it out to write the Plan"; + + for (const spelling of ["--run=false", "--run=true", "--run="]) { + yield* useWorkingDirectory(function* (dir) { + const { code, stdout, stderr } = yield* runCli( + ["prompt", REQUEST, spelling, "--journal", "trace.jsonl"], + { cwd: dir }, + ).join(); + + expect(code).toBe(1); + expect(stderr).toContain(REFUSAL); + // No approved Plan escaped: stdout is where one would have gone. + expect(stdout).toBe(""); + // No provider was reached — reaching one is what reports an agent as + // unavailable, and this command line never got that far. + expect(stderr).not.toContain("unavailable"); + // And nothing was written: not the journal it named, not an output + // file, not anything else. + expect(yield* until(readdir(dir))).toEqual([]); + }); + } + }); + + it("C1: --save is gone, and is refused as the unknown option it is", function* () { + // Nothing was released under the old spelling, so there is no alias and + // nothing to keep compatible with. Proven on a command line that is + // otherwise entirely valid: a case that also carried a second, earlier + // failure would pass whether or not `--save` still worked. + yield* useWorkingDirectory(function* (dir) { + const { code, stdout, stderr } = yield* runCli(["prompt", REQUEST, "--save", "out.md"], { + cwd: dir, + }).join(); + + expect(code).toBe(1); + expect(stderr).toContain("--save"); + // Not accepted, not silently dropped, and not mistaken for `--output`. + expect(yield* exists(join(dir, "out.md"))).toBe(false); + // And nothing downstream of the refusal ran: no catalog was rendered, no + // provider was reached, nobody was asked anything, no approved source + // was printed, and no journal exists. + expect(stdout).toBe(""); + expect(stderr).not.toContain("## Built-in components"); + expect(stderr).not.toContain("unavailable"); + expect(yield* until(readdir(dir))).toEqual([]); + }); + }); + + it("C1: inline source is refused before any phase begins", function* () { + // `-e` belongs to `xmd run`. `xmd prompt` is the command that *writes* a + // document, so a second one supplied on the command line is a + // contradiction — and one the parser used to drop in silence, leaving the + // caller watching a different document get generated. + for (const flag of ["-e", "--eval"]) { + yield* useWorkingDirectory(function* (dir) { + const { code, stdout, stderr } = yield* runCli( + [ + "prompt", + REQUEST, + flag, + "# supplied", + "--output", + "out.md", + "--run", + "--journal", + "trace.jsonl", + ], + { cwd: dir }, + ).join(); + + expect(code).toBe(1); + expect(stderr).toContain( + "unrecognized option for xmd prompt: --eval — inline documents are exclusive to xmd run", + ); + // Every later phase, unreached: no catalog was rendered, no provider + // was built, no review was asked, and neither named file was made. + expect(stdout).not.toContain("## Built-in components"); + expect(stderr).not.toContain("unavailable"); + expect(stdout).toBe(""); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* exists(join(dir, "trace.jsonl"))).toBe(false); + }); + } + }); + + it("C1: help needs no request and touches nothing", function* () { + yield* useWorkingDirectory(function* (dir) { + const { code, stdout, stderr } = yield* runCli( + [ + "prompt", + "--help", + "--output", + "out.md", + "--journal", + "trace.jsonl", + "--default-agent", + "xmd-nonexistent-agent", + ], + { cwd: dir }, + ).join(); + + expect(code).toBe(0); + expect(stdout).toContain("Usage: xmd prompt [OPTIONS] [request]"); + expect(stdout).toContain("Exactly one Prompt is required"); + expect(stdout).toContain("--props "); + expect(stdout).toContain("XMD_PROPS"); + // Where an approved Plan goes, and what changes that. + expect(stdout).toContain("The approved Plan is the result. By default it is written to"); + expect(stdout).toContain("--output "); + expect(stdout).toContain("Write the approved Plan there instead of to stdout"); + expect(stdout).toContain("--run"); + expect(stdout).toContain("Run the approved Plan instead of writing it"); + expect(stdout).toContain("--session "); + // The run-only flags are named as such rather than left to be + // discovered by a refusal. + expect(stdout).toContain("are\nrefused without --run"); + // The permission flags are the approved Plan's, and help says so rather + // than letting a caller believe they configure how the Plan is written. + expect(stdout).toContain("Permission flags configure the approved Plan"); + expect(stderr).not.toContain("unavailable"); + // No catalog was rendered, and neither file the options named was made. + expect(stdout).not.toContain("## Built-in components"); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* exists(join(dir, "trace.jsonl"))).toBe(false); + }); + + const program = yield* runCli(["--help"]).expect(); + expect(program.stdout).toMatch(/^\s+prompt\s/m); + + // A session is named or it is not selected, and the refusal happens in + // preflight, beside the request's own — a parser that read the empty value + // as absent would have used the generated name instead. + yield* useWorkingDirectory(function* (dir) { + const empty = yield* runCli(["prompt", REQUEST, "--session", ""], { cwd: dir }).join(); + expect(empty.code).toBe(1); + expect(empty.stderr).toContain("--session needs a name"); + expect(empty.stdout).not.toContain("## Built-in components"); + + // A flag that only configures running a Plan, without --run: refused + // before authorship and before anything reaches the filesystem. + for (const flag of [["--journal", "trace.jsonl"], ["--raw"], ["--deny-all"]]) { + const stray = yield* runCli(["prompt", REQUEST, ...flag], { cwd: dir }).join(); + expect(stray.code).toBe(1); + expect(stray.stderr).toContain(`${flag[0]} configures running the Plan`); + expect(stray.stderr).toContain("add --run"); + expect(stray.stdout).toBe(""); + expect(yield* exists(join(dir, "trace.jsonl"))).toBe(false); + } + // That the same flag is ordinary again once `--run` is present is fixed + // grammar, and is proven there rather than by an invocation that would + // have to reach a real agent to say so. + }); + }); + + it("C15: individual, aggregate and environment sources resolve and reach the run", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + yield* useEnvironment({ XMD_PROPS: '{"name":"FromEnv","count":7}' }); + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-name", "Ada", "--props-loud"]), + harness.deps, + ); + + expect(code).toBe(0); + // Individual CLI beats the aggregate environment; the switch is true; the + // property only the aggregate supplied is still there. + expect(yield* readTextFile(join(dir, "greeting.txt"))).toBe("name=Ada loud=true count=7"); + }); + }); + + it("C12: every refusal is nonzero and leaves no output file, journal or run", function* () { + const journalName = "trace.jsonl"; + + // Stop at review, through the command document's authored failure. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, join(dir, journalName)); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Stop" }); + + const code = yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + + expect(code).toBe(1); + expect(harness.executions).toHaveLength(0); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* exists(join(dir, journalName))).toBe(false); + }); + + // A turn that produced text and then failed. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, join(dir, journalName)); + harness.fake.script({ reply: PLAIN, stopReason: "refusal" }); + + const code = yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + + expect(code).toBe(1); + expect(harness.reviews).toHaveLength(0); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* exists(join(dir, journalName))).toBe(false); + }); + + // A terminal property-source failure. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, join(dir, journalName)); + harness.fake.script({ reply: PLAIN }); + + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-absent", "x"], "out.md"), + harness.deps, + ); + + expect(code).toBe(1); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* exists(join(dir, journalName))).toBe(false); + }); + + // An approved run's journal holds the document's events and no authorship. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const journal = join(dir, journalName); + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, journal); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + + const trace = yield* readTextFile(journal); + expect(trace).toContain("__root__"); + expect(trace).toContain(PLAIN.trim()); + // Nothing about generation is in it: not the request, not a turn, not a + // review, not a repair. + expect(trace).not.toContain(REQUEST); + expect(trace).not.toContain("agent_prompt"); + expect(trace).not.toContain("elicit"); + }); + }); + + it("C11: the approved Plan is the result, and where it goes is the caller's", function* () { + // Default: the exact source on stdout, and nothing runs. Written with + // `process.stdout.write`, so what a pipe receives is the bytes and not a + // line the command added. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const written: string[] = []; + const original = process.stdout.write.bind(process.stdout); + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + process.stdout.write = original; + }); + process.stdout.write = ((chunk: string | Uint8Array) => { + written.push(typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk)); + return true; + }) as typeof process.stdout.write; + return yield* runPrompt(writing(dir, [REQUEST]), harness.deps); + }); + + expect(code).toBe(0); + // Byte for byte, once, with nothing around it. + expect(written.join("")).toBe(PLAIN); + // And nothing ran: no execution, so no journal and no document effects. + expect(harness.executions).toHaveLength(0); + expect(yield* until(readdir(dir))).toEqual([]); + }); + + // `--output`: the same bytes in the file, a quiet stdout, and still no run. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + const written: string[] = []; + const original = process.stdout.write.bind(process.stdout); + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + process.stdout.write = original; + }); + process.stdout.write = ((chunk: string | Uint8Array) => { + written.push(typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk)); + return true; + }) as typeof process.stdout.write; + return yield* runPrompt( + { + ...writing(dir, [REQUEST, "--props-name", "Ada"], "plan.md"), + argv: ["prompt", REQUEST, "--props-name", "Ada", "--output", "plan.md"], + }, + harness.deps, + ); + }); + + expect(code).toBe(0); + expect(written.join("")).toBe(""); + expect(yield* readTextFile(join(dir, "plan.md"))).toBe(GREETER); + // The Plan was not run, so what it would have written is not there. + expect(harness.executions).toHaveLength(0); + expect((yield* until(readdir(dir))).sort()).toEqual(["plan.md"]); + }); + + // `--run`: the Plan runs, and stdout is the Plan's own to use. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + const written: string[] = []; + const original = process.stdout.write.bind(process.stdout); + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + process.stdout.write = original; + }); + process.stdout.write = ((chunk: string | Uint8Array) => { + written.push(typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk)); + return true; + }) as typeof process.stdout.write; + return yield* runPrompt(command(dir, [REQUEST, "--props-name", "Ada"]), harness.deps); + }); + + expect(code).toBe(0); + // The source itself was never printed. + expect(written.join("")).not.toContain("props:"); + // The document ran, and did what it says. + expect(yield* readTextFile(join(dir, "greeting.txt"))).toBe("name=Ada loud=false count="); + }); + }); + + it("C11: the approved bytes are created exclusively, before the run", function* () { + // Created before execution, and byte for byte. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + const run = executor(dir); + const seen: boolean[] = []; + harness.deps.execute = function* (approved) { + seen.push(yield* exists(join(dir, "out.md"))); + return yield* run(approved); + }; + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-name", "Ada"], "out.md"), + harness.deps, + ); + + expect(code).toBe(0); + // The output file already existed when the document started. + expect(seen).toEqual([true]); + // Source only: no diagnostics, no decision, no wrapper. + expect(yield* readTextFile(join(dir, "out.md"))).toBe(GREETER); + }); + + // An existing path is left exactly as it is, and stops the run. + yield* useWorkingDirectory(function* (dir, profileRoot) { + yield* writeTextFile(join(dir, "out.md"), "keep me\n"); + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + + expect(code).toBe(1); + expect(yield* readTextFile(join(dir, "out.md"))).toBe("keep me\n"); + expect(harness.executions).toHaveLength(0); + }); + + // Without the option, no generated source file is created at all. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST, "--props-name", "Ada"]), harness.deps); + + expect(code).toBe(0); + // Only what the document itself wrote. + expect((yield* until(readdir(dir))).sort()).toEqual(["greeting.txt"]); + }); + }); + + it("C15: the approved source runs as an ordinary document under ", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const journal = join(dir, "trace.jsonl"); + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, journal); + harness.fake.script({ reply: GREETER }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-name", "Ada"], "out.md"), + harness.deps, + ); + expect(code).toBe(0); + + // The identity the run reports is the deliberate one. + expect(yield* readTextFile(journal)).toContain(""); + // Relative filesystem operations resolved the contextual cwd, not the + // identity, so the document's own write landed beside the output file. + expect(yield* exists(join(dir, "greeting.txt"))).toBe(true); + expect(yield* exists(join(dir, "out.md"))).toBe(true); + }); + + // A runtime failure is an ordinary run failure: nonzero, with the file the + // caller asked for still on disk to hand-edit. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: FAILS_AT_RUN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + + expect(code).toBe(1); + expect(yield* readTextFile(join(dir, "out.md"))).toBe(FAILS_AT_RUN); + // The failure did not send the run back to generation or review. + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.reviews).toHaveLength(1); + }); + }); + + it("C15: a failing test in the approved document reports as a run reports it", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: FAILING_TEST }); + harness.script({ decision: "Approve" }); + + const written = console.error; + const lines: string[] = []; + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + console.error = written; + }); + console.error = (...parts: unknown[]) => { + lines.push(parts.map((part) => String(part)).join(" ")); + }; + return yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + }); + + expect(code).toBe(1); + // Byte for byte what `xmd run` prints for this failure: the heading and + // the blank line above it are how a failed suite is told apart from an + // ordinary error, and printing only the message would lose both. + expect(lines.at(-1)).toBe("\ntests failed: 1 test(s) failed in "); + + // A runtime failure is not a candidate defect: the agent was asked once + // and the person was asked once, and neither was asked again. + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.reviews).toHaveLength(1); + // The file was already written, and a failing run leaves it to hand-edit. + expect(yield* readTextFile(join(dir, "out.md"))).toBe(FAILING_TEST); + }); + }); + + it("C15: one Agent resolution serves generation and the execution after it", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const reads: string[] = []; + yield* useRecordedEnvironment(reads, { DEFAULT_AGENT_NAME: "settled-agent" }); + + // What a dispatch settles, once, for the whole invocation. + const settled = yield* resolveAgentStack( + { + agentProvider: "acpx", + defaultAgent: undefined, + approveAll: false, + approveReads: false, + denyAll: true, + }, + undefined, + ); + if (!settled.ok) { + throw settled.error; + } + const stack = settled.value; + expect(stack.defaultAgent).toBe("settled-agent"); + expect(timesRead(reads, "DEFAULT_AGENT_NAME")).toBe(1); + + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir, undefined, stack); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt({ ...command(dir, [REQUEST]), stack }, harness.deps); + + expect(code).toBe(0); + // The command document resolved the settled agent rather than a name of + // its own. + expect(harness.fake.ensured.map((input) => input.agent)).toEqual(["settled-agent"]); + // And nothing after it read the name again: authorship and the document + // installation that followed were both configured from the one answer, + // so they cannot disagree about which agent this invocation meant. + expect(timesRead(reads, "DEFAULT_AGENT_NAME")).toBe(1); + }); + }); + + it("C13: the deadline encloses every phase, and teardown gates what follows", function* () { + // Expiry is cancellation, so the proof is what cancelling the command does: + // the turn in flight is cancelled, the provider is dismantled, and no later + // phase begins. The barrier is what makes this a gate rather than a race — + // the turn it interrupts is known to be running. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.script({ reply: PLAIN, manual: true }); + + yield* scoped(function* () { + const running = yield* spawn(() => + runPrompt(command(dir, [REQUEST], "out.md"), harness.deps), + ); + yield* harness.fake.startedTurns(1); + yield* running.halt(); + }); + + expect(harness.fake.cancels).toBeGreaterThanOrEqual(1); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + }); + + // A teardown failure prevents the final admission, the output file and the run, + // whatever the command document selected. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.deps.execute = executor(dir); + harness.fake.closeFailure = new Error("the profile provider would not close"); + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST], "out.md"), harness.deps); + + expect(code).toBe(1); + expect(harness.executions).toHaveLength(0); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + }); + + // Nothing bounds an authoring turn: the exec and fetch defaults belong to + // the document, and the run deadline is the enclosing timebox above. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + // deno-lint-ignore require-yield + harness.deps.execute = function* () { + return Ok(undefined); + }; + harness.fake.script({ reply: PLAIN }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + expect(harness.fake.turns[0]?.timeoutMs).toBe(undefined); + }); + }); + + it("C13: the deadline is the whole prompt command's, as it is a run's", function* () { + // Read for prompt exactly as for run: a value that is not a duration fails + // the invocation before it prepares anything. + for (const flag of ["--timeout", "--timeout-exec", "--timeout-fetch"]) { + const { code, stderr } = yield* runCli(["prompt", REQUEST, `${flag}=abc`]).join(); + expect(code).toBe(1); + expect(stderr).toContain(flag); + } + + // And it bounds the command rather than only its final document: the + // deadline expires while the catalog is still being built, long before any + // document exists to bound. + const expired = yield* runCli(["prompt", REQUEST, "--timeout=1ms"]).join(); + expect(expired.code).toBe(1); + expect(expired.stderr).toContain("exceeded its --timeout of 1ms and was cancelled"); + }); + }, +); diff --git a/packages/cli/tests/prompt-command-document.test.ts b/packages/cli/tests/prompt-command-document.test.ts new file mode 100644 index 00000000..452991e3 --- /dev/null +++ b/packages/cli/tests/prompt-command-document.test.ts @@ -0,0 +1,186 @@ +/** + * The packaged prompt command document, executed as itself. + * + * This runs the exact Markdown the CLI ships — read through the packaged + * loader, not copied into a fixture — so what it proves is what a release does. + * The seams around it are deterministic: a scriptable ACP runtime for the one + * Agent turn, a scripted Elicitation answer for the review, and a test-only + * validator in the place the prompt profile declares the production one. + * + * The include list is empty on purpose. Repository component search must not be + * able to supply `Loop`, `If`, `Return`, `Fail`, `CodeBlock` or the validator: + * the policy under test is the one in the packaged document, resolved against + * first-party declarations only. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped } from "effection"; +import type { Operation } from "effection"; +import { + agentIdentityComponents, + collect, + Elicitation, + installAgentComponents, + registerAgentProvider, + retainedSource, +} from "@executablemd/core"; +import type { ElicitationRequest, Json } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import { createAcpxProvider } from "@executablemd/acp"; + +import { PROMPT_COMMAND_DOCUMENT, readPackagedDocument } from "../src/packaged-document.ts"; +import { AGENT, useWorkingDirectory } from "./support/prompt-harness.ts"; +import { createFakeAcp, makeRegistry, makeStore } from "./support/fake-acp.ts"; + +/** + * A candidate whose exact bytes are worth preserving. + * + * Leading and trailing blank lines, interior indentation, and a five-backtick + * run: enough that trimming, re-fencing, or reconstructing it through the + * presentation would all be visible in the assertion. + */ +const CANDIDATE = [ + "", + "---", + "returns: { type: string }", + "---", + "", + "# A greeting", + "", + "This document explains itself before it does anything.", + "", + "`````markdown", + " ", + "`````", + "", + '', + "", +].join("\n"); + +/** What the command document asked the validator about, in order. */ +interface CommandRun { + validated: string[]; + reviews: ElicitationRequest[]; + prompts: string[]; + value: Json | undefined; + failure: string | undefined; +} + +function* runDocument(): Operation { + const source = yield* readPackagedDocument(PROMPT_COMMAND_DOCUMENT); + const fake = createFakeAcp(); + fake.script({ reply: CANDIDATE }); + + const validated: string[] = []; + const reviews: ElicitationRequest[] = []; + let value: Json | undefined; + let failure: string | undefined; + + yield* scoped(function* () { + const acpx = createAcpxProvider({ + createRuntime: fake.create, + sessionStore: makeStore(), + agentRegistry: makeRegistry({ [AGENT]: `${AGENT}-cmd` }), + }); + yield* registerAgentProvider("acpx", acpx); + yield* installAgentComponents({ + defaultAgent: AGENT, + permissionMode: "deny-all", + rootProvider: { factory: acpx, options: { defaultAgent: AGENT, permissionMode: "deny-all" } }, + }); + + yield* Elicitation.around( + { + // deno-lint-ignore require-yield + *elicit([request], _next) { + reviews.push(request); + return { decision: "Approve" }; + }, + }, + { at: "min" }, + ); + + try { + value = yield* collect( + yield* executeInstalled( + { + ...retainedSource("", source), + stream: new InMemoryStream(), + includes: [], + props: { + request: "ask me for my age and write the result to a file", + syntax: "## Built-in components\n\n### ``\n", + session: "prompt-command-regression", + }, + }, + [ + { + components: [ + ...agentIdentityComponents(), + { + name: "CheckDraft", + origin: "test", + forms: ["self-closing"] as const, + props: { + type: "object", + properties: { source: { type: "string" } }, + required: ["source"], + additionalProperties: false, + }, + returns: { + type: "object", + properties: { + valid: { type: "boolean" }, + diagnostics: { type: "object" }, + }, + required: ["valid", "diagnostics"], + additionalProperties: false, + }, + // The deterministic seam only, standing where the command + // declares its own validator. It records what it was asked + // about and says yes, so what this case observes is the + // program's control flow rather than validation's answers. + factory: () => + // deno-lint-ignore require-yield + function* checkDraft(props: Record) { + validated.push(String(props.source)); + return { valid: true, diagnostics: {} }; + }, + }, + ], + }, + ], + ), + ); + } catch (error) { + failure = error instanceof Error ? error.message : String(error); + } + }); + + return { validated, reviews, prompts: fake.prompts, value, failure }; +} + +describe("the packaged prompt command document", () => { + it("C2: returns the approved candidate's exact bytes and never reaches exhaustion", function* () { + const run = yield* useWorkingDirectory(function* () { + return yield* runDocument(); + }); + + // Approving the first valid candidate is one turn and one question. A + // repair or revision turn here would mean the loops ran when they had + // nothing to fix. + expect(run.prompts).toHaveLength(1); + expect(run.reviews).toHaveLength(1); + + // The validator saw the Agent's complete close value, once, unaltered. + expect(run.validated).toEqual([CANDIDATE]); + + // The document settled with a value rather than an authored failure. Before + // the control-flow correction this was the ten-draft exhaustion message: a + // `` selects a value but does not end the body, so the unconditional + // `` after the Session ran and won over every approval. + expect(run.failure).toBe(undefined); + expect(run.value).toBe(CANDIDATE); + }); +}); diff --git a/packages/cli/tests/prompt.test.ts b/packages/cli/tests/prompt.test.ts new file mode 100644 index 00000000..fb6c7d2d --- /dev/null +++ b/packages/cli/tests/prompt.test.ts @@ -0,0 +1,1352 @@ +/** + * Tier PR — `xmd prompt` writing a Plan, as the host and the packaged prompt + * command document perform it together (specs/prompt-command-spec.md). + * + * The policy under test is `src/documents/prompt-command.md`, reached the way + * the command reaches it: `runPrompt` builds the prompt profile, the packaged + * document runs inside it, and every observation here is of that document's + * behaviour rather than of a TypeScript loop standing in for it. + * + * The seams are deterministic. The ACPX runtime is the scriptable fake, the + * review provider is a scripted `Elicitation` handler, the executor records what + * it was handed, and the contextual working directory is a temporary one. + * Nothing here starts an agent, opens a browser or reaches a network. + * + * Every refusal is proven by the phases that stayed at zero — turns not sent, + * reviews not asked, executions not handed anything — rather than by output + * nobody produced. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, scoped, spawn, until } from "effection"; +import type { Operation } from "effection"; +import { ensureDir, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { readdir } from "node:fs/promises"; +import { join, sep } from "node:path"; +import { API, stat } from "@executablemd/runtime"; +import { Elicitation } from "@executablemd/core"; +import type { ElicitationRequest } from "@executablemd/core"; + +import { runPrompt } from "../src/prompt.ts"; +import type { PromptCommand } from "../src/prompt.ts"; +import { + DEFAULT_PROFILE_ROOT, + profileDirectoryFor, + PROMPT_INSTRUCTIONS, +} from "../src/prompt-profile.ts"; +import { scanPromptArgs } from "../src/prompt-args.ts"; +import type { AgentStack } from "../src/agent-stack.ts"; +import { + AGENT, + createPromptHarness, + useProfileRoot, + useWorkingDirectory, +} from "./support/prompt-harness.ts"; +import { makeStore } from "./support/fake-acp.ts"; +import type { PromptHarness } from "./support/prompt-harness.ts"; + +const REQUEST = "write a greeting"; + +/** A document that validates and runs. */ +const VALID = "Hello from the agent.\n"; + +/** A document that resolves no such component. */ +const UNRESOLVED = "\n"; + +/** A root whose own source cannot be read: the frontmatter never closes. */ +const BROKEN_SOURCE = ["---", "props: [", "---", "", "hi", ""].join("\n"); + +/** A root whose two declared properties generate one option. */ +const COLLIDING = [ + "---", + "props:", + " type: object", + " properties:", + " firstName: { type: string }", + " first_name: { type: string }", + "---", + "", + "hi", + "", +].join("\n"); + +/** A root that declares one required scalar property. */ +const REQUIRES_NAME = [ + "---", + "props:", + " type: object", + " properties:", + " name: { type: string }", + " required: [name]", + " additionalProperties: false", + "---", + "", + "Hello, {props.name}!", + "", +].join("\n"); + +/** The same document with `name` declared a switch instead. */ +const NAME_IS_BOOLEAN = [ + "---", + "props:", + " type: object", + " properties:", + " name: { type: boolean }", + " additionalProperties: false", + "---", + "", + "Hello, {props.name}!", + "", +].join("\n"); + +/** One document under two schemas that bind `--props-count` the same way. */ +function counting(type: "number" | "string"): string { + return [ + "---", + "props:", + " type: object", + " properties:", + ` count: { type: ${type} }`, + " additionalProperties: false", + "---", + "", + "Counting to {props.count}.", + "", + ].join("\n"); +} + +/** A Plan whose effect is visible on the filesystem if anything runs it. */ +const WRITES_A_FILE = ['the draft ran', ""].join("\n"); + +/** The Agent configuration a dispatch settles once and hands to both consumers. */ +const STACK: AgentStack = { + provider: "acpx", + defaultAgent: AGENT, + permissionMode: "deny-all", +}; + +/** + * One invocation, asking for the approved Plan to be run. + * + * `--run` is the default here because these cases are about what reaches the + * execution: the modes that write the Plan instead have their own cases, and + * name the mode they mean. + */ +function command(dir: string, args: string[], stack: AgentStack = STACK): PromptCommand { + const argv = ["prompt", ...args, "--run"]; + return { argv, scan: scanPromptArgs(argv), include: [dir], run: true, stack }; +} + +/** + * What every turn that asks for a Plan has to say, on its own. + * + * Pinned as one block rather than as sentences found anywhere in the document: a + * repair or a revision that quietly dropped a clause would still satisfy a + * search for the words somewhere, and a replacement Plan is only complete if the + * message asking for it said so. + */ +const PLAN_REQUIREMENTS = [ + "Every Plan is complete on its own:", + "", + "- optional frontmatter, and then one descriptive level-one Markdown heading as", + " the first body content;", + "- the Prompt's complete sequence, written as readable steps;", + "- every outcome the Prompt asked for;", + "- those steps in an order that makes sense; and", + "- each XMD component beside the prose describing the action it performs.", +].join("\n"); + +/** Every session key the fake was asked to establish, deduplicated in order. */ +function sessions(harness: PromptHarness): string[] { + return [...new Set(harness.fake.ensured.map((input) => input.sessionKey))]; +} + +/** The decisions one review request offered. */ +function decisions(request: ElicitationRequest): unknown { + const properties = request.schema.properties; + if (typeof properties !== "object" || properties === null) { + return undefined; + } + const decision = (properties as Record).decision; + if (typeof decision !== "object" || decision === null) { + return undefined; + } + return (decision as Record).enum; +} + +/** + * A review provider that looks at the profile directory before it answers. + * + * Installed in place of the harness's own, so the observation happens inside the + * profile's scope — the only moment the directory exists. + */ +function watching( + harness: PromptHarness, + observe: (workdir: string) => Operation, +): () => Operation { + return function* () { + yield* Elicitation.around( + { + *elicit([request], _next) { + harness.reviews.push(request); + yield* observe(String(harness.fake.created[0]?.cwd)); + return { decision: "Approve" }; + }, + }, + { at: "min" }, + ); + }; +} + +/** Whether a path is there at all. */ +function* exists(path: string): Operation { + return (yield* stat(path)).exists; +} + +/** What one message's `json` fence holds, parsed back. */ +interface FencedDiagnostics { + validation?: { + version?: number; + outcome?: string; + diagnostics?: { code?: string }[]; + }; +} + +/** + * The complete JSON a message carried, read out of its fence. + * + * Parsed rather than string-matched: what matters is that ``'s serialized + * text reached `` whole, and a structure that parses back to the same + * shape is what says so. + */ +function fencedJson(message: string): FencedDiagnostics { + const fence = /```json\n([\s\S]*?)\n```/.exec(message); + if (fence === null) { + throw new Error(`no json fence in: ${message}`); + } + return JSON.parse(fence[1]); +} + +/** How many times one exact string appears. */ +function occurrences(haystack: string, needle: string): number { + return haystack.split(needle).length - 1; +} + +/** What the command wrote to stderr while it ran. */ +function* reported(body: () => Operation): Operation<{ value: T; lines: string[] }> { + const written = console.error; + const lines: string[] = []; + const value = yield* scoped(function* (): Operation { + yield* ensure(() => { + console.error = written; + }); + console.error = (...parts: unknown[]) => { + lines.push(parts.map((part) => String(part)).join(" ")); + }; + return yield* body(); + }); + return { value, lines }; +} + +describe( + "Tier PR — xmd prompt writing a Plan", + { sanitizeOps: false, sanitizeResources: false }, + () => { + it("C2, C3, C14: the packaged program's own words ask for the document", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + // A repository TypeScript component, so the catalog has to state the one + // thing it honestly cannot know without importing the module. + yield* writeTextFile(join(dir, "Widget.ts"), "export default function Widget() {}\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + + // Exactly one catalog, built with the invocation's own includes. + expect(harness.catalogCalls).toEqual([[dir]]); + + // The turn is the shipped Markdown's, word for word: the sentences below + // exist nowhere in TypeScript, so a host that chose its own policy could + // not produce them. + const turn = harness.fake.prompts[0]; + expect(turn).toContain("Create one complete XMD Plan from this Prompt"); + // C14: the complete Plan requirement, stated in this turn rather than + // referred to from somewhere else in the document. + expect(turn).toContain(PLAN_REQUIREMENTS); + // The worked example travels with it, unparsed, and is a titled Plan — + // the shape being asked for is the shape being shown. + expect(turn).toContain('{answer.age}'); + expect(turn).toContain("# Ask for and save your age"); + expect(turn).toContain("keep every outcome it asked for"); + // The request travels inside it, byte for byte. + expect(turn).toContain(REQUEST); + // So does the host's catalog, including the renderer's honest statement + // about a component nobody imported. + expect(turn).toContain("## Built-in components"); + expect(turn).toContain("### ``"); + expect(turn).toContain("### ``"); + expect(turn).toContain("This component is a repository TypeScript module."); + + // The host owns the instruction layer, and owns only the shape of an + // answer: the catalog and the request are the program's to send. + expect(harness.fake.ensured[0]?.sessionOptions?.systemPrompt).toBe(PROMPT_INSTRUCTIONS); + expect(PROMPT_INSTRUCTIONS).not.toContain("Built-in components"); + + // C3: one is one turn. Nothing repaired, nothing retried. + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.fake.turns).toHaveLength(1); + expect(harness.reviews).toHaveLength(1); + }); + }); + + it("C3, C14: every turn that asks for a Plan states the whole requirement", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + // A draft that fails its check, then one that passes, then a revision. + // Three turns, one of each kind that produces a Plan. + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Request changes", feedback: "say it differently" }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + + const [initial, repair, revision] = harness.fake.prompts; + expect(harness.fake.prompts).toHaveLength(3); + expect(initial).toContain("Create one complete XMD Plan from this Prompt"); + expect(repair).toContain("Send one complete replacement Plan that resolves every problem"); + expect(revision).toContain("say it differently"); + + // Each of them carries the requirement itself. A turn that only said + // "keep what the last draft had" would leave a missing or wrong title + // missing or wrong, and none of these do. + for (const turn of [initial, repair, revision]) { + expect(turn).toContain(PLAN_REQUIREMENTS); + } + + // And a replacement is told to write the title the Plan needs rather + // than to carry the previous one forward. + for (const turn of [repair, revision]) { + expect(turn).toContain( + "Write the title the Plan needs rather than the one the last draft had", + ); + expect(turn).toContain("add it if\nit was missing"); + expect(turn).not.toContain("Keep its\nlevel-one title"); + } + }); + }); + + it("C4: one Session carries every turn, and --session names it", function* () { + // Initial, repair and revision turns all land in one conversation, and a + // second invocation places a different one in a directory of its own. + const keys: string[] = []; + const directories: string[] = []; + for (const _invocation of [0, 1]) { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: UNRESOLVED }); + harness.script({ decision: "Request changes", feedback: "try plain text" }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + // A draft, three repairs, a revision: five turns, one session. + expect(harness.fake.prompts).toHaveLength(5); + expect(sessions(harness)).toHaveLength(1); + keys.push(sessions(harness)[0]); + const workdir = String(harness.fake.created[0]?.cwd); + directories.push(workdir); + // This suite reaches no directory this host would use for real. + expect(workdir.startsWith(`${profileRoot}${sep}`)).toBe(true); + expect(workdir.startsWith(DEFAULT_PROFILE_ROOT)).toBe(false); + }); + } + expect(keys[0]).not.toBe(keys[1]); + // Different conversations, so different directories: a generated name + // reaches a location nothing else does. + expect(directories[0]).not.toBe(directories[1]); + + // `--session` replaces the generated name, so two invocations name one + // conversation in one directory — and one ACPX store is what turns that + // from equal keys into an actually continued session. The root is shared + // deliberately, and by this case alone. + yield* useProfileRoot(function* (profileRoot) { + const store = makeStore(); + const named: string[] = []; + const namedDirectories: string[] = []; + const materializations: (string | undefined)[] = []; + const survived: boolean[] = []; + for (const _invocation of [0, 1]) { + yield* useWorkingDirectory(function* (dir) { + const harness = createPromptHarness({ profileRoot, store }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt( + { ...command(dir, [REQUEST]), session: "ada" }, + harness.deps, + ); + expect(code).toBe(0); + named.push(sessions(harness)[0]); + namedDirectories.push(String(harness.fake.created[0]?.cwd)); + materializations.push(harness.fake.ensured[0]?.materialization); + // A named conversation's directory outlives the invocation, because + // its identity is what the next `--session ada` derives. + survived.push(yield* exists(profileDirectoryFor(profileRoot, "ada"))); + }); + } + expect(named[0]).toBe(named[1]); + expect(namedDirectories[0]).toBe(namedDirectories[1]); + expect(survived).toEqual([true, true]); + // The name never reaches the path, and the digest is what the host + // derived from it. + expect(namedDirectories[0]).toBe(profileDirectoryFor(profileRoot, "ada")); + expect(namedDirectories[0]).not.toContain("ada"); + + // The second invocation continued the record the first established + // rather than placing a second one: the store holds one, and only the + // first ensure asked for a session to be materialized by its first turn. + expect([...store.records.keys()]).toEqual([named[0]]); + expect(materializations).toEqual(["first-turn-acceptance", undefined]); + }); + }); + + it("C4, C13: a default session's directory belongs to its invocation", function* () { + // It exists and is empty while the turn runs, and it is gone once the + // profile has torn down — before anything the host does with what was + // approved. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + const seen: { workdir?: string; entries?: string[] } = {}; + harness.deps.installElicitation = watching(harness, function* (workdir) { + seen.workdir = workdir; + seen.entries = yield* until(readdir(workdir)); + }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(0); + expect(seen.entries).toEqual([]); + expect(seen.workdir?.startsWith(`${profileRoot}${sep}`)).toBe(true); + // Handed back non-recursively when the conversation ended, and the root + // it lived under is still there for the next one. + expect(yield* exists(String(seen.workdir))).toBe(false); + expect(yield* exists(profileRoot)).toBe(true); + // The approved Plan still ran: cleanup is not a failure. + expect(harness.executions).toHaveLength(1); + }); + + // Stopping, a turn that failed, and a cancelled command each hand the + // directory back the same way a success does. + for (const ending of [ + { name: "stop", drive: (harness: PromptHarness) => harness.script({ decision: "Stop" }) }, + { + name: "a failed turn", + drive: (harness: PromptHarness) => { + harness.fake.script({ reply: VALID, stopReason: "refusal" }); + }, + }, + ]) { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + if (ending.name === "stop") { + harness.fake.script({ reply: VALID }); + } + ending.drive(harness); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(1); + expect(harness.executions).toHaveLength(0); + // That exact leaf, and the root it lived under, both answer the + // question: the directory this ending made is gone, and nothing else + // was made in its place. + expect(yield* exists(String(harness.fake.created[0]?.cwd))).toBe(false); + expect(yield* until(readdir(profileRoot))).toEqual([]); + }); + } + + // Cancellation: the turn in flight is interrupted, and the ensure that + // hands the directory back runs on the way out like every other one. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID, manual: true }); + + yield* scoped(function* () { + const running = yield* spawn(() => runPrompt(command(dir, [REQUEST]), harness.deps)); + yield* harness.fake.startedTurns(1); + yield* running.halt(); + }); + + expect(yield* exists(String(harness.fake.created[0]?.cwd))).toBe(false); + expect(yield* until(readdir(profileRoot))).toEqual([]); + }); + + // A failure between making the leaf and using it still hands it back. The + // claim is taken before the `mkdir`, so there is no window in which a + // directory exists that nothing is responsible for. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + // deno-lint-ignore require-yield + harness.deps.installElicitation = function* () { + throw new Error("this host could not install a review provider"); + }; + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST]), harness.deps), + ); + + expect(value).toBe(1); + expect(lines.join("\n")).toContain("could not install a review provider"); + // Nothing was built after it, and no empty leaf was left behind. + expect(harness.fake.created).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + expect(yield* until(readdir(profileRoot))).toEqual([]); + }); + + // Establishment failing for a reason of its own leaves nothing behind and + // says what it found. The claim is already taken here, so the release runs + // and finds no directory it was ever given — which is the one case where + // an absent directory is the ordinary answer rather than interference. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const blocked = join(profileRoot, "not-a-directory"); + yield* writeTextFile(blocked, "in the way\n"); + + const harness = createPromptHarness({ profileRoot: blocked }); + harness.fake.script({ reply: VALID }); + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST]), harness.deps), + ); + + expect(value).toBe(1); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain("could not establish"); + // What was in the way is untouched, and no phase after it began. + expect(yield* readTextFile(blocked)).toBe("in the way\n"); + expect(harness.fake.created).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + }); + + // A leaf that disappears under a live conversation is interference, not a + // tidy exit. The command fails terminally rather than shrugging at an + // absent directory, and nothing it would have done next happens. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + harness.deps.installElicitation = watching(harness, function* (workdir) { + yield* rm(workdir, { recursive: true, force: true }); + }); + + const { value, lines } = yield* reported(() => + runPrompt({ ...command(dir, [REQUEST]), output: "out.md" }, harness.deps), + ); + + expect(value).toBe(1); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain("was made for this conversation and is already gone"); + expect(lines[0]).not.toContain("does not validate"); + // The Plan was approved and still reached nothing: no admission that + // could have delivered it, no output file, no execution and so no journal. + expect(harness.reviews).toHaveLength(1); + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* until(readdir(dir))).toEqual([]); + expect(harness.executions).toHaveLength(0); + }); + + // The other outcome of the one attempt. A directory this invocation was + // given empty and did not leave empty is preserved and the command fails: + // something wrote there while the conversation ran, and this host + // authorized nothing to. The draft was approved first, so what is being + // observed is a Plan that would otherwise have been written and run. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + let workdir: string | undefined; + let planted: string | undefined; + harness.deps.installElicitation = watching(harness, function* (directory) { + workdir = directory; + planted = join(directory, "stowaway.txt"); + yield* writeTextFile(planted, "not this command's doing\n"); + }); + + const { value, lines } = yield* reported(() => + runPrompt({ ...command(dir, [REQUEST]), output: "out.md" }, harness.deps), + ); + + // Terminal, and said once: the attempt happens once and either settles + // the directory or ends the command. + expect(value).toBe(1); + expect(lines).toHaveLength(1); + expect(lines[0]).toContain("was empty when this conversation started"); + expect(lines[0]).toContain("its contents were left alone"); + // The one line is the directory's, not admission's: the attempt settles + // before the host looks at what was approved, so a run that had reached + // admission would have reported that instead. + expect(lines[0]).not.toContain("does not validate"); + + // Preserved whole — the directory and what appeared in it. A recursive + // removal would have taken both. + expect(harness.reviews).toHaveLength(1); + expect(yield* exists(String(workdir))).toBe(true); + expect(yield* readTextFile(String(planted))).toBe("not this command's doing\n"); + + // And nothing after the failure began: no output file, and no execution — so no + // journal, which only an execution creates. + expect(yield* exists(join(dir, "out.md"))).toBe(false); + expect(yield* until(readdir(dir))).toEqual([]); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C13: a default directory is handed back before final admission", function* () { + // The same veto C10 proves, watched from the other side: by the time the + // host reports what it decided about the approved bytes, the conversation + // and its directory are both already gone. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const widget = join(dir, "Widget.md"); + yield* writeTextFile(widget, "A widget.\n"); + const draft = ["# Uses a widget", "", "", ""].join("\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: draft }); + let workdir: string | undefined; + harness.deps.installElicitation = watching(harness, function* (directory) { + workdir = directory; + yield* rm(widget, { force: true }); + }); + + const events: string[] = []; + const written = console.error; + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + console.error = written; + }); + console.error = () => { + events.push("reported"); + }; + return yield* runPrompt(command(dir, [REQUEST]), harness.deps); + }); + + expect(code).toBe(1); + // Admission ran, and it ran after the directory was handed back: the + // observation below is taken at the moment the host reported its + // decision. + expect(events).toEqual(["reported"]); + expect(yield* exists(String(workdir))).toBe(false); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C5: the prompt profile's ceiling is the host's, and no flag widens it", function* () { + // This session's own directory, empty, no MCP servers and no native tools + // — observed while the command document is still running, because that is + // the only moment the claim is about. + yield* useWorkingDirectory(function* (dir, profileRoot) { + yield* writeTextFile(join(dir, "secret.txt"), "the caller's tree\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + const seen: { cwd?: string; entries?: string[]; refusals: string[] } = { refusals: [] }; + harness.deps.installElicitation = function* () { + yield* Elicitation.around( + { + *elicit([request], _next) { + harness.reviews.push(request); + const cwd = harness.fake.created[0]?.cwd; + seen.cwd = cwd; + seen.entries = cwd === undefined ? undefined : yield* until(readdir(cwd)); + // The command document's own capabilities, asked for from inside + // its scope. It decides what to write; it writes nothing. + for (const [name, ask] of [ + ["a directory", () => API.Files.operations.temporaryDirectory()], + ["a command", () => API.Process.operations.exec({ command: ["true"] })], + ["the network", () => API.Fetch.operations.fetch("http://localhost")], + ] as const) { + try { + yield* ask(); + seen.refusals.push(`${name} was granted`); + } catch (error) { + seen.refusals.push(error instanceof Error ? error.message : String(error)); + } + } + return { decision: "Approve" }; + }, + }, + { at: "min" }, + ); + }; + + const code = yield* runPrompt( + { ...command(dir, [REQUEST]), session: "ceiling" }, + harness.deps, + ); + expect(code).toBe(0); + + // Not the caller's working directory: this session's, and empty while + // the conversation ran. + expect(seen.cwd).not.toBe(dir); + expect(seen.cwd).toBe(profileDirectoryFor(profileRoot, "ceiling")); + expect(seen.entries).toEqual([]); + // Stated rather than omitted: this host configures no MCP server and + // allows no native tool on a fresh session. + expect(harness.fake.created[0]?.mcpServers).toEqual([]); + expect(harness.fake.ensured[0]?.sessionOptions?.allowedTools).toEqual([]); + // And the command document is given nothing to act with. + expect(seen.refusals).toEqual([ + "xmd prompt asked for a directory, which the prompt profile grants to nothing", + "xmd prompt asked for a command, which the prompt profile grants to nothing", + "xmd prompt asked for the network, which the prompt profile grants to nothing", + ]); + }); + + // Something already in a named session's directory is a refusal, not a + // cleanup. It happens before the provider exists, so no session is placed + // and no turn is started — and what was there is still there afterwards. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const occupied = profileDirectoryFor(profileRoot, "occupied"); + yield* ensureDir(occupied); + yield* writeTextFile(join(occupied, "someone-elses.txt"), "not mine to delete\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + + const { value, lines } = yield* reported(() => + runPrompt({ ...command(dir, [REQUEST]), session: "occupied" }, harness.deps), + ); + + expect(value).toBe(1); + expect(lines.join("\n")).toContain("is not empty"); + expect(lines.join("\n")).toContain("name a different --session"); + expect(lines.join("\n")).toContain(occupied); + // Nothing downstream of the refusal happened. + expect(harness.fake.created).toHaveLength(0); + expect(harness.fake.ensured).toHaveLength(0); + expect(harness.fake.started).toBe(false); + expect(harness.fake.prompts).toHaveLength(0); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + // The contents were left exactly as they were found. + expect(yield* readTextFile(join(occupied, "someone-elses.txt"))).toBe( + "not mine to delete\n", + ); + }); + + // `--approve-all` configures the approved document. A native permission + // request while the Plan is being written is still denied, privately, and + // the turn it belongs to fails. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID, requestsTool: "Bash" }); + + const code = yield* runPrompt( + command(dir, [REQUEST], { ...STACK, permissionMode: "approve-all" }), + harness.deps, + ); + + expect(code).toBe(1); + expect(harness.fake.decisions).toEqual(["reject_once"]); + // The denial ended the command: nobody was asked and nothing ran. + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C5: two cases' profile roots cannot see or remove one another", function* () { + // Roots are made per scope and named by a UUID, so one case's cleanup + // cannot reach another's directory even while both are live. + yield* useProfileRoot(function* (mine) { + const marker = join(mine, "mine.txt"); + yield* writeTextFile(marker, "still here\n"); + + yield* useWorkingDirectory(function* (dir, profileRoot) { + expect(profileRoot).not.toBe(mine); + expect(profileRoot.startsWith(`${mine}${sep}`)).toBe(false); + expect(mine.startsWith(`${profileRoot}${sep}`)).toBe(false); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + // The other root is untouched, and this one holds nothing afterwards. + expect(yield* readTextFile(marker)).toBe("still here\n"); + expect(yield* until(readdir(profileRoot))).toEqual([]); + }); + + expect(yield* readTextFile(marker)).toBe("still here\n"); + }); + }); + + it("C6: a candidate is inert until the approved document runs", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: WRITES_A_FILE }); + harness.script({ decision: "Stop" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(1); + // Validated, presented, and never run: the document's own write is the + // effect that would have happened if anything had executed it. + expect(harness.reviews).toHaveLength(1); + expect((yield* until(readdir(dir))).includes("drafted.txt")).toBe(false); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C7: candidate defects earn a repair turn; caller defects escape", function* () { + // A defect the agent authored: the root's own frontmatter. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: BROKEN_SOURCE }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(0); + expect(harness.fake.prompts).toHaveLength(2); + expect(harness.fake.prompts[1]).toContain("That Plan has problems"); + // `` captured the whole serialized value, and the fence it went + // into holds every byte of it: the wrapper's removal changed where the + // text is bound, not what it says. Parsed back rather than matched, so + // what is proven is that the complete structure survived the trip. + const repaired = fencedJson(harness.fake.prompts[1]); + expect(repaired.validation?.version).toBe(1); + expect(repaired.validation?.outcome).toBe("invalid"); + expect( + repaired.validation?.diagnostics?.some((entry) => entry.code === "source-invalid"), + ).toBe(true); + expect(harness.fake.prompts[1]).toContain("source-invalid"); + // The whole versioned value, as data the program serialized. + expect(harness.fake.prompts[1]).toContain('"version": 1'); + expect(harness.executions).toHaveLength(1); + }); + + // A defect the agent authored: two properties generating one option. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: COLLIDING }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(0); + expect(harness.fake.prompts).toHaveLength(2); + expect(harness.fake.prompts[1]).toContain("generated-binding-collision"); + expect(harness.fake.prompts[1]).toContain("--props-first-name"); + // Carried as this command's own finding, not as a core diagnostic code. + expect(harness.fake.prompts[1]).not.toContain("DocumentValidationCode"); + }); + + // A defect the caller wrote: an option the candidate never declares. It + // raises out of the validator, so the program never sees it as feedback. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-nothing", "here"]), + harness.deps, + ); + + expect(code).toBe(1); + // One turn, no repair: the agent cannot fix a command line. + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + }); + + // A defect the caller wrote: aggregate JSON that is not JSON. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + + const code = yield* runPrompt(command(dir, [REQUEST, "--props", "{oops"]), harness.deps); + + expect(code).toBe(1); + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + }); + + // A defect the caller wrote: a value this candidate's schema rejects. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: NAME_IS_BOOLEAN }); + + const code = yield* runPrompt( + command(dir, [REQUEST, "--props-name=not-a-boolean"]), + harness.deps, + ); + + expect(code).toBe(1); + expect(harness.fake.prompts).toHaveLength(1); + expect(harness.reviews).toHaveLength(0); + expect(harness.executions).toHaveLength(0); + }); + + // A revision that changes what the command line means is the caller's + // failure too, and it is caught before the candidate is presented. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: REQUIRES_NAME }); + harness.script({ decision: "Request changes", feedback: "make it shout" }); + harness.fake.script({ reply: NAME_IS_BOOLEAN }); + + const code = yield* runPrompt(command(dir, [REQUEST, "--props-name", "Ada"]), harness.deps); + + expect(code).toBe(1); + expect(harness.reviews).toHaveLength(1); + expect(harness.executions).toHaveLength(0); + expect(harness.fake.prompts).toHaveLength(2); + }); + }); + + it("C8: one base draft, three repairs, and ten presentations", function* () { + // Three repairs are available, and the fourth draft is what a person sees. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: UNRESOLVED }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(0); + expect(harness.fake.prompts).toHaveLength(4); + expect(harness.reviews).toHaveLength(1); + expect(decisions(harness.reviews[0])).toEqual(["Approve", "Request changes", "Stop"]); + }); + + // A fourth invalid candidate is repair-exhausted: it reaches review with + // its diagnostics, and there is no value that would approve it. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + for (const _draft of [0, 1, 2, 3]) { + harness.fake.script({ reply: UNRESOLVED }); + } + harness.script({ decision: "Stop" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(1); + expect(harness.fake.prompts).toHaveLength(4); + expect(harness.reviews).toHaveLength(1); + expect(decisions(harness.reviews[0])).toEqual(["Request changes", "Stop"]); + expect(harness.reviews[0].message).toContain( + "The coding agent used all three repair attempts", + ); + // The presentation carries the same complete captured JSON the repair + // turns did, through the same `` binding. + const shown = fencedJson(harness.reviews[0].message); + expect(shown.validation?.version).toBe(1); + expect( + shown.validation?.diagnostics?.some((entry) => entry.code === "component-unresolved"), + ).toBe(true); + expect(harness.executions).toHaveLength(0); + }); + + // Ten presentations: nine revisions, and a tenth round with nothing left + // to revise into. Each revision starts its own repair budget. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + for (const round of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) { + harness.fake.script({ reply: VALID }); + if (round < 10) { + harness.script({ decision: "Request changes", feedback: `round ${round}` }); + } + } + harness.script({ decision: "Stop" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + + expect(code).toBe(1); + expect(harness.reviews).toHaveLength(10); + // Every draft validated first time, so each round cost exactly one turn. + expect(harness.fake.prompts).toHaveLength(10); + expect(harness.fake.prompts[1]).toContain("round 1"); + for (const round of [0, 1, 2, 3, 4, 5, 6, 7, 8]) { + expect(decisions(harness.reviews[round])).toEqual(["Approve", "Request changes", "Stop"]); + } + // The last presentation offers no revision, because there is no eleventh + // round to revise into. + expect(decisions(harness.reviews[9])).toEqual(["Approve", "Stop"]); + // One conversation held all ten. + expect(sessions(harness)).toHaveLength(1); + }); + }); + + it("C9: arbitrary source cannot close the presentation, and stopping is authored", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + // A document that holds a fence of its own, and a run of five backticks. + const fenced = [ + "Here is a block:", + "", + "```bash", + "echo hi", + "```", + "", + "and `````five````` backticks.", + "", + ].join("\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: fenced }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + + const message = harness.reviews[0].message; + // The fence is longer than every run inside the candidate, so nothing in + // the source can close it. + expect(message).toContain("``````markdown\n"); + expect(message).toContain(`\n${fenced}\n\`\`\`\`\`\``); + }); + + // Stop reaches the command document's own ``, with the message the + // shipped Markdown wrote. Nothing about it is host policy. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Stop" }); + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST]), harness.deps), + ); + + expect(value).toBe(1); + expect(lines.join("\n")).toContain( + "xmd prompt stopped at your request. Nothing was output or run.", + ); + expect(harness.executions).toHaveLength(0); + }); + + // Exhaustion is reachable, and it is a different ending. Ten presentations + // that never validated, the last offering nothing but a way out, and the + // sentence says there was never a Plan to approve rather than that + // somebody decided to stop. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + for (const round of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) { + // A base draft and its three repairs, none of which validates. + for (const _draft of [0, 1, 2, 3]) { + harness.fake.script({ reply: UNRESOLVED }); + } + harness.script( + round < 10 + ? { decision: "Request changes", feedback: `round ${round}` } + : { decision: "Stop" }, + ); + } + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST], STACK), harness.deps), + ); + + expect(value).toBe(1); + expect(harness.reviews).toHaveLength(10); + // Rounds one to nine could be sent back; the tenth had one choice, and + // taking it is what reaches the exhaustion ending. + expect(decisions(harness.reviews[8])).toEqual(["Request changes", "Stop"]); + expect(decisions(harness.reviews[9])).toEqual(["Explain what went wrong", "Stop"]); + expect(lines).toHaveLength(1); + expect(lines[0]).toBe( + "xmd prompt reviewed ten drafts without an approved Plan. Nothing was output or run.", + ); + // Stopping asks the coding agent nothing: forty drafting turns and no + // forty-first. + expect(harness.fake.prompts).toHaveLength(40); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C3, C9: the last invalid draft can be explained rather than only stopped", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + for (const round of [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) { + for (const _draft of [0, 1, 2, 3]) { + harness.fake.script({ reply: UNRESOLVED }); + } + harness.script( + round < 10 + ? { decision: "Request changes", feedback: `round ${round}` } + : { decision: "Explain what went wrong" }, + ); + } + // The explanation the coding agent gives is prose, and stays prose. + const explanation = [ + "Every draft used , which this profile does not offer.", + "", + "Say which of the available components should do the work, or describe the", + "outcome without naming a component: ", + ].join("\n"); + harness.fake.script({ reply: explanation }); + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST]), harness.deps), + ); + + expect(value).toBe(1); + // Exactly one turn more than the forty drafting turns, in the same + // conversation. It is not another draft: no eleventh review, and the + // ten-draft limit is not reopened. + expect(harness.fake.prompts).toHaveLength(41); + expect(harness.reviews).toHaveLength(10); + expect(sessions(harness)).toHaveLength(1); + + // It carries the final problems, which were produced after the agent's + // last draft and have not appeared in the conversation — and nothing + // else. The Session already holds the Prompt, the catalog and every + // draft, so none of them is resent. + const asked = harness.fake.prompts[40]; + expect( + fencedJson(asked).validation?.diagnostics?.some( + (entry) => entry.code === "component-unresolved", + ), + ).toBe(true); + expect(asked).toContain("Do not create another Plan"); + expect(asked).not.toContain(REQUEST); + expect(asked).not.toContain("## Built-in components"); + expect(asked).not.toContain(UNRESOLVED.trim()); + + // What it said is reported as the coding agent's words, and the command + // ends. Nothing about the explanation is treated as a Plan. + expect(lines).toHaveLength(1); + expect(lines[0]).toContain("reviewed ten drafts without an approved Plan"); + expect(lines[0]).toContain(explanation); + expect(lines[0]).toContain("Nothing was output or run."); + expect(harness.executions).toHaveLength(0); + }); + }); + + it("C10, C11: the approved bytes are what runs, and props are theirs", function* () { + // The command line is unchanged across a revision, and the schema that + // resolves it is the approved document's rather than the first draft's. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: counting("number") }); + harness.script({ decision: "Request changes", feedback: "count in words" }); + harness.fake.script({ reply: counting("string") }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST, "--props-count", "7"]), harness.deps); + + expect(code).toBe(0); + // Resolved under the approved bytes: a props object kept from the first + // candidate would carry the number 7. + expect(harness.executions[0]?.props).toEqual({ count: "7" }); + expect(harness.executions[0]?.root.source).toBe(counting("string")); + expect(harness.executions[0]?.root.path).toBe(""); + }); + + // Nothing is stripped. A reply wrapped in a fence is not a document, so it + // earns repairs and a review — and what is shown is exactly what arrived. + yield* useWorkingDirectory(function* (dir, profileRoot) { + const wrapped = ["```md", "Hello.", "```", ""].join("\n"); + const harness = createPromptHarness({ profileRoot }); + for (const _draft of [0, 1, 2, 3]) { + harness.fake.script({ reply: wrapped }); + } + harness.script({ decision: "Stop" }); + + yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(harness.reviews[0].message).toContain(wrapped); + }); + }); + + it("C10: final admission vetoes after the profile has torn down", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + // A repository component the draft uses. It exists while the command + // document runs, so the same production validator that answers + // finds the draft sound. + const widget = join(dir, "Widget.md"); + yield* writeTextFile(widget, "A widget.\n"); + const draft = ["# Uses a widget", "", "", ""].join("\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: draft }); + const events: string[] = []; + harness.deps.installElicitation = function* () { + // Registered inside the profile's scope, so it runs while that scope + // is being torn down — after the provider is gone and before the host + // looks at what was approved. + yield* ensure(function* () { + events.push("teardown"); + yield* rm(widget, { force: true }); + }); + yield* Elicitation.around( + { + // deno-lint-ignore require-yield + *elicit([request], _next) { + harness.reviews.push(request); + return { decision: "Approve" }; + }, + }, + { at: "min" }, + ); + }; + + const written = console.error; + const lines: string[] = []; + const code = yield* scoped(function* (): Operation { + yield* ensure(() => { + console.error = written; + }); + console.error = (...parts: unknown[]) => { + events.push("reported"); + lines.push(parts.map((part) => String(part)).join(" ")); + }; + return yield* runPrompt(command(dir, [REQUEST]), harness.deps); + }); + + // The draft was sound enough to approve, and the approved bytes are + // unchanged — only the tree they resolve against moved. + expect(harness.reviews).toHaveLength(1); + expect(harness.reviews[0].message).toContain(""); + + // The gate is effective on its own: the same bytes now fail, because the + // host validates them again rather than trusting what the document + // concluded while its own scope was still standing. + expect(code).toBe(1); + expect(lines.join("\n")).toContain("the approved document does not validate"); + expect(lines.join("\n")).toContain("component-unresolved"); + expect(lines.join("\n")).toContain("Widget"); + + // And it is ordered: teardown finished first, which is the only reason + // the component was missing when the second validation ran. + expect(events).toEqual(["teardown", "reported"]); + + // Nothing after the veto happened. + expect(harness.executions).toHaveLength(0); + expect((yield* until(readdir(dir))).sort()).toEqual([]); + }); + }); + + it("C14: an interleaved Plan survives approval and execution byte for byte", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + // What the shipped instruction asks for: the request restated in prose a + // reader was written for, with each component beside the sentences that + // describe what it does. + const plan = [ + "# Your age", + "", + "You asked to be asked for your age and to have it written down, so this", + "document does exactly that, in that order.", + "", + "First, the question.", + "", + '', + "", + "Then the answer goes to a file beside this document.", + "", + '{answer.age}', + "", + ].join("\n"); + + const harness = createPromptHarness({ profileRoot }); + harness.fake.script({ reply: plan }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt( + command(dir, ["ask me for my age and write it to a file"]), + harness.deps, + ); + + expect(code).toBe(0); + // The prose reached the person who approved it, inside the presentation + // fence and unaltered. + expect(harness.reviews[0].message).toContain(plan); + // And the exact bytes are what runs: nothing trimmed, re-fenced or + // reflowed between approval and execution. + expect(harness.executions[0]?.root.source).toBe(plan); + }); + }); + + it("C3, C9: what you read says each thing once, however many rounds it took", function* () { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + // Round one: a draft that cannot be repaired, presented with its problems. + for (const _draft of [0, 1, 2, 3]) { + harness.fake.script({ reply: UNRESOLVED }); + } + harness.script({ decision: "Request changes", feedback: "use plain text" }); + // Round two: a draft that passes, and is approved. + harness.fake.script({ reply: VALID }); + harness.script({ decision: "Approve" }); + + const code = yield* runPrompt(command(dir, [REQUEST]), harness.deps); + expect(code).toBe(0); + + const [exhausted, approvable] = harness.reviews.map((review) => review.message); + // The heading each presentation carries appears once in it, and the + // second presentation carries neither the first draft nor its problems: + // a loop that accumulated its explanation would show both. + for (const message of [exhausted, approvable]) { + expect(occurrences(message, "### Original Prompt")).toBe(1); + expect(occurrences(message, "### Draft Plan")).toBe(1); + } + expect(occurrences(exhausted, "### Problems that remain")).toBe(1); + expect(approvable).not.toContain("Problems that remain"); + expect(approvable).not.toContain("NoSuchComponent"); + expect(exhausted).not.toContain(VALID.trim()); + + // The repair turns say what they need once each, and carry only the + // problems of the draft they are repairing. + for (const repair of harness.fake.prompts.slice(1, 4)) { + expect(occurrences(repair, "That Plan has problems")).toBe(1); + expect(occurrences(repair, "Send one complete replacement Plan")).toBe(1); + } + // The revision turn says what changed, once, and asks for a whole + // document rather than repeating the original brief. + const revision = harness.fake.prompts[4]; + expect(occurrences(revision, "use plain text")).toBe(1); + expect(revision).not.toContain("Create one complete XMD Plan from this Prompt"); + }); + + // Stopping prints one sentence, once, whether it happened on the first + // presentation or the tenth: an approvable Plan existed and you chose to + // stop, and ten rounds of it do not accumulate into a different sentence. + for (const rounds of [1, 10]) { + yield* useWorkingDirectory(function* (dir, profileRoot) { + const harness = createPromptHarness({ profileRoot }); + for (const round of Array.from({ length: rounds }, (_, i) => i + 1)) { + harness.fake.script({ reply: VALID }); + if (round < rounds) { + harness.script({ decision: "Request changes", feedback: `round ${round}` }); + } + } + harness.script({ decision: "Stop" }); + + const { value, lines } = yield* reported(() => + runPrompt(command(dir, [REQUEST]), harness.deps), + ); + + expect(value).toBe(1); + // One line, from the document's own , and no accumulated + // repetition of it however many rounds preceded it. + expect(lines).toHaveLength(1); + expect(occurrences(lines[0], "xmd prompt ")).toBe(1); + expect(lines[0]).toContain("stopped at your request"); + }); + } + }); + }, +); diff --git a/packages/cli/tests/support/fake-acp.ts b/packages/cli/tests/support/fake-acp.ts index ee3e5dd2..4a751e09 100644 --- a/packages/cli/tests/support/fake-acp.ts +++ b/packages/cli/tests/support/fake-acp.ts @@ -29,7 +29,10 @@ import type { ProbeCapableRuntime, } from "@executablemd/acp"; -export function makeStore(): AcpxSessionStore & { records: Map } { +/** A session store whose records a case can read back. */ +export type FakeStore = AcpxSessionStore & { records: Map }; + +export function makeStore(): FakeStore { const records = new Map(); return { records, @@ -82,6 +85,14 @@ export interface ScriptedTurn { * the turns that had already committed. */ readonly manual?: boolean; + /** + * The stop reason this turn settles with. + * + * `end_turn` unless a case says otherwise. Anything else is a turn the + * provider reports as failed while keeping whatever text it streamed, which is + * how a partial reply is driven without an adapter that misbehaves on purpose. + */ + readonly stopReason?: string; /** * The App Server turn this reply is, as an adapter that names its turns says * it: on this exact response's own `_meta`. @@ -100,10 +111,24 @@ export interface FakeAcp { readonly ensured: AcpRuntimeEnsureInput[]; /** Every prompt text this provider sent, in order. */ readonly prompts: string[]; + /** Every turn this provider started, whole, in order. */ + readonly turns: AcpRuntimeTurnInput[]; /** Every permission decision this fake was answered with. */ readonly decisions: string[]; /** Whether an agent process would have been started at all. */ readonly started: boolean; + /** Every handle this runtime was told to close, by the reason it was given. */ + readonly closes: string[]; + /** How many turns were cancelled rather than allowed to settle. */ + readonly cancels: number; + /** + * Fail every close. + * + * A provider whose teardown fails is the case a host has to survive without + * acting on what the failed scope produced, and only a close that rejects + * makes the provider report one. + */ + closeFailure?: Error; script(turn: ScriptedTurn): void; /** * Settles once `count` turns have been started. @@ -170,11 +195,14 @@ export function createFakeAcp(): FakeAcp { const created: AcpRuntimeOptions[] = []; const ensured: AcpRuntimeEnsureInput[] = []; const prompts: string[] = []; + const turns: AcpRuntimeTurnInput[] = []; const decisions: string[] = []; + const closes: string[] = []; const waiting: Array<{ count: number; settle: () => void }> = []; const reading: Array<{ count: number; settle: () => void }> = []; let consumed = 0; let started = false; + let cancels = 0; function announceTurn(): void { for (const barrier of [...waiting]) { @@ -199,10 +227,15 @@ export function createFakeAcp(): FakeAcp { created, ensured, prompts, + turns, decisions, + closes, get started(): boolean { return started; }, + get cancels(): number { + return cancels; + }, script(turn) { scripted.push(turn); }, @@ -285,6 +318,7 @@ export function createFakeAcp(): FakeAcp { }, startTurn(input: AcpRuntimeTurnInput): AcpRuntimeTurn { prompts.push(input.text); + turns.push(input); announceTurn(); const turn = scripted.shift() ?? { reply: "" }; const settled = withResolvers(); @@ -348,7 +382,7 @@ export function createFakeAcp(): FakeAcp { void asked.then(() => { settled.resolve({ status: "completed", - stopReason: "end_turn", + stopReason: turn.stopReason ?? "end_turn", ...(turn.turnId === undefined ? {} : { _meta: { codex: { turnId: turn.turnId } } }), }); }); @@ -387,6 +421,7 @@ export function createFakeAcp(): FakeAcp { }, result: run(() => settled.operation), cancel(): Promise { + cancels += 1; settled.resolve({ status: "cancelled" }); released.resolve(); return Promise.resolve(); @@ -402,8 +437,9 @@ export function createFakeAcp(): FakeAcp { cancel(): Promise { return Promise.resolve(); }, - close(): Promise { - return Promise.resolve(); + close(input?: { reason?: string }): Promise { + closes.push(input?.reason ?? ""); + return fake.closeFailure ? Promise.reject(fake.closeFailure) : Promise.resolve(); }, }; }, diff --git a/packages/cli/tests/support/prompt-harness.ts b/packages/cli/tests/support/prompt-harness.ts new file mode 100644 index 00000000..59b7d630 --- /dev/null +++ b/packages/cli/tests/support/prompt-harness.ts @@ -0,0 +1,231 @@ +/** + * The deterministic seams `xmd prompt` is proven against. + * + * Every phase the command owns is driven in process: the ACPX runtime is the + * scriptable fake, the review provider is a scripted `Elicitation` handler, the + * catalog and the execution are recorded, and the contextual working directory + * is a temporary one. No live agent, browser, or network belongs in this + * evidence. + * + * Each recorder is a tripwire as well as a fake. A refusal is proven by the + * phases that stayed at zero, never by empty output — a command that printed + * nothing and still opened a session would pass such a check. + */ + +import { Elicitation } from "@executablemd/core"; +import type { ElicitationRequest, SyntaxCatalog } from "@executablemd/core"; +import { Ok } from "effection"; +import type { Operation, Result } from "effection"; +import { ensure, scoped } from "effection"; +import { ensureDir, rm } from "@effectionx/fs"; +import { randomUUID } from "node:crypto"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { API, useHostFiles } from "@executablemd/runtime"; +import { syntaxCatalog } from "../../src/syntax.ts"; +import type { PromptDependencies, PromptExecution } from "../../src/prompt.ts"; +import { createFakeAcp, makeRegistry, makeStore } from "./fake-acp.ts"; +import type { FakeAcp, FakeStore } from "./fake-acp.ts"; + +/** The agent every prompt case drives, and the command it resolves to. */ +export const AGENT = "scripted-agent"; + +/** + * One review answer, scripted. + * + * The values are the words a person reads and the words the provider answers + * with: the workflow keeps no internal spelling behind them. + */ +export interface ScriptedReview { + decision: "Approve" | "Request changes" | "Stop" | "Explain what went wrong"; + feedback?: string; + /** Answer with this instead, to drive a response the schema rejects. */ + raw?: unknown; +} + +export interface PromptHarness { + fake: FakeAcp; + /** Every catalog request, by the includes it was made with. */ + catalogCalls: string[][]; + /** Every review request a provider was asked, in order. */ + reviews: ElicitationRequest[]; + /** Every approved execution that reached the executor. */ + executions: PromptExecution[]; + /** What the executor answers with, once each, then `Ok`. */ + executionResults: Result[]; + /** Review answers, taken in order. Running out is a test defect, not a case. */ + script(review: ScriptedReview): void; + /** The dependencies `runPrompt` is driven with. */ + deps: PromptDependencies; +} + +export function createPromptHarness(options: { + /** + * Where this harness keeps its profile session directories. + * + * Required, and always a tree the case itself created: a suite that fell back + * to the host default would be reading and removing directories under the + * developer's own home, and two cases running close together could not tell + * whose was whose. + */ + profileRoot: string; + /** Replace the catalog entirely, for a case about catalog failure. */ + catalog?: (includes: readonly string[]) => Operation; + /** + * The ACPX session store this invocation reads and writes. + * + * One store shared by two harnesses is two invocations of the command against + * one provider's memory, which is the only way to observe whether a named + * session is continued or created a second time. + */ + store?: FakeStore; +}): PromptHarness { + const fake = createFakeAcp(); + const catalogCalls: string[][] = []; + const reviews: ElicitationRequest[] = []; + const executions: PromptExecution[] = []; + const executionResults: Result[] = []; + const answers: ScriptedReview[] = []; + + const harness: PromptHarness = { + fake, + catalogCalls, + reviews, + executions, + executionResults, + script(review) { + answers.push(review); + }, + deps: { + acp: { + createRuntime: fake.create, + sessionStore: options.store ?? makeStore(), + agentRegistry: makeRegistry({ [AGENT]: `${AGENT}-cmd` }), + }, + *catalog(includes) { + catalogCalls.push([...includes]); + return yield* (options.catalog ?? syntaxCatalog)(includes); + }, + profileRoot: options.profileRoot, + *installElicitation() { + yield* Elicitation.around( + { + // deno-lint-ignore require-yield + *elicit([request], _next) { + reviews.push(request); + const answer = answers.shift(); + if (answer === undefined) { + throw new Error("the case scripted no answer for this review"); + } + if (answer.raw !== undefined) { + return answer.raw; + } + return { + decision: answer.decision, + ...(answer.feedback === undefined ? {} : { feedback: answer.feedback }), + }; + }, + }, + { at: "min" }, + ); + }, + // deno-lint-ignore require-yield + *execute(approved) { + executions.push(approved); + return executionResults.shift() ?? Ok(undefined); + }, + }, + }; + return harness; +} + +/** + * A temporary directory that is also the contextual working directory. + * + * Both, because the two answer different questions: session placement and + * `--output` resolve the contextual one, while a real file has to live + * somewhere. + */ +export function* useWorkingDirectory( + body: (dir: string, profileRoot: string) => Operation, +): Operation { + const dir = join(tmpdir(), `xmd-prompt-${randomUUID()}`); + // A sibling rather than a child: the working directory is what the approved + // document writes into and what several cases read back, and a profile root + // inside it would show up in those listings. + const profileRoot = `${dir}-profile`; + yield* ensureDir(dir); + yield* ensureDir(profileRoot); + return yield* scoped(function* () { + yield* ensure(() => rm(dir, { recursive: true, force: true })); + // Recursive, and safe because it is: everything under this root was created + // by this scope, so nothing here can reach a directory another case or a + // real invocation owns. + yield* ensure(() => rm(profileRoot, { recursive: true, force: true })); + yield* API.Env.around({ + // deno-lint-ignore require-yield + *cwd() { + return dir; + }, + }); + // The provider `API.Files` has no host default for, installed exactly where + // the runtime entrypoint installs it: a document that reaches the + // filesystem must reach the caller's, or fail. + yield* useHostFiles(); + return yield* body(dir, profileRoot); + }); +} + +/** + * Environment values this scope answers with, delegating every other name. + * + * Delegation matters: the agent stack reads `DEFAULT_AGENT_NAME` through the + * same Api, and answering `undefined` for everything would make a case about + * `XMD_PROPS` quietly also be a case about the default agent. + */ +export function* useEnvironment(values: Record): Operation { + yield* useRecordedEnvironment([], values); +} + +/** + * The same environment, with every name it was asked recorded in order. + * + * What a resolution costs is visible only as the reads it performs: a value + * settled once and handed to both consumers reads its name once, while two + * consumers each settling their own read it twice and agree only by accident. + */ +export function* useRecordedEnvironment( + reads: string[], + values: Record, +): Operation { + yield* API.Env.around({ + *env([name], next) { + reads.push(name); + if (Object.hasOwn(values, name)) { + return values[name]; + } + return yield* next(name); + }, + }); +} + +/** How many times one name was read. */ +export function timesRead(reads: readonly string[], name: string): number { + return reads.filter((read) => read === name).length; +} + +/** + * A profile-session root this scope creates, owns and removes whole. + * + * Owning it is what makes recursive removal safe: everything under it was made + * by this scope, so nothing here can reach a directory another case — or a real + * invocation — is using. + */ +export function* useProfileRoot(body: (root: string) => Operation): Operation { + const root = join(tmpdir(), `xmd-prompt-profile-${randomUUID()}`); + yield* ensureDir(root); + return yield* scoped(function* () { + yield* ensure(() => rm(root, { recursive: true, force: true })); + return yield* body(root); + }); +} diff --git a/packages/cli/tests/syntax-cli.test.ts b/packages/cli/tests/syntax-cli.test.ts index f57f5c07..10da2db1 100644 --- a/packages/cli/tests/syntax-cli.test.ts +++ b/packages/cli/tests/syntax-cli.test.ts @@ -384,10 +384,13 @@ describe("Tier SX — the command line", { sanitizeOps: false, sanitizeResources ); }); - it("SX11: adds no prompt command", function* () { + it("SX11: the catalog is inspection, and prompt is the command that writes with it", function* () { const { stdout } = yield* runCli(["--help"]).expect(); expect(stdout).toContain("syntax"); - expect(stdout).not.toMatch(/^\s+prompt\s/m); + // `xmd prompt` is the other reader of this catalog: it feeds the same + // structured value to a generator rather than spawning this command and + // parsing what it printed. + expect(stdout).toMatch(/^\s+prompt\s/m); }); }); diff --git a/scripts/build-npm.ts b/scripts/build-npm.ts index 385edd08..dfa43408 100644 --- a/scripts/build-npm.ts +++ b/scripts/build-npm.ts @@ -27,6 +27,7 @@ import { build } from "jsr:@deno/dnt@0.42.3"; import { copyFile, emptyDir, + ensureDir, exists, fromFileUrl, readTextFile, @@ -34,12 +35,29 @@ import { writeTextFile, } from "@effectionx/fs"; import { listWorkspacePaths } from "./lib/workspace.ts"; -import { join } from "node:path"; +import { join, sep } from "node:path"; // Recursive directory copy and temp-dir creation are not part of @effectionx/fs. -import { cp, mkdtemp } from "node:fs/promises"; +import { cp, mkdtemp, readdir } from "node:fs/promises"; import { tmpdir } from "node:os"; import { z } from "npm:zod@^4"; +/** + * The documents a package ships, as package-relative paths. + * + * `src/documents/` and nothing else. Packages keep test documents and scenario + * fixtures under `src/` as well, so publishing every Markdown found there would + * ship a package full of fixtures; being in this one directory is what declares + * a document part of the product. + */ +function* packagedDocuments(pkgDir: URL): Operation { + const documents = new URL("src/documents/", pkgDir); + if (!(yield* exists(documents))) { + return []; + } + const names = yield* until(readdir(fromFileUrl(documents), { recursive: true })); + return names.map((name) => `src/documents/${name.split(sep).join("/")}`); +} + const ExportsSchema = z.union([z.string(), z.record(z.string(), z.string())]); const DenoJsonSchema = z.object({ @@ -312,6 +330,17 @@ function* buildPackage(pkgArg: string, version: string, ctx: BuildContext): Oper yield* copyFile(license, new URL("LICENSE", outDir)); } + // A package that executes its own Markdown ships that Markdown beside its + // emitted module. dnt emits the module graph and nothing else, so an asset no + // TypeScript imports would be absent from the published package while the + // source checkout kept working — the command would find nothing at runtime, + // on Node and Bun only. + for (const asset of yield* packagedDocuments(pkgDir)) { + const target = new URL(`esm/${asset}`, outDir); + yield* ensureDir(fromFileUrl(new URL(".", target))); + yield* copyFile(new URL(asset, pkgDir), target); + } + ctx.built.add(denoJson.name); const provenance = ctx.localSiblings && workspaceDeps.length > 0 ? " (local siblings — not publishable)" : ""; diff --git a/scripts/tests/cli-npm-bin.test.ts b/scripts/tests/cli-npm-bin.test.ts index 1be7be83..74955cff 100644 --- a/scripts/tests/cli-npm-bin.test.ts +++ b/scripts/tests/cli-npm-bin.test.ts @@ -101,5 +101,14 @@ describe("npm CLI package", { sanitizeOps: false, sanitizeResources: false }, () // the whole of what a built npm bin has to get right for one. expect(run.stdout).toContain("You chose to approve the review."); expect(run.stdout).not.toContain("ERROR"); + + // The Markdown this package executes itself ships beside the module that + // reads it. dnt emits the module graph only, so an asset nothing imports is + // absent from the package unless the build copies it — and the command + // would then find no program to run, on Node and Bun while Deno stayed + // green. + expect(yield* readTextFile(path.join(OUT_DIR, "esm/src/documents/prompt-command.md"))).toBe( + yield* readTextFile(path.join(ROOT, PKG_DIR, "src/documents/prompt-command.md")), + ); }); }); diff --git a/scripts/tests/packaged-document.test.ts b/scripts/tests/packaged-document.test.ts new file mode 100644 index 00000000..a70f3c5e --- /dev/null +++ b/scripts/tests/packaged-document.test.ts @@ -0,0 +1,55 @@ +/** + * Every packaged Markdown document reaches every build that ships it. + * + * The npm build discovers these documents, so it needs no list. `deno compile` + * names files one at a time, and it is named twice — in `deno task build` and in + * the release matrix. A document added beside its module and forgotten in either + * place produces a binary that runs until the moment it looks for its own + * program, which is the failure this refuses to let reach a release. + */ +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { readTextFile } from "@effectionx/fs"; +import type { Operation } from "effection"; +import { readdir } from "node:fs/promises"; +import { until } from "effection"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const ROOT = fileURLToPath(new URL("../../", import.meta.url)); + +/** Every document a package ships, repository-relative. */ +function* packagedDocuments(): Operation { + const found: string[] = []; + for (const member of yield* until(readdir(path.join(ROOT, "packages")))) { + const documents = path.join(ROOT, "packages", member, "src/documents"); + let entries: string[]; + try { + entries = yield* until(readdir(documents, { recursive: true })); + } catch { + continue; + } + for (const entry of entries) { + found.push(`packages/${member}/src/documents/${entry.split(path.sep).join("/")}`); + } + } + return found.sort(); +} + +describe("packaged documents reach every build", () => { + it("is embedded by both compile sites", function* () { + const documents = yield* packagedDocuments(); + // The suite is only meaningful while at least one exists; an empty sweep + // would pass while proving nothing. + expect(documents.length).toBeGreaterThan(0); + + const denoJson = yield* readTextFile(path.join(ROOT, "deno.json")); + const release = yield* readTextFile(path.join(ROOT, ".github/workflows/release.yml")); + const [buildTask = ""] = denoJson.split("\n").filter((line) => line.includes('"build":')); + + for (const document of documents) { + expect(buildTask).toContain(`--include ${document}`); + expect(release).toContain(`--include ${document}`); + } + }); +}); diff --git a/specs/acp-client-spec.md b/specs/acp-client-spec.md index 2d3fa717..dcec3b24 100644 --- a/specs/acp-client-spec.md +++ b/specs/acp-client-spec.md @@ -477,9 +477,9 @@ own session store and one empty working directory per session, both disposable. ## Command-line configuration -`xmd run` configures the agent stack; the options are exclusive to it, and -`xmd test` rejects them, driving agents through the deterministic test-agent -stack instead. +`xmd run` and `xmd prompt` configure the agent stack; the options belong to +those two commands, and `xmd test` rejects them, driving agents through the +deterministic test-agent stack instead. That division also applies to a nested run profile. The outer `xmd test` invocation supplies no live Agent configuration to a child and no contextual @@ -498,7 +498,7 @@ Agent provider from the outer test. The permission options are mutually exclusive. -`xmd run` also takes the three timeout options, which are exclusive to it in the +Both commands also take the three timeout options, which belong to them in the same way: | Option | Establishes | @@ -531,6 +531,83 @@ The default agent resolves in order, each entry overriding the ones above it: The installed provider belongs to the run's `DocumentExecution` scope and closes during its teardown. +### The `xmd prompt` prompt profile + +`xmd prompt` resolves that configuration once and uses it for the prompt command +document that writes the Plan, and — when `--run` asks for one — for the run that +follows. Without `--run` the approved Plan is written rather than executed, so +the second consumer never appears and the resolved configuration is used once. +The provider +name, the default agent with its `DEFAULT_AGENT_NAME` precedence, the permission +mode and the host's own machine-session assembly are settled before any catalog +is built or any document executes, so an unknown provider or an incompatible +pair of permission flags fails first. It is the settled value that reaches both +consumers, not the flags that produced it: `DEFAULT_AGENT_NAME` is read once per +invocation, and authorship and a run of the Plan cannot reach different +conclusions from one command line. + +The prompt profile takes the provider name and the default agent from that +answer, and nothing else. Its ceiling is the host's, assembled for that one +document and not readable from the command line: + +| The profile's provider gets | Stated as | +| --- | --- | +| one host-owned directory per logical session | `agentCwd`, `~/.xmd/prompt/sessions/`, required empty | +| no MCP servers | `mcpServers: []`, an empty set rather than an omission | +| no native tools on a fresh session | `newSessionOptions.allowedTools: []` | +| a private refusal of every native permission request | `permissions: "strict"` | + +`"strict"` answers the request inside the provider: the request is denied, the +turn it belongs to fails, and no public Agent handler is consulted or can +intervene. So `--approve-all`, `--approve-reads` and `--deny-all` cannot widen a +ceiling that has nothing in it; they configure the approved document later. A +provider that cannot establish this ceiling refuses before session +materialization or a turn. + +The command document itself is given no Files, command, service or XMD-mediated +network capability, and the host decides for that whole execution that a failing +`` ends it — so a turn that streamed text and then failed presents +nothing. + +The profile's working directory is derived from the logical session name rather +than shared or freshly made: `~/.xmd/prompt/sessions/`, with the +digest in the path and never the name. A session's key includes the directory it +lives in, so a shared location would put two conversations in one ambient +directory while a fresh one would leave `--session` unable to name a conversation +that already exists. It is created empty and required to be empty before the +provider is constructed or a session is materialized; a non-empty one is a +terminal refusal rather than something to clean. Nothing this profile grants can +write there. + +An explicitly named session keeps its directory afterwards, because that is what +the next invocation derives the same session identity from, and no cleanup +applies to it. An invocation-unique default keeps nothing: its release is +registered before the directory is created, and once the profile has torn down, +on every ending, exactly one cleanup is attempted — the leaf is removed +non-recursively if it is still the empty directory that was handed over, and if +it has gained content or disappeared it is left as found and the command fails +terminally. Which applies is a trusted host value — +whether `--session` was written — and where the directories live is a host +dependency no caller or document can select. + +Only the profile's provider receives `newSessionOptions.systemPrompt`. It carries +one fixed statement — that an answer belongs to the message that asked for it — +and nothing else. Which shape any particular message wants, a Plan or an +explanation of why there is not one, is that message's own business, and every +message is the command document's to send in its own turns +([`xmd prompt`](./prompt-command-spec.md)). ACPX applies those options when it +creates a session and ignores them when it reuses a record. Without `--session` +the command places a name unique to the invocation, so each invocation is a fresh +conversation created under those instructions; `--session ` selects an +existing one under ordinary continuation semantics. + +That document's scope closes before the final admission and before whichever +result the caller asked for: the source on stdout, the `--output` file, the run, +or the file followed by the run. A teardown failure fails the command and no later phase happens. The +executed program is an ordinary `xmd run` document with its own root provider and +its own lifetime: it inherits neither the assistant session nor its instruction +layer. It exists only under `--run`. + ### Availability Installing a provider starts nothing. The **first** Agent API use validates the diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index fe2cbb79..990a9c66 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -8237,6 +8237,28 @@ identity, and no temporary file is created: the text is captured inside the durable root import, so the journal holds it and a replay restores it without reading anything. +Text a host generated reports `` on the same terms. `xmd prompt` runs a +packaged first-party command document that turns a Prompt into a Plan — asking a +coding agent for a complete root, having the host check it, and having a person +approve it; the host then validates the returned bytes again. What happens to +them is the caller's choice: by default the exact source goes to stdout, and +nothing runs; `--output` writes those bytes to a file instead; and only `--run` +executes `retainedSource("", source)`, with `--output --run` writing the +file first. So the identity says where the bytes came from, and a source position +reads `(:5:1)`. It is a deliberate identity rather than a path nobody +could read back, and it is the only thing that differs: an approved Plan that is +run goes through the ordinary supplied-source path, renders and returns exactly +as any other root, resolves the contextual working directory for every relative +operation, and creates its journal only when that execution starts — so the modes +that write a Plan create none. + +That command document is the other root the command executes, under its own +stable internal identity ``, on an invocation-owned in-memory +durable stream that is discarded when authorship is over. So nothing about +generation, repair or approval is journaled or replayed, and the two executions +share no scope. The command's complete contract is +[`xmd prompt`](./prompt-command-spec.md). + `inspectDocument(root)` loads and validates the root definition and returns what it declares — without executing the document or creating a journal: @@ -10005,7 +10027,7 @@ itself, so an execution starting anywhere fails the row. Defined in §5.3. | SX4–SX6 | Renderers take a value | Both formats render from a supplied catalog with the filesystem refusing every call, twice with identical bytes, under the fixed category headings; every table cell is escaped, a prop name holding a pipe included | | SX7/SX8 | Includes | Repeated values select in caller order and replace the defaults; absent, the defaults apply | | SX9 | Failure | An unusable include exits 1, reports on stderr and prints no catalog | -| SX10/SX11 | Formats | Markdown by default, version-1 JSON with `--json`, and no `prompt` command | +| SX10/SX11 | Formats | Markdown by default, version-1 JSON with `--json`; the catalog is inspection, and `xmd prompt` is the command that writes with the same structured value | | SX12 | A package tree | Bare `xmd syntax` succeeds with the default includes in a repository whose `node_modules` holds directory links | ### Tier SM — `xmd syntax` end to end @@ -10019,6 +10041,29 @@ One checked-in Markdown suite runs the real command against a fixture directory. | SM6/SM7 | Nothing runs | A `.ts` fixture that throws at top level is origin-only and the command still succeeds; a fixture whose body writes a file leaves no file | | SM8–SM11 | Discovery | Direct, dotted and index paths; the two formats agree; repeated includes replace the defaults in caller order; the built-in category holds the run profile including `` | +### Tier PR — The `xmd prompt` command + +One root execution on every invocation — the packaged prompt command document, +which owns the visible generation, repair, review and failure policy — and a +second only under `--run`: the Plan it returned, which runs as any supplied root +does, behind a complete scope boundary. Without `--run` the approved Plan is the +result rather than a second execution: stdout by default, or an exclusively +created `--output` file. Defined in +[`xmd prompt`](./prompt-command-spec.md), whose acceptance table this points at +rather than restating. The ACPX runtime is a scriptable fake, the review provider +is a scripted `Elicitation` handler, and the contextual working directory is a +temporary one — no live agent, browser or network appears in this evidence, and +every refusal is proven by the phase tripwires that stayed at zero. + +| # | Test | Verify | +|---|------|--------| +| C1 | Fixed grammar and help | One Prompt preserved byte for byte; missing, repeated, empty and whitespace-only Prompts refused; individual options after the Prompt and aggregate props before it; a built-in option never read as a generated property's value; `--session` named or refused; every run-only flag refused without `--run` before any effect; generic help with no effects, explaining `--output` and `--run` | +| C2–C3 | The packaged document is the policy | The command executes the checked-in Markdown value root under ``, and the turn text is that document's own words; generation, repair, review, revision, approval, stopping, exhaustion and the final explanation are Markdown under visible headings, every Plan-producing turn states the complete Plan requirements for itself, `` stays one turn, and what a person reads says each thing once however many rounds it took | +| C4–C6 | Session and ceiling | One enclosing Session carries every turn, defaults differ per invocation and `--session` supplies the exact override; the prompt profile gives the assistant an empty host-owned directory, no MCP servers, no native tools and a private strict denial no permission flag widens; a draft is data until the approved Plan is delivered, and no draft effect happens before that | +| C7–C9 | Classification, bounds and presentation | Draft defects return structured facts and caller defects escape the checker; one base draft plus three repairs, and ten presentations with no revision on the last; arbitrary source cannot close ``, the review schemas expose exactly the friendly choices for each round and state, and stopping, exhaustion and the explanation ending each reach their own authored `` | +| C10–C13 | Admission and lifetime | The host revalidates after the command document has completely torn down and resolves props for the exact returned bytes; those bytes reach exactly one of stdout, an exclusively created `--output` file, a run, or a file and then a run; a journal exists only when `--run` begins and holds only the Plan's events; cancellation and teardown failure settle before anything later begins | +| C14–C15 | Result | The shipped generation, repair and revision instructions each carry the complete titled-Plan rule, and a titled Plan of prose interleaved with components survives approval byte for byte into stdout, a file and a run alike; a run reports `` and behaves as `xmd run` does | + ### Tier WB — The workflow component bundle in core Provider-neutral, and portable across every runtime. Defined in §5.3 and §8.1. diff --git a/specs/prompt-command-spec.md b/specs/prompt-command-spec.md new file mode 100644 index 00000000..e35aa57c --- /dev/null +++ b/specs/prompt-command-spec.md @@ -0,0 +1,635 @@ +# The `xmd prompt` command + +`xmd prompt` turns your **Prompt** — the request you typed, in ordinary language +— into a **Plan**: an Executable Markdown document that states what you asked +for in ordinary language and places the components that perform the work +alongside those words. + +```console +$ xmd prompt "ask me for my age and write the result to a file" +``` + +A Plan is not merely executable code. It preserves the Prompt's intent as +reader-facing prose interleaved with the components that fulfil it, beginning +with one descriptive level-one heading, so the request above should produce a +document shaped like this: + +```markdown +# Ask for and save your age + +Ask me for my age. + + + +Write it to a file. + +{answer.age} +``` + +and an execution organized by the same narrative: the title, the first sentence, +the interaction, the second sentence, the file. The Prompt need not be copied +literally — it may be divided, clarified and rewritten into natural prose — but +the descriptive title, every requested outcome, their meaningful ordering, and +their relationship to the components performing them are preserved. Every +Plan-producing turn — the first draft, each repair and each revision — states +that whole requirement for itself, so a replacement can add or correct a title +rather than only carry one forward. + +That command prints the approved Plan and runs nothing. `--output` writes it to a +file instead, `--run` executes it, and the two together write it and then run +it. + +The command writes no policy of its own. It always executes one root document: +the **prompt command document**, a checked-in first-party Markdown value root +that implements this conversion and its review workflow — what the coding agent +is asked, how many drafts may be repaired, what you are shown, and what happens +when you approve nothing. It executes a second root only under `--run`: the Plan +that document returned, through the same path `xmd run` uses for a supplied one, +with a complete scope boundary between the two. The prompt command document is +not itself a Plan. There is no second execution model, no second props model and +no second journal. + +## The flow + +```text +fixed command preflight + -> build the run-profile syntax catalog + -> execute the exact packaged prompt command document + -> one Session + -> generate, check, repair, review, revise, approve, explain or fail + -> Return the exact approved Plan source + -> await that execution and provider teardown + -> validate the returned source again + -> --output: exclusively create the file with the exact bytes + -> --run: execute retainedSource("", source) through the ordinary run + path; otherwise, with no --output, write the exact bytes to stdout +``` + +Each phase hands the next one a value. No phase after the first failure begins, +so a refused command line reaches no catalog, a failed turn reaches no review, +and a review that stopped reaches no stdout, no file and no run. + +Writing a Plan is a conversation, and a conversation is not a run: the prompt +command document runs on an invocation-owned in-memory durable stream that is +never written to `--journal`, persisted, reused or replayed. Only the approved +Plan's execution owns `--journal`. + +## Command grammar + +```console +xmd prompt [options] [--props- ]… +``` + +The request is one positional argument and is text for the agent, never a path. +It must hold at least one non-whitespace character; that is the only test +applied to it. Its original text is preserved and sent byte for byte, so leading +and trailing whitespace survives. + +No request, more than one request, an empty string and a whitespace-only string +each fail the invocation. There is no stdin form and no editor form: a request +nobody wrote is never guessed, and no token is silently dropped. + +`--` ends option parsing, so a request that begins with `-` is written after it: + +```console +xmd prompt -- "--this is the request" +``` + +`xmd prompt` takes the complete `xmd run -e` execution flag set — `--include`, +`--verbose`, `--journal`/`-j`, `--raw`, `--agent-provider`, `--default-agent`, +`--approve-all`, `--approve-reads`, `--deny-all`, `--no-secret-detection`, and +the three timeout options — plus `--output`, `--run` and `--session`. + +`--include`, `--agent-provider`, `--default-agent`, `--session` and `--timeout` +are always in use: they build the catalog, settle who writes the Plan, name the +conversation, admit properties and bound the whole command. + +Every other execution flag configures **running** a Plan and nothing else — +`--journal`/`-j`, `--raw`, `--verbose`/`-V`, `--timeout-exec`, `--timeout-fetch`, +the three permission flags, and `--secret-detection`/`--no-secret-detection`. +Without `--run` nothing runs for them to configure, so writing one is refused in +fixed preflight rather than accepted and ignored: + +```console +$ xmd prompt "…" --journal trace.jsonl +--journal configures running the Plan, and without --run this command writes the +Plan instead of running it — add --run, or drop --journal +``` + +The permission flags configure only the approved Plan; authorship does not +inherit the final run's permission mode, and no permission flag widens the +prompt profile's ceiling below. + +An option this command does not define is refused by name in the same preflight, +because the ordinary parser stops at the first option it does not recognize and +drops the rest — accepting a command line nobody honoured. `--save` is named in +particular: it was replaced by `--output` before release, so there is no alias, +and the refusal says where an approved Plan goes now. + +`--run` is a switch, and every valued spelling of it — `--run=false`, +`--run=true`, `--run=` — is refused there too: + +```console +$ xmd prompt "…" --run=false +--run does not take a value — write --run to execute the Plan or leave it out to +write the Plan +``` + +An option's name is read up to its first `=`, so such a token arrives under the +name of the switch. Taken as the switch it would establish the opposite of what +was written and satisfy the run-only gate on the way, which is how +`--run=false --journal ` came to be accepted by a command that then created +no journal. It establishes nothing and reaches no later phase. + +`-e`/`--eval` stays exclusive to `xmd run`. A prompt supplies a request, not a +document. Supplying one anyway is refused in the command's own preflight, with +`unrecognized option for xmd prompt: --eval — inline documents are exclusive to +xmd run`, before the catalog, the command document, the review, the file, the +journal or any execution exists. + +Those options are resolved into one Agent configuration once per invocation — +one `--agent-provider`, one `--default-agent` or `DEFAULT_AGENT_NAME`, one +permission mode — and that settled answer is what both the prompt profile and +the approved Plan's installation are configured from. Incompatible permission +flags and an unknown provider fail there, before the catalog is built. + +### The approved Plan is the result + +The Plan is what the command produces, and where it goes is the caller's choice: + +| Invocation | Result | +| --- | --- | +| `xmd prompt ""` | the exact approved source on **stdout**; nothing runs | +| `xmd prompt "" --output ` | the exact bytes in ``; no source on stdout; nothing runs | +| `xmd prompt "" --run` | the Plan runs; no source is printed, and stdout belongs to the Plan's own output | +| `xmd prompt "" --output --run` | the file is created first, and only a successful write is followed by the run | + +Stdout carries the approved source and nothing else: no fence, no label, no +delimiter and no newline this command added. What a caller reads is what the +coding agent wrote, so it can be piped into a file, a diff or another program. + +`--output ` is resolved against the contextual working directory and +**created exclusively**: an existing path is left exactly as it is, the command +fails, and nothing runs. There is no check-then-write — the exclusive create *is* +the check. The file holds the approved Plan and nothing else: no problems, no +decision, no wrapper. Without `--output`, no file is created anywhere. + +### `--session ` + +Use this logical assistant session instead of one unique to the invocation. The +name must be non-empty; `--session` with an empty value is refused by fixed +grammar rather than read as absent, so a caller who asked for a named session +never falls back to the generated one by accident. + +Ordinary provider session continuation applies when the configured provider +already holds that name. The prompt command document still supplies the current +request and the current catalog in this invocation's initial turn. + +### Help + +```console +xmd prompt --help +``` + +Help needs no request. It describes the Prompt, `--output`, `--run`, +`--session`, the aggregate `--props`/`XMD_PROPS` sources, that Plan-declared +individual options follow the Prompt, that stdout is where an approved Plan goes +by default, that the run-only flags are refused without `--run`, and that the +permission flags configure the approved Plan. It describes no individual property, +because the document that would declare one does not exist yet — answering +otherwise would mean generating a document in order to describe one. + +Help reads no catalog, contacts no provider, places no session, asks nobody +anything, creates no file and runs nothing. + +## The packaged prompt command document + +The host executes one immutable packaged Markdown value root, +`packages/cli/src/documents/prompt-command.md`, under the stable internal source +identity ``. Its declared return schema is exactly +`{ type: "string" }`. It is located from the CLI module's own URL — never from +the working directory and never through the component search path — so the same +policy is found whatever directory a person stands in, and no repository file can +answer for it ([release process](./release-process-spec.md)). + +The host supplies three fixed internal inputs as that root's props: + +- `request` — the original request text; +- `syntax` — the rendered syntax catalog for the current run profile and ordered + `--include` values; and +- `session` — the resolved logical assistant-session name. + +They are not Plan root props and consume none of the approved Plan's property +sources. + +The document contains one enclosing `` expansion whose body holds every +initial, repair and human-revision ``. A loop inside that one Session +creates no second placement; sibling Sessions are not used, because sibling +placements stay distinct even when their authored names match. + +The host owns the provider instruction layer and the Agent ceiling. The Markdown +document owns the text of each generation, repair and revision request: the +initial prompt preserves the Prompt, includes the host's catalog, and asks for +one complete replacement root as source only — written as a Plan, with every +requested outcome kept as reader-facing prose and each component placed +immediately after the sentences describing the action it performs. That +authorship rule is repeated in every repair and revision request, so the +narrative survives a draft being replaced. + +`` remains one Agent turn. It gains no hidden repair, retry, review or +approval behaviour. + +## The prompt profile + +The prompt profile is the trusted-host assembly used only for that exact root. It +supplies the document's inputs, a constrained Agent provider, Elicitation, the +fixed first-party components and the host-declared draft validator. It uses no +repository component search — that execution's include list is empty — and +exposes no custom root. + +### Agent authority under the prompt profile + +The assistant that writes a Plan is assembled separately from the final run +provider: + +- one host-owned directory dedicated to the logical prompt session, created + empty, required to be empty on the way in, and — for an invocation-unique + default session — handed back non-recursively on the way out; +- no additional directories; +- no MCP servers — stated as an empty set, not omitted; +- an empty requested native-tool allowlist on a fresh session; +- strict denial of every native permission request, answered inside the provider + without consulting an authored approval scope; and +- no Files, command, service or XMD-mediated network capability for the command + document itself. + +The provider may use its own transport to perform the model turn; that does not +grant the Agent a native network tool. + +`--approve-all`, `--approve-reads` and `--deny-all` do not change this ceiling. +They apply to the approved Plan later. A provider that cannot establish this +ceiling refuses before session materialization or a turn; there is no silent +downgrade. + +The profile's working directory is +`~/.xmd/prompt/sessions/`. It is dedicated to that +one logical session rather than shared by every invocation, so two conversations +never see one ambient directory. The leaf is the digest and never the name +itself: a logical session name is a caller's string, and a caller's string that +becomes a path is one that can escape a path. + +The digest also carries the identity `--session` needs. A generated +invocation-unique name reaches a location nothing else does; the same explicit +name reaches the same one, and because a session's key includes the directory it +lives in, that is what lets the provider continue the session it established. + +The directory is created empty and **required to be empty** before the provider +is constructed or a session is materialized. A non-empty one is a terminal, +actionable refusal naming the path: nothing is deleted or cleaned, because +whatever is in there was put there by something this host did not authorize. +Nothing this profile grants can write there, so what is created empty stays +empty. + +The two kinds of session then have different directory lifetimes, because only +one of them has an identity worth keeping: + +- an **invocation-unique default** directory is the command's own, held as a + scope-owned resource: the release is registered *before* the first filesystem + call that could create it, so a leaf this invocation made and then failed on is + still a leaf this invocation hands back. Exactly one cleanup is attempted, after + the command document and every provider, Prompt task and Elicitation resource + inside it has torn down, and it settles before final validation and every way a + Plan could leave the command. Success, stopping, a failed turn and cancellation all reach + it. Once the directory has been established, it has exactly two outcomes: + - **still empty** — that exact leaf is removed, non-recursively; + - **anything else** — the directory and whatever is in it are preserved and the + command fails terminally. A leaf that has gained content, and one that has + disappeared, are both interference: something acted on a directory this host + authorized nothing to touch. Neither is a warning and neither is a silent + skip, and no final admission, stdout, file or execution follows. + + A directory establishment never handed over — refused as non-empty, or never + created at all — is a different question, already answered by what + establishment reported. The release then removes an empty leaf it did create + and otherwise leaves both the report and the directory's contents alone. +- an **explicitly named** directory survives the invocation and is not subject to + that cleanup at all, because a later `--session ` derives the same + location and therefore the same ACPX session identity from it. It is required + empty again on the way in every time, and its contents are never cleaned or + overwritten. + +Which of the two applies is a trusted host value — whether the caller wrote +`--session` — not something read back out of the name, and the command document +is told neither the directory nor which kind it is. + +Where those directories live is a host dependency as well. No flag, environment +variable, document prop or replaceable context selects it; production uses its +own default, and a harness that owns a temporary tree is given that tree +directly. + +The approved Plan later receives the caller-selected ordinary run Agent and +permission configuration. It inherits neither the assistant Session nor its +instruction layer. + +### The assistant Session + +Without `--session`, the host generates a logical name unique to the invocation. +The exact name is supplied to the command document, and its one enclosing +`` materializes only at the first consuming ``. Every turn +within this invocation uses that one Session. + +A turn produces a draft only from its complete successful close value. The host +decides, for the whole command document execution, that a failing `` ends +it: a failed, cancelled, unavailable or protocol-invalid turn discards its +partial text and reaches no human review, result or final execution, and the +document cannot opt out of that. + +That execution closes after approval or failure. The host observes its result +only after every Prompt task, Agent provider resource, Elicitation resource and +other child has completed teardown. A teardown failure wins over a selected Plan +and prevents final validation and every way a Plan could leave the command. + +### A draft is data + +An Agent reply is an inert string while the Plan is being written. The command +document may bind it, pass it to the validator, serialize its problems, present +it with `` and return it. It never evaluates the draft and never +dynamically imports it. Only after approval, teardown and final host validation +may those exact bytes enter ordinary execution. + +## Host-declared draft validation + +The prompt profile declares one internal value component to the execution: + +```md + +``` + +Canonical execution supplies its invocation identity, so repository resolution +cannot replace it. Its result is a closed candidate assessment: either +`{ valid: true, diagnostics: {} }`, or `{ valid: false, diagnostics }` where +`diagnostics` carries the complete versioned `DocumentValidation` when core +produced one and the structured generated-binding diagnostic when property +binding failed. + +The component performs no candidate execution. For each source it: + +1. validates the supplied root declaration without executing it; +2. once a usable root props schema exists, derives that schema's bindings; +3. checks every frozen supplied individual-option signature before consuming + tokens; +4. resolves the original CLI and invocation-environment property sources under + that schema; and +5. calls `validateDocument()` with those exact props, the caller's includes and + the run profile's identity-component declarations. + +## Generated document properties + +An `xmd prompt` invocation resolves root props exactly as `xmd run` does — same +sources, same precedence, same decoding ([Root Document +Props](./root-document-props-spec.md)) — with one difference: the schema comes +from the *candidate*, and there may be several candidates. + +```console +xmd prompt "greet someone" --props-name Ada --props-loud +``` + +The aggregate `--props` may be written before the request, because its meaning +never depends on a document. An individual `--props-*` option before the request +fails in preflight, before any catalog, Agent, elicitation, file, journal or +document operation. + +The original argv is the command line source for every candidate. `XMD_PROPS` +and the candidate's `XMD_PROPS_*` variables are read through the contextual +runtime environment. No resolved props object is carried from one candidate to +the next, or from any candidate to the approved bytes. + +### Frozen signatures + +An individual option's **signature** is its generated option name, its token +arity — a bare switch or one value — and its accumulation behaviour — scalar +last-wins or repeated array. The first candidate that successfully binds a +supplied option freezes that option's signature. + +Before a later candidate extracts anything, every frozen supplied option is +compared with that candidate's binding. A removed option, a changed arity or a +changed accumulation is a terminal caller-source failure. The comparison happens +*before* extraction, so a switch that became a value option cannot reach forward +and read the `--raw`, `--include` or other built-in option written after it. + +A built-in option is never read as a generated property's value. A value that +really begins with `-` is written in the unambiguous inline form: + +```console +xmd prompt "" --props-name=-Ada +``` + +### Candidate failures and caller failures + +These are **repairable candidate failures**. The agent authored them, so the +assessment answers `valid: false` and the command document may ask again: + +- source, frontmatter, target, root-props and return-declaration diagnostics; +- a generated binding-name collision — two declared properties producing one + option or one environment variable; +- missing required root props and every other `props-invalid` result; and +- every other definite document diagnostic. + +These are **terminal caller-source failures**. The caller wrote them, so they +raise out of the validator and end the command document immediately, with no +repair turn, no presentation, no approval, no stdout, no file, no final journal +and no final execution: + +- a supplied individual option the usable candidate schema does not declare; +- malformed aggregate CLI or environment JSON; +- an individual CLI or environment value the candidate schema cannot decode; +- an extra positional the candidate's binding arity exposes; and +- a later candidate removing a frozen option or changing its signature. + +The split is about authorship, not severity. A collision is repairable because +the candidate authored both colliding names; an undeclared option is terminal +because only the caller supplied it. The command document has no way to catch a +terminal failure and no way to recategorize it as draft feedback. + +An opaque `not-statically-checkable` invocation is not a diagnostic and does not +by itself make a candidate invalid. + +## The policy the prompt command document owns + +The following is the shipped program's behaviour, not the host's. It is stated +here because it is what a caller sees; it is changed by editing +`prompt-command.md`, and nothing in TypeScript decides it. Prose quality is that +document's instructions and your review, never a hidden TypeScript validation +rule: `` reports structural facts and executes nothing. + +**A complete titled Plan.** Every Plan begins with one descriptive level-one +Markdown heading, as the first body content after optional frontmatter, naming +what the Plan produces. The generation, repair and revision instructions each +require it, together with the Prompt's sequence of readable steps in a +meaningful order and each component beside the step it performs, so the +structure survives every replacement. It is an authorship and human-review +requirement: `` does not enforce it, and a titleless Plan is +something you send back rather than something the checker refuses. + +**Automatic repair.** The initial draft and every human-requested revision each +start a fresh repair budget: the base draft is attempt one, at most three repair +turns may replace it, every repair prompt carries the complete structured +diagnostics, and every answer must be another complete replacement Plan. The +fourth draft with problems is repair-exhausted and goes to human review with its +diagnostics. No fence is stripped, no Markdown substring is extracted and no +patch is applied. + +**Human review.** At most ten draft presentations: the initial review plus at +most nine revisions. The choices are the words shown, and they are the values the +provider answers with — there is no internal spelling behind them: + +| Round | Draft | Choices | +| --- | --- | --- | +| 1–9 | passed its check | **Approve**, **Request changes**, **Stop** | +| 1–9 | problems remain | **Request changes**, **Stop** | +| 10 | passed its check | **Approve**, **Stop** | +| 10 | problems remain | **Explain what went wrong**, **Stop** | + +**Request changes** requires non-empty feedback, sends one complete-replacement +request through the same enclosing Session, and resets the three-turn repair +budget. + +**The explanation turn.** **Explain what went wrong** is offered only on a tenth +draft that still has problems, and makes exactly one more `` in the same +enclosing Session. The Session already holds the original Prompt, the catalog, +every draft, every earlier diagnostic and every revision request, so nothing is +resent: the turn carries only the final diagnostics, which were produced after +the agent's last draft and have not appeared in the conversation. It asks for a +brief explanation and explicitly not another Plan. + +That turn is not a draft, a repair, a revision or a review round; it cannot +reopen the ten-draft limit; its answer is inert text that is never interpreted as +XMD; and it is reported to you and then ends the command with no approved +source, file, journal or execution. It is subject to the host's failed-turn +policy like every other turn, so a turn that fails ends the command immediately. + +**Presentation.** The review message presents the exact draft with ``, +whose fence is longer than every backtick run the draft holds, so draft text +cannot close it and is never interpreted. An invalid draft is followed by its +complete JSON problems, serialized and captured by ``. Prose outside +`` addresses you; text inside `` instructs the assistant. + +**Failure.** **Approve** selects the draft, and the branch after the Session +returns its source unchanged. Stopping reaches `` with one of two +authored messages, and which one depends on whether an approvable Plan ever +existed: + +- **Stop** on a tenth draft that still has problems is **exhaustion** — ten + drafts were reviewed and none was approved; +- **Explain what went wrong**, offered only there, makes one more turn (below) + and then ends the same way; +- every other **Stop**, including one on a tenth draft that could have been + approved, is the ordinary ending: you decided to stop. + +The branch after the Session is only an unexpected-no-decision fallback and says +so; exhaustion is decided inside review, not duplicated there. Failure is authored in Markdown +rather than hidden in the host or represented by a missing-`` accident. + +The command document's rendered output is not command output. Everything you see +while a Plan is written reaches you through Elicitation, and that document's +successful public result is the approved Plan source and nothing else. + +## Final admission and the result + +After the command document has completely torn down, the host treats the +returned string as untrusted again. It repeats the draft check and property resolution using the exact returned +source, the original raw CLI and invocation-environment property sources, the +individual-option signatures frozen while it was written, the caller's ordered +includes and the ordinary run profile declarations. + +A final caller-source failure or document validation failure exits non-zero +before any result at all: no stdout, no file and no execution. It does not +re-enter the command document or ask for a repair. The final resolved props +belong to those exact source bytes; no props object from an earlier draft is +reused. + +`--output` then exclusively creates the target with those exact bytes. With +`--run`, and only then, the host executes `retainedSource("", source)` +through the ordinary supplied-source run path, with the same includes, output, +value result, secret detection, Agent configuration, permission mode and +timeouts `xmd run -e` uses. No temporary Markdown file is created. Without +`--run` and without `--output`, the exact bytes go to stdout. + +The `` identity affects positions and diagnostics only. The contextual +working directory still resolves relative filesystem operations, repository +components and includes. The executed program receives a fresh ordinary document +Agent provider and inherits neither the assistant Session nor its instruction +layer. + +`--journal` is created when the final execution starts, and only then — so +source-only and `--output` invocations create no journal at all, and neither does +any ending before admission. That journal contains no command document, assistant +Session, Agent turn, draft check, repair or human-review event. Rendered output, a value root's JSON +result, runtime failure reporting and exit codes are byte-for-byte ordinary +`xmd run` behaviour — including a failing `` boundary, which is reported +under its own `tests failed:` heading rather than as a bare message. A runtime +failure does not send the command back to authorship; `--output` has already +completed, and the Plan is there to hand-edit. + +## Timeouts + +`--timeout` bounds the whole command: preflight, catalog construction, the +command document's execution, Elicitation, its teardown, final validation, and +whichever result the caller asked for — stdout, the file, the run, or the file +and then the run. Expiry is Effection cancellation, so structured teardown +completes before the failure is reported. A teardown failure prevents every later phase. + +`--timeout-exec` and `--timeout-fetch` configure the final document's effects +only, exactly as under `xmd run`. Nothing bounds an authoring turn but the +command deadline. + +## Failures + +Every failure below exits non-zero, and each one stops the phases after it: + +| Failure | Reaches | +| --- | --- | +| a malformed command line, an unknown option, `--save`, or a valued `--run=…` | nothing | +| incompatible permission flags or an unknown `--agent-provider` | nothing | +| a catalog an include makes unreadable | no command document | +| a provider that cannot establish the prompt profile's ceiling | no session, no turn | +| a turn that did not complete | no review, stdout, file or run | +| a terminal caller-source failure | no repair, review, stdout, file or run | +| the command document's authored `` — stopping, exhaustion, or the ending after an explanation | no stdout, file or run | +| command document teardown | no final validation, stdout, file or run | +| final validation of the approved bytes | no stdout, file or run | +| an `--output` path that exists, or a write that fails | no stdout, no run | +| the Plan's own runtime failure under `--run` | nothing after it; the `--output` file stands | + +## Acceptance + +Tier PR. The evidence lives in `packages/cli/tests/prompt-args.test.ts` (fixed +grammar and signatures), `packages/cli/tests/prompt-command-document.test.ts` +(the packaged document executed as itself), `packages/cli/tests/prompt.test.ts` +(the host and the packaged document writing a Plan together) and +`packages/cli/tests/prompt-cli.test.ts` (the command lifecycle, filesystem, +journal and execution). + +The ACPX runtime is a scriptable fake, the review provider is a scripted +`Elicitation` handler, and the contextual working directory is a temporary one: +no live agent, browser or network appears in this evidence. Every refusal is +proven by the phase tripwires that stayed at zero rather than by output nobody +produced. + +| # | Criterion | Required observation | +| --- | --- | --- | +| C1 | Fixed grammar and help | Prompt cardinality, individual-property ordering, aggregate props before the Prompt, `--session` including its empty-value refusal, run-only flags refused without `--run` before any authorship or filesystem effect, and effect-free generic help that explains `--output` and `--run` | +| C2 | Exact packaged root | The command executes the checked-in Markdown value root under ``; the turn text is that document's own words, and no TypeScript authorship loop or custom root chooses policy | +| C3 | Visible policy | Generation, three-turn repair, ten-round review, revision, approval, stopping, exhaustion and the explanation turn are present in Markdown under visible headings; the generation, repair and revision instructions each require the descriptive title and the steps-beside-components structure; `` remains one turn | +| C4 | One Session | One enclosing Session expansion carries every turn; two default invocations get different profile directories and session keys, two `--session` invocations get the same directory and key with the raw name absent from the path, and two named invocations sharing one ACPX store continue the established record rather than placing a second | +| C5 | Prompt profile ceiling | This session's own host-owned directory, empty while the command document runs, no MCP servers, no native tools, strict private denial, no Files/command/network capability for the command document, and final-run permission flags that cannot widen any of it; pre-existing content in that directory refuses before any provider, session, turn, review, result or execution and is left untouched | +| C6 | Draft inertness | A draft is only data while the Plan is written, and no draft effect occurs before the final execution | +| C7 | Validation classification | Candidate failures return structured facts; caller-source failures escape immediately; frozen signatures are checked before token extraction | +| C8 | Bounds | One base plus three automatic repairs per draft, and no more than ten human presentations, with no revision offered on the last | +| C9 | Safe presentation and authored failure | Arbitrary source cannot close ``; the review schemas expose exactly the friendly choices for each round-and-state; an ordinary **Stop**, a tenth-round exhaustion and the explanation ending reach their distinct authored `` messages, and the closing fallback says no approval was reached rather than repeating exhaustion | +| C10 | Final gate | The host revalidates after the command document has completely torn down — a component removed during that teardown makes the unchanged approved bytes fail admission, with no stdout, file, journal or execution — and resolves props for the exact returned bytes | +| C11 | Exact bytes | Approval, stdout, the exclusive `--output` create and `` execution each receive the Agent close value without rewriting or fence removal; `--output --run` writes before it runs, and an existing destination prevents the run | +| C12 | Journal separation | Authorship uses only disposable in-memory history; a journal exists only when `--run` begins the final execution, and it begins with the approved Plan | +| C13 | Lifetime | Cancellation and every teardown failure settle before final validation, stdout, the file or execution; a default session's directory is empty while its turn runs and gone after teardown on success, stopping, a failed turn and cancellation alike; and one that gained content or vanished under the conversation is preserved as found while the command fails terminally, with no admission, result, journal or execution after it | +| C14 | Narrative preservation | The shipped generation, repair and revision instructions carry the narrative-plus-components rule, the assistant receives it with the request and the catalog, and a scripted Plan of prose interleaved with components returns byte for byte after approval | +| C15 | Ordinary run | Under `--run` the approved source keeps normal cwd, includes, props, output/value, permission, timeout and failure behaviour, and an `--output` file written before it stands | diff --git a/specs/release-process-spec.md b/specs/release-process-spec.md index a5a83ccc..005ede27 100644 --- a/specs/release-process-spec.md +++ b/specs/release-process-spec.md @@ -92,7 +92,8 @@ documents at the revision it checks. prerelease marker — so a forgotten bump is visible where the release was made, then refuses to build. On a valid tag it compiles `packages/cli/src/compiled.ts` per target with - `--include packages/code-review-agent` and attaches the binaries and + `--include packages/code-review-agent --include packages/cli/src/documents/prompt-command.md` + and attaches the binaries and sha256 checksums to the tag's GitHub Release. That module is the compiled-binary entrypoint: it installs the `API.Env.command` adapter that relaunches the binary as itself, which a source entrypoint cannot do. Between @@ -543,3 +544,50 @@ A publishable `@executablemd/web` is an atomic configuration state: public inclusion of `generated/client-bundle.ts` in both published artifacts. The configuration elements change atomically; the package is never published without its browser asset and never published while private. + +## 9. Packaged documents (`@executablemd/cli`) + +A document-backed command executes first-party Markdown through the ordinary XMD +engine rather than a TypeScript policy. A package declares which Markdown it +ships by putting it in `src/documents/`; every other Markdown under `src/` — +test documents, scenario fixtures — stays out of the product. `xmd prompt` is +the first such command, and `packages/cli/src/documents/prompt-command.md` is +the document that implements it: the checked-in Markdown is the deployed +artifact and the single source of truth, not a generated string mirror of one. + +The command locates it from its own module URL — never from the contextual +working directory, and never through the component search path. Both are +answerable by whatever directory a person is standing in, and which policy the +command runs is not a thing a repository file may decide. + +Every build therefore keeps the asset beside its module, at the same relative +path: + +- **source checkout** — the file as committed; +- **`deno compile`** — embedded by `--include packages/cli/src/documents/prompt-command.md`, + in `deno task build` and in `release.yml`'s matrix compile; +- **npm (dnt)** — copied by `scripts/build-npm.ts`, which copies each package's + `src/documents/` into `esm/src/documents/`, preserving relative location. dnt + emits the module graph and nothing else, so an asset no TypeScript imports is + absent from the published package unless the build copies it. That failure is + invisible under Deno and reaches only Node and Bun. + +A missing asset fails loudly, naming the path it looked at, rather than +selecting different behavior. + +The checks that hold this together, each proving a different build: + +- `packages/cli/tests/packaged-document.test.ts` reads the document from a + temporary working directory and compares it to the committed bytes. It runs + under Deno, Node and Bun, which is what makes it evidence rather than one + runtime's opinion. +- `scripts/tests/cli-npm-bin.test.ts` builds the real package and asserts + `esm/src/documents/prompt-command.md` is byte-identical to the source. +- `scripts/tests/packaged-document.test.ts` holds the two `deno compile` sites + to the documents that exist, because that list is the one thing no build + discovers for itself. + +Adding another packaged document needs no npm-build change — `build-npm.ts` +copies the directory. Only `deno compile` names files individually, so a new +document must be added to `deno task build` and to `release.yml`, which +`scripts/tests/packaged-document.test.ts` enforces. diff --git a/specs/root-document-props-spec.md b/specs/root-document-props-spec.md index 62f8d963..dab78b1b 100644 --- a/specs/root-document-props-spec.md +++ b/specs/root-document-props-spec.md @@ -354,9 +354,73 @@ than another spelling of the same one. ## Command Scope -Root document props belong to `xmd run`, and so does the inline root document. -`xmd test` accepts neither `--props`, `--props-*`, `XMD_PROPS`, `XMD_PROPS_*`, -nor `--eval`/`-e`. +Root document props belong to `xmd run` and `xmd prompt`. The inline root +document belongs to `xmd run` alone. `xmd test` accepts neither `--props`, +`--props-*`, `XMD_PROPS`, `XMD_PROPS_*`, nor `--eval`/`-e`. + +## Generated documents + +`xmd prompt` resolves root props from the same sources, in the same order, with +the same decoding ([`xmd prompt`](./prompt-command-spec.md)). What differs is +where the schema comes from: the document is written by an agent while the +command runs, and one invocation may see several drafts of it. + +That command executes two roots. The first is the packaged prompt command +document, whose own props — the request, the rendered syntax catalog and the +assistant-session name — are host-supplied internal inputs. They are not draft +props, they are declared by no command-line option, and they consume none of the +sources below. + +Individual options follow the **request** rather than a document path, for the +same reason they follow the path under `xmd run` — the schema that gives them +meaning arrives after them: + +```console +xmd prompt "greet someone" --props-name Ada --props-loud +``` + +An individual `--props-*` written before the request fails in preflight, before +any catalog, agent, elicitation, save, journal or document operation. The +aggregate `--props` may be written before it, because its meaning never depends +on a document. + +`xmd prompt --help` describes the Prompt, `--output`, `--run`, `--session`, the +aggregate sources and where individual options go. It names no individual option and no +default: the document that would declare them does not exist yet, and generating +one in order to describe it is not what help does. + +Every draft is bound afresh. The original argv is the command-line source for +all of them, the environment is read for each candidate's own bindings, and no +resolved props object is carried from one draft to the next. + +An individual option's **signature** is its generated option name, its token +arity — a bare switch or one value — and its accumulation behavior — scalar +last-wins or repeated array. The first draft that successfully binds a supplied +option freezes that option's signature, and every later draft is compared with +it before a single token is extracted. A removed option, a changed arity or a +changed accumulation fails the command. Checking before extraction is what stops +a switch that became a value option from reading the `--raw` or `--include` +written after it; a built-in option is never a generated property's value, and a +value that really begins with `-` is written `--props-name=-value`. + +Failures divide by who authored them. A draft's own defect — an unreadable +declaration, a collision between two properties generating one option or one +variable, a missing required property, any other document diagnostic — is +repairable: the host's assessment answers `valid: false` with the structured +findings, and the command document may ask for a replacement. A defect the caller +wrote — an option no draft declares, malformed aggregate JSON, a value the schema +cannot decode, an extra positional a draft's arity exposes, or a signature change +— raises out of that assessment instead, ending the command document with no +repair, no review, and no approved Plan on stdout, in a file or in a run. That +document cannot catch it and cannot recategorize it as feedback for an agent that +could not have caused it. + +The props the approved Plan is admitted with are resolved once more, after the +command document has completely torn down, from those exact returned bytes and +the original unchanged sources. Admission happens in every mode — before stdout, +before an `--output` file and before a run alike — and a failure there produces +none of them. No props object any draft produced is reused, so a revision that +changed a property's declared type changes what a run receives. ## Targeted roots