diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 76a11687..bf4e8329 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -186,6 +186,32 @@ jobs: fi uv run python scripts/check_version_gates.py --release-if-newer-than "$base" + - name: Release-scope check (release PRs only) + # Proves the new changelog entry covers every PR the tag will CONTAIN, + # not just the scope collected when the release PR was opened. Those + # differ whenever a feature PR merges while the release PR is open -- + # a structural window, since a release PR stays open for as long as its + # CI runs. It bit v0.91.0: #625 merged nine minutes before the release + # PR and landed inside the tag with no release note. `changelog-check` + # cannot see it (it proves every released VERSION has an entry, never + # that an entry covers every COMMIT under the tag). + # + # Armed exactly like the vNEXT gate above -- only for a PR that RAISES + # the version. The check needs tags and real history, which the default + # shallow checkout lacks, so the deepening is done ONLY when the version + # differs; the script fails open (warns, exits 0) if git still cannot + # answer, so an ordinary PR can never go red because of this. + if: github.event_name == 'pull_request' + run: | + base=$(git show "origin/$GITHUB_BASE_REF:pyproject.toml" 2>/dev/null \ + | sed -n 's/^version = "\(.*\)"/\1/p' | head -1) + head=$(sed -n 's/^version = "\(.*\)"/\1/p' pyproject.toml | head -1) + if [ -n "$base" ] && [ "$base" != "$head" ]; then + echo "version changed ($base -> $head); deepening checkout for the scope check" + git fetch --unshallow --tags --quiet 2>/dev/null || git fetch --tags --quiet || true + fi + uv run python scripts/check_release_scope.py --only-if-newer-than "$base" + - name: Error-code enum check # Rejects raw error_code="LITERAL" string literals (must use ErrorCode). run: uv run python scripts/check_error_codes.py diff --git a/CLAUDE.md b/CLAUDE.md index 27a52cc3..928454d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -197,7 +197,7 @@ The one exception is a **beta/pre-release** (below): there the bump deliberately rides the feature branch, because the pre-release tag and GitHub Release are cut from that branch -- the branch temporarily *is* the release PR. -### Beta / pre-release versions (since 0.43.3) +### Beta / pre-release versions Beta and release-candidate versions follow **PEP 440**: `0.44.0b1`, `0.44.0rc1`, ... -- **not** the SemVer `-beta.1` form (hatchling + uv require PEP 440 syntax in `pyproject.toml`). Three independent gates keep stable users safe from accidentally landing on a beta: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bbd712df..42fc196b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -625,10 +625,18 @@ silent-drift risks summarized in the entry and the release notes must cover each of them, and nothing else. 2. **Edit `pyproject.toml`** -- bump `version = "X.Y.Z"`. Single source of truth; everything else derives from it. This is the release PR's defining change -- if you are doing this in a feature PR, stop and read the section intro above. 3. **Add a changelog entry** to `src/keboola_agent_cli/changelog.py` -- ONE entry for the new version, covering **every PR merged since the last release** (step 1), no exceptions. CI fails (`make changelog-check`) if this is missing. Author it as the file's docstring describes: **one logical change per bullet** (split the release into several list items rather than one mega-paragraph), each starting with a recognised prefix (`BREAKING:`, `New:`, `Fix:`, `Change:`, `Note:`, `Security:`, ...), carrying its `(#PR)` reference, and leading with a self-contained first sentence. `kbagent changelog` shows only that first sentence per version by default (the rest is revealed by `--full`), so a buried headline or a single wall-of-text bullet reads as an unscannable blob. The first sentence is also **capped at 160 characters**, enforced by `tests/test_changelog_render.py::TestLiveChangelogHeadlines::test_newest_release_notes_are_not_truncated` (so `make check` in step 12 catches it) -- past the cap the default view and the release page show it cut mid-clause. Write a short self-contained first sentence and put the detail in the sentences after it; 2 of 0.90.0's 13 bullets needed exactly this rewrite. -4. **Replace every `vNEXT` placeholder** left behind by the feature PRs with the version being released, then verify none survive: +4. **Replace every `vNEXT` placeholder** left behind by the feature PRs with the version being released. Do it mechanically -- never by hand, and never with a repo-wide `sed`: ```bash + make vnext-resolve VERSION=X.Y.Z make vnext-check ``` + `vnext-resolve` reuses the same scanner `vnext-check` does, so it rewrites + exactly the live gates and leaves every backticked mention of the token + alone -- including a line that carries both at once, which a line-level + `sed` corrupts. It refuses any `VERSION` that disagrees with + `pyproject.toml` (bump that first, in step 2): `packaging` happily parses + `v0.91` and `0.91`, so a typo can look valid and then be stamped into every + gate in the tree at once. A leftover `(since vNEXT)` ships agents a gate no installed version can ever satisfy -- strictly worse than no gate, because they then refuse a command the user has. The release PR is the only place it can be fixed. @@ -651,12 +659,61 @@ silent-drift risks summarized in the > numeric gates -- `docs/sdk.md` writes 14 genuine ones as `` `0.66.0+` ``, > where backticks are ordinary typography rather than quotation. - While resolving, keep version tags **out of markdown headings**: a - `### Foo *(since vNEXT)*` heading changes its generated anchor slug at - every release, breaking each inbound `#foo-...` link (this bit 0.90.0 -- + Version tags must stay **out of markdown headings**: a + `### Foo *(since vNEXT)*` heading changes its generated anchor slug when the + placeholder resolves, breaking each inbound `#foo-...` link (this bit 0.90.0 -- the What's-new section's link broke the moment the placeholder resolved). Put the tag on the section's first body line instead; the gate checks scan whole files, not just headings, so nothing is lost. + + **This is CI-enforced on EVERY PR**, not just at release time -- a `vNEXT` + inside an ATX heading in a `.md` file fails `make version-gate-check` + (already part of `make check`). It is deliberately armed everywhere rather + than only under `--release`, because the rule used to be a hand-run + `grep -rn '^##.*vNEXT' plugins/` at release time and that grep **lost a + merge race in 0.91.0**: PR #697 ran it two minutes before #694 and #696 + landed headings of their own, so all three shipped and had to be cleaned up + after the tag. Any rule of the form "run this grep when releasing" loses + that race eventually, because a release is exactly when parallel branches + converge. Already-numeric headings are *not* flagged -- a resolved tag never + changes again, so its slug is stable. +4b. **Retire gates below the floor** (periodic, not every release): + ```bash + make gate-floor-report # what is below the current floor + ``` + A version gate earns its place only while some live install predates it. + kbagent self-updates on startup, so that population shrinks to roughly + nothing: pip/uv installs upgrade themselves, and only a standalone binary + (brew/choco/apt/dnf, which self-update is disabled for), an explicit + `KBAGENT_AUTO_UPDATE=false`, a dev tree, or a pip install stranded below + 0.62.0 by the #424 rename can sit on an old version. Meanwhile the stale + gate keeps making the agent refuse a command the user actually has -- which + this file already calls strictly worse than no gate. + + The two failure modes are asymmetric, and that is the whole argument for + pruning: a **kept-too-long** gate fails silently and permanently (the user + never learns the command exists), while a **removed-too-early** gate fails + loudly and self-correctingly (`No such command 'x'`, and `kbagent context` / + `--help` on the user's own install are authoritative anyway). + + **The floor is 0.80.0** as of the 0.91.0 cleanup. Retiring a gate means + deleting the *tag*, never the content -- the guidance under it is almost + always still true, and 0.91.0's pass kept every word while removing 223 tags. + + Four things are deliberately out of scope: + + - `changelog.py` -- the historical record; the version IS the content. + - `src/**/*.py` except `commands/context.py` -- developer comments + (`# DEPRECATED (since 0.43.4)`) are provenance, and no agent reads them. + - `X+` written inside a sentence -- often load-bearing prose + (`created by < 0.66.1 stay dormant until re-run on 0.66.1+`). + - **Safety gates, at any age.** Keep the tag wherever not knowing the + version causes silent data loss or a false assurance rather than an error + message -- e.g. `sync pull --force` (pre-0.53.0 it silently stranded local + edits), the `sync status` / `doctor` plaintext-secret audit (a false + all-clear on a leaked credential), the manage-token default-deny, and the + `--deny-writes` firewall. + 5. **Run `make version-sync`** -- propagates the new version to `plugins/kbagent/.claude-plugin/plugin.json`. The pre-commit hook does this automatically on `git commit`, but running it explicitly lets you eyeball the diff. 6. **Run `make skill-gen`** -- regenerates the decision table in `SKILL.md`. Idempotent if no commands changed since the previous release. 7. **Add a curated What's-new entry** to `web/frontend/src/whatsnew.ts` when the release ships anything UI-visible -- a `WhatsNewRelease` element keyed by the **exact** new version, newest first. This is the reel the web UI shows once per version; it is deliberately *not* derived from `changelog.py` (see `docs/web-server.md` > "What's-new popup"). Skipping it does not error anywhere: `whatsNewFor` falls back to the previous release's reel, which returning users have already dismissed -- so the release's UI work ships **dark**. A release with no UI-visible changes correctly adds nothing. Only the release PR can write this entry (a feature PR cannot know the version), which is why it lives in this checklist and not the per-command one. @@ -670,17 +727,33 @@ silent-drift risks summarized in the 12. **Run `make check`** -- lint + format + skill freshness + version sync + changelog completeness + error-code enum + full test suite. 13. **Run `make test-e2e`** if any command changed since the last release -- requires `E2E_API_TOKEN` and `E2E_URL`. 14. **Open the release PR** -- link the merged PRs it covers (step 1) and list every plugin file you touched in the description so reviewers can spot what was missed. Plugin files do not auto-show up in CI failures the way Python files do; reviewers are the second line of defence. -15. **Merge via `gh pr merge`, then tag -- the tag push IS the release.** Never push directly to `main` (protected). The only manual action after the merge is: +15. **Re-verify the scope against the commit you are about to tag:** + ```bash + make release-scope-check # in the release PR, before merging + make release-scope-check SCOPE_ARGS="--head origin/main --ignore-pr " + ``` + Step 1 collected the scope when the release PR was *opened*; this proves + the changelog entry covers every PR the **tag will actually contain**. The + two differ whenever a feature PR merges while the release PR is open -- + which is a structural window, not bad luck, since a release PR stays open + for as long as its CI runs. It shipped in v0.91.0: #625 merged nine minutes + before the release PR did, landing inside the tag's tree with no release + note, and was caught only because the tag happened to be deferred. + `make changelog-check` cannot see this: it proves every *released version* + has an entry, never that an entry covers every *commit* under the tag. + Run before merging and nothing needs ignoring -- the release PR's own + number is not in the log until its merge commit exists. +16. **Merge via `gh pr merge`, then tag -- the tag push IS the release.** Never push directly to `main` (protected). The only manual action after the merge is: ```bash git fetch origin && git tag v && git push origin v ``` The tag must point at the release PR's merge commit on `main` -- the pipeline's `gate` job fails the whole release if the tag's `pyproject.toml` disagrees with the tag name. Pushing it triggers `.github/workflows/release-kbagent.yml`, which does **everything else**: re-runs the gates, renders the release notes from `changelog.py` (`scripts/gen_release_notes.py` -- never write them by hand), publishes to PyPI, freezes the native binaries for all platforms, packages deb/rpm, creates the GitHub Release with every asset attached and fills its body, and updates Homebrew/Chocolatey/WinGet. Do **not** pre-create the GitHub Release by hand: the pipeline keeps a hand-written body untouched, which silently discards the changelog-rendered notes. -16. **Verify the publish** -- the pipeline guards against half-releases, but both guards exist because each failure shipped once (v0.66.1 went out with an empty body, v0.64.0 without a wheel), so look anyway: +17. **Verify the publish** -- the pipeline guards against half-releases, but both guards exist because each failure shipped once (v0.66.1 went out with an empty body, v0.64.0 without a wheel), so look anyway: ```bash gh run watch $(gh run list --workflow release-kbagent.yml --limit 1 --json databaseId --jq '.[0].databaseId') ``` then confirm `gh release view v` shows a non-empty body rendered from the changelog and both wheels (`keboola_cli-*` + legacy `keboola_agent_cli-*`) among the assets. A `skipped` winget job is normal; any red job is a real signal. -17. **After the tag: merge the ai-kit publish PR.** The `ai-kit-marketplace` job opens +18. **After the tag: merge the ai-kit publish PR.** The `ai-kit-marketplace` job opens `chore(kbagent): publish vX.Y.Z` against `keboola/ai-kit`, bumping the `kbagent` entry in the `keboola-claude-kit` marketplace to this tag. Until that PR merges, `/plugin install kbagent@keboola-claude-kit` still serves the PREVIOUS version -- diff --git a/Makefile b/Makefile index 53df0c70..9662e3dc 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ .DEFAULT_GOAL := help -.PHONY: help install install-server sync test test-unit test-integration test-e2e test-e2e-local test-e2e-invite test-e2e-feature test-e2e-stream test-e2e-auth test-file test-cov lint lint-fix format format-check typecheck typecheck-warn skill-check skill-gen version-sync version-check version-gate-check changelog changelog-check check-error-codes check-sentinel-guards loc-check loc-report loc-baseline command-sync-check gen-command-reference endpoints-gen endpoints-check check clean hooks web-install web-dev-backend web-dev-frontend web-build web-clean +.PHONY: help install install-server sync test test-unit test-integration test-e2e test-e2e-local test-e2e-invite test-e2e-feature test-e2e-stream test-e2e-auth test-file test-cov lint lint-fix format format-check typecheck typecheck-warn skill-check skill-gen version-sync version-check version-gate-check vnext-check vnext-resolve gate-floor-report release-scope-check changelog changelog-check check-error-codes check-sentinel-guards loc-check loc-report loc-baseline command-sync-check gen-command-reference endpoints-gen endpoints-check check clean hooks web-install web-dev-backend web-dev-frontend web-build web-clean help: ## Show this help message @grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-18s\033[0m %s\n", $$1, $$2}' @@ -124,6 +124,16 @@ version-gate-check: ## Reject a (since vX.Y.Z) / X.Y.Z+ marker naming an unrelea vnext-check: ## Reject an unresolved version-gate placeholder -- run in the RELEASE PR uv run python scripts/check_version_gates.py --release +vnext-resolve: ## Rewrite every live vNEXT gate to pyproject's version (RELEASE PR step 4) + @test -n "$(VERSION)" || { echo "usage: make vnext-resolve VERSION=X.Y.Z"; exit 2; } + uv run python scripts/check_version_gates.py --resolve $(VERSION) + +gate-floor-report: ## List version gates below the retirement floor (default 0.80.0) + uv run python scripts/check_version_gates.py --list-below $(or $(FLOOR),0.80.0) + +release-scope-check: ## Prove the changelog entry covers every PR the tag will contain + uv run python scripts/check_release_scope.py $(SCOPE_ARGS) + check-sentinel-guards: ## Reject an unguarded kbc-session:// sentinel path (silent-drift gate) uv run python scripts/check_sentinel_guards.py diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md index eca9df62..1c06a7c3 100644 --- a/docs/TUTORIAL.md +++ b/docs/TUTORIAL.md @@ -139,7 +139,7 @@ Flags worth knowing: The command is **idempotent**: running it again skips projects that are already registered. Safe to re-run after adding new project IDs. -**Security note (since v0.29.0)**: `KBC_MANAGE_API_TOKEN` is **ignored +**Security note**: `KBC_MANAGE_API_TOKEN` is **ignored by default** -- the env var is read only when the top-level `--allow-env-manage-token` flag is passed. Without the flag, kbagent prompts on stdin (hidden input). kbagent never accepts the token as a @@ -810,7 +810,7 @@ footguns the platform does not surface as errors: platform transitions `created -> stopped -> starting -> running`, so a naive poll that exits on `stopped` reports a phantom failure. -`kbagent data-app` (since 0.27.0) encodes all four in the service layer, +`kbagent data-app` encodes all four in the service layer, so the `--json` output you see at the CLI is what would have happened if you had done everything right at the raw HTTP level. The eight subcommands -- `list`, `detail`, `create`, `deploy`, `start`, `stop`, @@ -923,7 +923,7 @@ kbagent --json data-app create \ deploy. To retrieve it: ```bash -# Manage API token: interactive prompt by default (since v0.29.0). For CI, +# Manage API token: interactive prompt by default. For CI, # add `--allow-env-manage-token` and set KBC_MANAGE_API_TOKEN in env. kbagent --json data-app password \ --project prod --app-id 12345678 \ diff --git a/plugins/kbagent/skills/kbagent/references/agent-tasks-cli-workflow.md b/plugins/kbagent/skills/kbagent/references/agent-tasks-cli-workflow.md index 1cf42b19..ccc1b906 100644 --- a/plugins/kbagent/skills/kbagent/references/agent-tasks-cli-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/agent-tasks-cli-workflow.md @@ -55,7 +55,7 @@ Every task carries an `action` envelope with `type` + `params`: `cli` accepts `claude`, `codex`, or `gemini`. The chosen CLI must be on the server's `PATH` when the task fires (cron or `agent run`). -**`extra_args` are ignored unless the serve operator opts in (since v0.60.2).** +**`extra_args` are ignored unless the serve operator opts in.** They are passed verbatim to the AI CLI and can disable its safety rails, so `kbagent serve` drops them with a warning unless it was started with a truthy `KBAGENT_ALLOW_AI_EXTRA_ARGS` (e.g. `KBAGENT_ALLOW_AI_EXTRA_ARGS=1`). The @@ -98,7 +98,7 @@ kbagent agent list kbagent agent show ``` -> **ID forms (since v0.44.0):** every subcommand that takes a task/run ID +> **ID forms:** every subcommand that takes a task/run ID > accepts it positionally (`agent show `) or via a named flag > (`--id` / `--task-id`, plus `--run-id` for `run-detail` / `run-events`) -- > matching the rest of the CLI (`--job-id`, `--config-id`, ...). Examples diff --git a/plugins/kbagent/skills/kbagent/references/agent-tasks-rest-workflow.md b/plugins/kbagent/skills/kbagent/references/agent-tasks-rest-workflow.md index 7cdaf5dd..168b693b 100644 --- a/plugins/kbagent/skills/kbagent/references/agent-tasks-rest-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/agent-tasks-rest-workflow.md @@ -66,7 +66,7 @@ Every task has an `action` block with `type` + `params`: `cli` accepts `claude`, `codex`, or `gemini`. The chosen CLI must be on the server's `PATH`. -**`extra_args` are ignored unless the serve operator opts in (since v0.60.2).** +**`extra_args` are ignored unless the serve operator opts in.** They are forwarded verbatim to the AI CLI and can disable its safety rails, so `kbagent serve` drops them with a warning unless it was started with a truthy `KBAGENT_ALLOW_AI_EXTRA_ARGS`. The `["--print"]` above takes effect only when diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index efd7ba75..71784fe2 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -5,8 +5,8 @@ All commands support `--json` for structured output. Multi-project flags (`--pro ## Setup & Info - `init [--from-global] [--project ALIAS ...]` -- create local `.kbagent/` workspace in current directory; `--project ALIAS` (repeatable) copies only the named project(s) from the global config and implies `--from-global` - `doctor` -- health check for CLI config (no `--fix` since v0.85.0 -- it only installed the MCP server) -- `version [--beta]` -- show kbagent version info and update status (kbagent only since v0.85.0; no `dependencies` key). On a standalone binary the payload carries additive `kbagent.install_channel` + `kbagent.upgrade_hint` keys and `upgrade_command` holds the channel's command (empty for a hand-unpacked archive) (since v0.79.0). `--beta` (since v0.42.0) reports the latest pre-release (beta / rc) instead of the latest stable. Env override: `KBAGENT_INCLUDE_PRERELEASE=1` -- `update [--beta]` -- self-update to latest version. `--beta` (since v0.42.0) opts into pre-release versions (PEP 440 betas / rc, e.g. `0.43.0b1`). Default behaviour: GitHub's `/releases/latest` endpoint filters prereleases server-side, so the startup auto-update hook never silently lands on a beta. Resolver-level opt-in (`--prerelease=allow` for uv, `--pre` for pip) is added automatically when `--beta` is set. **Standalone (PyInstaller) binaries refuse the self-update** and report their own channel's command instead -- a uv/pip reinstall would install a second, unrelated kbagent rather than upgrade the packaged one (since v0.79.0) +- `version [--beta]` -- show kbagent version info and update status (kbagent only since v0.85.0; no `dependencies` key). On a standalone binary the payload carries additive `kbagent.install_channel` + `kbagent.upgrade_hint` keys and `upgrade_command` holds the channel's command (empty for a hand-unpacked archive). `--beta` reports the latest pre-release (beta / rc) instead of the latest stable. Env override: `KBAGENT_INCLUDE_PRERELEASE=1` +- `update [--beta]` -- self-update to latest version. `--beta` opts into pre-release versions (PEP 440 betas / rc, e.g. `0.43.0b1`). Default behaviour: GitHub's `/releases/latest` endpoint filters prereleases server-side, so the startup auto-update hook never silently lands on a beta. Resolver-level opt-in (`--prerelease=allow` for uv, `--pre` for pip) is added automatically when `--beta` is set. **Standalone (PyInstaller) binaries refuse the self-update** and report their own channel's command instead -- a uv/pip reinstall would install a second, unrelated kbagent rather than upgrade the packaged one - `changelog [--limit N] [--full]` -- show recent changelog (default: last 5 versions, one-line summary per version; `--full` / `-v` expands every note). After auto-update, "What's new" is printed automatically (summarised). Manual trigger: `KBAGENT_UPDATED_FROM=0.17.0 kbagent version` - `context` -- print full CLI reference for AI agents @@ -72,7 +72,7 @@ walkthrough and troubleshooting. - `project current` -- print the effective default project and its source (`env` / `pin` / `none`). Reports both the env override AND the persisted pin so misconfigurations are visible. Returns `{"alias": null, "source": "none"}` when neither is set - `project info --project NAME` -- show detailed project metadata. Also carries `auth_mode`, rendered as an `Auth` row above the Token rows (on a session project those rows describe the rotating access token). `project current`, `project add` and `project edit` deliberately do **not** carry `auth_mode` -- `current` answers which alias is effective (and reports a `KBAGENT_PROJECT` override that may name an alias absent from the config), the other two are write confirmations. The same keys appear over HTTP on `/projects`, `/projects/status` and `/projects/{alias}/info` (since v0.80.0) -## Project Members & Invitations (since v0.29.0) +## Project Members & Invitations All seven commands authenticate via `KBC_MANAGE_API_TOKEN` (Manage API), not the project's Storage token. Allowed roles are exactly `admin`, `guest`, `readOnly`, `share` -- the API self-reports this list in its 400 validation error and `constants.PROJECT_ROLES` mirrors it. @@ -105,7 +105,7 @@ The `permissions` subcommands persist a write/destructive policy to config.json ## Billing (PAYG Credits) (since v0.84.2) - `billing credits [--project ALIAS ...]` -- read-only PAYG credit balance (`GET /credits` on `billing.{stack}`, plain Storage token). Fans out across all registered projects in parallel by default; `--project` (repeatable) narrows. Per-project failures degrade individually and are collected in `errors`, never abort the run. A project without the `pay-as-you-go` `owner.features` flag never calls the billing host (NXDOMAIN on some non-PAYG stacks) -- it gets an `error_code: PAYG_NOT_AVAILABLE` entry instead. `--json` emits `{"credits": [...], "errors": [...]}`. Rows carry the API's native unit (`consumed`/`remaining` credits) plus derived `*_minutes` fields (1 credit = 60 minutes, matching the Keboola UI). Gives the current balance only -- purchase history / Stripe invoice IDs are not reachable with a project token (issue #594 primary ask, still open; that data lives on `connection.{stack}` `/pay-as-you-go/billing/*`). See [billing-workflow.md](billing-workflow.md) for the full shape of the invoice-history gap and why it must not be worked around. -## Feature Flags (since v0.48.0) +## Feature Flags Requires a **super-admin** Manage API token (same kind as `org setup`). Same default-deny token policy: interactive hidden prompt by default, or `--allow-env-manage-token` + `KBC_MANAGE_API_TOKEN` for CI. `--project ALIAS` resolves the stack URL (and, for project ops, the numeric `project_id`) from config -- the alias is the only handle you pass. - `feature list --project ALIAS` -- the stack-wide feature catalogue (`GET /manage/features`). Returns `{alias, stack_url, features: [{name, title, description, type, ...}]}`. Only `name` is a stable identifier; extra fields pass through unmodified. - `feature project-show --project ALIAS` -- features assigned to a project, read from the project object's `features` array. Returns `{alias, project_id, project_name, features: [...]}`. @@ -116,8 +116,8 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def - `feature user-remove --project ALIAS --email EMAIL --feature NAME [--dry-run] [--yes]` -- disable a feature on a user (`DELETE /manage/users/{email}/features/{name}`). ## Component Discovery -- `component sync-action ACTION_NAME --component-id ID --project ALIAS (--config-id ID [--row-id ID] | --config-data JSON|@file|-) [--branch ID] [--timeout N]` (since 0.73.0) -- run a synchronous component action (`testConnection`, `getTables`, ...) on the `sync-actions.{stack}` service. `ACTION_NAME` is freeform (component-defined; discover via `component detail` `synchronous_actions`). `--row-id` shallow-merges the row over the root config at TOP level only (row `parameters`/`storage` replace root wholesale -- NOT deep merge; MCP `run_sync_action` parity). `--config-data` sends explicit `configData` verbatim. Response is action-specific pass-through. Ports the `run_sync_action` MCP tool. **Since 0.89.0 (#620)** the ROOT configuration's `authorization` and `runtime` blocks are forwarded into `configData` too -- root only (a `--row-id` never overrides them), and only when non-empty. `authorization.oauth_api.id` is the OAuth broker reference the sync-actions service resolves and decrypts, so on 0.88.0 and earlier every sync action on an OAuth / Service-Account component (`keboola.ex-linkedin-ads`, ...) failed with an opaque empty-body 400. -- `config examples --component-id ID [--project NAME] [--row]` (since 0.73.0) -- sample root/row configurations from the AI-service component detail. `--json` emits `{component_id, root_examples, row_examples}`; `--row` limits to row examples. Ports the `get_config_examples` MCP tool. +- `component sync-action ACTION_NAME --component-id ID --project ALIAS (--config-id ID [--row-id ID] | --config-data JSON|@file|-) [--branch ID] [--timeout N]` -- run a synchronous component action (`testConnection`, `getTables`, ...) on the `sync-actions.{stack}` service. `ACTION_NAME` is freeform (component-defined; discover via `component detail` `synchronous_actions`). `--row-id` shallow-merges the row over the root config at TOP level only (row `parameters`/`storage` replace root wholesale -- NOT deep merge; MCP `run_sync_action` parity). `--config-data` sends explicit `configData` verbatim. Response is action-specific pass-through. Ports the `run_sync_action` MCP tool. **Since 0.89.0 (#620)** the ROOT configuration's `authorization` and `runtime` blocks are forwarded into `configData` too -- root only (a `--row-id` never overrides them), and only when non-empty. `authorization.oauth_api.id` is the OAuth broker reference the sync-actions service resolves and decrypts, so on 0.88.0 and earlier every sync action on an OAuth / Service-Account component (`keboola.ex-linkedin-ads`, ...) failed with an opaque empty-body 400. +- `config examples --component-id ID [--project NAME] [--row]` -- sample root/row configurations from the AI-service component detail. `--json` emits `{component_id, root_examples, row_examples}`; `--row` limits to row examples. Ports the `get_config_examples` MCP tool. - `component list [--project NAME] [--type TYPE] [--query "text"]` -- list/search components (AI-powered with `--query`) - `component detail --component-id ID [--project NAME]` -- show component schema, docs URL, examples. **Since 0.90.0** a component the AI Service does not index (private/deprecated: `keboola.mcp-server-tool`, `keboola.data-apps`) no longer errors -- it falls back to the project's Storage component catalog. `documentation_source` (`"ai_service"` vs `"storage_catalog"`) is on BOTH paths and tells them apart; the fallback has NO configuration examples (`examples_count`/`row_examples_count` always 0), so read `documentation_source` before treating 0 as "this component ships none". `NOT_FOUND` is still raised when both sources miss. **Since 0.90.0** omitting `--project` really does use the first configured project -- on <= 0.89.x it failed with `CONFIG_ERROR: Project 'None' not found` despite the help text, so pass `--project` explicitly there. See `gotchas.md`. @@ -154,7 +154,7 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def ## Job History - `job list [--project NAME] [--component-id ID] [--config-id ID] [--status STATUS] [--limit N] [--offset N] [--sort-by FIELD] [--sort-order asc|desc]` -- list jobs (default 50, max 500). `--offset` (0.88.0+) pages past `--limit`; `--sort-by` accepts `startTime` (default), `endTime`, `createdTime`, `durationSeconds`, `id` and `--sort-order` `asc`/`desc` (default `desc`) -- an unknown value exits 2 locally rather than returning an arbitrarily ordered page. Without `--project` the fan-out over every registered project is merged **globally** by the same `--sort-by`/`--sort-order` (0.90.1+, #675) -- one chronological feed, missing values last in both directions, deterministic `(project_alias, id)` tiebreak. Before 0.90.1 rows came back grouped alias-then-id, which destroyed the chronological order the fan-out exists to produce; anything wanting per-project blocks must group them itself - `job detail --project NAME --job-id ID [--log-tail-lines N]` -- full job detail with timing and result message. `--log-tail-lines N` (0.88.0+) attaches the job's last N events as `logTail` -- this is how you read the logs of a job that ALREADY finished (`job run` only tails the run it started). Off by default (0 = no extra API call); an events-endpoint failure degrades to an empty tail, it never hides the detail -- `job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--mode run|debug] [--variable-values-id ID] [--no-variables] [--poll-strategy exponential|fixed] [--log-tail-lines N] [--idempotency-key KEY] [--force-rerun]` -- run a job, optionally wait for completion (branch-aware). `--idempotency-key KEY` (since v0.63.0) makes a replayed run safe: on a second `job run` with the same key, a prior still-running or non-failed job is returned (JSON gains `idempotent_replay: true`, human mode prints a note) instead of creating a duplicate side effect; a prior FAILED run is re-run. Dedup is **client-side** (the Queue API has no server idempotency token -- verified against the live spec) and persisted to `/job_idempotency.json`, so it is scoped to one machine. Reusing a key for a *different* component/config exits with `INVALID_ARGUMENT` rather than returning the wrong job; `--force-rerun` ignores the stored entry and always creates a fresh job. For configs with linked `keboola.variables` (root-level `configuration.variables_id`), kbagent auto-resolves a `variableValuesId` so transformations bind to the deployed values row. `--variable-values-id` overrides; `--no-variables` skips resolution. `NO_VARIABLE_ROWS` when the linked variables config has zero rows -- fix via `kbagent config variables-set`. `--mode debug` (since v0.43.6) sets the Queue API job `mode` body field to `"debug"`: the component runs with the same configuration + inputs but the worker redirects output to a Storage File tagged `debug-` instead of writing to destination buckets. Use for dry-runs, reproducing a failing job on a production config without touching downstream tables, or harvesting the worker's output bytes (download via `storage file-download --tag debug-`) to feed into VCR fixtures or component test cases. Default `--mode run` is unchanged. Invalid values (`--mode anything-else`) exit 2 at the Click choice gate before any wire call. Under `--wait`, polls with an exponential curve (2s x 30 -> 5s x 48 -> 15s); `--poll-strategy fixed` keeps a constant 1s interval. On FAILED/WARNING/TERMINATED, the last `--log-tail-lines` events (default 200, **0 disables -- recommended for automation pipelines**) are attached as `logTail` in the JSON result (or `details.logTail` on errors). If `--timeout` expires, kbagent issues `kill_job` on the remote and exits **7** (`JOB_TIMEOUT_TERMINATED`) with the cancelled `details.job` + `details.logTail`; if the kill itself fails, exits **4** (`QUEUE_JOB_TIMEOUT`, `retryable=true`). Use jq pattern `.error.details.logTail? // .data.logTail? // []` to pick up the tail regardless of exit code. +- `job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--mode run|debug] [--variable-values-id ID] [--no-variables] [--poll-strategy exponential|fixed] [--log-tail-lines N] [--idempotency-key KEY] [--force-rerun]` -- run a job, optionally wait for completion (branch-aware). `--idempotency-key KEY` makes a replayed run safe: on a second `job run` with the same key, a prior still-running or non-failed job is returned (JSON gains `idempotent_replay: true`, human mode prints a note) instead of creating a duplicate side effect; a prior FAILED run is re-run. Dedup is **client-side** (the Queue API has no server idempotency token -- verified against the live spec) and persisted to `/job_idempotency.json`, so it is scoped to one machine. Reusing a key for a *different* component/config exits with `INVALID_ARGUMENT` rather than returning the wrong job; `--force-rerun` ignores the stored entry and always creates a fresh job. For configs with linked `keboola.variables` (root-level `configuration.variables_id`), kbagent auto-resolves a `variableValuesId` so transformations bind to the deployed values row. `--variable-values-id` overrides; `--no-variables` skips resolution. `NO_VARIABLE_ROWS` when the linked variables config has zero rows -- fix via `kbagent config variables-set`. `--mode debug` sets the Queue API job `mode` body field to `"debug"`: the component runs with the same configuration + inputs but the worker redirects output to a Storage File tagged `debug-` instead of writing to destination buckets. Use for dry-runs, reproducing a failing job on a production config without touching downstream tables, or harvesting the worker's output bytes (download via `storage file-download --tag debug-`) to feed into VCR fixtures or component test cases. Default `--mode run` is unchanged. Invalid values (`--mode anything-else`) exit 2 at the Click choice gate before any wire call. Under `--wait`, polls with an exponential curve (2s x 30 -> 5s x 48 -> 15s); `--poll-strategy fixed` keeps a constant 1s interval. On FAILED/WARNING/TERMINATED, the last `--log-tail-lines` events (default 200, **0 disables -- recommended for automation pipelines**) are attached as `logTail` in the JSON result (or `details.logTail` on errors). If `--timeout` expires, kbagent issues `kill_job` on the remote and exits **7** (`JOB_TIMEOUT_TERMINATED`) with the cancelled `details.job` + `details.logTail`; if the kill itself fails, exits **4** (`QUEUE_JOB_TIMEOUT`, `retryable=true`). Use jq pattern `.error.details.logTail? // .data.logTail? // []` to pick up the tail regardless of exit code. - `job terminate --project NAME (--job-id ID [--job-id ...] | --status any|created|waiting|processing [--component-id ID] [--config-id ID] [--branch ID] [--limit N]) [--dry-run] [--yes]` -- kill running Queue API jobs. Use to stop runaway loops or clean up pile-ups from repeated `job run` calls. Two modes: by ID (single/batch) or by filter (`--status any` catches every killable state). Response partitions IDs into `killed / already_finished / not_found / failed`; safe to re-run idempotently. Kill is async -- poll `job detail` for `isFinished=true`. ## Storage @@ -168,16 +168,16 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def - `storage download-table --project NAME --table-id ID [--output FILE] [--columns COL ...] [--limit N] [--where-column COL --where-value VAL ... [--where-operator eq|neq]] [--changed-since WHEN] [--changed-until WHEN] [--branch ID]` -- export table to CSV (branch-aware). `--where-column` + `--where-value` (repeatable, OR within the set) + `--where-operator eq|neq` filter rows server-side; `--changed-since`/`--changed-until` (unix ts or strtotime like `-2 days`) filter by import time -- the credential-only, no-workspace way to pull a filtered/incremental slice (0.62.0+) - `storage add-column --project NAME --table-id ID --column COL:TYPE[(length)] [--not-null] [--default VALUE] [--branch ID]` -- add a single column to an existing table (0.62.0+). Same `name:TYPE(length)` grammar as `create-table --column`; a bare `name` adds an untyped STRING column. Synchronous endpoint (no job to wait on). `--not-null` needs an empty table or a `--default`. Mirror of `delete-column` - `storage delete-table --project NAME --table-id ID [--table-id ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete tables, --force cascade-deletes aliased tables (branch-aware) -- `storage truncate-table --project NAME --table-id ID [--table-id ...] [--dry-run] [--yes] [--branch ID]` (since v0.32.0) -- delete all rows while preserving table schema, primary key, descriptions, sharing edges, and downstream dependents. Batch via repeated `--table-id`. Endpoint is uniformly async-via-job on every branch (returns a queued `tableRowsDelete` job; client polls via `_wait_for_storage_job` before returning). Idempotent (truncating an empty table is a no-op). Use when re-seeding a table without losing the schema contract +- `storage truncate-table --project NAME --table-id ID [--table-id ...] [--dry-run] [--yes] [--branch ID]` -- delete all rows while preserving table schema, primary key, descriptions, sharing edges, and downstream dependents. Batch via repeated `--table-id`. Endpoint is uniformly async-via-job on every branch (returns a queued `tableRowsDelete` job; client polls via `_wait_for_storage_job` before returning). Idempotent (truncating an empty table is a no-op). Use when re-seeding a table without losing the schema contract - `storage delete-column --project NAME --table-id ID --column COL [--column ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete columns from a table (branch-aware) - `storage delete-bucket --project NAME --bucket-id ID [--bucket-id ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete buckets (branch-aware) -- `storage swap-tables --project NAME --table-id ID --target-table-id ID --branch ID [--dry-run] [--yes]` (since v0.28.0) -- swap two storage tables in any branch, including the default/production branch (POST `/tables/{id}/swap`). Both tables exchange physical positions; aliases are NOT transferred (they keep pointing at the same physical position and therefore expose the OTHER table's data after the swap). Service refuses without a branch (active branch via `branch use` works too). Use to flip a typed rebuild ("data_change_log") into the original name ("data") without touching downstream config references -- `storage clone-table --project NAME --table-id ID --branch ID [--dry-run]` (since v0.52.0) -- pull (clone) a production table into a dev branch (POST `/tables/{id}/pull`, operationName `devBranchTablePull`). On `storage-branches` projects a dev branch reads prod tables transparently until the first write, so an in-branch schema mutation (`swap-tables`, dropping a column) fails with a misleading "bucket not found" until the table is materialized branch-local; `clone-table` does that. One-way (default -> branch). Service refuses without a branch (active branch via `branch use` works too). Permission class `write` -- `storage snapshot-create --project NAME --table-id ID [--description D] [--branch ID]` (since v0.75.0) -- create a point-in-time snapshot of a table (data + columns + primary key; async `tableSnapshotCreate` job, client polls to completion). The receipt carries `snapshot_id` -- restores are addressed by it. Permission class `write` -- `storage snapshots --project NAME --table-id ID [--limit N] [--branch ID]` (since v0.75.0) -- list a table's snapshots (`id`, `createdTime`, `description`, `creatorToken`). Read command: production endpoint by default -- `storage snapshot-detail --project NAME --snapshot-id ID` (since v0.75.0) -- one snapshot's detail. Snapshot IDs are global (not table-scoped); the detail embeds the source `table` object (id, columns, primaryKey), so it traces a bare snapshot ID back to its origin table -- `storage table-from-snapshot --project NAME --snapshot-id ID --bucket-id ID --name NAME [--branch ID] [--dry-run]` (since v0.75.0) -- create a NEW table from an existing snapshot (issue #512; restore). Restores data, columns, and primary key into `--bucket-id` under `--name` via the classic `tables-async` endpoint (NOT `tables-definition` -- that is why it is not a `create-table` flag). `--name` is REQUIRED (API rejects empty). No overwrite: restoring onto an existing table name fails -- restore under a new name, verify, then `swap-tables` or `delete-table` yourself. Permission class `write` -- `storage snapshot-delete --project NAME --snapshot-id ID [--snapshot-id ...] [--dry-run] [--yes]` (since v0.75.0) -- delete snapshots (destructive: forecloses restores; source tables untouched). Batch-tolerant, exit 1 when any ID failed +- `storage swap-tables --project NAME --table-id ID --target-table-id ID --branch ID [--dry-run] [--yes]` -- swap two storage tables in any branch, including the default/production branch (POST `/tables/{id}/swap`). Both tables exchange physical positions; aliases are NOT transferred (they keep pointing at the same physical position and therefore expose the OTHER table's data after the swap). Service refuses without a branch (active branch via `branch use` works too). Use to flip a typed rebuild ("data_change_log") into the original name ("data") without touching downstream config references +- `storage clone-table --project NAME --table-id ID --branch ID [--dry-run]` -- pull (clone) a production table into a dev branch (POST `/tables/{id}/pull`, operationName `devBranchTablePull`). On `storage-branches` projects a dev branch reads prod tables transparently until the first write, so an in-branch schema mutation (`swap-tables`, dropping a column) fails with a misleading "bucket not found" until the table is materialized branch-local; `clone-table` does that. One-way (default -> branch). Service refuses without a branch (active branch via `branch use` works too). Permission class `write` +- `storage snapshot-create --project NAME --table-id ID [--description D] [--branch ID]` -- create a point-in-time snapshot of a table (data + columns + primary key; async `tableSnapshotCreate` job, client polls to completion). The receipt carries `snapshot_id` -- restores are addressed by it. Permission class `write` +- `storage snapshots --project NAME --table-id ID [--limit N] [--branch ID]` -- list a table's snapshots (`id`, `createdTime`, `description`, `creatorToken`). Read command: production endpoint by default +- `storage snapshot-detail --project NAME --snapshot-id ID` -- one snapshot's detail. Snapshot IDs are global (not table-scoped); the detail embeds the source `table` object (id, columns, primaryKey), so it traces a bare snapshot ID back to its origin table +- `storage table-from-snapshot --project NAME --snapshot-id ID --bucket-id ID --name NAME [--branch ID] [--dry-run]` -- create a NEW table from an existing snapshot (issue #512; restore). Restores data, columns, and primary key into `--bucket-id` under `--name` via the classic `tables-async` endpoint (NOT `tables-definition` -- that is why it is not a `create-table` flag). `--name` is REQUIRED (API rejects empty). No overwrite: restoring onto an existing table name fails -- restore under a new name, verify, then `swap-tables` or `delete-table` yourself. Permission class `write` +- `storage snapshot-delete --project NAME --snapshot-id ID [--snapshot-id ...] [--dry-run] [--yes]` -- delete snapshots (destructive: forecloses restores; source tables untouched). Batch-tolerant, exit 1 when any ID failed - `storage describe-bucket --project NAME --bucket-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a bucket description (stored as `KBC.description` in bucket metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage bucket-detail` - `storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a table description (stored as `KBC.description` in table metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage table-detail` - `storage describe-column --project NAME --table-id ID --column NAME=DESCRIPTION [--column ...] [--branch ID]` -- set one or more column descriptions. *(since v0.88.0)* Writes through the native `PUT /v2/storage/branch/{branch}/tables/{id}/definition` endpoint (the one the web UI uses; async `tableDefinitionUpdate` storage job) with `isDescriptionSystemManaged: false`, so the next component run's Output Mapping cannot overwrite the text. The backend mirrors the value into `columnMetadata` `KBC.description`, so the Keboola UI, the MCP server (`get_tables`) and the Snowflake `COMMENT` / BigQuery column description all see it. Unknown column names are rejected BEFORE any write (behavior change -- the pre-0.88.0 flat-metadata write accepted typos silently). Legacy flat `KBC.column.{name}.description` entries on the same table are migrated in the same write and then deleted. Read back in `storage table-detail` under `column_details[].description` @@ -194,14 +194,14 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def - `storage load-file --project NAME --file-id ID --table-id ID [--incremental] [--delimiter D] [--enclosure E] [--branch ID]` -- import a Storage File into a table (CSV) - `storage unload-table --project NAME --table-id ID [--columns COL ...] [--limit N] [--tag TAG ...] [--download] [--output FILE|DIR] [--file-type csv|parquet] [--branch ID]` -- export a table to a Storage File. `--file-type parquet` produces sliced Parquet; `--download` saves each slice as its own file under `./{project}/{table_id}.parquet/` (default) together with `_manifest.json` -## Data Streams (OTLP) (since v0.50.0) +## Data Streams (OTLP) Uses the per-project Storage token (no manage token). Control plane = `stream.` (derived from `connection.`). The OTLP ingest endpoint (`stream-in./otlp///`) is returned in `source.otlp.url` with the secret in the path -- **masked by default, `--reveal` to print it**. `create-source --type otlp` auto-provisions the logs/metrics/traces sinks (bucket `in.c-otlp-`) so data lands; `--no-sinks` opts out. See `stream-workflow.md`. - `stream list --project NAME [--branch ID]` -- list sources (id, name, type, secret-free base endpoint) - `stream create-source --project NAME --name NAME [--type otlp|http] [--branch ID] [--if-not-exists] [--no-sinks] [--reveal]` -- create a source; for OTLP auto-creates 3 sinks (idempotent); polls the async task and returns the endpoint. `--if-not-exists` returns an existing same-named source as `status=skipped` - `stream detail [SOURCE_ID | --name NAME] --project NAME [--branch ID] [--reveal]` -- base + per-signal endpoints (`/v1/logs|/v1/traces|/v1/metrics`), protocol `http/protobuf`, destination bucket/tables (from sinks). Secret masked unless `--reveal` - `stream delete SOURCE_ID --project NAME [--branch ID] [--dry-run] [--yes|--force]` -- delete a source (destructive; async task polled to completion) -## Scoped Storage Tokens (since v0.66.0) +## Scoped Storage Tokens Mint, revoke, and rotate scoped Storage API tokens (the Keboola single-bucket-write pattern -- a device/component gets a token that can write to exactly one bucket). Uses the per-project Storage token from config (no manage token). **`token create` requires a master (admin) Storage token** -- `canManageTokens` alone is not enough there (a non-master token carrying it makes the API answer a generic 500, issue #599), so it fails fast with `MISSING_MASTER_TOKEN` exit 3 on a pre-flight check before any write. That defect is create-only: `token list` / `token delete` / `token refresh` need just `canManageTokens` and are deliberately unguarded. The token secret is displayed **once** on create/refresh and is never retrievable again. The importable SDK `Client(url, token)` mirrors this surface: `create_scoped_token` / `delete_token` / `refresh_token` (plus the stream-source primitives `create_stream_source` / `get_stream_source` / `list_stream_sources` / `delete_stream_source`) -- dicts on `.raw`, typed `ScopedTokenResult` / `StreamSourceResult` on the facade. See `sdk.md`. - `token list --project NAME` -- list the project's tokens (GET `/v2/storage/tokens`): id, description, created, expires (with an expired marker), master flag, and the token that created each one. This is how you find the `--token-id` that `delete` / `refresh` need. Secret values are stripped from every row before output, including under `--json` -- on a project carrying the `force-decrypted-token` feature the API embeds live secrets in the listing, and reproducing them would break the "revealed once, at mint" rule for every token at once. (since v0.86.0) - `token list --project NAME --with-last-used` -- adds `lastUsed`, `lastUsedEvent` and `lastUsedStatus` per token and sorts dormant-first, turning the listing into a revocation worklist. **Opt-in: one extra API call per token** (parallel fan-out), so do not reach for it when you only need an id. Storage tokens carry no `lastUsed` of their own, so it is derived from `GET /v2/storage/tokens/{id}/events` narrowed server-side to `q=token.id:{id}`. `lastUsedStatus` is `used` | `never` | `unknown` | `error` -- see gotchas before acting on it, `never` and `unknown` are NOT interchangeable and dev-branch activity is invisible. `--columns` (repeatable) selects and orders the human table (`id`, `description`, `created`, `refreshed`, `expires`, `master`, `created_by`, `last_used`, `last_used_event`); it does not affect `--json`. (since v0.88.0) @@ -252,33 +252,33 @@ Bucket sharing + linking across projects in the same organization. `sharing edge Lifecycle for `keboola.data-apps`. Combines Storage API (config body, git block, encrypted secrets, runtime size) with Data Science API (`/apps` -- deployment record, state, URL, configVersion). The CLI encapsulates the §9 redeploy contract so callers cannot pin to the empty-shell v2; see `data-app-workflow.md` for the gotcha inventory and recipes. Since v0.33.0 the JSON output envelope's data-app id key is `app_id` (renamed from bare `id` for symmetry with the `--app-id` input flag); `config_id` is unchanged. - `data-app list [--project NAME ...] [--branch ID]` -- list data apps across projects (Data Science index merged with Storage names). Since v0.43.9 filters out workspace/sandbox deployments (`componentId=keboola.sandboxes`, `type=snowflake`/`bigquery`) that the Data Science `/apps` collection also returns, so the listing matches the Apps UI. Envelope carries `component_id` per app. - `data-app detail --project NAME --app-id ID [--branch ID]` -- merged view (state, desired, url, configVersion, slug, git block with PAT redacted) -- `data-app create --project ALIAS --name NAME --slug SLUG (--git-repo URL | --use-managed-git-repo) [--git-public/--no-git-public] [--git-username USER] [--git-pat-env VAR | --git-pat-file PATH | --git-pat-encrypted KBC::Project...] [--auth password|public] [--size tiny|small|medium|large] [--auto-suspend SECONDS] [--type python-js|python|streamlit|r|...] [--workspace/--no-workspace] [--branch ID] [--no-deploy] [--wait] [--timeout SECONDS] [--keep-on-failure] [--dry-run]` -- POST shell + encrypt PAT + PUT Storage config (with auto-injected `parameters.id`) + PATCH deploy with the §9 trio. Cleanup-in-finally on failure unless `--keep-on-failure`. Default `--auth password` mints a 20-char hex simpleAuth password (retrievable via `data-app password`). **Exactly one git source required.** `--use-managed-git-repo` (since 0.65.0) provisions an EMPTY Keboola-hosted repo (POST `useManagedGitRepo:true`), writes NO `parameters.dataApp.git` block, and forces `--no-deploy` (empty repo, nothing to run); mutually exclusive with `--git-repo` and all `--git-*`/PAT flags. Managed-repo deploy WORKS with no credential wiring (verified live -- tic-tac-toe deployed and serving from a Keboola-managed repo). Full flow to a RUNNING app: `git-credentials-create --type http_token --permissions readWrite` -> `git push` your code to the managed repo URL (`data-app git-repo` shows it) -> `data-app deploy`. The platform injects the clone credentials at deploy time, so nothing extra is wired into the config; the minted credential is only used to authenticate YOUR push. **`--workspace` (since 0.87.0) is ON by default** and writes `runtime.workspace.enabled: true` -- the single switch that makes the platform provision the app's ephemeral workspace and inject `WORKSPACE_ID` / `QUERY_SERVICE_URL` / `KBC_WORKSPACE_MANIFEST_PATH`. Every app that reads Storage needs it; before 0.87.0 kbagent never wrote it, so such an app deployed, reported `state=running`, passed its health probe and read nothing, with NO platform-side diagnostic (check `config detail` -> `configuration.runtime`; a `Missing env vars: WORKSPACE_ID` log line, if any, comes from the app's own code, so its absence rules nothing out). Pass `--no-workspace` only for an app that never touches Storage -- it omits the key entirely (no `enabled: false`), leaving the body identical to 0.86.0. The block is a sibling of `runtime.backend`. Not gated on any project feature. Retrofit an existing app with `config update --merge --set 'runtime.workspace.enabled=true'` then `data-app deploy`. +- `data-app create --project ALIAS --name NAME --slug SLUG (--git-repo URL | --use-managed-git-repo) [--git-public/--no-git-public] [--git-username USER] [--git-pat-env VAR | --git-pat-file PATH | --git-pat-encrypted KBC::Project...] [--auth password|public] [--size tiny|small|medium|large] [--auto-suspend SECONDS] [--type python-js|python|streamlit|r|...] [--workspace/--no-workspace] [--branch ID] [--no-deploy] [--wait] [--timeout SECONDS] [--keep-on-failure] [--dry-run]` -- POST shell + encrypt PAT + PUT Storage config (with auto-injected `parameters.id`) + PATCH deploy with the §9 trio. Cleanup-in-finally on failure unless `--keep-on-failure`. Default `--auth password` mints a 20-char hex simpleAuth password (retrievable via `data-app password`). **Exactly one git source required.** `--use-managed-git-repo` provisions an EMPTY Keboola-hosted repo (POST `useManagedGitRepo:true`), writes NO `parameters.dataApp.git` block, and forces `--no-deploy` (empty repo, nothing to run); mutually exclusive with `--git-repo` and all `--git-*`/PAT flags. Managed-repo deploy WORKS with no credential wiring (verified live -- tic-tac-toe deployed and serving from a Keboola-managed repo). Full flow to a RUNNING app: `git-credentials-create --type http_token --permissions readWrite` -> `git push` your code to the managed repo URL (`data-app git-repo` shows it) -> `data-app deploy`. The platform injects the clone credentials at deploy time, so nothing extra is wired into the config; the minted credential is only used to authenticate YOUR push. **`--workspace` (since 0.87.0) is ON by default** and writes `runtime.workspace.enabled: true` -- the single switch that makes the platform provision the app's ephemeral workspace and inject `WORKSPACE_ID` / `QUERY_SERVICE_URL` / `KBC_WORKSPACE_MANIFEST_PATH`. Every app that reads Storage needs it; before 0.87.0 kbagent never wrote it, so such an app deployed, reported `state=running`, passed its health probe and read nothing, with NO platform-side diagnostic (check `config detail` -> `configuration.runtime`; a `Missing env vars: WORKSPACE_ID` log line, if any, comes from the app's own code, so its absence rules nothing out). Pass `--no-workspace` only for an app that never touches Storage -- it omits the key entirely (no `enabled: false`), leaving the body identical to 0.86.0. The block is a sibling of `runtime.backend`. Not gated on any project feature. Retrofit an existing app with `config update --merge --set 'runtime.workspace.enabled=true'` then `data-app deploy`. - `data-app deploy --project NAME --app-id ID [--config-version N] [--wait] [--timeout SECONDS] [--branch ID]` -- the §9 redeploy contract. Default reads latest Storage version; `--config-version` pins an older version (rollback). Since 0.65.0: omits `configVersion` for a PURE managed repo (no git block -- deploys from `app.managedGitRepoId`, and the platform injects the clone credentials) and pins the LATEST Storage `configVersion` when a git block is present (external repos). An explicit `--config-version` always wins. - `data-app start --project NAME --app-id ID [--wait] [--timeout SECONDS]` -- wake an auto-suspended app at the currently-pinned version. Distinct from deploy: does NOT bump configVersion. - `data-app stop --project NAME --app-id ID [--wait] [--timeout SECONDS]` -- stop a running app (URL and Storage config preserved). - `data-app delete --project NAME --app-id ID [--yes]` -- destructive, cascades to Storage config; URL retired permanently. - `data-app password --project NAME --app-id ID` -- read the simpleAuth password. Manage token via interactive prompt by default, or `--allow-env-manage-token` + `KBC_MANAGE_API_TOKEN` for CI on 0.29.0+. Auto-generated, not rotatable -- delete + recreate to mint a new one. - `data-app logs --project NAME --app-id ID [--lines N] [--since ISO8601]` -- tail container logs (Data Science `/apps/{id}/logs/tail`). Plain-text body covering the full spin-up trace ([TIMING] git_clone, Cloning into /app, uv install, supervisord, runtime stack traces). Default `--lines 500`; pass `--lines 0` for the full current buffer (no server-side cap). `--lines` and `--since` are mutually exclusive on the server; `--since` requires a timezone (Z or +00:00). App must be running or recently-stopped — never-started apps return 400 "App X is not running" (recover with `data-app start` or `data-app deploy`). Closes the upstream `keboola-mcp-server` gap where `get_data_apps` hardcodes a 20-line cap; this CLI surface is unconstrained. The log buffer can echo runtime secrets the app printed to stdout/stderr — consider hygiene before piping `--json` output into AI agent context. -- `data-app runs --project NAME --app-id ID [--limit N]` (since 0.65.0) -- list deployment attempts newest-first (Data Science `/apps/{id}/runs`), each with `failure_reason` + `startup_logs`. Captures setup-phase failures (e.g. git-clone errors) that produce NO container logs, so unlike `data-app logs` it works on never-started / failed apps where `data-app logs` returns HTTP 400. This is the way to find WHY a deploy reverted to stopped. Auth: ordinary project storage token only. +- `data-app runs --project NAME --app-id ID [--limit N]` -- list deployment attempts newest-first (Data Science `/apps/{id}/runs`), each with `failure_reason` + `startup_logs`. Captures setup-phase failures (e.g. git-clone errors) that produce NO container logs, so unlike `data-app logs` it works on never-started / failed apps where `data-app logs` returns HTTP 400. This is the way to find WHY a deploy reverted to stopped. Auth: ordinary project storage token only. - `data-app secrets-set --project ALIAS --app-id ID --secret '#KEY=VALUE' [--secret ...] [--secrets-file PATH] [--branch ID] [--allow-plaintext-on-encrypt-failure] [--dry-run] [--no-hint-next]` -- encrypt and write `#`-prefixed secrets to `parameters.dataApp.secrets`. Per-project KMS encryption, fail-closed. Read-modify-write at the service layer (NOT Storage `merge=True` -- shallow). Runtime exposes each key as an env var with `#` stripped, `-` -> `_`, uppercased. Adding bumps the Storage version; the running container keeps the OLD config until the next `data-app deploy`. - `data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]` -- list secret keys + derived runtime env-var names. Never echoes encrypted ciphertext in full. `--show-fingerprint` opt-in for a short ciphertext fingerprint. - `data-app secrets-get --project ALIAS --app-id ID --key 'KEY' [--branch ID]` -- show ONE key from `parameters.dataApp.secrets`. The leading `#` is OPTIONAL (since v0.43.9); the block holds both encrypted secrets (`#`) and plain unencrypted env-var values, and `secrets-list` enumerates both. For an ENCRYPTED secret it stays metadata-only (`encrypted: true`, `value: null`, fingerprint/prefix) -- the decrypted plaintext is NEVER echoed (Encryption API is one-way). For a PLAIN value it returns the literal value (`encrypted: false`), which is already visible via `config detail`. NOT_FOUND on absent key (exact match, no `#KEY`<->`KEY` fuzzing); never enumerates siblings. -- `data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run]` -- destructive (can break a running app at next deploy). Leading `#` OPTIONAL (since v0.43.9): removes both encrypted secrets and plain env-var keys. Idempotent: missing keys exit 0 with `removed: 0`. +- `data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run]` -- destructive (can break a running app at next deploy). Leading `#` OPTIONAL: removes both encrypted secrets and plain env-var keys. Idempotent: missing keys exit 0 with `removed: 0`. - `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures. Since 0.88.0 (#636) the two `setup.sh` rules (`golden-rule.setup-sh-no-pip`, `golden-rule.setup-sh-uv-sync`) match against **comment-stripped code**: a comment reading `# never pip install` no longer BLOCKS, and a comment merely mentioning `uv sync` no longer satisfies the uv-sync rule -- `data-app git-repo --project NAME --app-id ID` (since 0.63.3) -- show the clone URLs (`ssh_url` / `https_url`) of the app's configured git repo + `is_managed_git_repo` (sandboxes-service `GET /apps/{id}/git-repo`). Read-only, project storage token only. **GOTCHA**: returns 409 `no Git repository configured` until the app has been DEPLOYED at least once -- the git block is synced from the Storage config into the Data Science app record at deploy time; a `--no-deploy` app has no git repo from the service's point of view. -- `data-app git-credentials --project NAME --app-id ID` (since 0.63.3) -- list the credentials of the app's MANAGED git repo (`id`, `type`, `permissions`, `name`, `owner_admin_id`, `created_at`). The secret is NEVER returned here. Needs an admin storage token; external repos have none. -- `data-app git-credentials-create --project NAME --app-id ID --type ssh_key|http_token --permissions readOnly|readWrite [--public-key KEY | --public-key-file PATH] [--name LABEL] [--yes]` (since 0.63.3) -- mint a git credential for the app's MANAGED git repo. `ssh_key` requires a public key; `http_token` returns a ONE-TIME secret (shown once, never retrievable again -- mirrors `data-app password`). Needs an admin storage token. Apps from `data-app create --git-repo` are EXTERNAL => 409 `no managed Git repository`. Confirmation unless `--yes`/`--json`. For a managed-repo app this credential authenticates YOUR `git push` of the code; the deploy itself uses the platform's injected clone credentials -- no further wiring needed. +- `data-app git-repo --project NAME --app-id ID` -- show the clone URLs (`ssh_url` / `https_url`) of the app's configured git repo + `is_managed_git_repo` (sandboxes-service `GET /apps/{id}/git-repo`). Read-only, project storage token only. **GOTCHA**: returns 409 `no Git repository configured` until the app has been DEPLOYED at least once -- the git block is synced from the Storage config into the Data Science app record at deploy time; a `--no-deploy` app has no git repo from the service's point of view. +- `data-app git-credentials --project NAME --app-id ID` -- list the credentials of the app's MANAGED git repo (`id`, `type`, `permissions`, `name`, `owner_admin_id`, `created_at`). The secret is NEVER returned here. Needs an admin storage token; external repos have none. +- `data-app git-credentials-create --project NAME --app-id ID --type ssh_key|http_token --permissions readOnly|readWrite [--public-key KEY | --public-key-file PATH] [--name LABEL] [--yes]` -- mint a git credential for the app's MANAGED git repo. `ssh_key` requires a public key; `http_token` returns a ONE-TIME secret (shown once, never retrievable again -- mirrors `data-app password`). Needs an admin storage token. Apps from `data-app create --git-repo` are EXTERNAL => 409 `no managed Git repository`. Confirmation unless `--yes`/`--json`. For a managed-repo app this credential authenticates YOUR `git push` of the code; the deploy itself uses the platform's injected clone credentials -- no further wiring needed. ## The `tool` group (REMOVED in v0.85.0 -- epic #390 phase 3) `tool list` / `tool call` and `agent --type mcp_tool` no longer exist. Every catalog tool has a native command -- look an old tool name up in `docs/mcp-migration.md` and run its replacement. Surviving `mcp_tool` agent tasks are inert tombstones: they never run, `agent list` flags them, and `doctor` reports them as FAIL. `keboola-mcp-server` itself is unaffected (a separate distribution kbagent no longer installs or updates -- refresh it with `uv tool install --upgrade --prerelease=allow keboola-mcp-server`). -## SQL Transformations (since v0.73.0) +## SQL Transformations Ports the `create_sql_transformation` / `update_sql_transformation` tools (#396). See [transformation-workflow.md](transformation-workflow.md) for the show-before-edit recipe. - `transformation create --project NAME --name NAME (--sql 'SELECT ...' | --sql-file PATH) [--created-table NAME ...] [--component-id ID] [--description D] [--branch ID] [--dry-run]` -- component id from project `default_backend` (snowflake/bigquery; else pass `--component-id`); SQL split one statement per `script[]` element into a single block `Blocks`/code `Code`; each `--created-table T` maps to `out.c-.` (bucket derived from the transformation NAME -- renaming later breaks the match). - `transformation show --project NAME --config-id ID [--component-id ID] [--branch ID]` -- block/code tree with synthetic positional ids `b{i}`/`b{i}.c{j}` + storage. Probes all SQL transformation components when `--component-id` omitted. **Always show before edit** -- ids renumber after structural ops. - `transformation edit --project NAME --config-id ID --change-description TEXT (--op JSON ... | --op-file ops.json) [--storage JSON|@file|-] [--component-id ID] [--branch ID] [--dry-run]` -- 9 ops (`add_block`, `remove_block`, `rename_block`, `add_code`, `remove_code`, `rename_code`, `set_code`, `add_script`, `str_replace`) applied sequentially against batch-start ids. `--storage` REPLACES `configuration.storage` wholesale (include ALL mappings you want to keep). Unknown ids error with the list of valid ids. -## Documentation Q&A (since v0.73.0) +## Documentation Q&A - `docs query "QUESTION" [--project NAME]` -- natural-language answer from the Keboola documentation via the AI Service (server-side RAG, no local corpus). Returns answer text + source URLs; `--json` emits `{query, text, source_urls}`. Unlike `kai ask` it does NOT see project data, works with any token (no master-token / feature-flag requirement), and is the right tool for "how do I ..." questions. Ports the `docs_query` MCP tool. ## Kai (Keboola AI Assistant) @@ -291,7 +291,7 @@ Requires the project to be added with its **master ('owner') Storage API token** - `kai history [--project NAME] [--limit N]` -- list recent Kai chat sessions (default limit: 10) ## Flows (Conditional Flows -- keboola.flow only) -- `flow examples [--component-id keboola.flow|keboola.orchestrator]` (since 0.73.0) -- bundled example flow configs (vendored from keboola-mcp-server), fully offline; default `keboola.flow`; `keboola.orchestrator` examples are informational-only (kbagent cannot create/edit orchestrator flows). Ports the `get_flow_examples` MCP tool. +- `flow examples [--component-id keboola.flow|keboola.orchestrator]` -- bundled example flow configs (vendored from keboola-mcp-server), fully offline; default `keboola.flow`; `keboola.orchestrator` examples are informational-only (kbagent cannot create/edit orchestrator flows). Ports the `get_flow_examples` MCP tool. > Since 0.57.0 the `flow` group targets `keboola.flow` (Conditional Flows) ONLY; `keboola.orchestrator` is dropped and `--component-id` is removed from every subcommand. IDs are **strings**; phases use `next[].goto` (a phase id or `null`) + optional `condition`; tasks are typed (`job`/`notification`/`variable`). The old `dependsOn` template is invalid. Execute a flow with `kbagent job run --component-id keboola.flow --config-id ID`. See `flow-workflow.md`. - `flow list [--project NAME] [--branch ID] [--with-schedules]` -- list conditional flows (keboola.flow) across one or all projects. Legacy keboola.orchestrator configs are NOT listed; their total appears as `legacy_orchestrator_count` (+ a warning). `--with-schedules` enriches each row with `schedules: [{schedule_id, cron, timezone, enabled}, ...]` via one extra keboola.scheduler list call per project (not per flow) - `flow detail --project NAME --flow-id ID [--branch ID]` -- full phase/task breakdown; per-phase transitions (`→ goto [condition | default]`), typed-task badges, retry info; JSON is the raw body unchanged @@ -324,10 +324,10 @@ Requires the project to be added with its **master ('owner') Storage API token** - Exposed over `kbagent serve` as `GET /notifications`, `GET /notifications/{project}/{subscription_id}`, `POST /notifications/{project}`, `DELETE /notifications/{project}/{subscription_id}`, and `POST /notifications/{project}/{subscription_id}/replace-recipient` ## Sync (GitOps) -- `sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing]` -- initialize sync working directory; `--adopt-existing` (since v0.22.0) adopts a `.keboola/manifest.json` already written by the kbc Go CLI without overwriting (idempotent; validates `project_id` against the alias token) +- `sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing]` -- initialize sync working directory; `--adopt-existing` adopts a `.keboola/manifest.json` already written by the kbc Go CLI without overwriting (idempotent; validates `project_id` against the alias token) - `sync pull --project ALIAS [--all-projects] [--force] [--theirs] [--dry-run] [--with-samples] [--no-storage] [--no-jobs] [--job-limit N] [--branch ID]` -- download configs to local files. For large projects (>100 configs), automatically fetches jobs per-config when the grouped API limit is insufficient. `--force` is conflict-aware (since 0.53.0): a locally-modified config whose remote is unchanged is **preserved** (pending delta stays pushable, never silently re-stamped); a true merge conflict (local AND remote both changed since last pull) **aborts** the pull (exit 1, `SYNC_CONFLICT`; `--json` lists `details.conflicts`); local-untouched + remote-changed takes remote. `--theirs` (since v0.72.0) is the supported "discard local, take production" reconcile path: overwrites locally-modified configs/rows, restores deleted/missing files, resolves conflicts by taking remote (no abort, no manifest surgery). Since v0.72.0 plain pull also re-materializes a tracked config whose local dir was deleted (manifest<->disk invariant), so delete-dir-then-pull refetches. Config-level `isDisabled` round-trips (since v0.72.0) as sparse `is_disabled: true` in `_config.yml` -- absent key = enabled. `--branch` (0.47.0+) per-invocation dev-branch override, beats every other branch source. Ignored components (since 0.91.0): `keboola.sandboxes` + `keboola.mcp-server-tool` are always excluded, unioned with the manifest's `ignoredComponents` list; a component newly ignored has its manifest entry dropped and local directory removed, reported with pull action `"ignored"` (distinct from `"removed"` = genuinely deleted on remote). - `sync push --project ALIAS [--all-projects] [--dry-run] [--force] [--allow-plaintext-on-encrypt-failure] [--branch ID] [--no-name-drift-warnings]` -- push local changes (auto-encrypts secrets, fails if encryption fails). Fresh-CREATE writeback updates placeholder manifest entries in place (since 0.47.0) and propagates any `KBC.configuration.*` metadata via `set_config_metadata`. Fresh-CREATE variable binding (since 0.47.2): when a `keboola.variables` config + its values row are created alongside a transformation in the same push, the transformation's `variables_id` / `variables_values_id` placeholders are rebound to the assigned ULIDs and the row's `values` are hoisted even without a `_keboola` block, so `job run` succeeds with no post-push `config variables-set` step (unresolvable/ambiguous links surface a `variable_link` entry in `errors[]`, never a broken link). Never-fetched guard (since v0.72.0): a manifest entry with an empty `pull_hash` and no local files (pre-0.72 name-collision phantom) is **never** planned as a remote DELETE -- diff/push exclude it and report it under `never_fetched` with a warning (run `sync pull` to materialize); local deletion of a properly-pulled config still deletes on push. Adopted-by-id writeback (since v0.72.0): pushing an untracked file whose `_keboola.config_id` resolves on the branch also writes the manifest entry, so follow-up diffs are stable. `--branch` (0.47.0+) per-invocation override; when no `/` subtree exists on disk (since 0.47.2) the local default tree (`main/`) is promoted to the target branch (API writes still target the branch id); `--no-name-drift-warnings` (0.47.0+) drops the cosmetic warnings array. Branch-scoped since v0.89.0 (issue #649): push consumes the diff's changeset, so configs tracked on another branch's tree are never planned as creates -- they ride along on the result envelope under `orphaned` instead (see `sync diff`). **Since 0.91.0 (#686)** the manifest baseline `pull_config_hash` is stamped from the API response (or a read-back), not from the files on disk, so a pushed multi-statement SQL transformation -- or anything disabled in the UI whose local YAML lacks `is_disabled` -- no longer shows permanent phantom `REMOTE MODIFIED` drift; if the config cannot be read back after the write the baseline is left UNTOUCHED and a `warnings[]` entry says to run `sync pull` (never a disk-derived fallback). One legacy change is refused per-change with `SYNC_LEGACY_BOUNDARY`: a tree pulled before statement-boundary markers existed whose only difference from the remote is the lost boundaries (pushing it would collapse separate SQL statements into one) -- run `sync pull` for that project first. Ignored components (since 0.91.0) are filtered out on both sides of the diff push builds on, so a stale local directory for an ignored component (e.g. `keboola.mcp-server-tool`) is never classified as `DELETED` and can never be pushed as a remote deletion. -- `sync clone --source DIR --target ALIAS --target-dir DIR [--bucket-map FILE] [--variable-values FILE] [--instance-rename FILE] [--dry-run] [--branch ID]` -- clone a reference synced project into a **fresh** target project and parameterize it (since v0.63.0). Copies the reference tree at `--source` into `--target-dir`, applies declarative overrides from JSON/YAML files (`--bucket-map` `{old_bucket_id: new_bucket_id}` rewrites storage input/output table refs; `--variable-values` `{var_name: value}` overrides `keboola.variables` rows; `--instance-rename` `{old_path_prefix: new_path_prefix}` renames config dirs + manifest paths), re-points the manifest at the target project, and pushes. Because the reference's config ids do not exist in the fresh target, every config is CREATEd fresh and **keboola.flow task `configId`s + transformation variable links are remapped reference->ULID** by push Phase C/D (the push result carries `flow_task_remaps`). **Idempotent**: re-running with an existing `--target-dir` skips copy/overrides and just pushes, reporting `no_changes` / `created: 0`. Fails fast (`CONFIG_ERROR`) if the target already contains the reference's configs -- clone requires a fresh/empty target. `SyncService.clone_project(...)` returns a typed `CloneResult` for in-process SDK callers. Override files must be flat `{id: scalar}` mappings *(since v0.89.0)* -- a nested mapping, list, or null value is rejected with `CONFIG_ERROR` (exit 5) naming the key and its actual type. +- `sync clone --source DIR --target ALIAS --target-dir DIR [--bucket-map FILE] [--variable-values FILE] [--instance-rename FILE] [--dry-run] [--branch ID]` -- clone a reference synced project into a **fresh** target project and parameterize it. Copies the reference tree at `--source` into `--target-dir`, applies declarative overrides from JSON/YAML files (`--bucket-map` `{old_bucket_id: new_bucket_id}` rewrites storage input/output table refs; `--variable-values` `{var_name: value}` overrides `keboola.variables` rows; `--instance-rename` `{old_path_prefix: new_path_prefix}` renames config dirs + manifest paths), re-points the manifest at the target project, and pushes. Because the reference's config ids do not exist in the fresh target, every config is CREATEd fresh and **keboola.flow task `configId`s + transformation variable links are remapped reference->ULID** by push Phase C/D (the push result carries `flow_task_remaps`). **Idempotent**: re-running with an existing `--target-dir` skips copy/overrides and just pushes, reporting `no_changes` / `created: 0`. Fails fast (`CONFIG_ERROR`) if the target already contains the reference's configs -- clone requires a fresh/empty target. `SyncService.clone_project(...)` returns a typed `CloneResult` for in-process SDK callers. Override files must be flat `{id: scalar}` mappings *(since v0.89.0)* -- a nested mapping, list, or null value is rejected with `CONFIG_ERROR` (exit 5) naming the key and its actual type. - `sync diff --project ALIAS [--all-projects] [--branch ID]` -- 3-way diff (local vs base vs remote), detects conflicts. `--branch` (0.47.0+) per-invocation dev-branch override. Branch-scoped since v0.89.0 (issue #649): the local side is read from exactly ONE tree (the target branch's subtree, or `main/` when the target has none). Manifest entries belonging to another branch's tree -- what `sync pull --branch ` leaves behind when it re-targets the manifest -- are excluded from the changeset and reported under `orphaned` (`summary.orphaned` + details with `component_id`, `config_id`, `path`, `branch_id`, `branch_path`, `exists_on_target`, `reason`, `hint`); human mode previews the first 10. An orphaned FILE whose `_keboola.config_id` still resolves on the target is adopted (diffed as `unchanged`/`modified`), never re-created; same-tree id claims keep the #482/#497 fork-by-copy CREATE. Fix a non-zero `summary.orphaned` with `sync pull`. **Since 0.91.0 (#686)** a manifest entry without `metadata.config_hash_version` (written by a pre-0.91.0 kbagent) is compared leniently: a stored hash equal to the pre-0.91.0 hash of the SAME remote config counts as in sync, so the phantom `codes changed` entries disappear immediately; every other field is still pinned by that hash, so real remote drift is unaffected. One `sync pull` per project stamps the version and ends the leniency. Ignored components (since 0.91.0) -- `keboola.sandboxes`, `keboola.mcp-server-tool`, and anything listed in the manifest's `ignoredComponents` -- are excluded from BOTH sides of the comparison, so a stale local directory for one of them never shows up as `DELETED`. - `sync status [--directory DIR]` -- show locally modified/added/deleted configs. Also surfaces `plaintext_secret_warnings` (since 0.55.0): in-sync configs/rows whose `#`-secrets are still plaintext on the remote (a leftover from pre-0.54.0 writes; #378). Pending (un-pushed) edits are not flagged. Fix = re-push on >=0.54.0 + rotate (version history keeps the plaintext). - `sync branch-link --project ALIAS [--branch-id ID] [--branch-name NAME]` -- link git branch to Keboola dev branch @@ -341,11 +341,11 @@ Requires the project to be added with its **master ('owner') Storage API token** Talks to `apps-api.keboola.com`. **Reads are unrestricted; writes always require a human to type a random hex code on a real TTY (no `--yes`, no env bypass, exit 6 on non-TTY).** Use `--dry-run` for the agent-safe preview path. -`--role-hint` is **load-bearing** for `dev-portal patch` (since v0.51.1): `vendor` (default) → `PATCH /vendors/{vendor}/apps/{app}` (restricted schema, the common case); `admin` → `PATCH /admin/apps/{app}` (permissive schema, the only way to set `complexity`, `categories`, `category`, `features`, `forwardToken`, `forwardTokenDetails`, `injectEnvironment`, `processTimeout`, `requiredMemory`). A `vendor` identity with any of those 9 fields in the payload fails fast at preflight with the exact command to switch. +`--role-hint` is **load-bearing** for `dev-portal patch`: `vendor` (default) → `PATCH /vendors/{vendor}/apps/{app}` (restricted schema, the common case); `admin` → `PATCH /admin/apps/{app}` (permissive schema, the only way to set `complexity`, `categories`, `category`, `features`, `forwardToken`, `forwardTokenDetails`, `injectEnvironment`, `processTimeout`, `requiredMemory`). A `vendor` identity with any of those 9 fields in the payload fails fast at preflight with the exact command to switch. -`--password-stdin` (since v0.51.1) works in both TTY mode (hidden line-based prompt, Enter to confirm) and pipe mode (`echo $PASS | … --password-stdin`, reads to EOF). +`--password-stdin` works in both TTY mode (hidden line-based prompt, Enter to confirm) and pipe mode (`echo $PASS | … --password-stdin`, reads to EOF). -MFA login (since v0.51.1) sends `challenge: SOFTWARE_TOKEN_MFA` explicitly to fix a 404 on personal-account TOTP logins where the apps-api server silently rejects missing-challenge requests despite the spec calling it optional. Single attempt only; failure surfaces the actual server body with a stale-TOTP hint. +MFA login sends `challenge: SOFTWARE_TOKEN_MFA` explicitly to fix a 404 on personal-account TOTP logins where the apps-api server silently rejects missing-challenge requests despite the spec calling it optional. Single attempt only; failure surfaces the actual server body with a stale-TOTP hint. ### Identity management - `dev-portal identity add --alias A --username U [--password P | --password-stdin] [--role-hint vendor|admin] [--vendor V] [--portal-url URL]` -- store a portal login credential per-alias in `config.json` (0600 perms). `--role-hint` is validated (`vendor`/`admin`, case-folded) since v0.51.1. @@ -367,7 +367,7 @@ MFA login (since v0.51.1) sends `challenge: SOFTWARE_TOKEN_MFA` explicitly to fi - `dev-portal publish --app VENDOR.APP_ID [--identity A] [--dry-run]` -- publish the component (makes it visible in the UI). - `dev-portal deprecate --app VENDOR.APP_ID [--identity A] [--dry-run]` -- mark the component as deprecated. -## Semantic Layer (Metastore) (since v0.41.0) +## Semantic Layer (Metastore) -Reads `KBAGENT_SERVE_URL` + `KBAGENT_SERVE_TOKEN` env vars. The scheduler auto-injects these (plus `KBAGENT_CONFIG_DIR`) into every AI-agent / `cli_command` subprocess. Outside a serve subprocess context the command refuses to run with exit code 2. **Inside a scheduled-agent task, prefer `kbagent http get /openapi.json` then a typed call over forking another `kbagent` CLI -- the HTTP path always sees the operator's live config (not the global `~/.config/keboola-agent-cli/` one).** Manage Keboola metastore models -- datasets, metrics, relationships, constraints, glossary terms. Metastore URL derived from the stack URL by replacing `connection.` with `metastore.` (cloud/region-agnostic). Auth: same `X-StorageApi-Token` as Storage. Hidden alias: `kbagent sl ...` is equivalent to `kbagent semantic-layer ...`. See [semantic-layer-workflow.md](semantic-layer-workflow.md) for full recipes. @@ -376,9 +376,9 @@ Manage Keboola metastore models -- datasets, metrics, relationships, constraints - `semantic-layer model create --project P --name N [--description D] [--sql-dialect Snowflake]` -- create a new model. `--sql-dialect` defaults to `Snowflake`. Returns the new model UUID; subsequent commands accept either name or UUID via `--model`. - `semantic-layer model delete --project P --model M [--yes]` -- delete a model **and cascade-delete every child entity** (datasets, metrics, relationships, constraints, glossary terms) in `reversed(PUSH_ORDER)` (constraints first, datasets last) before the parent. Confirmation prompt unless `--yes`. **Cascade is unconditional in 0.43.4+** -- before that release the call only DELETEd the parent, silently leaking children pointing at the dead `modelUUID` and breaking subsequent `build` / `import` retries with HTTP 422 name collisions (closes #306). On any child-DELETE failure the parent is **preserved** and the response carries `details.cascade = {attempted, deleted, failures: [{type, id, name, error}], parent_deleted: False, model_uuid}` so the user can re-run after fixing the underlying error. Happy-path envelope adds `cascade.deleted` per-type counts. Legacy `orphaned_children` top-level key kept for back-compat (same shape, meaning flipped from "leaked" to "cascaded") but **deprecated -- removal scheduled for a future minor release**; new callers should read `cascade.deleted` instead. See [gotchas.md](gotchas.md) for the meaning-flip + deprecation note. - `semantic-layer show --project P [--model M] [--type T]` -- show a model's entities. `--type` filters to `dataset | metric | relationship | constraint | glossary`. Without `--type` prints a per-type count summary. `--model` is optional when the project has exactly one model. -- `semantic-layer search-context --project P [--pattern G ...] [--type model|dataset|metric|relationship|constraint|glossary|all] [--limit N]` (since 0.47.0) -- project-wide glob search across semantic-layer entity names. Mirrors the upstream `keboola-mcp-server search_semantic_context` MCP tool so a downstream caller can drop the MCP dependency for the pre-flight "is the model populated?" check. Patterns are case-sensitive `fnmatch`, repeatable (union); default `*`. Default `--type all` searches every CHILD type (`model` searches semantic models). `--limit N` short-circuits both per-type and outer loops. Envelope: `{project, contexts: [{id, type, name, description, attributes}], total_count}`; the `type` field is the CLI-friendly singular (no `semantic-` prefix). -- `semantic-layer schema --project P (--type model|dataset|metric|relationship|constraint|glossary[,TYPE...] | --all)` (since 0.73.0) -- live JSON Schema per semantic object type, fetched from the deployed metastore (never bundled, cannot drift). Exactly one of `--type`/`--all` (usage error otherwise); `--type` takes a comma-separated list, fan-out is parallel. The bare schema endpoint returns only a version LISTING -- the service resolves the `isDefault` version and fetches the real schema (a deliberate improvement over the upstream `get_semantic_schema` MCP tool, which passes the bare listing through). Envelope: `{project, schemas: [{type, schema, schema_version}]}`. -- `semantic-layer get-context --project P --context-id ID` (since 0.47.0) -- single-entry fetch by id, irrespective of type. Probes `semantic-model` first then every CHILD type (dataset / metric / relationship / constraint / glossary) until a 200 lands. 404 on any one type is non-terminal; only a full miss raises `NOT_FOUND` (exit 1). Non-404 errors (500, etc.) propagate immediately rather than being swallowed by the next probe. +- `semantic-layer search-context --project P [--pattern G ...] [--type model|dataset|metric|relationship|constraint|glossary|all] [--limit N]` -- project-wide glob search across semantic-layer entity names. Mirrors the upstream `keboola-mcp-server search_semantic_context` MCP tool so a downstream caller can drop the MCP dependency for the pre-flight "is the model populated?" check. Patterns are case-sensitive `fnmatch`, repeatable (union); default `*`. Default `--type all` searches every CHILD type (`model` searches semantic models). `--limit N` short-circuits both per-type and outer loops. Envelope: `{project, contexts: [{id, type, name, description, attributes}], total_count}`; the `type` field is the CLI-friendly singular (no `semantic-` prefix). +- `semantic-layer schema --project P (--type model|dataset|metric|relationship|constraint|glossary[,TYPE...] | --all)` -- live JSON Schema per semantic object type, fetched from the deployed metastore (never bundled, cannot drift). Exactly one of `--type`/`--all` (usage error otherwise); `--type` takes a comma-separated list, fan-out is parallel. The bare schema endpoint returns only a version LISTING -- the service resolves the `isDefault` version and fetches the real schema (a deliberate improvement over the upstream `get_semantic_schema` MCP tool, which passes the bare listing through). Envelope: `{project, schemas: [{type, schema, schema_version}]}`. +- `semantic-layer get-context --project P --context-id ID` -- single-entry fetch by id, irrespective of type. Probes `semantic-model` first then every CHILD type (dataset / metric / relationship / constraint / glossary) until a 200 lands. 404 on any one type is non-terminal; only a full miss raises `NOT_FOUND` (exit 1). Non-404 errors (500, etc.) propagate immediately rather than being swallowed by the next probe. - `semantic-layer validate --project P [--model M] [--deep]` -- structural validation. Basic mode runs local checks: duplicate names, dangling rel/metric refs, SUM-on-PCT (warning), constraint orphans (metrics in `metrics[]` that no longer exist), severity-suffix mismatches between API `severity` and the 4-band name suffix. `--deep` adds parallel Snowflake column-existence probes via the in-process StorageService: phantom dataset fields, phantom column refs in metric SQL, AGG-on-STRING errors. Response: `{valid: bool, deep: bool, errors: [{type, item, detail}], warnings: [...]}`. - `semantic-layer export --project P [--model M] [--output PATH]` -- snapshot the model to a self-describing JSON file (default `./sl_export_{model_name}_{YYYYMMDD_HHMMSS}.json`). Schema-versioned for round-trip via `import` / `diff`. - `semantic-layer diff (--project-a A | --file-a P) (--project-b B | --file-b P) [--model-a M] [--model-b M]` -- three-way diff: project<->project, project<->file, file<->file. Mutually exclusive per side: pass exactly one of `--project-a` / `--file-a`, ditto for B. Output groups changes per entity type: `added[] / removed[] / changed[{name, diff_keys[]}]`. @@ -387,7 +387,7 @@ Manage Keboola metastore models -- datasets, metrics, relationships, constraints - `semantic-layer add relationship --project P [--model M] --name N --from TABLE_ID --to TABLE_ID --on EXPR [--type left|inner]` -- add a join relationship. `--type` defaults to `left`. - `semantic-layer add constraint --project P [--model M] --name N --constraint-type T --rule "EXPR" --metrics M1,M2 [--severity error|warning|info]` -- add a constraint. `--constraint-type` is the closed enum `inequality|equality|range|composition|exclusion|temporal|conditional`. `--rule` is a **STRING expression** (e.g. `"value >= 0"`), NEVER a `{bounds: {min, max}}` object (sl-builder docs are wrong -- see [gotchas.md](gotchas.md)). `--metrics` is a comma-separated list of metric names that must already exist in the model. `--severity` defaults to `warning`. Name regex `^[a-z][a-z0-9_]*$`; the 4-band health convention lives in the name suffix `_critical / _warning / _healthy / _review`, distinct from the 3-value API `severity`. - `semantic-layer add glossary --project P [--model M] --term TERM [--definition D]` -- add a glossary term. -- `semantic-layer edit metric --project P [--model M] --name N [--new-name N2] [--new-sql SQL] [--new-dataset TABLE_ID] [--new-description D] [--yes]` -- edit a metric. The metastore has NO PATCH endpoint, so this is DELETE+POST. Rename CASCADES through every constraint whose `metrics[]` includes the old name (DELETE old constraint + POST new with updated `metrics[]`). Prints the old/new CODE_METRIC computed via `re.sub(r"[^A-Z0-9]+", "_", name.upper()).strip("_")`. `--yes` skips the confirm prompt. On POST failure the service re-POSTs `original_attrs` and reports rollback success/failure explicitly in the envelope. **Partial-state envelope (since v0.41.10)**: when metric rename succeeds but one or more dependent constraints fail to repoint, the response sets `partial_state: true` and `recovery_hint: ""` at the envelope's top level (was previously buried in `cascaded_constraints[i].status == 'failed'`). Human-mode CLI prints a bright red `PARTIAL STATE` banner above the per-entry list. Recovery recipe: `kbagent semantic-layer validate` to surface the dangling refs, then re-run each failed cascade via `edit constraint --new-metrics ...`. +- `semantic-layer edit metric --project P [--model M] --name N [--new-name N2] [--new-sql SQL] [--new-dataset TABLE_ID] [--new-description D] [--yes]` -- edit a metric. The metastore has NO PATCH endpoint, so this is DELETE+POST. Rename CASCADES through every constraint whose `metrics[]` includes the old name (DELETE old constraint + POST new with updated `metrics[]`). Prints the old/new CODE_METRIC computed via `re.sub(r"[^A-Z0-9]+", "_", name.upper()).strip("_")`. `--yes` skips the confirm prompt. On POST failure the service re-POSTs `original_attrs` and reports rollback success/failure explicitly in the envelope. **Partial-state envelope**: when metric rename succeeds but one or more dependent constraints fail to repoint, the response sets `partial_state: true` and `recovery_hint: ""` at the envelope's top level (was previously buried in `cascaded_constraints[i].status == 'failed'`). Human-mode CLI prints a bright red `PARTIAL STATE` banner above the per-entry list. Recovery recipe: `kbagent semantic-layer validate` to surface the dangling refs, then re-run each failed cascade via `edit constraint --new-metrics ...`. - `semantic-layer edit dataset --project P [--model M] --name N [--new-name N2] [--new-description D] [--new-grain G]` -- edit a dataset. No cascade -- metrics reference the dataset's tableId, not its name. - `semantic-layer edit constraint --project P [--model M] --name N [--new-name N2] [--new-rule "EXPR"] [--new-constraint-type T] [--new-severity error|warning|info] [--new-metrics M1,M2]` -- edit a constraint (DELETE+POST). Local validators enforce the name regex, constraintType enum, severity enum, and that every entry in `--new-metrics` exists in the model. - `semantic-layer edit relationship --project P [--model M] --name N [--new-name N2] [--new-from TABLE_ID] [--new-to TABLE_ID] [--new-on EXPR] [--new-type left|inner]` -- edit a relationship (DELETE+POST). No constraint cascade. Rollback on POST failure. @@ -399,10 +399,10 @@ Manage Keboola metastore models -- datasets, metrics, relationships, constraints - `semantic-layer remove glossary --project P [--model M] --term TERM [--yes]` -- destructive. Glossary entries aren't referenced by other entities; no orphan-check. - `semantic-layer import --project P --file PATH [--model M] [--types T,T,...] [--dry-run] [--yes] [--overwrite]` -- replay a snapshot. Default: SKIP on conflict (no surprise overwrites). `--overwrite` opts into DELETE+POST for conflicting items. `--types` filters to a subset (`datasets,metrics,relationships,glossary,constraints`). Dependency-ordered push: datasets -> metrics -> relationships -> glossary -> constraints. Response: `imported: {: {created, skipped, overwritten, failed: [{name, reason}]}}`. - `semantic-layer promote --from-project A --to-project B [--from-model M] [--to-model M] [--types T,T,...] [--dry-run] [--yes]` -- cross-project model copy with `modelUUID` rewrite to the target model's UUID. Classifies items NEW / IDENTICAL / CHANGED (deep-equality after stripping `modelUUID` + timestamps). Additive + overwrite only: NEVER deletes target items absent from source. Holds two MetastoreClients in try/finally. Response: per-type counts + `changes[]` with `diff_keys` and `failed[]`. -- `semantic-layer build --project P [--model M] --tables T,T,... [--name N] [--dry-run] [--keep-on-failure] [--output PATH]` -- non-interactive heuristic builder. **AI caveat**: the existing `ai_client` has no arbitrary-JSON endpoint, so `build` falls back to a deterministic heuristic synthesising one dataset + one COUNT(*) metric + one glossary entry per table (FQN derived; fields[] role-classified). Response carries `fallback_used: "heuristic"`. The push loop walks all 5 child types in dependency order -- this **fixes** the `sl-build` skill bug where `semantic-constraint` was silently dropped. `--model` omitted creates a new model (default name `kbagent_build_model` or `--name N`). **Rollback on push failure (since v0.41.10)**: every successfully-POSTed child is DELETEd in reverse PUSH_ORDER, and the model itself is DELETEd if we created it during this call. The wrapped `KeboolaApiError` carries `details.rollback={attempted, posted_children, deleted, failed_deletes, model_created_here, model_deleted, model_uuid}` so operators get full diagnostics. Pass `--keep-on-failure` to preserve the partial state for forensic inspection (mirrors `data-app create --keep-on-failure`); the wrapped error then carries `details.rollback.attempted=False, reason='keep_on_failure'`. +- `semantic-layer build --project P [--model M] --tables T,T,... [--name N] [--dry-run] [--keep-on-failure] [--output PATH]` -- non-interactive heuristic builder. **AI caveat**: the existing `ai_client` has no arbitrary-JSON endpoint, so `build` falls back to a deterministic heuristic synthesising one dataset + one COUNT(*) metric + one glossary entry per table (FQN derived; fields[] role-classified). Response carries `fallback_used: "heuristic"`. The push loop walks all 5 child types in dependency order -- this **fixes** the `sl-build` skill bug where `semantic-constraint` was silently dropped. `--model` omitted creates a new model (default name `kbagent_build_model` or `--name N`). **Rollback on push failure**: every successfully-POSTed child is DELETEd in reverse PUSH_ORDER, and the model itself is DELETEd if we created it during this call. The wrapped `KeboolaApiError` carries `details.rollback={attempted, posted_children, deleted, failed_deletes, model_created_here, model_deleted, model_uuid}` so operators get full diagnostics. Pass `--keep-on-failure` to preserve the partial state for forensic inspection (mirrors `data-app create --keep-on-failure`); the wrapped error then carries `details.rollback.attempted=False, reason='keep_on_failure'`. - `semantic-layer token --encrypt --project P --component-id C` -- encrypt the project's storage token for a transformation's `user_properties`. Builds `{"#metastore_token": }` from the project's already-stored Storage API token and delegates to the existing EncryptService. `--encrypt` is currently required; other modes are refused with `USAGE_ERROR` (exit 2). Output (human): the raw envelope ready to paste. JSON: full `{encrypted, component_id, project}`. -### Reference data (dimension members, e.g. a Chart of Accounts) (since 0.55.0) +### Reference data (dimension members, e.g. a Chart of Accounts) `semantic-reference-data` is a per-dimension member store: ONE record per dimension holding the full member list in a `members[]` array. The driving use case is a Chart of Accounts (the account list + all attributes) held in the metastore instead of a hardcoded Storage table. It is deliberately kept **outside** `build` / `export` / `diff` / cascade / `PUSH_ORDER` — its members come from `DIM_COA`, not from AI generation — so it has its own self-contained CRUD surface. Member field names mirror the `DIM_COA` columns 1:1 (snake_case: `account_code`, `account_name`, `parent_code`, `is_leaf`, `level_1_code`, `cf_category`, …). @@ -419,15 +419,15 @@ Manage Keboola metastore models -- datasets, metrics, relationships, constraints Reads `KBAGENT_SERVE_URL` + `KBAGENT_SERVE_TOKEN` env vars. The scheduler auto-injects these (plus `KBAGENT_CONFIG_DIR`) into every AI-agent / `cli_command` subprocess. Outside a serve subprocess context the command refuses to run with exit code 2. **Inside a scheduled-agent task, prefer `kbagent http get /openapi.json` then a typed call over forking another `kbagent` CLI -- the HTTP path always sees the operator's live config (not the global `~/.config/keboola-agent-cli/` one).** -## Agent Tasks (since v0.44.0) +## Agent Tasks CLI parity for the `/agents` REST surface. Reads/writes `/agents.json` -- the same on-disk format the in-process scheduler inside `kbagent serve` consumes. CRUD + ad-hoc `run` work offline; the cron loop that fires scheduled tasks still requires `kbagent serve` running. See [agent-tasks-cli-workflow.md](agent-tasks-cli-workflow.md) for full walkthroughs; [agent-tasks-rest-workflow.md](agent-tasks-rest-workflow.md) covers the REST/SSE form for AI-agent subprocesses. -**ID forms (since v0.44.0):** every subcommand that takes `TASK_ID` / `RUN_ID` accepts it positionally (`agent show TASK_ID`) **or** via a named flag (`--id` / `--task-id`, plus `--run-id` for `run-detail` / `run-events`). The flag aliases bring agent commands in line with the rest of the CLI, which identifies entities by flag everywhere else (`--job-id`, `--config-id`, `--app-id`, ...). Passing both forms with conflicting values is a usage error (exit 2). +**ID forms:** every subcommand that takes `TASK_ID` / `RUN_ID` accepts it positionally (`agent show TASK_ID`) **or** via a named flag (`--id` / `--task-id`, plus `--run-id` for `run-detail` / `run-events`). The flag aliases bring agent commands in line with the rest of the CLI, which identifies entities by flag everywhere else (`--job-id`, `--config-id`, `--app-id`, ...). Passing both forms with conflicting values is a usage error (exit 2). - `agent list` -- list all registered tasks (id / name / cron / type / state / last-run / next-run). - `agent show TASK_ID` -- full task detail including the action payload. -- `agent create --name N [--description D] [--cron CRON] [--manual] [--enabled/--disabled] (--type ai_agent --cli claude|codex|gemini --prompt P [--extra-arg ...] [--timeout SECONDS] | --type cli_command --argv ARG [--argv ARG ...] [--timeout SECONDS] | --from-file PATH|@path|-) [--trigger-task-id ID --trigger-on success|error|always]` -- persist a new task. Convenience flags cover the typical single-action case; `--from-file` accepts the full `{"type": ..., "params": ...}` JSON envelope. **`--extra-arg` / `extra_args` on an `ai_agent` task are honored only when `kbagent serve` runs with a truthy `KBAGENT_ALLOW_AI_EXTRA_ARGS` (since v0.60.2); otherwise they are dropped with a warning** -- see [gotchas.md](gotchas.md). +- `agent create --name N [--description D] [--cron CRON] [--manual] [--enabled/--disabled] (--type ai_agent --cli claude|codex|gemini --prompt P [--extra-arg ...] [--timeout SECONDS] | --type cli_command --argv ARG [--argv ARG ...] [--timeout SECONDS] | --from-file PATH|@path|-) [--trigger-task-id ID --trigger-on success|error|always]` -- persist a new task. Convenience flags cover the typical single-action case; `--from-file` accepts the full `{"type": ..., "params": ...}` JSON envelope. **`--extra-arg` / `extra_args` on an `ai_agent` task are honored only when `kbagent serve` runs with a truthy `KBAGENT_ALLOW_AI_EXTRA_ARGS`; otherwise they are dropped with a warning** -- see [gotchas.md](gotchas.md). - `agent update TASK_ID [--name N] [--description D] [--cron C] [--enabled/--disabled] [--manual/--auto] [--clear-trigger] [--trigger-task-id ID --trigger-on ...]` -- patch one or more fields. Omitted flags leave the field unchanged. `--manual` nulls `next_run_at`; `--auto` recomputes it from the cron expression. - `agent delete TASK_ID [--yes]` -- permanent removal. Run history on disk is preserved. - `agent run TASK_ID [--stream] [--runtime-prompt TEXT | --runtime-input JSON|@file|-]` -- trigger immediately. `--stream` prints one line per event in human mode, NDJSON in `--json` mode. `--runtime-prompt` appends ad-hoc text to an ai_agent's persisted prompt for this run only; `--runtime-input` merges arbitrary JSON into the action params (cli_command: appends to `params.argv`). @@ -436,7 +436,7 @@ CLI parity for the `/agents` REST surface. Reads/writes `/agents.jso - `agent run-events TASK_ID RUN_ID` -- replay the persisted ai_agent event timeline (only present for ai_agent runs from v0.10+). - `agent test [--type ... | --from-file PATH] [--stream] [--name N] [common action flags]` -- execute an action ad-hoc; nothing is persisted. Same dispatcher as the cron scheduler, useful for sanity-checking a prompt / argv before saving. - `agent cron-preview --cron "..." [--count N]` -- validate a cron expression and show the next N firings (UTC, capped at 20). -- `agent prompt-improve --goal "..." [--draft "..."] [--cli claude|codex|gemini] [--project ALIAS] [--extra-arg X ...] [--stream/--no-stream]` -- AI-polished single-shot prompt for an unattended agent task. The final `done` event's `data.prompt` carries the cleaned body ready to drop into `agent create --prompt ...`. `--extra-arg` is subject to the `KBAGENT_ALLOW_AI_EXTRA_ARGS` opt-in (since v0.60.2) -- see [gotchas.md](gotchas.md). +- `agent prompt-improve --goal "..." [--draft "..."] [--cli claude|codex|gemini] [--project ALIAS] [--extra-arg X ...] [--stream/--no-stream]` -- AI-polished single-shot prompt for an unattended agent task. The final `done` event's `data.prompt` carries the cleaned body ready to drop into `agent create --prompt ...`. `--extra-arg` is subject to the `KBAGENT_ALLOW_AI_EXTRA_ARGS` opt-in -- see [gotchas.md](gotchas.md). ## Utility - `init [--from-global] [--project ALIAS ...]` -- create local `.kbagent/` workspace (per-directory isolation); `--project ALIAS` (repeatable) copies only the named project(s) and implies `--from-global` @@ -458,7 +458,7 @@ CLI parity for the `/agents` REST surface. Reads/writes `/agents.jso |----------|---------| | `KBC_TOKEN` | Fallback for `--token`. Also the credential source for headless `__env__` mode (see `KBAGENT_PROJECT_FROM_ENV`) | | `KBC_STORAGE_API_URL` | Default stack URL. Also the stack source for headless `__env__` mode | -| `KBAGENT_PROJECT_FROM_ENV` | Set to `1`/`true`/`yes`/`on` to synthesize an in-memory project `__env__` from `KBC_TOKEN` + `KBC_STORAGE_API_URL` (since 0.50.0). Headless / token-only: no `project add`, no `config.json` on disk; token stays in memory (never persisted). Use `--project __env__`. Works for CLI and `kbagent serve`. Fails fast if creds missing | +| `KBAGENT_PROJECT_FROM_ENV` | Set to `1`/`true`/`yes`/`on` to synthesize an in-memory project `__env__` from `KBC_TOKEN` + `KBC_STORAGE_API_URL`. Headless / token-only: no `project add`, no `config.json` on disk; token stays in memory (never persisted). Use `--project __env__`. Works for CLI and `kbagent serve`. Fails fast if creds missing | | `KBC_MANAGE_API_TOKEN` | Manage API token (org setup, project refresh, data-app password). Default-DENY since 0.28.0: requires top-level `--allow-env-manage-token` to opt in, otherwise ignored with a warning. | | `KBAGENT_CONFIG_DIR` | Override config directory | | `KBAGENT_SERVE_URL` | Self-URL of `kbagent serve` (used by `kbagent http`; auto-injected into scheduled-agent subprocesses) | diff --git a/plugins/kbagent/skills/kbagent/references/data-app-workflow.md b/plugins/kbagent/skills/kbagent/references/data-app-workflow.md index b8bcf958..5bbe1d8f 100644 --- a/plugins/kbagent/skills/kbagent/references/data-app-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/data-app-workflow.md @@ -215,7 +215,7 @@ kbagent data-app deploy --project prod --app-id 12345678 \ (rollback). Subsequent deploys without the flag will jump back to the latest. -### Pre-flight repo validation (since v0.29.0) +### Pre-flight repo validation ```bash kbagent data-app validate-repo \ @@ -238,7 +238,7 @@ repo. Public repos: drop `--git-pat-env` and use `--git-public`. Total GitHub call budget per run is ≤5 (1 tree + ≤4 contents) regardless of repo size, so the 60/hour unauth limit rarely fires; pass a PAT for CI loops. -### Inspect the deployed-from git repo (since v0.63.3) +### Inspect the deployed-from git repo ```bash # Clone URLs + whether the repo is managed by Keboola: @@ -256,7 +256,7 @@ block is synced from the Storage config into the Data Science app record at deploy time, so a fresh `--no-deploy` app returns 409 "no Git repository configured". Run `data-app deploy` first. -### Manage git credentials for a managed repo (since v0.63.3) +### Manage git credentials for a managed repo ```bash # List credentials of a MANAGED git repo (the secret is never returned): @@ -281,7 +281,7 @@ returns 409 "no managed Git repository" for them. Both credential commands also need an **admin** storage token (`CanManageAppRepoCredentials`), unlike `git-repo` above which needs only the ordinary project storage token. -### Create an app on a Keboola-MANAGED git repo (since v0.65.0) +### Create an app on a Keboola-MANAGED git repo ```bash # 1. Provision the app + an EMPTY Keboola-hosted repo (no external URL). @@ -319,7 +319,7 @@ contract), so `data-app deploy` on a pure managed repo deploys straight from `app.managedGitRepoId`. If a deploy ever reverts to stopped, diagnose it with `data-app runs` (`failure_reason` + `startup_logs`). -### Manage app-runtime secrets (since v0.29.0) +### Manage app-runtime secrets ```bash # Set two secrets at once. Plaintext values; the CLI encrypts under @@ -341,7 +341,7 @@ kbagent data-app secrets-list --project prod --app-id 12345678 # -> #my-database-url -> env MY_DATABASE_URL # -> ADMIN_EMAILS -> env ADMIN_EMAILS (plain, unencrypted) -# Read one key. Leading '#' is OPTIONAL (since v0.43.9). Encrypted secret +# Read one key. Leading '#' is OPTIONAL. Encrypted secret # -> metadata only (NEVER decrypts). Plain value -> the literal value: kbagent data-app secrets-get --project prod --app-id 12345678 --key '#ANTHROPIC_API_KEY' # metadata only kbagent data-app secrets-get --project prod --app-id 12345678 --key ADMIN_EMAILS # shows value diff --git a/plugins/kbagent/skills/kbagent/references/dev-portal-workflow.md b/plugins/kbagent/skills/kbagent/references/dev-portal-workflow.md index 79fcb34c..b145b947 100644 --- a/plugins/kbagent/skills/kbagent/references/dev-portal-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/dev-portal-workflow.md @@ -25,7 +25,7 @@ Google Authenticator). `--password-stdin` works in both pipe mode (`echo $PASS | … --password-stdin`, reads to EOF) and TTY mode (hidden line-based prompt, Enter to confirm). -### `role_hint` is load-bearing (since v0.51.1) +### `role_hint` is load-bearing `--role-hint` is **not** a free-text label. It picks which apps-api endpoint kbagent uses for `dev-portal patch`: diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index 96820b5c..48891fa6 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -262,7 +262,7 @@ Versioning convention: denies everything (fail-closed). Re-run `kbagent permissions set` with `cli:read` if that was your setup. -## Native parity commands (since v0.73.0) +## Native parity commands - **These commands are the native answers to tools the removed passthrough used to expose**: `docs query`, `config examples`, `semantic-layer schema`, @@ -298,7 +298,7 @@ Versioning convention: token, no feature flag, no project data); `kai ask` sees project data but needs the master token + `agent-chat` feature. -## `token` group mints/rotates/revokes SCOPED Storage tokens; secret shown ONCE (since v0.66.0) +## `token` group mints/rotates/revokes SCOPED Storage tokens; secret shown ONCE - **`kbagent token create --project P --description D [--bucket-write B ...] [--bucket-read B ...] [--component-access ID ...] [--can-read-all-file-uploads] @@ -332,7 +332,7 @@ Versioning convention: retrievable again — persist only the `id` (to revoke/refresh later) and `expires`. Lost the secret? `token refresh --token-id ID` mints a new one. -## Files-upload permission: `canReadAllFileUploads` gates READING, not the UPLOAD (since v0.66.0) +## Files-upload permission: `canReadAllFileUploads` gates READING, not the UPLOAD - **Any valid Storage token can upload its OWN Files.** Uploading is NOT gated by `canReadAllFileUploads` or by `componentAccess`. `canReadAllFileUploads` controls @@ -347,7 +347,7 @@ Versioning convention: its `in.c-otlp-` sink bucket, then `token create --bucket-write in.c-otlp- --expires-in 3600` mints exactly the token the device holds. -## `stream create-source` uses a NORMAL Storage token — there is NO master-token gate (since v0.66.0) +## `stream create-source` uses a NORMAL Storage token — there is NO master-token gate - Creating a Data Streams source (`kbagent stream create-source` / `Client.create_stream_source`) authenticates with the ordinary project Storage @@ -547,7 +547,7 @@ Each fix is surfaced -- never silent -- as a push-envelope `warnings[]` entry wi (printed in human mode like any other push warning, structured under `warnings` in `--json`). `config update` keeps its own dedicated `normalizations` key. -## `sync push` fresh-CREATE writeback now updates placeholders in place (since v0.47.0) +## `sync push` fresh-CREATE writeback now updates placeholders in place Before v0.47.0, `kbagent sync push` always **appended** new `ManifestConfiguration` (and `ManifestConfigRow`) entries to `.keboola/manifest.json` on every CREATE. @@ -582,7 +582,7 @@ If a downstream consumer has been working around the duplication by post-processing the manifest, drop that workaround. The single-entry manifest is the new contract. -## `sync push` fresh-CREATE now resolves variable links, hoists row `values`, and `--branch` promotes the default tree (since v0.47.2) +## `sync push` fresh-CREATE now resolves variable links, hoists row `values`, and `--branch` promotes the default tree A transformation scaffolded alongside its sibling `keboola.variables` config + default-values row is now **runnable after a single `sync push`** — no post-push @@ -615,7 +615,7 @@ erroring with `Config file not found`. Source (where files are read) and target (where the API writes) are decoupled; API calls still target the branch id. When a per-branch subtree *does* exist, behaviour is unchanged. -## `sync push` / `sync pull` / `sync diff` accept `--branch ` for per-invocation dev-branch targeting (since v0.47.0) +## `sync push` / `sync pull` / `sync diff` accept `--branch ` for per-invocation dev-branch targeting The `--branch` override wins over every other branch source: `manifest.branches[0]`, `active_branch_id` (set by `kbagent branch use`), and the git-branching @@ -625,7 +625,7 @@ Useful for targeting a freshly-created dev branch without running `branch use` o to the manifest or to the config store, so subsequent commands without `--branch` fall back to the normal priority chain. -## `storage create-table --if-not-exists` returns `action: skipped` instead of raising on duplicate display name (since v0.47.0) +## `storage create-table --if-not-exists` returns `action: skipped` instead of raising on duplicate display name Opt-in flag (default `False`, so existing callers are unaffected). When set, catches the specific `STORAGE_JOB_FAILED` + "already has the same display name" @@ -649,7 +649,7 @@ detect "I hit a pre-existing table with a different shape". (Before v0.47.1 the skipped envelope re-echoed the request, so older installs must still call `kbagent storage table-detail` after a skip to get the real shape.) -## `sync push --no-name-drift-warnings` suppresses the cosmetic warnings array (since v0.47.0) +## `sync push --no-name-drift-warnings` suppresses the cosmetic warnings array When local directory names diverge from the canonical kbagent naming (e.g. FIIA's `var-07-fi-daily-date-refresh` pattern), `sync push` normally returns @@ -658,7 +658,7 @@ a `name_drift_warnings: [...]` array on the result envelope. The still runs, so a future operator who wants to audit can flip the flag off without losing data. -## `semantic-layer search-context` + `get-context` cover the upstream `search_semantic_context` / `get_semantic_context` parity (since v0.47.0) +## `semantic-layer search-context` + `get-context` cover the upstream `search_semantic_context` / `get_semantic_context` parity `kbagent semantic-layer search-context --project P [--pattern G ...] [--type T] [--limit N]` is project-wide (not model-scoped). Patterns are **case-sensitive `fnmatch`** against @@ -680,7 +680,7 @@ project's semantic model is populated before kicking off a downstream pipeline; the previous workaround (a `keboola-mcp-server` entry in `.mcp.json` solely for these two tools) can be dropped. -## `semantic-layer reference-data` holds a whole dimension as ONE record; `set` is PUT-replace, not append (since v0.55.0) +## `semantic-layer reference-data` holds a whole dimension as ONE record; `set` is PUT-replace, not append `semantic-reference-data` stores one record **per dimension** (e.g. a Chart of Accounts), with the full member list in a `members[]` array — NOT one @@ -757,7 +757,7 @@ Snowflake (`constants.QUERY_SERVICE_COMPATIBLE_LOGIN_TYPES`): - `snowflake-service-keypair` -- confirmed PASS - `snowflake-person-sso` -- confirmed PASS -- `snowflake-person-keypair` -- confirmed PASS (since v0.47.1) +- `snowflake-person-keypair` -- confirmed PASS - `snowflake-legacy-service` -- explicitly OFF the list (works on `connection.keboola.com` but FAILED on GCP us-east4 stack in the original #304 incident -- keep it off until cross-stack confirmation) @@ -766,7 +766,7 @@ Snowflake (`constants.QUERY_SERVICE_COMPATIBLE_LOGIN_TYPES`): BigQuery (`constants.QUERY_SERVICE_COMPATIBLE_LOGIN_TYPES_BIGQUERY`): -- `default` on BigQuery -- confirmed PASS (since v0.58.0). Every BigQuery +- `default` on BigQuery -- confirmed PASS. Every BigQuery workspace carries loginType `default` (the sandbox API exposes no Snowflake-style variants for BigQuery), and the Query Service runs SELECTs against it -- verified live against project 9621 on `connection.keboola.com`. @@ -779,7 +779,7 @@ confirmed-good whitelist". For an unknown loginType, `workspace list` renders it as `?` (yellow) in the QS column so callers know the policy is uncertain rather than confirmed-bad. -## `workspace query`: fast inline results vs `--full` CSV export (since v0.59.0) +## `workspace query`: fast inline results vs `--full` CSV export By default `workspace query` now reads the result set inline via the Query Service `GET /api/v1/queries/{job}/{stmt}/results` endpoint (JSON `columns` + @@ -810,7 +810,7 @@ interactive queries are markedly faster. `full=True` so the web UI's "Download CSV" stays complete; REST clients can pass `full=false` (+ `limit`) in the JSON body to opt into the fast path. -## Snowflake `workspace create` returns `private_key`, not password (since v0.47.1) +## Snowflake `workspace create` returns `private_key`, not password Headless `workspace create` on Snowflake requests `loginType: snowflake-person-keypair`, generates an RSA key pair locally, @@ -894,7 +894,7 @@ limit, transient 5xx), the detail call still succeeds and `storage_workspace_id` is set to `null` -- the annotation is UX, not a contract. -## `kbagent job run --mode debug` redirects output to a Storage File, not the destination buckets (since v0.43.6) +## `kbagent job run --mode debug` redirects output to a Storage File, not the destination buckets `kbagent job run` accepts `--mode run|debug` (default `run`). The flag is threaded straight into the Queue API job-creation body as `"mode": "..."`; @@ -930,7 +930,7 @@ Before this release the `mode` parameter existed on `KeboolaClient.create_job` but neither `JobService.run_job` nor `commands/job.py` exposed it, so every job created via `kbagent` hard-coded `mode: "run"` on the wire. -## Metastore duplicate-name POST returns 409 OR 500 -- both map to `ALREADY_EXISTS` (since v0.43.5) +## Metastore duplicate-name POST returns 409 OR 500 -- both map to `ALREADY_EXISTS` `MetastoreClient.post_item` normalises **both** the post-go-monorepo-PR#513 HTTP 409 (`"Object with this name already exists in this project"`) and the @@ -951,7 +951,7 @@ costs `MAX_RETRIES` round-trips before the normaliser fires; 409 is not, so post-fix duplicates resolve in a single round-trip. No code change needed in callers either way. -## `semantic-layer model delete` cascade-deletes children (since v0.43.4) +## `semantic-layer model delete` cascade-deletes children `kbagent semantic-layer model delete --project P --model M` used to DELETE only the parent `semantic-model` row, leaving every dataset / metric / @@ -999,7 +999,7 @@ follow-up. Scripts that scraped `orphaned_children` to detect the bug now see the same zeros they always wanted — but should switch to `cascade.deleted` ahead of v0.42.0. -## Web UI `Kai Chat` is gone — replaced by `Local AI` (since v0.41.9) +## Web UI `Kai Chat` is gone — replaced by `Local AI` The web UI dashboard tile / left-nav entry previously labelled **Kai Chat** has been replaced by **Local AI** (PR #301, follow-up to #291 @@ -1037,7 +1037,7 @@ a kbagent co-pilot. `POST /ai/chat/stream` (SSE) -- different wire protocol, different envelope. -## Dashboard `▶ run` button on scheduled agents uses BLOCKING `/agents/{id}/run`, NOT the SSE stream (since v0.41.9) +## Dashboard `▶ run` button on scheduled agents uses BLOCKING `/agents/{id}/run`, NOT the SSE stream The dashboard's Scheduled agents tile gained an inline `▶ run` button per row (issue #292). It fires `POST /agents/{task_id}/run` -- the @@ -1059,7 +1059,7 @@ Both endpoints persist the same `AgentRun` record on disk; the blocking endpoint returns it once the run completes, the SSE endpoint streams events and emits a final `done` SSE frame mirroring the same record. -## Semantic-layer constraint `rule` is a STRING, not an object (since v0.41.0) +## Semantic-layer constraint `rule` is a STRING, not an object - The `sl-builder` skill docs (in `04_AI_Kit/ai-kit/`) describe range constraints with `ruleExpression: {bounds: {min: 0, max: 100}}` -- @@ -1083,7 +1083,7 @@ events and emits a final `done` SSE frame mirroring the same record. object the CLI exits 2 / `VALIDATION_ERROR` with a hint pointing at this gotcha. -## Constraint name regex `^[a-z][a-z0-9_]*$` AND the 3-vs-4 severity split (since v0.41.0) +## Constraint name regex `^[a-z][a-z0-9_]*$` AND the 3-vs-4 severity split - Constraint NAMES must match `^[a-z][a-z0-9_]*$`: lowercase ASCII, digits, underscores; must start with a letter. UPPERCASE, hyphens, @@ -1104,7 +1104,7 @@ events and emits a final `done` SSE frame mirroring the same record. - `kbagent semantic-layer add constraint --severity` only accepts the 3 API values; the 4-band band lives in `--name` suffix. -## Metric rename auto-cascades through `CODE_METRIC` (since v0.41.0) +## Metric rename auto-cascades through `CODE_METRIC` - `kbagent semantic-layer edit metric --new-name NEW` does DELETE+POST on the metric and ALSO DELETE+POST on every constraint whose @@ -1148,7 +1148,7 @@ events and emits a final `done` SSE frame mirroring the same record. semantic-layer validate` to surface the dangling refs, then re-run each failed cascade via `edit constraint --new-metrics ...`. -## Removing a metric corrupts `DIM_METRIC_THRESHOLD` downstream (since v0.41.0) +## Removing a metric corrupts `DIM_METRIC_THRESHOLD` downstream - `kbagent semantic-layer remove metric --name N` runs a pre-deletion scan listing every constraint whose `metrics[]` includes N. Each @@ -1170,7 +1170,7 @@ events and emits a final `done` SSE frame mirroring the same record. keeps the constraint refs valid (and the CODE_METRIC alive in historical comparisons). -## `semantic-layer build` is a HEURISTIC fallback, not full AI (since v0.41.0) +## `semantic-layer build` is a HEURISTIC fallback, not full AI - The kbagent AI Service client (`ai_client.py`) only exposes `get_component_detail` and `suggest_components` as of v0.41.0 -- @@ -1196,7 +1196,7 @@ events and emits a final `done` SSE frame mirroring the same record. `04_AI_Kit/ai-kit/`. Bridge to that skill when the heuristic is not enough; the two are interoperable via the same metastore contract. -- **Field-type normalization (since v0.41.10)**: warehouse-native +- **Field-type normalization**: warehouse-native column types from Storage (`VARCHAR(255)`, `NUMBER(38,2)`, `STRING`, `TIMESTAMP_NTZ`, ...) are mapped to the metastore's closed lowercase set (`string`, `integer`, `decimal`, `boolean`, @@ -1225,7 +1225,7 @@ events and emits a final `done` SSE frame mirroring the same record. the model itself is NEVER deleted on rollback (only the children WE POSTed during this call get torn down). -## `kbagent http` works only inside `kbagent serve` subprocesses (since v0.40.0) +## `kbagent http` works only inside `kbagent serve` subprocesses - `kbagent http get/post/patch/delete ` is a thin self-call client against the running `kbagent serve`. It requires both @@ -1283,7 +1283,7 @@ events and emits a final `done` SSE frame mirroring the same record. - Agents on kbagent < 0.89.0: do NOT recommend `--output-dir` + `--push` together; use scaffold-only then `sync push`, or `--push --no-files`. -## `kbagent config new --push` is one-shot remote create; default is scaffold-only (since v0.33.0) +## `kbagent config new --push` is one-shot remote create; default is scaffold-only - **Pre-v0.33.0**, `kbagent config new` was scaffold-only -- it wrote boilerplate files to `--output-dir` (or stdout) and made **no Storage @@ -1548,7 +1548,7 @@ config, the retry fires, and the retry destroys it for good. DELETE specifically; idempotency here is a property of the endpoint, not of the method. -## `data-app` JSON output: key for the app's own id is `app_id` (since v0.33.0) +## `data-app` JSON output: key for the app's own id is `app_id` - Every `kbagent --json data-app ` envelope emits the data-app's own identifier under the key `app_id`. Prior to v0.33.0 the @@ -1572,7 +1572,7 @@ config, the retry fires, and the retry destroys it for good. If a future API shape change introduces snake_case wire keys, this helper will need a defensive alias pass -- not yet warranted. -## `project edit --new-alias` does NOT rewrite lineage caches (since v0.31.0) +## `project edit --new-alias` does NOT rewrite lineage caches - `kbagent project edit --project OLD --new-alias NEW` cascades the rename through `config.json` (`projects` dict key + `default_project` field if it @@ -1621,7 +1621,7 @@ config, the retry fires, and the retry destroys it for good. original command immediately afterwards. A failed or timed-out reinstall prints a copy-paste recovery command (#528/#530, v0.76.2). -## `storage swap-tables` is branch-scoped and aliases stay put (since v0.28.0) +## `storage swap-tables` is branch-scoped and aliases stay put - `kbagent storage swap-tables --project P --table-id A --target-table-id B --branch ` swaps two tables' physical positions @@ -1644,7 +1644,7 @@ config, the retry fires, and the retry destroys it for good. branch and run the real build + swap in the production (default) branch. Full procedure: `typify-table-workflow.md`. -## `storage clone-table` materializes a prod table into a dev branch (since v0.52.0) +## `storage clone-table` materializes a prod table into a dev branch - `kbagent storage clone-table --project P --table-id T --branch ` pulls a production table into a dev branch @@ -1683,7 +1683,7 @@ config, the retry fires, and the retry destroys it for good. - The only API path between the two table stores is `clone-table` (pull, default -> branch). There is no "push branch -> default". -## `storage truncate-table` preserves schema; endpoint is uniformly async-via-job (since v0.32.0) +## `storage truncate-table` preserves schema; endpoint is uniformly async-via-job - `kbagent storage truncate-table --project P --table-id T [--branch ID] [--dry-run] [--yes]` calls @@ -1770,7 +1770,7 @@ config, the retry fires, and the retry destroys it for good. flag. Use the Keboola UI to configure them after `data-app create`. Tracked as a follow-up issue. -## `data-app secrets-*` -- per-project KMS, idempotent remove, never decryptable (since v0.29.0) +## `data-app secrets-*` -- per-project KMS, idempotent remove, never decryptable - **Encryption is per-project KMS.** `kbagent data-app secrets-set` calls the project's Encryption API to wrap each plaintext value before @@ -1790,7 +1790,7 @@ config, the retry fires, and the retry destroys it for good. `parameters.dataApp` -- slug, git block, id back-pointer, `parameters` itself, and the top-level `runtime`/`authorization`/`storage`) is preserved bit-identical. -- **`data-app list` hides workspace/sandbox deployments (since v0.43.9).** +- **`data-app list` hides workspace/sandbox deployments.** The Data Science `GET /apps` collection returns EVERY deployment in the project, not just data apps -- interactive Snowflake/BigQuery workspaces (`componentId=keboola.sandboxes`, `type=snowflake`/`bigquery`, @@ -1810,8 +1810,8 @@ config, the retry fires, and the retry destroys it for good. ciphertext fingerprint, encryption prefix, `encrypted: true`, `value: null`. NOT_FOUND on an absent key never enumerates sibling keys. (Plain unencrypted values ARE returned in full -- see next entry.) -- **`secrets-get` / `secrets-remove` accept keys WITHOUT a leading `#` - (since v0.43.9).** The `parameters.dataApp.secrets` block holds BOTH +- **`secrets-get` / `secrets-remove` accept keys WITHOUT a leading `#`.** + The `parameters.dataApp.secrets` block holds BOTH `#`-prefixed encrypted secrets and plain unencrypted env-var config values (e.g. `ADMIN_EMAILS`, `SMTP_HOST`), and `secrets-list` enumerates both. Before 0.43.9 `get`/`remove` rejected any key without @@ -1838,11 +1838,11 @@ config, the retry fires, and the retry destroys it for good. - **Adding/removing a secret bumps the Storage version, but the running container keeps the OLD config until `data-app deploy` runs.** Same contract as any other `keboola.data-apps` config edit (see the - `(since v0.27.0)` entry below). The response includes a `next_step` + `` entry below). The response includes a `next_step` field with the exact redeploy command to run; suppress it with `--no-hint-next` for scripted callers. -## `data-app validate-repo` -- pre-flight against the Golden Rule, GitHub-only (since v0.29.0) +## `data-app validate-repo` -- pre-flight against the Golden Rule, GitHub-only - `kbagent data-app validate-repo --git-repo URL` walks the repo via the GitHub Contents + Trees API and verifies the documented "Golden Rule" @@ -1876,7 +1876,7 @@ config, the retry fires, and the retry destroys it for good. treats WARNs as failures (exit 1) for CI gating. - **Reading the build / runtime log** is now available via `kbagent data-app logs --project ALIAS --app-id ID [--lines N | - --since ISO8601]` (since v0.43.8). On `DATA_APP_BUILD_FAILED` / + --since ISO8601]`. On `DATA_APP_BUILD_FAILED` / `DATA_APP_DEPLOY_TIMEOUT`, fetch the container log tail directly from the CLI instead of opening the UI's Terminal Log tab. See the `data-app logs` section below for the mutex contract, the @@ -1918,7 +1918,7 @@ config, the retry fires, and the retry destroys it for good. - Storage tokens are unaffected: `KBC_TOKEN` (storage API) keeps resolving from env as before. -## `data-app deploy` is required after `config update` -- the running container does NOT auto-pick-up new config versions (since v0.27.0) +## `data-app deploy` is required after `config update` -- the running container does NOT auto-pick-up new config versions - `kbagent config update --component-id keboola.data-apps ...` bumps the Storage config version; the deployed container keeps running at the @@ -1939,7 +1939,7 @@ config, the retry fires, and the retry destroys it for good. at the currently-pinned version. It does NOT roll out new code or config -- use `data-app deploy` for that. -## Cross-project KMS ciphertext does NOT decrypt; re-encrypt per project (since v0.27.0) +## Cross-project KMS ciphertext does NOT decrypt; re-encrypt per project - The Encryption API's `KBC::Project*` ciphertext is bound to the **target project's KMS key**. A `#password` encrypted in project A @@ -1956,7 +1956,7 @@ config, the retry fires, and the retry destroys it for good. - Practical implication: you cannot copy-paste a `KBC::Project*` value from one project's `keboola.data-apps` config into another's. -## Transient `state == stopped` during initial data-app deploy is not a failure (since v0.27.0) +## Transient `state == stopped` during initial data-app deploy is not a failure - After `data-app create` (or any `data-app deploy --wait`), polling may observe `state == stopped` once for ~5-15s before the container @@ -1973,7 +1973,7 @@ config, the retry fires, and the retry destroys it for good. container after `autoSuspendAfterSeconds` of inactivity. Hit the URL to wake it (auto-restart triggers a 30-60s cold boot) or run `kbagent data-app start --app-id N`. -## `project invite` "already invited / already member" returns HTTP 400, not 422 (since v0.29.0) +## `project invite` "already invited / already member" returns HTTP 400, not 422 - Re-inviting a user the project already knows about returns HTTP **400** with one of two error strings: @@ -1987,7 +1987,7 @@ config, the retry fires, and the retry destroys it for good. is **wrong** for this API. If you write a parallel implementation, key off status_code 400 + the substring marker, not 422. -## `project member-set-role` is PATCH, not PUT (since v0.29.0) +## `project member-set-role` is PATCH, not PUT - The Manage API role-change endpoint is `PATCH /manage/projects/{id}/users/{userId}` with body `{"role": "..."}`. **PUT returns 404** ("resource not found") even @@ -1995,7 +1995,7 @@ config, the retry fires, and the retry destroys it for good. - The kbagent `ManageClient.update_project_member_role` method emits PATCH; any custom code re-implementing the call must do the same. -## `project invite --from-csv` order is not deterministic (since v0.29.0) +## `project invite --from-csv` order is not deterministic - Bulk invitation parallelises via `ThreadPoolExecutor` (default 8 workers). The `rows[]` array in the result is in completion order, not CSV order. @@ -2004,7 +2004,7 @@ config, the retry fires, and the retry destroys it for good. the command exits 0 with `failed > 0` reflected in the JSON summary. Mirror the `org setup` partial-success exit semantics. -## `default_bucket` is per-config and only an output prefix (since 0.26.0) +## `default_bucket` is per-config and only an output prefix - `kbagent config set-default-bucket` writes `configuration.storage.output.default_bucket`. The Storage API uses this @@ -2027,7 +2027,7 @@ config, the retry fires, and the retry destroys it for good. bucket key) says. The Storage `default_bucket` always wins for tables that don't pin their own `destination`. -## `config detail` has a bulk mode (since 0.23.0) +## `config detail` has a bulk mode - **Omit `--config-id`** to get every configuration under `--component-id` as `{"configs": [...], "errors": [...]}`. Each row is tagged with @@ -2048,7 +2048,7 @@ config, the retry fires, and the retry destroys it for good. `config detail --component-id X` returns every configuration body of component X. Different use cases, same underlying endpoint. -## `config list --include-rows` payload size warning (since 0.23.0) +## `config list --include-rows` payload size warning - The default `config list` response is summary-level: just name, description, component, last_modified, folder per config. Cheap and fast. @@ -2061,7 +2061,7 @@ config, the retry fires, and the retry destroys it for good. review across many projects). For just finding strings, prefer `config search` -- same endpoint, tighter response. -## `config detail --with-state` runtime-state fetch (since 0.23.0) +## `config detail --with-state` runtime-state fetch - The `state` dict on a configuration is mutable runtime data components persist between jobs (last sync cursors, auth refresh tokens, OAuth @@ -2082,7 +2082,7 @@ config, the retry fires, and the retry destroys it for good. never written state yet, or state was cleared). Treat `{}` as "no state", not an error. -## Variables: attach, don't manage (since 0.21.0) +## Variables: attach, don't manage - `keboola.variables` is an implementation detail. Use `kbagent config variables-set/get/clear` -- you never need to create, @@ -2102,7 +2102,7 @@ config, the retry fires, and the retry destroys it for good. - Full workflow + response shapes: see [variables-workflow.md](variables-workflow.md). -## `job run` auto-resolves variable values (since 0.21.0) +## `job run` auto-resolves variable values Transformations with linked `keboola.variables` used to run against empty strings unless the caller hand-wired a `variableValuesId` at the HTTP @@ -2129,7 +2129,7 @@ linked variables config. fired, so callers verify the binding without a second `job detail` round-trip. -## Sync: row deploy & manifest v3 (since 0.21.0) +## Sync: row deploy & manifest v3 - `sync push` **does** deploy config rows now (previously silently skipped). Row changes in the `pushed_details` array carry `"is_row": true` and @@ -2217,9 +2217,9 @@ unknown -- do not try to parse a fallback message. | 4 | Network error (timeout, unreachable) -- includes `QUEUE_JOB_TIMEOUT` (local gave up AND the remote-kill attempt failed; the remote job may still be running), `AUTH_FLOW_TIMEOUT`, and a session refresh that timed out or could not reach the auth service (`TIMEOUT` / `CONNECTION_ERROR`; a slow auth service is NOT a dead login -- re-run, do not re-login) (since v0.80.0) | | 5 | Configuration error (corrupt config, missing alias) | | 6 | Permission denied (blocked by firewall / `--deny-writes` / `--deny-destructive`) | -| 7 | `JOB_TIMEOUT_TERMINATED` -- `job run --timeout` elapsed AND the remote job was successfully cancelled (since 0.22.0). Scripts can distinguish "we killed it" from "it failed on its own" (exit 1) from "it's still running" (exit 4). | +| 7 | `JOB_TIMEOUT_TERMINATED` -- `job run --timeout` elapsed AND the remote job was successfully cancelled. Scripts can distinguish "we killed it" from "it failed on its own" (exit 1) from "it's still running" (exit 4). | -## `job run --wait` polling + log tail (since 0.22.0) +## `job run --wait` polling + log tail - Polling follows an exponential curve by default: **2s x 30 -> 5s x 48 -> 15s forever**. For a short test job or a test that needs fast turnaround, pass `--poll-strategy fixed` to force the legacy 1s fixed interval. - On terminal non-success (`error` / `warning` / `terminated`), kbagent fetches the last N Storage Events and attaches them as `logTail` on the response. Controlled by `--log-tail-lines N` (default 200, max 5000, `0` disables). @@ -2238,7 +2238,7 @@ unknown -- do not try to parse a fallback message. - Safe to run under either flag without mutating the saved policy -- useful when your agent needs a one-shot read-only run on a machine with a write-enabled config. - `permissions check OPERATION` reflects the EFFECTIVE policy (persisted policy MERGED with session flags) **(since v0.30.5)**. Pre-0.30.5 it consulted only the persisted policy, so an agent doing self-introspection (`kbagent --deny-writes permissions check branch.create`) got `allowed: true` despite the session flag denying that op at execution time. If your agent uses `permissions check` to gate destructive actions and may run against pre-0.30.5 installs, also re-check at execution-time exit codes (6 = denied) rather than trusting the dry probe alone. -## `storage create-table` native types + dev-branch materialize (since 0.25.0) +## `storage create-table` native types + dev-branch materialize - **Native types pass through to the Storage API.** `--column pk:VARCHAR(40)`, `--column amount:NUMERIC(18,2)`, `--column ts:TIMESTAMP_TZ`, @@ -2265,8 +2265,8 @@ unknown -- do not try to parse a fallback message. official Keboola Go CLI's `EnsureBucketExists`). Response includes `auto_created_bucket: true` when this happens. Production writes (no `--branch`) never materialize anything. -- **Auto-materialized buckets get `KBC.createdBy.branch.id` stamped** - (since 0.25.1). On projects with **branched storage** feature flag ON, +- **Auto-materialized buckets get `KBC.createdBy.branch.id` stamped.** + On projects with **branched storage** feature flag ON, the transformation runner's `output-mapping` rejects buckets without this system metadata with `bucket is not assigned to any development branch.` kbagent stamps it automatically; the metadata write is @@ -2311,7 +2311,7 @@ type inventory and examples. `include=` on the list route accepts no `definition` value, so reading the layout costs one `table-detail` request per table. -## `storage create-table --source-table-id` + partition/clustering are BigQuery-only (since 0.66.0) +## `storage create-table --source-table-id` + partition/clustering are BigQuery-only - **`--source-table-id` copies an existing table instead of building from `--column`.** The new table's schema is derived from the source and its rows are copied into the @@ -2338,7 +2338,7 @@ type inventory and examples. view) is queryable; a non-persisted alias (project lacks `bigquery-persisted-alias-views`) is rejected 422. -## Legacy fake-branch storage warning on `--branch` writes (since 0.25.2) +## Legacy fake-branch storage warning on `--branch` writes - **What it is.** Projects without the `storage-branches` feature flag use Keboola's legacy fake-branch storage. Writes via `kbagent storage @@ -2373,7 +2373,7 @@ type inventory and examples. user-facing command surface. See `storage-types-workflow.md` for the full fake-branch vs storage-branches mechanics. -## `sync init --adopt-existing` (since 0.22.0) +## `sync init --adopt-existing` - Adopts a `.keboola/manifest.json` written by the kbc Go CLI **in place** instead of overwriting. Idempotent; re-running is a no-op. - Validates `project_id` from the manifest against the token via `verify_token`. Mismatch exits 5 (`CONFIG_ERROR`) with guidance -- never silently adopts someone else's checkout. @@ -2416,7 +2416,7 @@ kbagent looks for configuration in this order: Use `kbagent init` to create a local `.kbagent/` workspace for per-directory isolation. -## `init --project` filters the copy; it does NOT select an existing project (since v0.59.0) +## `init --project` filters the copy; it does NOT select an existing project `--project ALIAS` means something different on `init` than on every other command. Everywhere else `--project` *selects an existing* project to act on; @@ -2784,9 +2784,9 @@ transparent -- no user action is normally required. - Version cache: checks the release endpoints at most once per hour - Skipped for: dev/editable installs, `update`/`version` commands - Never crashes the CLI -- update failures leave the current invocation running - and print a recovery command (since v0.76.2) + and print a recovery command -### Windows updates are deferred, not immediate (since v0.78.0) +### Windows updates are deferred, not immediate `uv tool install` recreates a tool environment by **removing** it and then building a fresh venv at the same path. It is not atomic and has no rollback. @@ -2937,7 +2937,7 @@ write descriptive metadata onto storage objects. Three behaviors are easy to mis endpoint, so the metadata entry is the authoritative source. `KBC.description` entries whose provider is not `user` (e.g. `system`) are ignored during read-back and the native field is used as fallback. -- **`storage bucket-detail` is dialect-aware** *(since v0.25.3)*. Output adapts +- **`storage bucket-detail` is dialect-aware** **. Output adapts to the bucket's backend: - **Snowflake**: `snowflake_database` / `snowflake_schema` and per-table `snowflake_path` quoted with `"DB"."schema"."table"`. @@ -2956,7 +2956,7 @@ write descriptive metadata onto storage objects. Three behaviors are easy to mis saved offline against a BQ project, treat the `snowflake_*` fields as garbage. The `f"sapi_{project_id}"` Snowflake fallback (when `backendPath` is missing) still fires for Snowflake buckets but no longer for BigQuery. -- **BigQuery `databaseName` is usually empty** *(since v0.25.3)*. On Keboola- +- **BigQuery `databaseName` is usually empty** **. On Keboola- managed BQ projects the Storage API returns `databaseName: ""`, so `bucket-detail` cannot construct a fully-qualified `project.dataset.table` path -- the resulting `bigquery_path` is dataset-qualified only @@ -3022,40 +3022,40 @@ CLI hides via its four-bucket response, but they matter when interpreting result subdirectory and there is no risk of name collisions. Override with `--output DIR` if you need a custom location. -## Flow: conditional flows only; `--component-id` removed (since v0.57.0) +## Flow: conditional flows only; `--component-id` removed -- **RESOLVED (since v0.57.0):** the old foot-gun where `flow new` defaulted to +- **RESOLVED:** the old foot-gun where `flow new` defaulted to `keboola.flow` but `flow detail/update/delete/schedule/...` defaulted to `keboola.orchestrator` is **gone**. The `flow` group now targets the single component `keboola.flow`, and `--component-id` has been **removed** from every `flow` subcommand. Passing it errors with "No such option". -- **`keboola.orchestrator` is dropped (since v0.57.0).** `flow list` does NOT +- **`keboola.orchestrator` is dropped.** `flow list` does NOT list orchestrator configs; it reports their total as `legacy_orchestrator_count` (+ a warning) so you can see why a legacy flow "disappeared". There is no migration command (cross-component migration is out of scope). -- **IDs are STRINGS (since v0.57.0).** `phase.id`, `task.id`, `next.id`, +- **IDs are STRINGS.** `phase.id`, `task.id`, `next.id`, `task.phase`, and `goto` are all JSON strings (`goto` is `string | null`). Integer ids fail Draft7 validation and are rejected with `INVALID_FLOW_DEFINITION`. -- **The old `dependsOn` phase-DAG template is invalid (since v0.57.0).** Phases +- **The old `dependsOn` phase-DAG template is invalid.** Phases use `next[].goto` (a phase id or `null` to end) with an optional `condition`; a phase with conditional transitions must end with a default (condition-less) transition. Tasks are typed (`job`/`notification`/`variable`). -- **`INVALID_FLOW_DAG` was renamed to `INVALID_FLOW_DEFINITION` (since v0.57.0).** +- **`INVALID_FLOW_DAG` was renamed to `INVALID_FLOW_DEFINITION`.** Update any code/string matching on the old error code. -- **Validation (since v0.57.0):** `kbagent flow validate --file @flow.yaml [--project ALIAS]`. +- **Validation:** `kbagent flow validate --file @flow.yaml [--project ALIAS]`. With `--project` it fetches the **live** JSON Schema from the stack and runs full structural + semantic checks; without `--project` it runs semantic-only and adds a note that structural validation was skipped (no schema source). Exit 0 valid, exit 2 on errors. Use it in a tight loop before `flow new`/`flow update`. -- **Schema is fetched live from the stack, NOT bundled (since v0.57.0).** The +- **Schema is fetched live from the stack, NOT bundled.** The conditional-flow JSON Schema is served by the stack's component registry and read at runtime via the AI Service `configurationSchema` for `keboola.flow` (the same path `config new --push` uses). There is nothing vendored, pinned, or to re-sync. `flow schema --full` therefore **requires `--project`** (plain `flow schema` is still the offline YAML template). -- **Graceful semantic-only degradation (since v0.57.0).** If the live schema +- **Graceful semantic-only degradation.** If the live schema fetch fails (network error, or the AI Service returns no `configurationSchema`), `flow new`/`flow update`/`flow validate --project` do **not** block: structural validation is skipped, the semantic checks still run (Storage does not validate @@ -3103,7 +3103,7 @@ The trade-off is deliberate: one big call avoids the O(unique-parents) round-tri scheduler configs that target the flow. Pair it with `--dry-run` to see the affected configs (cron + timezone) without calling `delete_config`. -## Flow: `schedule` activates on the Scheduler Service (since v0.66.1) +## Flow: `schedule` activates on the Scheduler Service - `kbagent flow schedule` registers the `keboola.scheduler` config with the **Scheduler Service** (`POST /schedules`) after writing it — writing the @@ -3118,13 +3118,13 @@ The trade-off is deliberate: one big call avoids the O(unique-parents) round-tri - `flow schedule-remove` (v0.66.1+) deregisters each schedule from the Scheduler Service before deleting its config, so removal stops the trigger. -## `search` is a top-level command, not `config search` (since v0.30.0) +## `search` is a top-level command, not `config search` `kbagent search QUERY` searches across **all item types** (tables, buckets, configs, flows, data apps, transformations) via the Storage API global-search endpoint. It is distinct from `kbagent config search --query Q` which scans only configuration JSON bodies. - `search --search-type config-based` delegates to `config search` internally but exposes the unified results shape. It matches **case-insensitively** (since v0.84.0) while `config search --query` stays case-sensitive unless `-i` is passed -- the same logical table is often mixed-case in a row name and upper-case in `storage.input.tables[].source`, so a case-sensitive body scan answers "is this referenced anywhere?" with a false no. - Options (`--type`, `--project`, `--limit`) must come AFTER the QUERY argument: `kbagent search "text" --type table --limit 10`. -## `config row-create` / `row-update` / `row-delete` lifecycle (since v0.30.0) +## `config row-create` / `row-update` / `row-delete` lifecycle Full CRUD for configuration rows is exposed as a separate `Rows` command panel: - `row-create` returns the new row object including `id`. Capture this ID for subsequent `row-update` / `row-delete` calls. @@ -3132,7 +3132,7 @@ Full CRUD for configuration rows is exposed as a separate `Rows` command panel: - `row-delete` is **destructive** (gated behind `--allow-destructive` if the session firewall is on). 404 from the API on a non-existent row surfaces as `NOT_FOUND` exit 1 — deletion is **not** treated as idempotent success. - `--json` mode auto-skips the interactive confirmation prompt on `row-delete`; in human mode pass `--yes` to skip. -## `project status` / `project list` expose `org_id` / `org_name`; `org_name` is Manage-API-only (since v0.40.3) +## `project status` / `project list` expose `org_id` / `org_name`; `org_name` is Manage-API-only `ProjectConfig` now persists `org_id` (int | None) and `org_name` (str | None); both are surfaced verbatim in `kbagent project status` and `kbagent project @@ -3158,7 +3158,7 @@ projects. The web UI Projects table renders `#` (e.g. `#73`) as a fallback when only the id is known, so any agent producing a human-readable project list should do the same — never render the bare null. -## `config oauth-url` requires a master Storage API token (since v0.30.0) +## `config oauth-url` requires a master Storage API token The OAuth wizard URL embeds a short-lived **child** Storage API token scoped to the target component. Minting this child token via `POST /v2/storage/tokens` @@ -3181,7 +3181,7 @@ requires a **master (admin) token** — `canManageTokens` alone is not enough working credential here. `token create` carries the same guard since v0.89.0 (`token refresh` does not — that endpoint has no such defect). -## `data-app logs` is the only unconstrained log surface (since v0.43.8) +## `data-app logs` is the only unconstrained log surface - The upstream `keboola-mcp-server` `get_data_apps` MCP tool hardcodes a 20-line cap on log output (`_fetch_logs(..., lines=20)` in @@ -3228,7 +3228,7 @@ requires a **master (admin) token** — `canManageTokens` alone is not enough values via `rich.markup.escape` per the `commands/config.py` precedent. -## `kbagent agent` CRUD works offline; cron firing needs `kbagent serve` running (since v0.44.0) +## `kbagent agent` CRUD works offline; cron firing needs `kbagent serve` running The `kbagent agent ` command tree reads/writes `/agents.json` directly via `AgentService`, so `agent list / show / create / update / delete / @@ -3253,7 +3253,7 @@ This is the SAME on-disk format the REST router writes via `POST /agents`, so CLI-created tasks are interchangeable with UI-created and REST-created tasks -- the only difference is who fires them. -## `agent` action helpers are shared between REST + CLI (since v0.44.0) +## `agent` action helpers are shared between REST + CLI The `validate_trigger` (cycle/self-loop check) and `merge_runtime_input` (per-action-type runtime input merge) helpers live in @@ -3263,7 +3263,7 @@ new action type, update **both** the runner dispatcher (`agent_runner.py`) **and** `merge_runtime_input` to keep CRUD parity. Tests in `tests/test_agent_service.py` + `tests/test_agent_cli.py` catch drift. -## `feature` command group: super-admin token, no per-project endpoint, opaque schema (since v0.48.0) +## `feature` command group: super-admin token, no per-project endpoint, opaque schema The `feature` group manages Keboola feature flags via the **Manage API**. Five things trip up callers: @@ -3319,7 +3319,7 @@ Reads (`dev-portal list`, `dev-portal get`) are unrestricted — peer-research patterns ("show me how MySQL and Postgres extractors configure themselves") are agent-friendly via `list --vendor` + `get --app`. -## Headless / token-only invocation: the `__env__` project (since v0.50.0) +## Headless / token-only invocation: the `__env__` project A daemon, container, or CI job that has only a token in its environment can run kbagent with **no `kbagent project add` and no `config.json` on disk**. Set all @@ -3358,14 +3358,14 @@ Gotchas: (`{projectId}-{tokenId}-{secret}`) but cannot fetch the real project name. Run `kbagent project status --project __env__` (or `project info`) to verify the token against the API and see the real name. -- **`KBC_STORAGE_API_URL` is forgiving (since v0.50.0).** A bare host +- **`KBC_STORAGE_API_URL` is forgiving.** A bare host (`connection.keboola.com`), a trailing slash, or a full project deep-link (`.../admin/projects/123/dashboard`) all normalize to `https://`. Same normalization applies to `project add --url` / `project edit --url`. Explicit `http://` / `file://` is still rejected; a bad URL fails fast with a clean config error (exit 5), not a traceback. -## `stream`: two hosts, secret-in-URL, no auto-sinks (since v0.50.0) +## `stream`: two hosts, secret-in-URL, no auto-sinks Data Streams has **two hosts**. The *control plane* is `stream.` (derived from `connection.`, same scheme as `ai.`/`queue.`) and is what the CLI @@ -3406,7 +3406,7 @@ so when `.forbidden()` fires Joi reuses the unrelated enum message instead of saying "this field is not allowed here". To set any of these you need an admin identity that routes the PATCH -to `PATCH /admin/apps/{app}` instead (since v0.51.1): +to `PATCH /admin/apps/{app}` instead: ``` kbagent dev-portal identity add --alias admin-keboola \ @@ -3417,12 +3417,12 @@ kbagent dev-portal patch --app keboola.ex-foo \ With `role_hint: vendor` (the default), kbagent now pre-flights the payload and fails fast with the same guidance instead of letting the -apps-api return the misleading 422 (since v0.51.1). The 9 forbidden +apps-api return the misleading 422. The 9 forbidden fields are documented in [keboola/developer-portal:src/lib/validation.js](https://github.com/keboola/developer-portal/blob/master/src/lib/validation.js) under `clientAppSchema()`. -## `dev-portal identity add`: MFA logins for TOTP accounts need the `challenge` field explicit (since v0.51.1) +## `dev-portal identity add`: MFA logins for TOTP accounts need the `challenge` field explicit The apiary spec calls `challenge` optional with default `SOFTWARE_TOKEN_MFA` on the second-step `POST /auth/login`, but in practice the server 404s @@ -3434,7 +3434,7 @@ the user` and mask the real first failure. The raised error includes the server response body and a hint about TOTP code rotation, so "stale code" can be distinguished from "wrong code" / "expired session". -## `dev-portal identity {add,edit} --password-stdin` works in both TTY and pipe mode (since v0.51.1) +## `dev-portal identity {add,edit} --password-stdin` works in both TTY and pipe mode Pre-0.51.1 the flag did `sys.stdin.read().strip()` unconditionally, which waits for EOF rather than Enter — pasting a password and pressing @@ -3464,7 +3464,7 @@ intentionally drop a local edit, delete the file (or the config dir) and pull. Applies at config and row granularity. `--all-projects` reports a per-project conflict as that project's error without aborting the rest of the batch. -## `ai_agent` `extra_args` need an opt-in env on the kbagent process (since v0.60.2) +## `ai_agent` `extra_args` need an opt-in env on the kbagent process `extra_args` on an `ai_agent` task are passed **verbatim** to the underlying AI CLI (claude/codex/gemini), so they can carry rail-disabling flags @@ -3480,14 +3480,14 @@ args** unless the env is set -- if your extra flags "do nothing", check the env on the kbagent process running the task, not the task definition. (Private advisory GHSA-777j-6p95-qv3m.) -Related (since v0.60.2): scheduled `ai_agent` subprocesses no longer inherit the +Related: scheduled `ai_agent` subprocesses no longer inherit the manage (`KBC_MANAGE_API_TOKEN`) or master (`KBC_MASTER_TOKEN*`) tokens from the serve environment -- an AI agent reaches Keboola via `kbagent http` or by forking `kbagent`, and never needs those super-admin credentials. `KBC_TOKEN` (storage) is retained, and `cli_command` tasks are unchanged. (Private advisory GHSA-wm54-r2hh-cxm9.) -## `data-app git-repo` needs a deployed app (since v0.63.3) +## `data-app git-repo` needs a deployed app The git-repo introspection command (sandboxes-service `GET /apps/{id}/git-repo`) returns **409 "App has no Git repository @@ -3622,7 +3622,7 @@ A `sync pull` without the right flags leaves column metadata empty in the local JSON. That does NOT mean Keboola has no metadata -- always re-fetch via `kbagent storage table-detail` when deciding about types. -### `Client` library: `query()` needs a provisioned workspace; `branch_id=None` costs a branch-list call (since v0.61.0) +### `Client` library: `query()` needs a provisioned workspace; `branch_id=None` costs a branch-list call The in-process library facade (`from keboola_agent_cli import Client`, 0.61.0+) is a thin wrapper, not a workspace manager. Three non-obvious behaviors: @@ -3641,7 +3641,7 @@ is a thin wrapper, not a workspace manager. Three non-obvious behaviors: coerce, so callers must cast (`int(row["x"])` etc.) for typed values. (Verified live against a Snowflake workspace; BigQuery behavior not yet verified.) -### `storage download-table` row filters send `whereValues[]` (array notation) (since v0.62.0) +### `storage download-table` row filters send `whereValues[]` (array notation) `--where-column` + `--where-value` + `--where-operator eq|neq` and `--changed-since` / `--changed-until` filter the export server-side. If you call @@ -3651,7 +3651,7 @@ the raw Storage API instead of the CLI, the values parameter is `whereValues[]` This is the credential-only, no-workspace way to pull a filtered/incremental slice -- `workspace query` with a `WHERE` clause needs a live workspace. -### `storage add-column --not-null` needs an empty table or `--default` (since v0.62.0) +### `storage add-column --not-null` needs an empty table or `--default` `storage add-column --column name:TYPE(length) [--not-null] [--default VALUE]` hits the SYNCHRONOUS Storage endpoint (no job to poll). `--not-null` on a table @@ -3659,7 +3659,7 @@ that ALREADY HAS ROWS is rejected by the backend with an API error (not a local validation error) unless you also pass `--default` -- the existing rows need a value for the new non-null column. Add `--default` when the table is non-empty. -### `job run --idempotency-key` is client-side dedup, scoped to one machine (since v0.63.0) +### `job run --idempotency-key` is client-side dedup, scoped to one machine The Keboola Queue API `POST /jobs` accepts **no** client-supplied idempotency / dedup token -- verified against the live OpenAPI spec (v1.3.8) and the server @@ -3678,7 +3678,7 @@ de-duplicates **client-side**: a `/job_idempotency.json` map of idempotency_store=JobIdempotencyStore(path))` -- the facade is config-dir-free, so you must supply the store path. -### The importable SDK is now typed (`py.typed` + result models) (since v0.63.0) +### The importable SDK is now typed (`py.typed` + result models) `keboola_agent_cli` ships a PEP 561 `py.typed` marker, so `mypy`/`ty`/IDEs treat the in-process library as typed. The high-traffic facade methods return pydantic @@ -3691,7 +3691,7 @@ snake_case field name, so `JobResult.model_validate(service_dict)` works on a service-layer dict directly. This is a typing/contract addition only; the dict shapes returned by the service layer and the `--json` CLI output are unchanged. -### `sync clone` needs a fresh target; flow/variable links remap automatically (since v0.63.0) +### `sync clone` needs a fresh target; flow/variable links remap automatically `sync clone` copies a reference synced tree into a **fresh** target project. Two things to internalise: @@ -3714,7 +3714,7 @@ things to internalise: the offending key and its actual type. Older versions silently stringified it (`str(dict)` → `"{'new': 'in.c-new'}"`) and pushed that as a "bucket ID". -### `search --regex` matches entity names only; `matched_columns` is textual-only (since v0.67.0) +### `search --regex` matches entity names only; `matched_columns` is textual-only `kbagent search --regex` opts into the Storage API `mode=regex` global-search path. Three things verified live against a real stack (2026-07-02): @@ -3778,7 +3778,7 @@ Four related sync-engine behaviors landed together (issues #466 / #467 / #472 / contacts the API, so it cannot see remote drift; treating it as a local-vs-production audit was the #466 trap. -## Table snapshots: restore is `tables-async`, `--name` is required, no overwrite (since v0.75.0) +## Table snapshots: restore is `tables-async`, `--name` is required, no overwrite `kbagent storage table-from-snapshot` (issue #512) creates a NEW table from an existing snapshot. Three traps, all verified live (us-east4.gcp, 2026-07-22): @@ -3801,7 +3801,7 @@ restores, source tables untouched; batch-tolerant, exit 1 on any failure). Snapshot create and restore are async storage jobs -- the CLI polls to completion, so the receipt's `table.rowsCount` is authoritative. -## The standalone binary never self-updates; recommend its own channel (since v0.79.0) +## The standalone binary never self-updates; recommend its own channel kbagent ships through two channels: a **Python distribution** (`uv tool install` / `pip`) and a **self-contained PyInstaller binary** with no Python runtime, @@ -3853,7 +3853,7 @@ upgraded in completely different ways, and the wrong advice is actively harmful. the GitHub release page. Note the **package** is `keboola-cli2` while the **binary** is `kbagent`; the PyPI distribution is a third name, `keboola-cli`. -## `--json` is written as UTF-8, independent of the console codepage (since v0.78.0) +## `--json` is written as UTF-8, independent of the console codepage `--json` output no longer goes through the terminal's text encoder. It is written straight to `sys.stdout.buffer` as UTF-8, so the bytes you parse never diff --git a/plugins/kbagent/skills/kbagent/references/library-workflow.md b/plugins/kbagent/skills/kbagent/references/library-workflow.md index 7d555643..b2cff9a2 100644 --- a/plugins/kbagent/skills/kbagent/references/library-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/library-workflow.md @@ -1,7 +1,7 @@ # Python Library Workflow (`from keboola_agent_cli import Client`) Besides the CLI and the `kbagent serve` daemon, kbagent ships a **stateless, -importable Python library** (since 0.61.0). It lets an in-process consumer -- a +importable Python library**. It lets an in-process consumer -- a Keboola Data App, a transformation, a hosted service -- run Query Service SQL and read/write Storage Files with **no CLI subprocess, no daemon, and no config-dir**. diff --git a/plugins/kbagent/skills/kbagent/references/member-workflow.md b/plugins/kbagent/skills/kbagent/references/member-workflow.md index 061c67cc..c38176f6 100644 --- a/plugins/kbagent/skills/kbagent/references/member-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/member-workflow.md @@ -1,4 +1,4 @@ -# Project Member & Invitation Workflow (since v0.29.0) +# Project Member & Invitation Workflow Closes the long-standing Manage API gap that forced every Keboola-internal automation (most recently the Cuesta-training orchestrator) to bypass kbagent diff --git a/plugins/kbagent/skills/kbagent/references/scaffold-workflow.md b/plugins/kbagent/skills/kbagent/references/scaffold-workflow.md index b92ff8c7..dd21d675 100644 --- a/plugins/kbagent/skills/kbagent/references/scaffold-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/scaffold-workflow.md @@ -1,6 +1,6 @@ # Creating New Configurations (Scaffold Workflow) -> **Two modes, one command (since v0.33.0):** +> **Two modes, one command:** > - `kbagent config new --output-dir DIR` (this workflow) -- generate > scaffold files to disk, edit them, then push later with > `kbagent sync push`. The "GitOps for configs" path. @@ -47,7 +47,7 @@ kbagent --json config new --component-id COMPONENT_ID --project ALIAS --name "Co # To disk (auto-detects kbc project structure, writes under main/ if applicable) kbagent config new --component-id COMPONENT_ID --project ALIAS --name "Config Name" --output-dir . -# One-shot remote create (since 0.33.0) -- no filesystem, just POST + return ID +# One-shot remote create -- no filesystem, just POST + return ID kbagent --json config new --component-id COMPONENT_ID --project ALIAS --name "Config Name" \ --push --no-files diff --git a/plugins/kbagent/skills/kbagent/references/semantic-layer-workflow.md b/plugins/kbagent/skills/kbagent/references/semantic-layer-workflow.md index d6e5a4d0..b7b89238 100644 --- a/plugins/kbagent/skills/kbagent/references/semantic-layer-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/semantic-layer-workflow.md @@ -230,7 +230,7 @@ success/failure in the response envelope's `rollback` field. If the rollback itself fails, the model is left in a partial state -- run `semantic-layer validate` immediately. -**Partial cascade state (since v0.41.10)**: the cascade has per-item +**Partial cascade state**: the cascade has per-item rollback only -- each constraint DELETE+POST rolls back individually. If the metric rename succeeds but M of N dependent constraints fail to repoint, the envelope sets `partial_state: true` at the top level @@ -390,7 +390,7 @@ inference, paired range constraints), the `sl-build` skill in `04_AI_Kit/ai-kit/` is the right tool. The two are interoperable via the same metastore contract; bridge between them as needed. -**Rollback on push failure (since v0.41.10)**: if a child POST fails +**Rollback on push failure**: if a child POST fails mid-push, the service walks the list of successfully-POSTed children in REVERSE PUSH_ORDER and DELETEs each one, then DELETEs the model itself if we created it during this call. The wrapped error carries @@ -495,7 +495,7 @@ Quick reminders: - **Duplicate-name POST -> 409 Conflict** (post go-monorepo PR #513) with `"Object with this name already exists in this project"`, or **500** with `"Failed to create meta object"` on legacy stacks. kbagent normalizes both - into `ErrorCode.ALREADY_EXISTS` (since v0.43.5). 409 is non-retryable so + into `ErrorCode.ALREADY_EXISTS`. 409 is non-retryable so the fix-deployed path avoids the `MAX_RETRIES` round-trips the 500 path still pays. - **DELETE -> 204** empty body. diff --git a/plugins/kbagent/skills/kbagent/references/storage-types-workflow.md b/plugins/kbagent/skills/kbagent/references/storage-types-workflow.md index 7fff28c8..420ad6a5 100644 --- a/plugins/kbagent/skills/kbagent/references/storage-types-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/storage-types-workflow.md @@ -86,7 +86,7 @@ against an unmaterialized bucket returns `Bucket not found`. The response surfaces this via `auto_created_bucket: true`. Production writes (no `--branch`) never materialize anything. -### Branched-storage metadata stamp (since 0.25.1) +### Branched-storage metadata stamp On projects with the **branched storage** feature flag enabled, the transformation runner's `output-mapping` library @@ -114,7 +114,7 @@ client-side workaround. Closes #224. -### Fake-branch vs `storage-branches`: when `--branch X` is a no-op for the runner (since 0.25.2) +### Fake-branch vs `storage-branches`: when `--branch X` is a no-op for the runner Keboola Storage has **two parallel branch-isolation models**: @@ -227,7 +227,7 @@ kbagent --json storage create-table \ the field in JSON mode; in human mode it is shown as a yellow note under the created-table banner. -## Promoting a typed rebuild back into the original name (since v0.28.0) +## Promoting a typed rebuild back into the original name Common pattern: an existing typeless table needs proper column types. AI agent profiles the data in a workspace, builds a typed copy via CTAS, and @@ -306,7 +306,7 @@ Rules: - The swap is symmetric; there is no rollback besides swapping again (or aborting the dev branch). -## BigQuery repartition via `create-table --source-table-id` (since 0.66.0) +## BigQuery repartition via `create-table --source-table-id` On **BigQuery** you can produce the repartitioned/re-clustered copy in a single call instead of hand-writing a CTAS transformation: `create-table` diff --git a/plugins/kbagent/skills/kbagent/references/sync-workflow.md b/plugins/kbagent/skills/kbagent/references/sync-workflow.md index e8836426..10f8bcf7 100644 --- a/plugins/kbagent/skills/kbagent/references/sync-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/sync-workflow.md @@ -2,7 +2,7 @@ Sync lets you manage Keboola configurations as local files with full git integration. -## Variable values deployment (since v0.21.0) +## Variable values deployment `sync push` now deploys **config rows**, not just parent configs. This unlocks the most common GitOps use case: deploying `keboola.variables` values through git. @@ -99,7 +99,7 @@ Related semantics (all since v0.72.0): - `sync status` is a LOCAL-only check (says so explicitly) -- use `sync diff` for a local-vs-production audit. -## Per-invocation dev-branch override (since v0.47.0) +## Per-invocation dev-branch override `sync push`, `sync pull`, and `sync diff` accept `--branch ` to target a dev branch for a single invocation. The override beats every other branch @@ -129,7 +129,7 @@ The override is per-invocation only — it does not write into the manifest or the config store, so a subsequent command without `--branch` falls back to the normal priority chain. -**Promote the default tree to a target branch (since v0.47.2).** When the +**Promote the default tree to a target branch.** When the target branch has no materialized `/` subtree on disk, `sync push --branch ` reads the **default tree** (`main/`) as the source and promotes it to the target branch, instead of failing with `Config file not found`. @@ -181,7 +181,7 @@ Before this landed, that production diff classified the whole orphaned `main/` tree as `added` with an empty `config_id`, and `sync push` would have created a duplicate of every production config (issue #649). -## Fresh-CREATE writeback (since v0.47.0) +## Fresh-CREATE writeback If you (or a tool like FIIA) seed `.keboola/manifest.json` with placeholder entries before the first `sync push`, the writeback updates each placeholder @@ -197,7 +197,7 @@ metadata API immediately after the create call. This was the previous "set folderName via `config set-folder` after push" workaround for fresh-create flows; from v0.47.0 a single push handles it. -**Variable links are resolved on fresh CREATE (since v0.47.2).** When the +**Variable links are resolved on fresh CREATE.** When the placeholder tree includes a `keboola.variables` config + default-values row and a transformation that cross-references them, one `sync push` now produces a *runnable* transformation: @@ -218,14 +218,14 @@ placeholder can't be matched and the binding is ambiguous (zero or >1 `variable_link` entry appears in the push `errors` array — never a silently broken link. A clean re-push reports `no_changes`. -`sync push --no-name-drift-warnings` (since v0.47.0) suppresses the +`sync push --no-name-drift-warnings` suppresses the cosmetic `name_drift_warnings` array on the result envelope. The detection still runs; only the report is dropped. Useful for downstream tools that already audit drift their own way (e.g. FIIA's `var-07-fi-daily-date-refresh` pattern legitimately differs from the canonical kbagent naming and the warnings are noise). -## Adopting an existing kbc Go CLI checkout (since v0.22.0) +## Adopting an existing kbc Go CLI checkout If you already have a `.keboola/manifest.json` produced by the official `kbc` Go CLI (keboola-as-code), `kbagent` can adopt it in place instead of @@ -414,7 +414,7 @@ Stored in `.keboola/branch-mapping.json`: - **Pull is idempotent**: re-running pull when nothing changed writes zero files - **Pull protects local edits**: locally-modified files are skipped by default -- **`--force` is conflict-aware (since 0.53.0)**: see below -- it no longer blindly overwrites +- **`--force` is conflict-aware**: see below -- it no longer blindly overwrites - **Push only sends local changes**: remote_modified and conflict changes are skipped - **Push records the API's own view of what it wrote (since 0.91.0, #686)**: the manifest baseline (`pull_config_hash`) comes from the API response (or a diff --git a/plugins/kbagent/skills/kbagent/references/transformation-workflow.md b/plugins/kbagent/skills/kbagent/references/transformation-workflow.md index 3bfe2fe0..cf797e0a 100644 --- a/plugins/kbagent/skills/kbagent/references/transformation-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/transformation-workflow.md @@ -1,4 +1,4 @@ -# SQL Transformation Workflow (since v0.73.0) +# SQL Transformation Workflow Native authoring/editing of SQL transformations -- the CLI port of the upstream `create_sql_transformation` / `update_sql_transformation` tools diff --git a/plugins/kbagent/skills/kbagent/references/typify-table-workflow.md b/plugins/kbagent/skills/kbagent/references/typify-table-workflow.md index e4bc484b..c7ac6d45 100644 --- a/plugins/kbagent/skills/kbagent/references/typify-table-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/typify-table-workflow.md @@ -490,6 +490,6 @@ The rehearsal branch is already gone (Phase 7); there is no merge step. + `workspace query` patterns. The `from-transformation` mode is the alternative to ad-hoc workspaces when you want to debug an existing config that is failing. -- `gotchas.md` -- "swap-tables aliases stay put (since v0.28.0)"; - "config update auto-normalizes script[] (since v0.28.0)" if the +- `gotchas.md` -- "swap-tables aliases stay put"; + "config update auto-normalizes script[]" if the Phase 3 Option B SQL transformation route is taken. diff --git a/plugins/kbagent/skills/kbagent/references/workspace-workflow.md b/plugins/kbagent/skills/kbagent/references/workspace-workflow.md index 53754579..481b7136 100644 --- a/plugins/kbagent/skills/kbagent/references/workspace-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/workspace-workflow.md @@ -168,7 +168,7 @@ kbagent --json workspace query \ --file query.sql ``` -## Fast inline results vs `--full` -- mind the result-set volume (since v0.59.0) +## Fast inline results vs `--full` -- mind the result-set volume `workspace query` has two ways to retrieve results. **Pick based on how many rows you actually need**, not by habit. @@ -254,7 +254,7 @@ kbagent --json storage bucket-detail --project ALIAS --bucket-id in.c-shared-dat ``` Prefer `sql_path` in agent code -- it is correctly quoted for the bucket's -backend without you having to branch on dialect yourself. (since v0.25.3) +backend without you having to branch on dialect yourself. **BigQuery FQN caveat**: on Keboola-managed BQ projects the Storage API returns `databaseName: ""`, so `bigquery_path` ends up dataset-qualified only @@ -274,11 +274,11 @@ workspace. - **Quoting** (dialect-specific): - **Snowflake**: converts unquoted identifiers to UPPERCASE. Always double-quote database, schema, and table names -- Keboola names are typically lowercase (e.g. `"sapi_901"."in.c-main"."users"`). - **BigQuery**: requires backticks (`` ` ``), not double quotes; the dataset name is normalized to underscores (e.g. `` `in_c_main`.`users` ``). - - Easiest path: read `tables[].sql_path` from `bucket-detail` -- it is already correctly quoted for the bucket's backend (since v0.25.3). + - Easiest path: read `tables[].sql_path` from `bucket-detail` -- it is already correctly quoted for the bucket's backend. - **Query Service**: uses Storage API token for auth -- no warehouse credentials needed in the query command. Backend-agnostic: runs SELECTs against **both Snowflake and BigQuery** workspaces (BigQuery since v0.58.0; the path was always identical, the gap was classification). BigQuery workspaces carry `login_type: "default"` and are `qs_compatible: true` from v0.58.0 -- earlier versions mislabeled them `false`. - **Transactional mode**: add `--transactional` to wrap SQL in a transaction -## Orphan detection + garbage collection (since v0.22.0) +## Orphan detection + garbage collection Workspaces are backed by `keboola.sandboxes` configs. When a config is deleted out-of-band (UI cleanup, another CLI, force-delete script), the workspace diff --git a/scripts/check_release_scope.py b/scripts/check_release_scope.py new file mode 100644 index 00000000..c0dd9e58 --- /dev/null +++ b/scripts/check_release_scope.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Prove the new changelog entry covers every PR the release tag will contain. + +``make changelog-check`` answers a different question: does every *released +version* have a changelog entry? It never asks whether that entry covers every +*commit* under the tag. The gap is not hypothetical -- it shipped in v0.91.0. + +PR #625 merged to ``main`` at 14:27, after the release PR had branched off and +before the release PR itself merged at 14:36. The changelog entry had been +authored against the earlier scope, so #625 -- a new plugin slash command plus +a rewritten onboarding flow -- sat inside the tag's tree with no release note +of any kind. It was caught only because the tag happened to be deferred. + +The window is structural: a release PR is open for as long as its CI takes, +and that is exactly when parallel feature PRs land. So the scope collected when +the release PR is *opened* is not the scope the tag will *contain*, and the +only trustworthy moment to compare them is immediately before tagging. + +Usage:: + + python scripts/check_release_scope.py # v..HEAD + python scripts/check_release_scope.py --base v0.90.1 + python scripts/check_release_scope.py --head origin/main + python scripts/check_release_scope.py --ignore-pr 699 # repeatable + +Run it in the release PR *before* merging and no PR needs ignoring: the release +PR's own number is not in the log until its merge commit exists. Run it after +merging (against ``origin/main``) and the release PR itself shows up as a miss +-- that is what ``--ignore-pr`` is for. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +from packaging.version import InvalidVersion, Version + +REPO_ROOT = Path(__file__).resolve().parents[1] + +# GitHub's squash-merge subject ends in ``(#N)``; the "Merge pull request #N" +# subject is the merge-commit form. A ``(#N)`` anywhere else in a subject is a +# cross-reference to another PR ("follow-up to (#686)"), not this commit's own, +# so only the trailing form counts. +_TRAILING_PR_RE = re.compile(r"\(#(\d+)\)\s*$") +_MERGE_COMMIT_RE = re.compile(r"^\S+\s+Merge pull request #(\d+)\b") + +# Any GitHub number cited in a changelog bullet -- a PR decoration like +# ``Fix (#686, #694):`` or an issue named in prose. Both are evidence the +# release notes account for the work. +_ANY_REF_RE = re.compile(r"#(\d+)") + + +def merged_pr_numbers(log_text: str) -> list[str]: + """Return the PR numbers in ``git log --oneline --first-parent`` output. + + Order is preserved and duplicates are dropped, so the report reads in the + same order as the log the caller can eyeball. + """ + found: dict[str, None] = {} + for line in log_text.splitlines(): + match = _TRAILING_PR_RE.search(line) or _MERGE_COMMIT_RE.match(line) + if match: + found.setdefault(match.group(1), None) + return list(found) + + +def referenced_pr_numbers(notes: list[str]) -> set[str]: + """Return every GitHub number cited anywhere in a release's changelog bullets.""" + return {ref for note in notes for ref in _ANY_REF_RE.findall(note)} + + +def missing_references(log_text: str, notes: list[str], ignore: frozenset[str]) -> list[str]: + """Return merged PR numbers that the changelog entry never mentions.""" + referenced = referenced_pr_numbers(notes) + return [pr for pr in merged_pr_numbers(log_text) if pr not in referenced and pr not in ignore] + + +def should_arm(base: str, head: str) -> bool: + """Whether this check applies: only a PR that RAISES the version is a release PR. + + Fails open on anything unreadable or malformed. A shallow CI checkout may + not be able to read the base branch's ``pyproject.toml`` at all, and an + ordinary feature PR must never be blocked by that -- the cost of a missed + arming is one manual ``make release-scope-check``, the cost of a false + arming is every PR in the repo going red. + """ + try: + return Version(head) > Version(base) + except (InvalidVersion, TypeError): + return False + + +class GitUnavailable(RuntimeError): + """Git could not answer -- typically a shallow CI checkout with no tags.""" + + +def _git(*args: str) -> str: + try: + return subprocess.run( + ["git", *args], cwd=REPO_ROOT, capture_output=True, text=True, check=True + ).stdout + except (subprocess.CalledProcessError, OSError) as exc: + raise GitUnavailable(f"git {' '.join(args)} failed") from exc + + +def _last_release_tag() -> str: + """The most recent tag reachable from HEAD, i.e. the release being built on.""" + return _git("describe", "--tags", "--abbrev=0", "--match", "v*").strip() + + +def _arg(name: str, default: str | None = None) -> str | None: + if name not in sys.argv: + return default + index = sys.argv.index(name) + 1 + if index >= len(sys.argv): + raise SystemExit(f"ERROR: {name} needs an argument") + return sys.argv[index] + + +def _ignored() -> frozenset[str]: + ignored: set[str] = set() + for index, token in enumerate(sys.argv): + if token == "--ignore-pr" and index + 1 < len(sys.argv): + ignored.add(sys.argv[index + 1].lstrip("#")) + return frozenset(ignored) + + +def main() -> int: + sys.path.insert(0, str(REPO_ROOT / "src")) + from keboola_agent_cli.changelog import CHANGELOG + + text = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + match = re.search(r'^version\s*=\s*"([^"]+)"', text, re.MULTILINE) + if not match: + print("ERROR: could not read version from pyproject.toml") + return 1 + version = match.group(1) + + notes = CHANGELOG.get(version) + if notes is None: + print(f"ERROR: pyproject.toml declares {version}, which has no changelog entry.\n") + print("Add the entry before checking its scope (release checklist step 3).") + return 1 + + arm_base = _arg("--only-if-newer-than") + if arm_base is not None and not should_arm(arm_base.strip(), version): + print( + f"Not a release PR (base {arm_base.strip() or ''} -> {version}); " + "release-scope check not armed." + ) + return 0 + + ignore = _ignored() + try: + base = _arg("--base") or _last_release_tag() + head = _arg("--head") or "HEAD" + log_text = _git("log", f"{base}..{head}", "--oneline", "--first-parent") + except GitUnavailable as exc: + # Loud, but never blocking: a shallow checkout without tags cannot + # answer this, and that must not turn into a red build on a PR whose + # content is fine. On a release PR the checklist runs it locally. + print(f"WARNING: release-scope check skipped -- {exc}.") + print(" A shallow checkout has no tags/history. Run 'make release-scope-check' locally.") + return 0 + absent = missing_references(log_text, notes, ignore) + + if absent: + print( + f"ERROR: {len(absent)} PR(s) merged in {base}..{head} are not in the {version} entry.\n" + ) + print( + "The tag will contain this work, but the release notes are rendered\n" + "from the changelog entry -- so it would ship with no note at all.\n" + "Add a bullet for each, or pass --ignore-pr N for the release PR itself.\n" + ) + for pr in absent: + subject = next( + (line for line in log_text.splitlines() if f"#{pr})" in line or f"#{pr} " in line), + "", + ) + print(f" #{pr} {subject.split(' ', 1)[-1] if subject else ''}") + return 1 + + covered = len(merged_pr_numbers(log_text)) + print( + f"Release scope OK: all {covered} PR(s) merged in {base}..{head} " + f"are referenced by the {version} changelog entry." + ) + if ignore: + print(f" (ignored: {', '.join('#' + pr for pr in sorted(ignore))})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/check_version_gates.py b/scripts/check_version_gates.py index 49d5c739..df80f0fa 100644 --- a/scripts/check_version_gates.py +++ b/scripts/check_version_gates.py @@ -155,6 +155,48 @@ def find_vnext_residue(paths: list[Path]) -> list[VnextResidue]: return residue +# An ATX markdown heading: 1-6 hashes followed by a space (CommonMark requires +# the space, so ``#tag`` is not a heading). Only ``.md`` files are considered -- +# ``src/**/*.py`` is scanned for gates too, but a ``#`` there opens a comment, +# which has no anchor slug to break. +HEADING_RE = re.compile(r"^ {0,3}#{1,6} ") + + +def find_heading_placeholders(paths: list[Path]) -> list[VnextResidue]: + """Return every markdown heading carrying a live ``vNEXT`` placeholder. + + Resolving the placeholder rewrites the heading, which rewrites its + generated anchor slug, which breaks every inbound ``#...`` link. Unlike + :func:`find_vnext_residue`, this is fatal on EVERY PR rather than only in + release mode: the rule used to be a hand-run grep at release time, and in + 0.91.0 that grep lost a merge race (PR #697 ran it two minutes before #694 + and #696 landed their own headings). Checking at authoring time is what + makes the race impossible. + + Numeric versions in headings are deliberately NOT flagged: an already + resolved tag never changes again, so its slug is stable, and flagging the + dozen historical ones would be noise with no inbound link at risk. + """ + flagged: list[VnextResidue] = [] + for path in paths: + if path.suffix != ".md": + continue + try: + rel = path.relative_to(REPO_ROOT).as_posix() + except ValueError: + rel = path.as_posix() + for lineno, line in enumerate( + path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1 + ): + if VNEXT_TOKEN not in line or not HEADING_RE.match(line): + continue + # Same quotation rule as the residue scan: a heading that merely + # names the token in backticks is prose about the placeholder. + if VNEXT_TOKEN in INLINE_CODE_RE.sub("", line): + flagged.append(VnextResidue(path=rel, line=lineno, text=line.strip())) + return flagged + + def collect_gates(paths: list[Path]) -> dict[str, list[tuple[str, int]]]: """Map each gated version to the ``(relative path, line number)`` naming it. @@ -178,6 +220,93 @@ def collect_gates(paths: list[Path]) -> dict[str, list[tuple[str, int]]]: return dict(gates) +# Matches the bare placeholder token. ``vNEXT+`` needs no special case: the +# token is replaced in place, so ``vNEXT+`` becomes ``0.91.0+`` on its own. +VNEXT_SUB_RE = re.compile(re.escape(VNEXT_TOKEN)) + + +def _replace_outside_code(line: str, version: str) -> tuple[str, int]: + """Substitute the placeholder only in the parts of *line* outside code spans. + + Rewriting whole lines is what makes a blanket ``sed`` unsafe: a line may + carry a quoted mention AND a live gate at once (CLAUDE.md's description of + the placeholder is exactly that), and only the live one may change. + """ + pieces: list[str] = [] + replaced = 0 + pos = 0 + for span in INLINE_CODE_RE.finditer(line): + chunk, count = VNEXT_SUB_RE.subn(version, line[pos : span.start()]) + pieces.append(chunk) + replaced += count + pieces.append(span.group(0)) # the code span itself is preserved verbatim + pos = span.end() + chunk, count = VNEXT_SUB_RE.subn(version, line[pos:]) + pieces.append(chunk) + replaced += count + return "".join(pieces), replaced + + +def resolve_vnext(paths: list[Path], version: str) -> list[VnextResidue]: + """Rewrite every live ``vNEXT`` gate in *paths* to *version*, in place. + + Returns one entry per rewritten LINE, carrying the text as it now reads. + Files with nothing to change are not written at all, so a release PR's + diff shows only the files that actually carry a gate. + + This is the mechanical form of release checklist step 4. The scanner + already tells a live gate from prose with perfect precision; having a + human apply that knowledge by hand across ~54 lines only adds error. + """ + try: + Version(version) + except Exception as exc: # packaging raises InvalidVersion + raise ValueError(f"{version!r} is not a valid PEP 440 version") from exc + + changed: list[VnextResidue] = [] + for path in paths: + try: + rel = path.relative_to(REPO_ROOT).as_posix() + except ValueError: + rel = path.as_posix() + original = path.read_text(encoding="utf-8") + if VNEXT_TOKEN not in original: + continue + out_lines: list[str] = [] + file_changed = False + for lineno, line in enumerate(original.splitlines(keepends=True), start=1): + if VNEXT_TOKEN not in line: + out_lines.append(line) + continue + new_line, count = _replace_outside_code(line, version) + out_lines.append(new_line) + if count: + file_changed = True + changed.append(VnextResidue(path=rel, line=lineno, text=new_line.strip())) + if file_changed: + path.write_text("".join(out_lines), encoding="utf-8") + return changed + + +def gates_below( + gates: dict[str, list[tuple[str, int]]], floor: str +) -> dict[str, list[tuple[str, int]]]: + """Return the gates naming a version older than *floor*, oldest version first. + + A gate only earns its place while some live install predates it. kbagent + self-updates on startup, so for a sufficiently old version that population + rounds to zero -- while the gate keeps making the agent refuse a command + the user actually has. Raising a floor and de-tagging below it is periodic + maintenance; this produces the worklist. + + The floor itself is NOT below the floor: it is the oldest version still + worth gating for. + """ + limit = Version(floor) + below = {v: locs for v, locs in gates.items() if Version(v) < limit} + return {v: below[v] for v in sorted(below, key=Version)} + + def resolve_paths() -> list[Path]: """Expand SCANNED_GLOBS into an ordered, de-duplicated file list.""" seen: dict[Path, None] = {} @@ -233,14 +362,87 @@ def main() -> int: paths = resolve_paths() gates = collect_gates(paths) residue = find_vnext_residue(paths) + heading_residue = find_heading_placeholders(paths) + + if "--resolve" in sys.argv: + index = sys.argv.index("--resolve") + 1 + if index >= len(sys.argv): + print("ERROR: --resolve needs a version argument (e.g. --resolve 0.91.0)") + return 1 + requested = sys.argv[index].strip() + shipped = _pyproject_version() + # Cross-check against pyproject rather than trusting the format alone: + # `packaging` accepts `v0.91` and `0.91`, so a typo can parse cleanly + # and then be stamped into every gate in the tree at once. + if requested.lstrip("v") != shipped: + print( + f"ERROR: --resolve {requested} disagrees with pyproject.toml ({shipped}).\n\n" + "Resolve gates to the version this tree actually ships. Bump\n" + "pyproject.toml first, then re-run.\n" + ) + return 1 + try: + applied = resolve_vnext(paths, shipped) + except ValueError as exc: + print(f"ERROR: {exc}") + return 1 + if not applied: + print(f"No unresolved '{VNEXT_TOKEN}' gates to rewrite.") + return 0 + print(f"Rewrote {len(applied)} '{VNEXT_TOKEN}' gate(s) to {shipped}:\n") + for gate in applied: + print(f" {gate.path}:{gate.line}") + still = find_heading_placeholders(resolve_paths()) + if still: # pragma: no cover - defensive; headings are fatal earlier + print(f"\nWARNING: {len(still)} placeholder(s) remain in headings.") + return 0 + + if "--list-below" in sys.argv: + index = sys.argv.index("--list-below") + 1 + if index >= len(sys.argv): + print("ERROR: --list-below needs a version argument (e.g. --list-below 0.80.0)") + return 1 + floor = sys.argv[index].strip().lstrip("v") + stale = gates_below(gates, floor) + total = sum(len(locs) for locs in stale.values()) + print(f"{total} gate(s) across {len(stale)} version(s) below the {floor} floor:\n") + for version in stale: + print(f" {version}") + for rel, lineno in stale[version]: + print(f" {rel}:{lineno}") + return 0 if "--list" in sys.argv: for version in sorted(gates, key=lambda v: [int(p) for p in v.split(".")]): mark = " " if version in CHANGELOG else " <-- UNKNOWN" print(f"{version:>10} {len(gates[version]):>3} marker(s){mark}") print(f"{VNEXT_TOKEN:>10} {len(residue):>3} marker(s) <-- unresolved placeholder") + print(f"{'in headings':>10} {len(heading_residue):>3} marker(s) <-- always fatal") return 0 + # Fatal in EVERY mode, unlike the plain residue below: a placeholder in a + # heading is never correct at any point in the release cycle, and deferring + # the complaint to the release PR is exactly how 0.91.0 shipped three of + # them (the hand-run grep in #697 raced #694 and #696). + if heading_residue: + print( + f"ERROR: {len(heading_residue)} '{VNEXT_TOKEN}' placeholder(s) inside a " + "markdown heading.\n" + ) + print( + "Resolving the placeholder rewrites the heading, which rewrites its\n" + "generated anchor slug and breaks every inbound '#...' link to the\n" + "section. Move the tag onto the section's first body line instead:\n" + "\n" + " ## Ignored components\n" + "\n" + f" *(since {VNEXT_TOKEN}, #689)*\n" + ) + for gate in heading_residue: + print(f" {gate.path}:{gate.line}") + print(f" {gate.text[:100]}") + return 1 + unknown = {v: locs for v, locs in gates.items() if v not in CHANGELOG} if unknown: print("ERROR: version gate names a version with no changelog entry.\n") diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index 642ef608..d3b6e99e 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -303,7 +303,7 @@ KBAGENT_PROJECT override that may name an alias absent from the config), and the other two are write confirmations. -### Project Members & Invitations (since v0.29.0) +### Project Members & Invitations Requires KBC_MANAGE_API_TOKEN (Manage API auth). Allowed roles: admin, guest, readOnly, share. @@ -352,7 +352,7 @@ NOT_FOUND is still raised when both sources miss. kbagent component sync-action ACTION_NAME --component-id ID --project ALIAS (--config-id ID [--row-id ID] | --config-data JSON|@file|-) [--branch ID] [--timeout N] - (since 0.73.0) Run a synchronous component action (testConnection, getTables, + Run a synchronous component action (testConnection, getTables, ...) on the dedicated sync-actions service. ACTION_NAME is freeform -- valid names are component-defined (see component detail synchronous_actions). --row-id shallow-merges the row over the root config at TOP level only @@ -470,7 +470,7 @@ Search config bodies for string/regex. Reports match location in JSON tree. Branch-aware. kbagent config examples --component-id ID [--project NAME] [--row] - (since 0.73.0) Sample root/row configurations for a component, straight from + Sample root/row configurations for a component, straight from the AI-service component detail (same data the UI shows). --row limits to row examples. --json emits {{component_id, root_examples, row_examples}} -- structured dicts, ideal as a starting point before config new / row-create. @@ -574,7 +574,7 @@ --mode run (default) writes to mapped output tables. --mode debug runs the component but redirects the output to a Storage File tagged `debug-` instead of into destination buckets -- safe for dry-runs and for reproducing a failing run on a production configuration - without touching production data. Invalid values exit 2 via Click choice gate (since v0.43.6). + without touching production data. Invalid values exit 2 via Click choice gate. --json response shapes by exit code: - exit 0 (success): {{status:"ok", data:{{..., logTail?:[...]}}}} - exit 1 (QUEUE_JOB_FAILED, remote job status=error): @@ -653,7 +653,7 @@ kbagent storage create-table --project NAME --bucket-id BUCKET_ID --name TABLE_NAME [--column col:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists] Create a typed table. --column repeatable. - - --if-not-exists (since 0.47.0): opt-in idempotency. On a duplicate-display-name failure, + - --if-not-exists: opt-in idempotency. On a duplicate-display-name failure, probe get-table-detail at the expected id and, if the table really exists, return `action: "skipped", skip_reason: "table already exists"` instead of raising. A different table with the same display name still surfaces the original error. Safe for parallel workers. @@ -978,7 +978,7 @@ /pay-as-you-go/billing/*, which does not accept a Storage token (issue #594, still open). Do not imply invoices are retrievable. -### Feature Flags (since v0.48.0) +### Feature Flags Requires a SUPER-ADMIN Manage API token (same kind as `org setup`). Same default-deny token policy: interactive hidden prompt by default; pass @@ -1016,7 +1016,7 @@ since 0.73.0 -- previously an error). kbagent flow examples [--component-id keboola.flow|keboola.orchestrator] - (since 0.73.0) Bundled example flow configurations (vendored from + Bundled example flow configurations (vendored from keboola-mcp-server), fully offline. Default keboola.flow (conditional); keboola.orchestrator serves legacy examples with an informational-only warning (kbagent cannot create or edit orchestrator flows). --json emits @@ -1417,12 +1417,12 @@ The supported way to reconcile a drifted tree with production (no manifest surgery). Since 0.72.0 plain pull also re-materializes a tracked config whose local dir is missing (manifest<->disk invariant), so delete-dir-then-pull refetches. Applies to rows too. - Config-level isDisabled round-trips (since 0.72.0) as sparse `is_disabled: true` in + Config-level isDisabled round-trips as sparse `is_disabled: true` in _config.yml -- absent key means enabled; pull writes it, diff surfaces drift, push sends it. --job-limit controls max recent jobs per config (default 5). For large projects, automatically falls back to per-config job fetching to ensure all configs get job history. Auto-detects renamed configs and renames local directories to match (uses git mv in git repos). - --branch (since 0.47.0): per-invocation dev-branch override. Same semantics as sync push/diff. + --branch: per-invocation dev-branch override. Same semantics as sync push/diff. Ignored components (since 0.91.0, #689): keboola.sandboxes + keboola.mcp-server-tool are always excluded, unioned with the manifest's ignoredComponents list (.keboola/manifest.json) -- a per-tree exclusion knob honored by pull/diff/push. A @@ -1441,7 +1441,7 @@ kbagent sync diff --project ALIAS [--all-projects] [--directory DIR] [--branch ID] 3-way diff: local vs pull-time snapshot vs remote. Detects conflicts. - --branch (since 0.47.0): per-invocation dev-branch override. Wins over + --branch: per-invocation dev-branch override. Wins over manifest.branches[0] / 'branch use' active branch / git-branching mapping. Requires exactly one --project. Branch-scoped (since v0.89.0, #649): the local side is read from exactly ONE tree -- @@ -1463,27 +1463,27 @@ kbagent sync push --project ALIAS [--all-projects] [--dry-run] [--force] [--allow-plaintext-on-encrypt-failure] [--branch ID] [--no-name-drift-warnings] Push local changes. Auto-encrypts secrets. Skips conflicts (pull first). Fails if encryption fails (plaintext secrets never pushed). Use escape hatch flag only if you know what you are doing. - Fresh-CREATE behavior (since 0.47.0): if the manifest contains a placeholder entry at + Fresh-CREATE behavior: if the manifest contains a placeholder entry at (component_id, path), the create path updates it in place (no manifest duplication) and propagates any KBC.configuration.* metadata via set_config_metadata. Re-pushes against the now-real config id are naturally idempotent. - Fresh-CREATE variable binding (since 0.47.2): when a keboola.variables config + its + Fresh-CREATE variable binding: when a keboola.variables config + its values row are created alongside a transformation in the same push, the transformation's variables_id / variables_values_id are rebound to the assigned ULIDs (not placeholder dirnames), the row's values are hoisted even when the scaffold row file has no _keboola block, and the row's placeholder parent is remapped before POST. job run then succeeds without a post-push config variables-set step. - --branch (since 0.47.0): per-invocation dev-branch override. Same semantics as sync diff. - When no / subtree exists on disk (since 0.47.2), the local default tree + --branch: per-invocation dev-branch override. Same semantics as sync diff. + When no / subtree exists on disk, the local default tree (main/) is read as the source and promoted to the target branch; API writes still target the branch id. - --no-name-drift-warnings (since 0.47.0): suppress the cosmetic name_drift_warnings + --no-name-drift-warnings: suppress the cosmetic name_drift_warnings array from the result envelope. - Never-fetched guard (since 0.72.0): a manifest entry with an empty pull_hash and no + Never-fetched guard: a manifest entry with an empty pull_hash and no local files (pre-0.72 name-collision phantom) is NEVER planned as a remote DELETE; diff/push exclude it and report it under never_fetched with a warning -- run sync pull to materialize it. Local deletion of a properly-pulled config still deletes on push. - Adopted-by-id writeback (since 0.72.0): pushing an untracked local file whose + Adopted-by-id writeback: pushing an untracked local file whose _keboola.config_id resolves on the branch (adopt-update, #482) now also writes the manifest entry, so follow-up diffs are stable and a later local delete is detected. Branch-scoped (since v0.89.0, #649): push consumes the diff's changeset, so configs @@ -1521,7 +1521,7 @@ --input accepts: inline JSON, @file.json (from file), or - (from stdin). Already-encrypted values (KBC:: prefix) pass through unchanged. -### Semantic Layer (Metastore) (since v0.41.0) +### Semantic Layer (Metastore) Manage Keboola metastore models: datasets, metrics, relationships, constraints, glossary terms. Metastore URL derived from stack URL by replacing `connection.` @@ -1542,12 +1542,12 @@ Without --type prints a per-type count summary. kbagent semantic-layer schema --project P (--type model|dataset|metric|relationship|constraint|glossary[,TYPE...] | --all) - (since 0.73.0) Live JSON Schema per semantic object type, fetched from the + Live JSON Schema per semantic object type, fetched from the deployed metastore (never bundled -- cannot drift). Exactly one of --type/--all. --json emits {{project, schemas: [{{type, schema}}]}}. kbagent semantic-layer search-context --project P [--pattern G ...] [--type model|dataset|metric|relationship|constraint|glossary|all] [--limit N] - (since 0.47.0) Project-wide glob search across semantic-layer entity names. + Project-wide glob search across semantic-layer entity names. Mirrors the upstream keboola-mcp-server search_semantic_context tool so a downstream caller can verify the model is populated without an MCP dependency. Patterns are case-sensitive fnmatch, repeatable (union). Default pattern is "*". @@ -1555,7 +1555,7 @@ Returns {{project, contexts: [{{id, type, name, description, attributes}}], total_count}}. kbagent semantic-layer get-context --project P --context-id ID - (since 0.47.0) Single-entry fetch by id, irrespective of type. Probes model first, + Single-entry fetch by id, irrespective of type. Probes model first, then datasets/metrics/relationships/constraints/glossary in order; raises NOT_FOUND if no type matches (exit 1). @@ -1573,7 +1573,7 @@ Three-way diff: project<->project, project<->file, file<->file. Output groups changes per entity type: added, removed, changed (with diff_keys). - kbagent semantic-layer reference-data list|get|set|delete ... (since 0.55.0) + kbagent semantic-layer reference-data list|get|set|delete ... Dimension-member records (semantic-reference-data): one record per dimension holding the full member list in a members[] array (e.g. a Chart of Accounts). Deliberately OUTSIDE build/export/diff/cascade. @@ -1605,7 +1605,7 @@ --yes skips the confirm prompt. `edit relationship` accepts --new-from / --new-to / --new-on / --new-type (left|inner). `edit glossary` accepts --new-term (destructive cascade; requires --yes in non-TTY) / --new-definition. - Partial-state envelope (since v0.41.10): when metric rename succeeds but + Partial-state envelope: when metric rename succeeds but one or more dependent constraints fail to repoint, the response sets `partial_state: true` at the top level + `recovery_hint: ""` pointing at `semantic-layer validate` + manual `edit constraint @@ -1640,7 +1640,7 @@ dependency order (fixes the long-standing sl-build skill bug where semantic-constraint was silently dropped). On push failure rolls back every successfully-POSTed child in reverse order + deletes the model - if we created it (since v0.41.10); pass --keep-on-failure to preserve + if we created it; pass --keep-on-failure to preserve the partial state for forensic inspection (mirrors data-app create). kbagent semantic-layer token --encrypt --project P --component-id C @@ -1695,7 +1695,7 @@ --from-file for the full {{"type":..., "params":...}} JSON envelope when prompts/args grow large. --extra-arg on an ai_agent task is honored only when the kbagent process (serve, or this `agent` run) - has a truthy KBAGENT_ALLOW_AI_EXTRA_ARGS env (since 0.60.2); + has a truthy KBAGENT_ALLOW_AI_EXTRA_ARGS env; otherwise the args are dropped with a warning. kbagent agent update TASK_ID [--name N] [--description D] [--cron C] @@ -1739,7 +1739,7 @@ the chosen AI CLI with a meta-prompt; the final `done` event's `data.prompt` carries the cleaned body ready to paste into `agent create --prompt ...`. --extra-arg follows the same - KBAGENT_ALLOW_AI_EXTRA_ARGS opt-in as `agent create` (since 0.60.2). + KBAGENT_ALLOW_AI_EXTRA_ARGS opt-in as `agent create`. See agent-tasks-cli-workflow.md skill reference for full walkthroughs. @@ -1789,7 +1789,7 @@ kbagent kai history [--project NAME] [--limit N] List recent Kai chat sessions. Default limit: 10. -### SQL Transformations (since v0.73.0) +### SQL Transformations kbagent transformation create --project NAME --name NAME (--sql 'SELECT ...' | --sql-file PATH) [--created-table NAME ...] [--component-id ID] [--description D] [--branch ID] [--dry-run] Create a SQL transformation. Component id derived from the project @@ -1814,7 +1814,7 @@ mapping you want to keep. --dry-run previews the resulting tree + op summary without writing. -### Documentation Q&A (since v0.73.0) +### Documentation Q&A kbagent docs query "QUESTION" [--project NAME] Answer a natural-language question from the Keboola documentation via the @@ -1823,7 +1823,7 @@ `kai ask` this does NOT see project data -- it is documentation-only, works with any token, and is the right tool for "how do I ..." questions. -### Developer Portal (since v0.49.0) +### Developer Portal The `dev-portal` command group talks to `apps-api.keboola.com` (the Keboola Developer Portal) and lets component developers register and update components @@ -1844,7 +1844,7 @@ --username admin@keboola.com --role-hint admin --password-stdin kbagent dev-portal identity use vendor-keboola - **`role_hint` is load-bearing (since v0.51.1)**: `vendor` (default) routes + **`role_hint` is load-bearing**: `vendor` (default) routes `dev-portal patch` to `PATCH /vendors/{{vendor}}/apps/{{app}}` (restricted schema); `admin` routes it to `PATCH /admin/apps/{{app}}` (permissive schema). The admin endpoint is the **only** way to set the 9 fields @@ -1856,7 +1856,7 @@ "must be one of: easy, medium, hard"; that message is a known apps-api bug -- the field is actually `forbidden()`, not enum-validated). - **`--password-stdin` (since v0.51.1)** works on TTY (hidden line-based + **`--password-stdin`** works on TTY (hidden line-based prompt, Enter to confirm) AND on a pipe (`echo $PASS | … --password-stdin`, reads to EOF). Pre-0.51.1 the flag hung interactively because it always waited for EOF. @@ -1931,7 +1931,7 @@ kbagent doctor Health checks (no --fix since 0.85.0 -- it only installed the MCP server). - Inside a sync working tree, the sync_secrets check (since 0.55.0) warns about + Inside a sync working tree, the sync_secrets check warns about in-sync configs that still hold plaintext #-secrets (#378); skipped outside a sync tree. The mcp_tool_tasks check FAILs on agent tasks still using the removed `--type mcp_tool`; recreate them as `--type cli_command`. @@ -1941,7 +1941,7 @@ the latest available; flags any staleness. Since 0.85.0 it reports kbagent only -- keboola-mcp-server is a separate distribution kbagent no longer tracks, so there is no `dependencies` key in --json. - --beta (since 0.42.0) reports the latest pre-release (beta / rc) instead + --beta reports the latest pre-release (beta / rc) instead of the latest stable. Same env override: KBAGENT_INCLUDE_PRERELEASE=1. kbagent update [--beta] @@ -1958,7 +1958,7 @@ one. `version --json` then carries kbagent.install_channel and kbagent.upgrade_hint; upgrade_command is empty for a hand-unpacked archive. - --beta (since 0.42.0) opts into pre-release versions (PEP 440 betas/rc, + --beta opts into pre-release versions (PEP 440 betas/rc, e.g. 0.43.0b1). Without --beta the auto-update path uses GitHub's /releases/latest endpoint, which excludes prereleases server-side -- stable users never silently land on a beta. Set @@ -2036,7 +2036,7 @@ KBAGENT_CONFIG_DIR Override config directory KBAGENT_PROJECT Override the pinned default project for this shell/session (beats pin, loses to --project) KBAGENT_PROJECT_FROM_ENV Set to "1" (or true/yes/on) to synthesize an in-memory project under the - reserved alias __env__ from KBC_TOKEN + KBC_STORAGE_API_URL (since 0.50.0). + reserved alias __env__ from KBC_TOKEN + KBC_STORAGE_API_URL. Headless / token-only mode: no `project add`, no config.json on disk. Use `--project __env__` (or rely on it as the sole/default project). The token lives in memory only -- it is NEVER persisted, even if a write op runs. diff --git a/tests/test_check_release_scope.py b/tests/test_check_release_scope.py new file mode 100644 index 00000000..071bd8b0 --- /dev/null +++ b/tests/test_check_release_scope.py @@ -0,0 +1,125 @@ +"""Unit tests for the release-scope audit (``scripts/check_release_scope.py``). + +The check exists because of a real miss in v0.91.0. PR #625 merged to ``main`` +after the release PR had branched but before it merged, so it ended up INSIDE +the tag's tree and OUTSIDE the changelog entry the release notes are rendered +from. Nothing caught it: ``make changelog-check`` proves that every released +version has an entry, never that an entry covers every commit under the tag. +Tagging would have shipped a new plugin command with no release note. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +_SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "check_release_scope.py" +_spec = importlib.util.spec_from_file_location("check_release_scope", _SCRIPT_PATH) +assert _spec is not None and _spec.loader is not None +check_release_scope = importlib.util.module_from_spec(_spec) +sys.modules["check_release_scope"] = check_release_scope +_spec.loader.exec_module(check_release_scope) + +merged_prs = check_release_scope.merged_pr_numbers +referenced_prs = check_release_scope.referenced_pr_numbers +missing = check_release_scope.missing_references + + +class TestMergedPrNumbers: + """PR numbers come from the squash-merge subject GitHub writes.""" + + def test_extracts_trailing_pr_reference(self) -> None: + log = "b5d4be39 docs(plugin): polish workspace-load guidance (#698)\n" + assert merged_prs(log) == ["698"] + + def test_ignores_a_commit_without_a_pr_reference(self) -> None: + """Local commits on the release branch carry no ``(#N)`` and are not PRs.""" + log = "4523f6d3 chore(release): 0.91.0\n1e584ac0 fix: something (#42)\n" + assert merged_prs(log) == ["42"] + + def test_only_a_trailing_reference_counts(self) -> None: + """A ``(#N)`` mid-subject is a cross-reference, not this commit's PR.""" + log = "abc1234 fix(sync): follow-up to (#686) behaviour change\n" + assert merged_prs(log) == [] + + def test_handles_a_merge_commit_subject(self) -> None: + log = "fc7ae7db Merge pull request #627 from keboola/feat/publish\n" + assert merged_prs(log) == ["627"] + + def test_preserves_order_and_deduplicates(self) -> None: + log = "a1 x (#5)\nb2 y (#7)\nc3 z (#5)\n" + assert merged_prs(log) == ["5", "7"] + + +class TestReferencedPrNumbers: + """Any ``#N`` anywhere in the release's changelog bullets counts as covered.""" + + def test_reads_the_prefix_decoration(self) -> None: + notes = ["New (#692): workspace load now clones."] + assert referenced_prs(notes) == {"692"} + + def test_reads_a_multi_pr_decoration(self) -> None: + notes = ["Fix (#686, #694, #696): push stamps API-derived baselines."] + assert referenced_prs(notes) == {"686", "694", "696"} + + def test_reads_a_reference_from_mid_sentence(self) -> None: + """Issue numbers cited in prose count too -- both are GitHub numbers.""" + notes = ["New: the write path fixes what the #600 audit finds."] + assert referenced_prs(notes) == {"600"} + + +class TestMissingReferences: + """The actual gate: every merged PR must appear in the new entry.""" + + def test_fully_covered_release_reports_nothing(self) -> None: + log = "a1 feat: x (#692)\nb2 fix: y (#694)\n" + notes = ["New (#692): x.", "Fix (#694): y."] + assert missing(log, notes, ignore=frozenset()) == [] + + def test_reports_a_merged_pr_absent_from_the_notes(self) -> None: + """This is the v0.91.0 / #625 miss, reproduced.""" + log = "a1 feat: setup (#625)\nb2 fix: y (#694)\n" + notes = ["Fix (#694): y."] + assert missing(log, notes, ignore=frozenset()) == ["625"] + + def test_ignored_pr_is_skipped(self) -> None: + """The release PR itself is in the log once merged, never in its own notes.""" + log = "a1 chore(release): 0.91.0 (#699)\nb2 fix: y (#694)\n" + notes = ["Fix (#694): y."] + assert missing(log, notes, ignore=frozenset({"699"})) == [] + + def test_reports_every_miss_not_just_the_first(self) -> None: + log = "a1 x (#1)\nb2 y (#2)\nc3 z (#3)\n" + assert missing(log, notes := ["Fix (#2): y."], ignore=frozenset()) == ["1", "3"] + assert notes # guard against the walrus being optimised away by a rewrite + + +class TestArmingAndFailOpen: + """CI must arm this only on a release PR, and never block an ordinary one. + + The check needs tags and real history (``git log v..HEAD``), which a + default shallow CI checkout does not have. Deepening every PR run to buy a + check that only matters on release PRs is the wrong trade, so the script + decides whether it applies and degrades to a warning when git cannot answer. + """ + + def test_not_armed_when_the_version_is_unchanged(self) -> None: + assert check_release_scope.should_arm(base="0.91.0", head="0.91.0") is False + + def test_not_armed_when_the_branch_trails_a_released_main(self) -> None: + """A stale feature branch behind main must not look like a release PR.""" + assert check_release_scope.should_arm(base="0.91.0", head="0.90.1") is False + + def test_armed_when_the_pr_raises_the_version(self) -> None: + assert check_release_scope.should_arm(base="0.90.1", head="0.91.0") is True + + def test_armed_for_a_pre_release_bump(self) -> None: + assert check_release_scope.should_arm(base="0.90.1", head="0.91.0b1") is True + + def test_unreadable_base_does_not_arm(self) -> None: + """Fail open: an unreadable base must never block an ordinary PR.""" + assert check_release_scope.should_arm(base="", head="0.91.0") is False + + def test_malformed_version_does_not_arm(self) -> None: + assert check_release_scope.should_arm(base="not-a-version", head="0.91.0") is False diff --git a/tests/test_check_version_gates.py b/tests/test_check_version_gates.py index 5e7c3c79..412601e4 100644 --- a/tests/test_check_version_gates.py +++ b/tests/test_check_version_gates.py @@ -22,6 +22,8 @@ collect = check_version_gates.collect_gates residue = check_version_gates.find_vnext_residue +headings = check_version_gates.find_heading_placeholders +resolve_vnext = check_version_gates.resolve_vnext def _write(tmp_path: Path, name: str, body: str) -> Path: @@ -362,3 +364,233 @@ def test_double_backticks_do_not_hide_numeric_gates(self, tmp_path: Path) -> Non """The GATE_RE asymmetry survives: code spans are never stripped for versions.""" f = _write(tmp_path, "mod.py", '"""Device-enrollment primitives (``0.66.0+``)."""\n') assert list(collect([f])) == ["0.66.0"] + + +class TestHeadingPlaceholders: + """A ``vNEXT`` inside a markdown heading is fatal on EVERY PR, not just a release. + + Resolving the placeholder rewrites the heading text, which rewrites the + generated anchor slug, which breaks every inbound ``#...`` link. The rule + predates this check as prose in CONTRIBUTING.md plus a hand-run + ``grep -rn '^##.*vNEXT' plugins/`` at release time -- and that grep lost a + merge race in 0.91.0: PR #697 ran it two minutes before #694 and #696 + landed their own headings, so all three shipped and had to be cleaned up + after the fact. Checking at authoring time is what makes the race + impossible. + """ + + def test_atx_heading_with_placeholder_is_flagged(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "intro\n\n## Ignored components (since vNEXT, #689)\n") + found = headings([f]) + assert len(found) == 1 + assert found[0].line == 3 + + def test_emphasised_tag_in_heading_is_flagged(self, tmp_path: Path) -> None: + """``### Foo *(since vNEXT)*`` is the exact shape #697 had to clean up.""" + f = _write(tmp_path, "g.md", "### What's-new popup *(since vNEXT)*\n") + assert len(headings([f])) == 1 + + def test_placeholder_on_a_body_line_is_not_a_heading(self, tmp_path: Path) -> None: + """The prescribed fix -- tag on the first body line -- must stay legal.""" + f = _write(tmp_path, "g.md", "## Ignored components\n\n*(since vNEXT, #689)*\n") + assert headings([f]) == [] + + def test_python_comment_is_not_a_heading(self, tmp_path: Path) -> None: + """``src/**/*.py`` is scanned for gates, but ``#`` there is a comment. + + A Python comment has no anchor slug, so flagging it would be a pure + false positive -- and CLAUDE.md's command block is full of them. + """ + f = _write(tmp_path, "mod.py", "# workspace load (since vNEXT): auto-decides\n") + assert headings([f]) == [] + + def test_heading_quoting_the_token_is_prose(self, tmp_path: Path) -> None: + """Same inline-code rule as the residue scan: backticks mean quotation.""" + f = _write(tmp_path, "g.md", "## How the `vNEXT` placeholder works\n") + assert headings([f]) == [] + + def test_hash_without_a_space_is_not_a_heading(self, tmp_path: Path) -> None: + """``#tag`` is not ATX -- CommonMark requires a space after the hashes.""" + f = _write(tmp_path, "g.md", "#vNEXT (since vNEXT)\n") + assert headings([f]) == [] + + def test_live_repository_has_no_placeholder_headings(self) -> None: + """The real tree must stay clean -- this is the check's whole point.""" + assert headings(check_version_gates.resolve_paths()) == [] + + +class TestHeadingCheckIsFatalOutsideRelease: + """The heading rule must fail a FEATURE PR -- that is what closes the race. + + ``find_vnext_residue`` is deliberately advisory outside ``--release``, + because a feature PR is supposed to carry placeholders. A placeholder in a + *heading* is different: it is never correct, at any point in the cycle, so + it has to fail the PR that writes it. + """ + + def _run(self, monkeypatch, tmp_path: Path, body: str, argv: list[str]) -> int: + f = _write(tmp_path, "g.md", body) + monkeypatch.setattr(check_version_gates, "resolve_paths", lambda: [f]) + monkeypatch.setattr(sys, "argv", ["check_version_gates.py", *argv]) + return check_version_gates.main() + + def test_heading_placeholder_fails_a_plain_run(self, monkeypatch, tmp_path: Path) -> None: + rc = self._run(monkeypatch, tmp_path, "## Ignored components (since vNEXT)\n", []) + assert rc == 1 + + def test_body_line_placeholder_still_passes_a_plain_run( + self, monkeypatch, tmp_path: Path + ) -> None: + """The advisory-residue behaviour a feature PR relies on is untouched.""" + rc = self._run(monkeypatch, tmp_path, "## Ignored components\n\n*(since vNEXT)*\n", []) + assert rc == 0 + + +class TestResolveVnext: + """``make vnext-resolve VERSION=X`` rewrites live gates and nothing else. + + Before this existed, the release PR resolved 54 placeholders by hand off + the checker's own output. That is a machine's job: the scanner already + separates a live gate from prose with perfect precision, so a human doing + the edit can only introduce error -- and a blanket ``sed`` provably does, + because the process docs legitimately quote the token. + """ + + def test_live_gate_is_rewritten(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "- `--flag` (since vNEXT) does a thing\n") + changed = resolve_vnext([f], "0.91.0") + assert len(changed) == 1 + assert f.read_text(encoding="utf-8") == "- `--flag` (since 0.91.0) does a thing\n" + + def test_prose_inside_backticks_is_left_alone(self, tmp_path: Path) -> None: + """CLAUDE.md documents the placeholder; a blanket sed corrupts that.""" + body = "tag it with the literal placeholder **`vNEXT`** -- `(since vNEXT)`.\n" + f = _write(tmp_path, "g.md", body) + assert resolve_vnext([f], "0.91.0") == [] + assert f.read_text(encoding="utf-8") == body + + def test_mixed_line_rewrites_only_the_live_token(self, tmp_path: Path) -> None: + """The case a line-level rewrite gets wrong -- one quoted, one live.""" + f = _write(tmp_path, "g.md", "`vNEXT` is the placeholder; (since vNEXT) is live\n") + changed = resolve_vnext([f], "0.91.0") + assert len(changed) == 1 + expected = "`vNEXT` is the placeholder; (since 0.91.0) is live\n" + assert f.read_text(encoding="utf-8") == expected + + def test_vnext_plus_form_is_rewritten(self, tmp_path: Path) -> None: + """``vNEXT+`` is the other documented placeholder shape.""" + f = _write(tmp_path, "g.md", "- **vNEXT+**: resolves to the first project\n") + resolve_vnext([f], "0.91.0") + assert "0.91.0+" in f.read_text(encoding="utf-8") + + def test_python_docstring_gate_is_rewritten(self, tmp_path: Path) -> None: + """``src/**/*.py`` carries agent-facing gates too, so it must resolve.""" + f = _write(tmp_path, "mod.py", '"""Does a thing (since vNEXT)."""\n') + assert len(resolve_vnext([f], "0.91.0")) == 1 + assert "(since 0.91.0)" in f.read_text(encoding="utf-8") + + def test_is_idempotent(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "- `--flag` (since vNEXT)\n") + resolve_vnext([f], "0.91.0") + assert resolve_vnext([f], "0.91.0") == [] + + def test_reports_every_rewritten_location(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "## H\n\n(since vNEXT) one\nplain\n(since vNEXT) two\n") + changed = resolve_vnext([f], "0.91.0") + assert [c.line for c in changed] == [3, 5] + + def test_file_without_a_placeholder_is_not_touched(self, tmp_path: Path) -> None: + """No rewrite means no mtime churn on 900+ scanned files.""" + f = _write(tmp_path, "g.md", "nothing to see\n") + before = f.stat().st_mtime_ns + assert resolve_vnext([f], "0.91.0") == [] + assert f.stat().st_mtime_ns == before + + def test_rejects_a_malformed_version(self, tmp_path: Path) -> None: + """A garbled VERSION= must not be written into every gate in the tree. + + Note ``packaging`` is lenient about shapes that merely LOOK wrong -- + ``v0.91`` and ``0.91`` both parse. Format validation therefore cannot + catch a typo'd-but-parseable version; that is what the pyproject + cross-check in ``main()`` is for (see + :class:`TestResolveModeGuardsTheVersion`). + """ + f = _write(tmp_path, "g.md", "(since vNEXT)\n") + try: + resolve_vnext([f], "0.91.0.banana") + except ValueError: + pass + else: # pragma: no cover - the assert below reports the miss + raise AssertionError("expected ValueError for a malformed version") + assert "vNEXT" in f.read_text(encoding="utf-8") + + +class TestResolveModeGuardsTheVersion: + """``--resolve X`` must refuse any X that is not what pyproject ships. + + Format validation cannot catch this: ``v0.91`` and ``0.91`` are both valid + PEP 440. But resolving gates to a version the release is not actually + shipping recreates the exact bug the gate exists to prevent -- an agent + refusing a command the user has -- across the whole tree at once, and the + ``version-gate-check`` that would notice runs against CHANGELOG keys, not + against pyproject. + """ + + def _run(self, monkeypatch, tmp_path: Path, version: str) -> tuple[int, Path]: + f = _write(tmp_path, "g.md", "- `--flag` (since vNEXT)\n") + monkeypatch.setattr(check_version_gates, "resolve_paths", lambda: [f]) + monkeypatch.setattr(sys, "argv", ["check_version_gates.py", "--resolve", version]) + return check_version_gates.main(), f + + def test_version_matching_pyproject_is_applied(self, monkeypatch, tmp_path: Path) -> None: + shipped = check_version_gates._pyproject_version() + rc, f = self._run(monkeypatch, tmp_path, shipped) + assert rc == 0 + assert f"(since {shipped})" in f.read_text(encoding="utf-8") + + def test_version_disagreeing_with_pyproject_is_refused( + self, monkeypatch, tmp_path: Path + ) -> None: + rc, f = self._run(monkeypatch, tmp_path, "9.9.9") + assert rc == 1 + assert "vNEXT" in f.read_text(encoding="utf-8"), "nothing may be rewritten on refusal" + + def test_malformed_version_is_refused_without_touching_files( + self, monkeypatch, tmp_path: Path + ) -> None: + rc, f = self._run(monkeypatch, tmp_path, "0.91.0.banana") + assert rc == 1 + assert "vNEXT" in f.read_text(encoding="utf-8") + + +class TestGatesBelowFloor: + """``--list-below`` is the worklist generator for retiring stale gates. + + A gate only earns its place while some live install predates it. kbagent + self-updates on startup, so the population a very old gate protects rounds + to zero -- while the gate itself keeps making the agent refuse a command + the user actually has, which the gate's own docs call strictly worse than + no gate. Periodically raising a floor and de-tagging below it needs a + worklist, and hand-grepping one is how the heading rule got missed. + """ + + def test_returns_only_gates_below_the_floor(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "a (since 0.23.0)\nb (since 0.85.0)\n") + found = check_version_gates.gates_below(collect([f]), "0.80.0") + assert list(found) == ["0.23.0"] + + def test_floor_itself_is_not_below_the_floor(self, tmp_path: Path) -> None: + """The floor is the oldest version we still gate for -- inclusive.""" + f = _write(tmp_path, "g.md", "a (since 0.80.0)\n") + assert check_version_gates.gates_below(collect([f]), "0.80.0") == {} + + def test_carries_the_locations_through(self, tmp_path: Path) -> None: + f = _write(tmp_path, "g.md", "x\na (since 0.23.0)\n") + found = check_version_gates.gates_below(collect([f]), "0.80.0") + assert found["0.23.0"][0][1] == 2 + + def test_orders_versions_oldest_first(self, tmp_path: Path) -> None: + """Oldest first: the safest de-tagging starts at the far end.""" + f = _write(tmp_path, "g.md", "a (since 0.30.0)\nb (since 0.9.0)\nc (since 0.23.0)\n") + found = check_version_gates.gates_below(collect([f]), "0.80.0") + assert list(found) == ["0.9.0", "0.23.0", "0.30.0"]