From cdbb16ff0cc10868302a117b80d9e908205e353c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Maxmili=C3=A1n=20Ottomansk=C3=BD?= <114772250+ottomansky@users.noreply.github.com> Date: Thu, 23 Apr 2026 13:59:20 +0200 Subject: [PATCH 1/7] feat(0.22.0): project pin + firewall flags (FIIA P0-4) (#203) * feat(0.22.0): project pin + firewall flags (FIIA P0-4) Adds an explicit project-pin UX + session-only firewall sugar that together close the FIIA P0-4 gap. New surface: - `kbagent project use ` -- pins default_project in config.json. The field already existed (auto-set on first add, auto-rotated on remove) but had no explicit CLI verb. Now it does. - `kbagent project current` -- prints effective alias + source (env / pin / none). Reports both the env override AND the persisted pin so misconfigurations surface loudly instead of silently. - Top-level `--deny-writes` / `--deny-destructive` flags -- synthesize a session-only PermissionPolicy and merge with any persisted policy. Never written to config.json. `--deny-writes` covers the wide net (write+destructive+admin); `--deny-destructive` narrowly covers data destruction so callers can keep build-up capabilities while blocking tear-down. - `KBAGENT_PROJECT` env var overrides the persisted pin for a single shell. Resolution precedence for single-project ops: `--project` > `KBAGENT_PROJECT` > pin > sole-project > fail-hard with a repair-friendly CONFIG_ERROR. - `ProjectService.resolve_pinned_alias()` and the thin `commands._helpers.resolve_project_alias()` helper -- public contract for future single-project write commands to adopt without retrofitting every read-path fan-out. Wire notes: - `project.use` registered as 'write', `project.current` as 'read' in the permission engine's OPERATION_REGISTRY so persisted policies gate them correctly. - `_apply_firewall_flags()` builds a fresh PermissionPolicy and never mutates the persisted deny list. - Stale pin (default_project pointing at a deleted alias) now raises a repair-friendly CONFIG_ERROR instead of silently fanning out. * fix(repl): forward --deny-writes / --deny-destructive into REPL subcommands Review caught a silent-elevation bug: starting the CLI with `kbagent --deny-writes` and then entering the REPL (or typing `repl`) let inner commands re-parse WITHOUT the firewall flags, restoring full write/destructive access. The same PR5 firewall flag that a user opted into was silently dropped the moment they landed at the prompt. Root cause: `_run_repl()` only forwarded four flags (--json, --verbose, --no-color, --config-dir) when rebuilding argv for each inner command; the two new session-firewall flags were never threaded through. Fix: - Extend `_run_repl(deny_writes, deny_destructive)` and re-append both flags on every inner invocation unless the user already typed them. - Thread the flags from `cli.py::main()` into both REPL launch paths (implicit no-subcommand launch + explicit `kbagent repl`) via `ctx.obj["deny_writes"]` / `ctx.obj["deny_destructive"]`. - Banner advertises the active firewall so users can see what's on. Tests (`TestReplFirewallPropagation`): - `test_deny_writes_appended_to_inner_argv` -- guard against the regression by asserting every rebuilt argv carries --deny-writes. - `test_deny_destructive_appended_to_inner_argv` -- same for destructive. - `test_no_duplicate_when_user_retypes_flag` -- idempotence guard. Also closes other review-loop gaps (non-REPL): - `test_project_help` now asserts `use` and `current` subcommands. - `test_root_help_lists_firewall_flags` guards the global flags. - `test_list_includes_project_use_and_current` guards the registry. - `test_hint_on_non_api_project_use/current` cover --hint graceful exit. - `test_deny_writes_composes_with_persisted_deny_mode` validates the runtime composition under default-deny mode. * fix(test): use callback signature for firewall-flag help assertion The original `test_root_help_lists_firewall_flags` rendered `kbagent --help` via CliRunner and string-matched `--deny-writes` / `--deny-destructive`. CI's narrow default terminal width made Rich's typer help panel truncate the long option names into `...`, failing the assertion even though the flags are registered correctly. Switch to inspecting `cli.main()` function signature. The signature is the source of truth for registered options and is independent of terminal width. Still catches the regression (accidental flag removal). * fix(permissions): show/list now reflect session firewall flags UX gap surfaced in the review loop: `permissions show` / `permissions list` previously inspected ONLY the persisted policy. A user running `kbagent --deny-writes permissions show` (or inside a REPL launched with those flags) saw misleading output: "No permission policy configured" even though their writes were in fact blocked for the invocation. Fix: - `permissions list` now builds the engine from the MERGED policy (`_apply_firewall_flags(persisted, deny_writes=..., deny_destructive=...)`). The allowed/denied column matches what the command will actually do. Human-mode footer advertises active session flags. - `permissions show` reports both layers in the JSON envelope: { "active": bool, # true if persisted OR session present "persisted": {mode, allow, deny} | null, "session_flags": [...], # legacy top-level keys preserved when persisted != null "mode": ..., "allow": ..., "deny": ... } Human-mode prints a "Session firewall:" line whenever flags are active. Legacy JSON consumers that read `.mode` / `.allow` / `.deny` keep working. - `permissions show` with no persisted policy + no session flags still prints the legacy "No permission policy configured" line (empty-state contract preserved for existing tests). Tests added: - `test_list_reflects_session_deny_writes` -- project.add flips to denied when --deny-writes is active. - `test_show_reports_session_flags_without_persisted_policy` -- active=true even without a persisted policy. - `test_show_reports_session_flags_alongside_persisted` -- both layers reported + legacy keys still present. - `test_show_human_mode_no_policy_no_session` -- empty-state regression. * docs(cli): clarify --deny-destructive is NARROW; admin ops need --deny-writes Iter-4 review flagged a doc/flag-help ambiguity: the `--deny-destructive` help string said "destructive (delete/kill/reset) operations" which a user could reasonably read as covering 'project remove' and 'org setup'. The actual behavior (correctly) leaves admin-classified ops allowed, because 'admin' is a separate risk category from 'destructive' in the OPERATION registry. Only --deny-writes catches the wide net. Fixes: - `cli.py`: spell out WIDE vs NARROW in each flag's help string. Explicitly call out 'project remove' / 'org setup' as NOT blocked by --deny-destructive. - `context.py`: mirror the same WIDE vs NARROW contract in the AI-agent reference text (global flags section). Tests added (`TestFirewallFlags`): - `test_deny_destructive_allows_admin_ops` -- contract lock: running `kbagent --deny-destructive project remove prod` succeeds (exit != 6). - `test_deny_writes_blocks_admin_ops` -- complement: --deny-writes IS the wide net and DOES block admin ops (exit 6). Test hygiene (`TestPermissionsShow`): - `test_show_reports_session_flags_without_persisted_policy` now also asserts the legacy `mode`/`allow`/`deny` top-level keys are ABSENT when there is no persisted policy. Documents the new shape contract for downstream JSON consumers. * docs(references): add Session firewall flags section to permissions-workflow.md Document --deny-writes (WIDE) vs --deny-destructive (NARROW) semantics, REPL forwarding behavior, and the additive-merge relationship with any persisted policy (session flags never relax what is persisted, never land in config.json). Cross-references the project-pin UX. * docs(gotchas): document KBAGENT_PROJECT env var semantics and precedence Record the three non-obvious rules: empty string = unset (Unix convention), pointing to an unregistered alias fails hard with CONFIG_ERROR (exit 5), and the full precedence chain --project > KBAGENT_PROJECT > persisted pin > sole-project fallback > fail. * chore(hints): clarify project use/current are local-only (no --hint) Both commands operate purely on the local ConfigStore (no HTTP or service call to render), so registering HintRegistry entries would be meaningless. Add a single-line comment at each command so future contributors don't rediscover this by accident. * refactor(cli): promote _apply_firewall_flags to public apply_firewall_flags The function is imported from commands/permissions.py, which made the underscore prefix misleading (it looked private but wasn't). Drop the leading underscore at the definition, both call sites (cli.py, commands/permissions.py), and the test suite so the API boundary is honest. --- CLAUDE.md | 4 +- plugins/kbagent/.claude-plugin/plugin.json | 2 +- plugins/kbagent/skills/kbagent/SKILL.md | 2 + .../kbagent/references/commands-reference.md | 7 + .../skills/kbagent/references/gotchas.md | 26 + .../references/permissions-workflow.md | 71 +++ pyproject.toml | 2 +- src/keboola_agent_cli/changelog.py | 8 + src/keboola_agent_cli/cli.py | 85 ++- src/keboola_agent_cli/commands/_helpers.py | 37 ++ src/keboola_agent_cli/commands/context.py | 22 +- src/keboola_agent_cli/commands/permissions.py | 105 +++- src/keboola_agent_cli/commands/project.py | 83 +++ src/keboola_agent_cli/commands/repl.py | 36 +- src/keboola_agent_cli/constants.py | 4 + src/keboola_agent_cli/permissions.py | 2 + .../services/project_service.py | 140 +++++ tests/test_cli.py | 539 ++++++++++++++++++ tests/test_e2e.py | 348 +++++++++++ tests/test_helpers.py | 146 +++++ tests/test_permissions_cli.py | 92 +++ tests/test_repl.py | 129 +++++ tests/test_services.py | 175 ++++++ 23 files changed, 2035 insertions(+), 30 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5db8defb..e355b962 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -233,7 +233,7 @@ Note: `SKILL.md` instructs Claude to run `kbagent context` as its first step, wh ## All CLI Commands ``` -# Global options: --json, --verbose, --no-color, --config-dir, --hint client|service +# Global options: --json, --verbose, --no-color, --config-dir, --hint client|service, --deny-writes, --deny-destructive kbagent project add --project NAME --url URL --token TOKEN kbagent project list @@ -244,6 +244,8 @@ kbagent project refresh --project ALIAS [--dry-run] [--force] [--yes] [--token-d kbagent project refresh --all [--dry-run] [--force] [--yes] [--token-description DESC] [--token-expires-in N] kbagent project description-get --project NAME kbagent project description-set --project NAME [--text STR | --file PATH | --stdin] +kbagent project use ALIAS +kbagent project current kbagent config list [--project NAME] [--component-type TYPE] [--component-id ID] [--branch ID] kbagent config detail --project NAME --component-id ID --config-id ID [--branch ID] diff --git a/plugins/kbagent/.claude-plugin/plugin.json b/plugins/kbagent/.claude-plugin/plugin.json index ef33fbf1..d4cd469c 100644 --- a/plugins/kbagent/.claude-plugin/plugin.json +++ b/plugins/kbagent/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "kbagent", - "version": "0.21.2", + "version": "0.22.0", "description": "AI-friendly interface to Keboola Connection projects — explore configs, jobs, lineage, call MCP tools, manage dev branches, and debug SQL in workspaces", "author": { "name": "Keboola", diff --git a/plugins/kbagent/skills/kbagent/SKILL.md b/plugins/kbagent/skills/kbagent/SKILL.md index 06540e47..f2569306 100644 --- a/plugins/kbagent/skills/kbagent/SKILL.md +++ b/plugins/kbagent/skills/kbagent/SKILL.md @@ -81,6 +81,8 @@ When working inside a git repository or project directory, run `kbagent init` (o | Edit an existing Keboola project connection | `kbagent project edit --project ALIAS` | | Test connectivity to connected Keboola projects | `kbagent project status` | | Refresh expired or invalid Storage API tokens | `kbagent project refresh` | +| Pin as the default project for subsequent commands | `kbagent project use ` | +| Show the effective default project | `kbagent project current` | | Get the Keboola dashboard project description | `kbagent project description-get --project PROJECT` | | Set the Keboola dashboard project description (markdown) | `kbagent project description-set --project PROJECT` | | Set up projects and register them in the kbagent config | `kbagent org setup --url URL` | diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 7217f944..3f3f2aae 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -18,6 +18,13 @@ All commands support `--json` for structured output. Multi-project flags (`--pro - `project status [--project NAME]` -- test connectivity and response time - `project description-get --project NAME` -- read the dashboard project description (KBC.projectDescription on the default branch). Returns `{"description": ""}` if not set, not an error - `project description-set --project NAME [--text STR | --file PATH | --stdin]` -- set the dashboard project description (markdown). Pass exactly one of `--text`, `--file`, or `--stdin`. Writes to `KBC.projectDescription` on the default branch -- always the main branch, regardless of any active dev branch +- `project use ALIAS` -- pin `ALIAS` as the persistent default project. Stored as `default_project` in config.json. Overridden at runtime by `KBAGENT_PROJECT=ALIAS` (env, beats pin) and by `--project ALIAS` (CLI flag, beats both) +- `project current` -- print the effective default project and its source (`env` / `pin` / `none`). Reports both the env override AND the persisted pin so misconfigurations are visible. Returns `{"alias": null, "source": "none"}` when neither is set + +## Permission flags (top-level, session-only) +- `--deny-writes` -- block all write/destructive/admin operations for this single invocation. Merges with any persisted permission policy; never written to config.json. Exit code 6 (PERMISSION_DENIED) on blocked operations +- `--deny-destructive` -- block only destructive operations (delete-table, delete-bucket, terminate-job, etc.) for this invocation. Pure-write ops like create-table stay allowed. Use this when you want to keep build-up capabilities but lock out tear-downs +- Both flags compose: `kbagent --deny-writes --deny-destructive ...` is the safest read-only run ## Organization - `org setup --org-id ID --url URL [--dry-run] [--yes]` -- bulk-onboard all projects from an org (org admin, needs `KBC_MANAGE_API_TOKEN`) diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index e8ab84f4..08db2036 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -169,6 +169,32 @@ kbagent looks for configuration in this order: Use `kbagent init` to create a local `.kbagent/` workspace for per-directory isolation. +## `KBAGENT_PROJECT` environment variable + +Lets callers override the default project for one shell/session without editing +`config.json`. A few non-obvious rules: + +- **Empty string counts as unset.** `KBAGENT_PROJECT=""` (or a value consisting + only of whitespace) is treated exactly like the variable not being set at + all. This follows the standard Unix shell convention and prevents a stray + `export KBAGENT_PROJECT=` from silently breaking every subsequent command. +- **Points to an unregistered alias -> hard fail.** If the env var names an + alias that is NOT in your configured projects, write-ops (the ones that + consult the pin) fail with `CONFIG_ERROR` and exit code 5. Repair either by + running `kbagent project use ` and unsetting the env var, or by + `unset KBAGENT_PROJECT`. The CLI will not fall back silently to the persisted + pin -- that would mask a misconfiguration. +- **Precedence for resolving the target project** (highest wins): + 1. `--project ` CLI flag (explicit per-command) + 2. `KBAGENT_PROJECT` env var + 3. Persisted pin (`default_project` in `config.json`, set via + `kbagent project use `) + 4. Sole-project fallback (if exactly one project is configured) + 5. Hard fail with `CONFIG_ERROR` (no ambiguous defaulting) +- `kbagent project current` reports which of (2) or (3) is active and flags + when the env var points to an unregistered alias, so you can diagnose + precedence issues without reading the source. + ## config update vs MCP update_config For updating configuration content, prefer `kbagent config update` over MCP's `update_config` tool: diff --git a/plugins/kbagent/skills/kbagent/references/permissions-workflow.md b/plugins/kbagent/skills/kbagent/references/permissions-workflow.md index fd430e08..70763fab 100644 --- a/plugins/kbagent/skills/kbagent/references/permissions-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/permissions-workflow.md @@ -102,6 +102,77 @@ kbagent --json permissions list | `sync.*` | All sync subcommands (glob) | | `tool:create_*` | MCP tools matching glob pattern | +## Session firewall flags + +Two top-level flags let an operator harden a single invocation WITHOUT editing +the persisted policy in `config.json`. They are session-only, additive, and +evaluated alongside any persisted policy. + +```bash +# Wide net: blocks writes + destructive + admin +kbagent --deny-writes + +# Narrow net: blocks only data-destructive ops (pure writes still allowed) +kbagent --deny-destructive + +# Both (equivalent to --deny-writes here, since wide subsumes narrow) +kbagent --deny-writes --deny-destructive +``` + +### `--deny-writes` (WIDE) + +Appends `cli:write` and `tool:write` to the deny list. The `cli:write` pattern +intentionally spans the **write + destructive + admin** categories, so this one +flag blocks everything that mutates state -- config create/update/delete, branch +delete, project add/remove/edit, org setup, storage writes, sync push, and every +MCP write tool (`create_*`, `update_*`, `delete_*`, `add_*`, `set_*`, `remove_*`). + +Use this when you want a strict read-only session without touching the persisted +policy. Blocked operations exit with code 6 (`PERMISSION_DENIED`). + +### `--deny-destructive` (NARROW) + +Appends `cli:destructive` and `tool:destructive` to the deny list. This pattern +matches **only** operations categorized as destructive (data destruction) -- +`branch.delete`, `workspace.delete`, `config.delete`, `storage.delete-table`, +`storage.delete-bucket`, `storage.delete-column`, `job.terminate`, and MCP +`delete_*` / `remove_*` tools. + +Pure-write operations (e.g. `storage create-bucket`, `config update`) and admin +operations (e.g. `project remove`, `org setup`) are **still allowed**. Use this +when an agent needs to create/modify resources but must not be able to destroy +existing data. + +### REPL forwarding + +When invoked as `kbagent --deny-writes repl` (or `--deny-destructive`), the +flags propagate into every subcommand run inside the REPL session, so each +inner invocation picks them up automatically. A duplicate-append guard prevents +the flag from being injected twice if a user also types it explicitly on a REPL +line. + +### Relationship to persisted policy + +The session flags merge **additively** with the persisted policy for the +duration of the invocation: + +- The persisted `mode`, `allow` list, and existing `deny` entries are preserved + unchanged. Only the flag-implied deny patterns are appended (deduped). +- Session flags **can only add more deny entries** -- they NEVER relax the + persisted policy. Running `kbagent --deny-writes` against a policy that + already denies everything does not re-open anything. +- The merged policy lives in memory for this process only. It is never written + to `config.json`, so subsequent invocations without the flag revert to the + persisted policy alone. +- `kbagent permissions list` and `kbagent permissions show` render the + **effective** policy (persisted merged with session flags) so you can verify + what is actually active right now. The `session_flags` field in the JSON + output of `permissions show` surfaces which flags are in play. + +For the complementary project-pin UX (`kbagent project use `, which +persists a default project so you can drop `--project` from subsequent +commands), see the project management section of the skill. + ## Defense in depth (`--read-only`) `kbagent init --read-only` applies three layers of protection: diff --git a/pyproject.toml b/pyproject.toml index e37cb260..d622abf9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "keboola-agent-cli" -version = "0.21.2" +version = "0.22.0" description = "AI-friendly CLI for managing Keboola projects" readme = "README.md" requires-python = ">=3.12" diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 1514b530..8c3cdb75 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -8,6 +8,14 @@ # Ordered newest-first. Each value is a list of brief one-line descriptions. CHANGELOG: dict[str, list[str]] = { + "0.22.0": [ + "New: `kbagent project use ` -- pin a project as the default for subsequent commands. Persists `default_project` in config.json (the field already existed; now there is an explicit CLI verb to set it).", + "New: `kbagent project current` -- print the effective default project and its source (env / pin / none). Reports both the env override and the persisted pin so misconfigurations are visible, not silent.", + "New: `KBAGENT_PROJECT` env var overrides the persisted pin for a single shell/session. Resolution precedence for single-project ops: explicit `--project` > `KBAGENT_PROJECT` > pin > sole-project fallback > fail-hard with CONFIG_ERROR.", + "New: top-level `--deny-writes` / `--deny-destructive` flags synthesize a session-only firewall that merges with any persisted permission policy. Never written to config.json. `--deny-writes` blocks the wide net (write+destructive+admin); `--deny-destructive` is narrower and blocks only data destruction.", + "New: `ProjectService.resolve_pinned_alias()` plus `commands._helpers.resolve_project_alias()` -- single-project alias resolution contract for write/destructive commands. Public API for future PRs to adopt; FIIA P0-4 acceptance criterion.", + "Fix: stale pin (default_project pointing at a deleted alias) now raises a repair-friendly CONFIG_ERROR with `kbagent project use ` guidance instead of silently fanning out.", + ], "0.21.2": [ "Fix: `kbagent config search` now scans `rows[].configuration` in addition to the top-level configuration body (#196) -- queries like `--query '\"incremental\": false'` previously returned zero matches for row-based components (Snowflake/MySQL/BigQuery writers, DB extractors, Google Sheets) because the service only fetched `include=configuration`; match paths are now reported as `rows[N].configuration.parameters.`", "Fix: `kbagent storage tables` now accepts zero-or-more `--project` flags and queries all connected projects in parallel (#198) -- matches the multi-project behaviour of `storage buckets`, `config list`, `job list`; JSON envelope now returns `{tables: [...], errors: [...]}` with per-row `project_alias`; `--branch` still requires exactly one `--project`", diff --git a/src/keboola_agent_cli/cli.py b/src/keboola_agent_cli/cli.py index aeff431e..8a685a37 100644 --- a/src/keboola_agent_cli/cli.py +++ b/src/keboola_agent_cli/cli.py @@ -30,6 +30,7 @@ from .config_store import ConfigStore, resolve_config_dir from .constants import EXIT_PERMISSION_DENIED from .errors import PermissionDeniedError +from .models import PermissionPolicy from .output import OutputFormatter from .permissions import PermissionEngine from .services.branch_service import BranchService @@ -92,6 +93,58 @@ app.add_typer(encrypt_app, name="encrypt", rich_help_panel=_DEV) +def apply_firewall_flags( + persisted: PermissionPolicy | None, + *, + deny_writes: bool, + deny_destructive: bool, +) -> PermissionPolicy | None: + """Merge --deny-writes / --deny-destructive into the active policy for this invocation. + + Session-only: does NOT touch config.json. If neither flag is set, the + persisted policy is returned unchanged (possibly None). + + Merge semantics: + - A fresh session policy synthesized from the flags uses mode='allow' + so everything is allowed unless matched by the deny list. + - When a persisted policy already exists, the flag-implied deny patterns + are appended to its deny list (dedup); the mode is preserved. This is + strictly additive -- adding a flag never relaxes the persisted policy. + """ + if not deny_writes and not deny_destructive: + return persisted + + extra_deny: list[str] = [] + if deny_writes: + # cli:write pattern intentionally spans write+destructive+admin + # (permissions._matches_pattern lines 175-178). tool:write spans + # tool write+destructive. Wide net: --deny-writes blocks anything + # that mutates state. + extra_deny.extend(["cli:write", "tool:write"]) + if deny_destructive: + # cli:destructive narrowly matches only ops categorized 'destructive' + # (data destruction). Admin and pure-write are left allowed by design: + # the two flags exist precisely so callers can opt into the narrower + # block without forfeiting writes (e.g. allow create-bucket, block + # delete-bucket). + extra_deny.extend(["cli:destructive", "tool:destructive"]) + + if persisted is None: + return PermissionPolicy(mode="allow", allow=[], deny=extra_deny) + + # Preserve persisted mode, allow list; extend deny list without duplicates. + merged_deny = list(persisted.deny) + for pattern in extra_deny: + if pattern not in merged_deny: + merged_deny.append(pattern) + + return PermissionPolicy( + mode=persisted.mode, + allow=list(persisted.allow), + deny=merged_deny, + ) + + @app.callback() def main( ctx: typer.Context, @@ -123,6 +176,21 @@ def main( help="Show equivalent Python code instead of executing. " "Values: 'client' (direct API usage, default) or 'service' (uses CLI config).", ), + deny_writes: bool = typer.Option( + False, + "--deny-writes", + help="Session-only firewall: block write, destructive, AND admin " + "operations (the wide net -- project add/remove/edit, org setup, " + "storage writes and deletes, etc.). Merges with any persisted policy.", + ), + deny_destructive: bool = typer.Option( + False, + "--deny-destructive", + help="Session-only firewall: block ONLY data-destructive operations " + "(storage delete-table/delete-bucket/delete-column, job terminate, " + "branch delete, etc.). Admin ops like 'project remove' and 'org setup' " + "are NOT blocked -- use --deny-writes for the wide net.", + ), ) -> None: """Global options applied to all commands.""" from .auto_update import maybe_auto_update, show_post_update_changelog @@ -199,10 +267,17 @@ def main( try: config = config_store.load() - permission_engine = PermissionEngine(config.permissions) + persisted_policy = config.permissions except Exception: - # Config may be invalid (e.g. corrupted JSON) -- skip permission check - permission_engine = PermissionEngine(None) + # Config may be invalid (e.g. corrupted JSON) -- skip persisted policy + persisted_policy = None + + session_policy = apply_firewall_flags( + persisted_policy, + deny_writes=deny_writes, + deny_destructive=deny_destructive, + ) + permission_engine = PermissionEngine(session_policy) # Resolve hint mode hint_mode = None @@ -218,6 +293,8 @@ def main( ctx.obj["permission_engine"] = permission_engine ctx.obj["verbose"] = verbose ctx.obj["no_color"] = effective_no_color + ctx.obj["deny_writes"] = deny_writes + ctx.obj["deny_destructive"] = deny_destructive ctx.obj["config_store"] = config_store ctx.obj["project_service"] = project_service ctx.obj["component_service"] = component_service @@ -289,5 +366,7 @@ def main( verbose=verbose, no_color=effective_no_color, config_dir=config_dir, + deny_writes=deny_writes, + deny_destructive=deny_destructive, ) raise typer.Exit() diff --git a/src/keboola_agent_cli/commands/_helpers.py b/src/keboola_agent_cli/commands/_helpers.py index dc082a0e..39e7e575 100644 --- a/src/keboola_agent_cli/commands/_helpers.py +++ b/src/keboola_agent_cli/commands/_helpers.py @@ -153,6 +153,43 @@ def check_cli_permission(ctx: typer.Context, group_name: str) -> None: raise typer.Exit(code=EXIT_PERMISSION_DENIED) from None +def resolve_project_alias( + ctx: typer.Context, + formatter: OutputFormatter, + explicit: str | None, +) -> str: + """Resolve the effective project alias for a single-project operation. + + Precedence (first match wins): + 1. ``explicit`` (typically the CLI ``--project`` flag) + 2. ``KBAGENT_PROJECT`` env var + 3. Persisted pin (``config.default_project`` set by ``kbagent project use``) + 4. Sole registered project when exactly one exists (convenience) + 5. Exit code 5 with a CONFIG_ERROR if none of the above resolves + + Use this from write/destructive command paths where implicit fan-out + (``resolve_projects(None)`` returning every project) would be surprising + or unsafe. Read paths should keep their existing fan-out behavior. + + Args: + ctx: Typer context (must contain ``project_service``). + formatter: Output formatter for structured error emission. + explicit: The value of the CLI --project flag, or None. + + Returns: + The resolved project alias (guaranteed to be registered). + """ + from ..errors import ConfigError as _ConfigError + + service = get_service(ctx, "project_service") + try: + alias, _source = service.resolve_pinned_alias(explicit=explicit) + except _ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + return alias + + def validate_branch_requires_project( formatter: OutputFormatter, branch: int | None, diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index 1b1eea7a..ad492c46 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -43,11 +43,13 @@ ## Global Flags - --json / -j JSON output (always use for programmatic parsing) - --verbose / -v Verbose output - --no-color Disable colors (auto-disabled in non-TTY) - --config-dir Override config directory path - --hint MODE Generate Python code instead of executing (MODE: client or service) + --json / -j JSON output (always use for programmatic parsing) + --verbose / -v Verbose output + --no-color Disable colors (auto-disabled in non-TTY) + --config-dir Override config directory path + --hint MODE Generate Python code instead of executing (MODE: client or service) + --deny-writes Session-only firewall: block the WIDE NET -- every write, destructive, AND admin op (project add/remove/edit, org setup, all storage mutations) + --deny-destructive Session-only firewall: NARROW -- block only data-destructive ops in Keboola (delete-table/bucket/column, terminate-job, branch delete). Admin ops (project remove, org setup) stay allowed -- use --deny-writes for those ## All Commands @@ -83,6 +85,15 @@ Set the dashboard project description. Pass exactly one of --text, --file, or --stdin. Writes KBC.projectDescription to the default branch. + kbagent project use ALIAS + Pin ALIAS as the default project. Persists to config.json. + Env var KBAGENT_PROJECT=ALIAS overrides the pin for a single shell/session; + an explicit --project flag overrides both. + + kbagent project current + Print the effective default project and its source (env / pin / none). + Resolution order for single-project operations: --project > KBAGENT_PROJECT > pin. + ### Component Discovery kbagent component list [--project NAME] [--type TYPE] [--query "search"] @@ -500,6 +511,7 @@ KBC_MASTER_TOKEN Master token for sharing ops (global fallback) KBC_MASTER_TOKEN_* Per-project master token (e.g. KBC_MASTER_TOKEN_PROD) KBAGENT_CONFIG_DIR Override config directory + KBAGENT_PROJECT Override the pinned default project for this shell/session (beats pin, loses to --project) KBAGENT_MAX_PARALLEL_WORKERS Max concurrent threads for multi-project ops (default 10, max 100) KBAGENT_AUTO_UPDATE Set to "false" to disable automatic update on startup KBAGENT_UPDATED_FROM Set to an older version to trigger "What's new" display on next run diff --git a/src/keboola_agent_cli/commands/permissions.py b/src/keboola_agent_cli/commands/permissions.py index b281b65f..29b81d66 100644 --- a/src/keboola_agent_cli/commands/permissions.py +++ b/src/keboola_agent_cli/commands/permissions.py @@ -98,12 +98,28 @@ def permissions_list( help="Filter by risk category: read, write, destructive, admin", ), ) -> None: - """List all operations with their risk category and current allowed/denied status.""" + """List all operations with their risk category and current allowed/denied status. + + The allowed/denied column reflects the EFFECTIVE policy for this + invocation -- i.e. the persisted policy merged with any top-level + session flags like ``--deny-writes`` / ``--deny-destructive``. This + matches what a command will actually do right now. + """ + from ..cli import apply_firewall_flags + formatter = get_formatter(ctx) config_store: ConfigStore = get_service(ctx, "config_store") config = config_store.load() - engine = PermissionEngine(config.permissions) + deny_writes = bool(ctx.obj.get("deny_writes")) if ctx.obj else False + deny_destructive = bool(ctx.obj.get("deny_destructive")) if ctx.obj else False + effective_policy = apply_firewall_flags( + config.permissions, + deny_writes=deny_writes, + deny_destructive=deny_destructive, + ) + + engine = PermissionEngine(effective_policy) ops = engine.list_operations() if formatter.json_mode: @@ -116,44 +132,101 @@ def permissions_list( formatter.err_console.print( "\n[dim]No permission policy active. All operations are allowed.[/dim]" ) + elif deny_writes or deny_destructive: + active_flags = [] + if deny_writes: + active_flags.append("--deny-writes") + if deny_destructive: + active_flags.append("--deny-destructive") + formatter.err_console.print( + f"\n[dim]Session firewall active: {' '.join(active_flags)} (not persisted).[/dim]" + ) @permissions_app.command("show") def permissions_show( ctx: typer.Context, ) -> None: - """Show the current active permission policy.""" + """Show the current active permission policy. + + Reports both the PERSISTED policy (from config.json) and any SESSION + firewall layered on top via top-level ``--deny-writes`` / + ``--deny-destructive`` flags. Session flags are shown but are never + written to config.json -- they apply to this invocation only. + """ formatter = get_formatter(ctx) config_store: ConfigStore = get_service(ctx, "config_store") config = config_store.load() - if config.permissions is None: + deny_writes = bool(ctx.obj.get("deny_writes")) if ctx.obj else False + deny_destructive = bool(ctx.obj.get("deny_destructive")) if ctx.obj else False + session_flags: list[str] = [] + if deny_writes: + session_flags.append("--deny-writes") + if deny_destructive: + session_flags.append("--deny-destructive") + + persisted = config.permissions + + if persisted is None and not session_flags: if formatter.json_mode: - formatter.output({"active": False, "message": "No permission policy configured"}) + formatter.output( + { + "active": False, + "message": "No permission policy configured", + "session_flags": [], + } + ) else: formatter.console.print("No permission policy configured. All operations are allowed.") return - policy_data = { - "active": True, - "mode": config.permissions.mode, - "allow": config.permissions.allow, - "deny": config.permissions.deny, + policy_data: dict[str, Any] = { + "active": persisted is not None or bool(session_flags), + "persisted": ( + None + if persisted is None + else { + "mode": persisted.mode, + "allow": persisted.allow, + "deny": persisted.deny, + } + ), + "session_flags": session_flags, } + # Keep legacy top-level keys when a persisted policy exists so existing + # JSON consumers that read policy_data["mode"] / ["allow"] / ["deny"] + # remain compatible. Clients that need the new session-layer view read + # ``session_flags`` and ``persisted``. + if persisted is not None: + policy_data["mode"] = persisted.mode + policy_data["allow"] = persisted.allow + policy_data["deny"] = persisted.deny + if formatter.json_mode: formatter.output(policy_data) - else: + return + + if persisted is not None: mode_desc = ( "default-allow (everything allowed unless denied)" - if config.permissions.mode == "allow" + if persisted.mode == "allow" else "default-deny (everything denied unless allowed)" ) formatter.console.print(f"[bold]Mode:[/bold] {mode_desc}") - if config.permissions.allow: - formatter.console.print(f"[bold]Allow:[/bold] {', '.join(config.permissions.allow)}") - if config.permissions.deny: - formatter.console.print(f"[bold]Deny:[/bold] {', '.join(config.permissions.deny)}") + if persisted.allow: + formatter.console.print(f"[bold]Allow:[/bold] {', '.join(persisted.allow)}") + if persisted.deny: + formatter.console.print(f"[bold]Deny:[/bold] {', '.join(persisted.deny)}") + else: + formatter.console.print("[dim]No persisted permission policy (config.json is clean).[/dim]") + + if session_flags: + formatter.console.print( + f"[bold yellow]Session firewall:[/bold yellow] {' '.join(session_flags)} " + "[dim](active for this invocation only; not persisted)[/dim]" + ) @permissions_app.command("set") diff --git a/src/keboola_agent_cli/commands/project.py b/src/keboola_agent_cli/commands/project.py index 0f537dc4..210c3800 100644 --- a/src/keboola_agent_cli/commands/project.py +++ b/src/keboola_agent_cli/commands/project.py @@ -473,6 +473,89 @@ def project_refresh( formatter.output(result, _format_refresh_result) +# ── Project pin (default project) ───────────────────────────────────── + + +@project_app.command("use") +def project_use( + ctx: typer.Context, + alias: str = typer.Argument(..., help="Project alias to pin as default"), +) -> None: + """Pin as the default project for subsequent commands. + + The pin persists in config.json. ``KBAGENT_PROJECT`` overrides it for a + single invocation; an explicit ``--project`` flag overrides both. + """ + # No --hint: local-only ConfigStore mutation; no client or service call to render. + formatter = get_formatter(ctx) + service = get_service(ctx, "project_service") + + try: + result = service.use_project(alias=alias) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + + def _human(c: Console, d: dict[str, Any]) -> None: + previous = d.get("previous") + if previous and previous != d["alias"]: + c.print( + f"[bold green]Pinned:[/bold green] default project is now " + f"[bold]{d['alias']}[/bold] (was [dim]{previous}[/dim])" + ) + else: + c.print( + f"[bold green]Pinned:[/bold green] default project is [bold]{d['alias']}[/bold]" + ) + env_override = d.get("env_override") + if env_override and env_override != d["alias"]: + c.print( + f"[yellow]Note:[/yellow] KBAGENT_PROJECT='{env_override}' is set " + "and overrides this pin for the current shell." + ) + + formatter.output(result, _human) + + +@project_app.command("current") +def project_current(ctx: typer.Context) -> None: + """Show the effective default project. + + Reports whether the value comes from the ``KBAGENT_PROJECT`` env var + (``env``) or the persisted pin (``pin``). Prints nothing but a hint if + neither is set. + """ + # No --hint: local-only ConfigStore read; no client or service call to render. + formatter = get_formatter(ctx) + service = get_service(ctx, "project_service") + + result = service.current_project() + + def _human(c: Console, d: dict[str, Any]) -> None: + alias = d.get("alias") + source = d.get("source") + if alias is None: + c.print( + "[dim](no default project set)[/dim] -- pass --project, set " + "KBAGENT_PROJECT, or run 'kbagent project use '" + ) + return + if source == "env": + c.print(f"[bold cyan]{alias}[/bold cyan] [dim](source: KBAGENT_PROJECT env var)[/dim]") + if d.get("env_points_to_configured_project") is False: + c.print( + f"[yellow]Warning:[/yellow] '{alias}' is NOT in your " + "configured projects. Commands that use this pin will fail." + ) + pinned = d.get("pinned") + if pinned: + c.print(f"[dim] (pinned in config: {pinned}, overridden)[/dim]") + else: + c.print(f"[bold cyan]{alias}[/bold cyan] [dim](source: pinned default)[/dim]") + + formatter.output(result, _human) + + # ── Project description (dashboard KBC.projectDescription) ──────────── diff --git a/src/keboola_agent_cli/commands/repl.py b/src/keboola_agent_cli/commands/repl.py index bfbb0894..3770af74 100644 --- a/src/keboola_agent_cli/commands/repl.py +++ b/src/keboola_agent_cli/commands/repl.py @@ -67,8 +67,21 @@ def _get_history_path() -> Path: return config_dir / "repl_history" -def _run_repl(json_mode: bool, verbose: bool, no_color: bool, config_dir: str | None) -> None: - """Main REPL loop.""" +def _run_repl( + json_mode: bool, + verbose: bool, + no_color: bool, + config_dir: str | None, + deny_writes: bool = False, + deny_destructive: bool = False, +) -> None: + """Main REPL loop. + + Global flags from the outer invocation are re-applied on every command + executed inside the REPL. This includes the session-only firewall flags + ``--deny-writes`` / ``--deny-destructive`` -- dropping them here would + silently elevate the REPL above the policy the user started it with. + """ from ..cli import app as typer_app # Build command tree for completion @@ -94,7 +107,15 @@ def _run_repl(json_mode: bool, verbose: bool, no_color: bool, config_dir: str | # Show banner sys.stderr.write(f"\nkbagent v{__version__} -- interactive mode\n") sys.stderr.write("Type 'help' for commands, 'exit' to quit.\n") - sys.stderr.write(f"Global flags: --json={json_mode}, --verbose={verbose}\n\n") + sys.stderr.write(f"Global flags: --json={json_mode}, --verbose={verbose}\n") + if deny_writes or deny_destructive: + active = [] + if deny_writes: + active.append("--deny-writes") + if deny_destructive: + active.append("--deny-destructive") + sys.stderr.write(f"Session firewall: {' '.join(active)} (active for all commands)\n") + sys.stderr.write("\n") while True: try: @@ -141,6 +162,13 @@ def _run_repl(json_mode: bool, verbose: bool, no_color: bool, config_dir: str | full_argv.append("--no-color") if config_dir and "--config-dir" not in argv: full_argv.extend(["--config-dir", config_dir]) + # Session firewall flags: re-applied on every REPL invocation so the + # user's opt-in policy survives across prompts. Forgetting to forward + # these would silently restore write/destructive access inside the REPL. + if deny_writes and "--deny-writes" not in argv: + full_argv.append("--deny-writes") + if deny_destructive and "--deny-destructive" not in argv: + full_argv.append("--deny-destructive") full_argv.extend(argv) # Prevent recursive REPL @@ -176,4 +204,6 @@ def repl_command(ctx: typer.Context) -> None: verbose=ctx.obj.get("verbose", False), no_color=ctx.obj.get("no_color", False), config_dir=None, # Already resolved in ctx + deny_writes=ctx.obj.get("deny_writes", False), + deny_destructive=ctx.obj.get("deny_destructive", False), ) diff --git a/src/keboola_agent_cli/constants.py b/src/keboola_agent_cli/constants.py index 32118767..22210e4b 100644 --- a/src/keboola_agent_cli/constants.py +++ b/src/keboola_agent_cli/constants.py @@ -91,6 +91,10 @@ ENV_CONFIG_DIR: str = "KBAGENT_CONFIG_DIR" LOCAL_CONFIG_DIR_NAME: str = ".kbagent" +# --- Project Pin --- +# Overrides the persisted `default_project` pin for a single invocation/session. +ENV_KBAGENT_PROJECT: str = "KBAGENT_PROJECT" + # --- Environment Variable Names --- ENV_MAX_PARALLEL_WORKERS: str = "KBAGENT_MAX_PARALLEL_WORKERS" ENV_KBC_TOKEN: str = "KBC_TOKEN" diff --git a/src/keboola_agent_cli/permissions.py b/src/keboola_agent_cli/permissions.py index 1955f61c..6f63e47d 100644 --- a/src/keboola_agent_cli/permissions.py +++ b/src/keboola_agent_cli/permissions.py @@ -22,6 +22,8 @@ "project.refresh": "admin", "project.description-get": "read", "project.description-set": "write", + "project.use": "write", + "project.current": "read", # Config browsing & management "config.list": "read", "config.detail": "read", diff --git a/src/keboola_agent_cli/services/project_service.py b/src/keboola_agent_cli/services/project_service.py index 1b43bc7a..73019fa1 100644 --- a/src/keboola_agent_cli/services/project_service.py +++ b/src/keboola_agent_cli/services/project_service.py @@ -3,9 +3,11 @@ Orchestrates config persistence and API calls without knowing about CLI or HTTP details. """ +import os import time from typing import Any +from ..constants import ENV_KBAGENT_PROJECT from ..errors import ConfigError, KeboolaApiError, mask_token from ..models import ProjectConfig from .base import BaseService @@ -249,3 +251,141 @@ def get_status(self, aliases: list[str] | None = None) -> list[dict[str, Any]]: results.sort(key=lambda r: r.get("alias", "")) return results + + def use_project(self, alias: str) -> dict[str, Any]: + """Pin an alias as the persistent default project. + + The pin is stored as ``config.default_project`` in config.json. + It is overridden at runtime by the ``KBAGENT_PROJECT`` env var and by + explicit ``--project`` flags. + + Args: + alias: The project alias to pin. + + Returns: + Dict with the new pin, previous pin, and source. + + Raises: + ConfigError: If the alias does not exist. + """ + config = self._config_store.load() + if alias not in config.projects: + raise ConfigError(f"Project '{alias}' not found.") + + previous = config.default_project or None + config.default_project = alias + self._config_store.save(config) + + env_override = os.environ.get(ENV_KBAGENT_PROJECT) + return { + "alias": alias, + "previous": previous, + "source": "pin", + "env_override": env_override or None, + } + + def current_project(self) -> dict[str, Any]: + """Report the effective default project and its source. + + Resolution: + - If ``KBAGENT_PROJECT`` is set, it wins (source=env). + - Otherwise the persisted pin wins (source=pin). + - If neither is set, ``alias`` is ``None``. + + The env override is reported even when it points at a project that is + not (yet) registered in config.json -- callers get the true effective + alias plus an ``env_points_to_configured_project`` flag to reason about + it. This avoids silently masking misconfigurations. + + Returns: + Dict with keys: alias, source ('env' | 'pin' | 'none'), pinned, + env_override, env_points_to_configured_project. + """ + config = self._config_store.load() + pinned = config.default_project or None + # Treat KBAGENT_PROJECT="" the same as unset (Unix shell convention: + # empty env is commonly produced by `unset` substitutes / blank + # exports). Strict rejection would surprise CI users who export it + # conditionally. Apply consistently in resolve_pinned_alias(). + env_value = os.environ.get(ENV_KBAGENT_PROJECT) + env_override = env_value if env_value else None + + if env_override is not None: + return { + "alias": env_override, + "source": "env", + "pinned": pinned, + "env_override": env_override, + "env_points_to_configured_project": env_override in config.projects, + } + + return { + "alias": pinned, + "source": "pin" if pinned else "none", + "pinned": pinned, + "env_override": None, + "env_points_to_configured_project": None, + } + + def resolve_pinned_alias(self, explicit: str | None = None) -> tuple[str, str]: + """Resolve the effective project alias for a single-project operation. + + Precedence (first match wins): + 1. ``explicit`` argument (typically the CLI ``--project`` flag) + 2. ``KBAGENT_PROJECT`` env var + 3. Persisted ``default_project`` pin + 4. If exactly one project is registered, fall back to it (source=sole) + 5. Fail hard with ConfigError + + This is the single-project analog of ``resolve_projects()`` (which + fans out to all projects). Use this from write/destructive command + paths where fan-out would be surprising or unsafe. + + Args: + explicit: Explicit alias from a CLI flag, or None. + + Returns: + Tuple of (alias, source). + + Raises: + ConfigError: If the resolved alias is not registered, or if none + can be resolved. + """ + config = self._config_store.load() + + if explicit: + if explicit not in config.projects: + raise ConfigError(f"Project '{explicit}' not found.") + return explicit, "explicit" + + env_value = os.environ.get(ENV_KBAGENT_PROJECT) + if env_value: + if env_value not in config.projects: + raise ConfigError( + f"{ENV_KBAGENT_PROJECT}='{env_value}' points to a project " + "that is not registered. Use 'kbagent project add' or " + "unset the env var." + ) + return env_value, "env" + + pinned = config.default_project + if pinned: + if pinned not in config.projects: + raise ConfigError( + f"Pinned default project '{pinned}' is not registered. " + "Run 'kbagent project use ' to repair." + ) + return pinned, "pin" + + if len(config.projects) == 1: + (sole,) = config.projects.keys() + return sole, "sole" + + if not config.projects: + raise ConfigError("No projects configured. Run 'kbagent project add' first.") + + raise ConfigError( + "Multiple projects configured and no default pinned. " + "Pass --project , set KBAGENT_PROJECT, or run " + "'kbagent project use '." + ) diff --git a/tests/test_cli.py b/tests/test_cli.py index 86055d81..0f4eab6f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -470,6 +470,521 @@ def test_project_status_human(self, tmp_path: Path) -> None: assert "Project Status" in result.output +class TestProjectUse: + """Tests for `kbagent project use ` (pin default project).""" + + def _seed(self, config_dir: Path, *aliases: str) -> None: + """Seed a ConfigStore with one or more projects via the live add path.""" + mock_client = make_mock_client() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockService, + patch.dict(os.environ, {"KBC_TOKEN": TEST_TOKEN}), + ): + store = ConfigStore(config_dir=config_dir) + MockStore.return_value = store + MockService.return_value = ProjectService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + for alias in aliases: + runner.invoke(app, ["project", "add", "--project", alias]) + + def test_project_use_pins_alias(self, tmp_path: Path) -> None: + """project use ALIAS persists default_project to config.json.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod", "stage") + + # default_project should be the first-added (prod); now pin stage. + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + store = ConfigStore(config_dir=config_dir) + MockStore.return_value = store + result = runner.invoke(app, ["--json", "project", "use", "stage"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output)["data"] + assert data["alias"] == "stage" + assert data["previous"] == "prod" + # Verify persistence: re-load the store and check default_project. + persisted = ConfigStore(config_dir=config_dir).load() + assert persisted.default_project == "stage" + + def test_project_use_unknown_alias_exit_5(self, tmp_path: Path) -> None: + """project use on an unregistered alias returns exit code 5.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "use", "does-not-exist"]) + + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["status"] == "error" + assert data["error"]["code"] == "CONFIG_ERROR" + + def test_project_use_human_mode_confirms_pin(self, tmp_path: Path) -> None: + """project use in human mode prints the new pin.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["project", "use", "prod"]) + + assert result.exit_code == 0 + assert "prod" in result.output + assert "Pinned" in result.output or "pinned" in result.output + + def test_project_current_with_pin(self, tmp_path: Path) -> None: + """project current reports the persisted pin when no env is set.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {}, clear=False), + ): + os.environ.pop("KBAGENT_PROJECT", None) + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "current"]) + + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert data["alias"] == "prod" + assert data["source"] == "pin" + assert data["env_override"] is None + + def test_project_current_env_overrides_pin(self, tmp_path: Path) -> None: + """KBAGENT_PROJECT wins over persisted pin; env presence is reported.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod", "stage") + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBAGENT_PROJECT": "stage"}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "current"]) + + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert data["alias"] == "stage" + assert data["source"] == "env" + assert data["pinned"] == "prod" + assert data["env_points_to_configured_project"] is True + + def test_project_current_env_points_to_unknown(self, tmp_path: Path) -> None: + """Unregistered KBAGENT_PROJECT value is still reported, with a flag.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBAGENT_PROJECT": "mystery"}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "current"]) + + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert data["alias"] == "mystery" + assert data["source"] == "env" + assert data["env_points_to_configured_project"] is False + + def test_project_current_human_mode_with_pin(self, tmp_path: Path) -> None: + """project current in human mode prints alias + source label.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {}, clear=False), + ): + os.environ.pop("KBAGENT_PROJECT", None) + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["project", "current"]) + + assert result.exit_code == 0 + assert "prod" in result.output + # Rich output should mention the source. + assert "pin" in result.output.lower() + + def test_project_current_human_mode_env_warns_unknown(self, tmp_path: Path) -> None: + """Human-mode project current warns when env points to unregistered alias.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBAGENT_PROJECT": "ghost"}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["project", "current"]) + + assert result.exit_code == 0 + assert "ghost" in result.output + # Output should warn the alias is not registered. + assert "Warning" in result.output or "NOT" in result.output + + def test_project_use_blocked_by_persisted_deny_writes(self, tmp_path: Path) -> None: + """A persisted policy denying cli:write must block project use (it's a write op).""" + from keboola_agent_cli.models import PermissionPolicy + + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod", "stage") + + # Persist a default-allow policy that denies cli:write. + store = ConfigStore(config_dir=config_dir) + cfg = store.load() + cfg.permissions = PermissionPolicy(mode="allow", allow=[], deny=["cli:write"]) + store.save(cfg) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "use", "stage"]) + + assert result.exit_code == 6, result.output + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_hint_on_non_api_project_use(self, tmp_path: Path) -> None: + """project use is purely local -- --hint must exit cleanly, not crash.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--hint", "client", "project", "use", "prod"]) + # Should exit 0 with a clear message -- no hint available for local ops. + assert result.exit_code == 0 + + def test_hint_on_non_api_project_current(self, tmp_path: Path) -> None: + """project current is purely local -- --hint must exit cleanly, not crash.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--hint", "service", "project", "current"]) + assert result.exit_code == 0 + + def test_project_current_allowed_under_deny_writes(self, tmp_path: Path) -> None: + """project current is classified read, so cli:write deny must NOT block it.""" + from keboola_agent_cli.models import PermissionPolicy + + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir, "prod") + + store = ConfigStore(config_dir=config_dir) + cfg = store.load() + cfg.permissions = PermissionPolicy(mode="allow", allow=[], deny=["cli:write"]) + store.save(cfg) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "current"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["status"] == "ok" + + def test_project_current_none_set(self, tmp_path: Path) -> None: + """With no projects and no env, current reports alias=None source=none.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {}, clear=False), + ): + os.environ.pop("KBAGENT_PROJECT", None) + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--json", "project", "current"]) + + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert data["alias"] is None + assert data["source"] == "none" + + +class TestFirewallFlags: + """Tests for top-level --deny-writes / --deny-destructive session flags.""" + + def _seed(self, config_dir: Path) -> None: + mock_client = make_mock_client() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockService, + patch.dict(os.environ, {"KBC_TOKEN": TEST_TOKEN}), + ): + store = ConfigStore(config_dir=config_dir) + MockStore.return_value = store + MockService.return_value = ProjectService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + runner.invoke(app, ["project", "add", "--project", "prod"]) + + def test_deny_writes_blocks_project_add(self, tmp_path: Path) -> None: + """--deny-writes blocks project.add (admin is a superset of cli:write).""" + config_dir = tmp_path / "config" + config_dir.mkdir() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBC_TOKEN": TEST_TOKEN}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + [ + "--deny-writes", + "--json", + "project", + "add", + "--project", + "foo", + "--url", + "https://connection.keboola.com", + "--token", + TEST_TOKEN, + ], + ) + + assert result.exit_code == 6 + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_deny_writes_allows_read(self, tmp_path: Path) -> None: + """--deny-writes must not block read operations.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke(app, ["--deny-writes", "--json", "project", "list"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["status"] == "ok" + + def test_deny_destructive_blocks_delete_table(self, tmp_path: Path) -> None: + """--deny-destructive blocks storage.delete-table at the permission callback.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + [ + "--deny-destructive", + "--json", + "storage", + "delete-table", + "--project", + "prod", + "--table-id", + "in.c-x.y", + "--yes", + ], + ) + + assert result.exit_code == 6 + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_deny_destructive_allows_admin_ops(self, tmp_path: Path) -> None: + """--deny-destructive must NOT block admin-tier ops (project.remove, org.setup). + + Documented semantics: --deny-destructive is NARROW (data destruction + only). Admin operations fall through to --deny-writes, which is the + wide net. This test locks the contract so a future registry change + can't widen --deny-destructive's scope silently. + """ + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + # project.remove is classified 'admin'; --deny-destructive must + # NOT block it (permission gate exits 6 if blocked). The command + # will succeed in removing 'prod' since the seed registers it. + result = runner.invoke( + app, + [ + "--deny-destructive", + "--json", + "project", + "remove", + "--project", + "prod", + ], + ) + + assert result.exit_code != 6, ( + f"--deny-destructive incorrectly blocked admin op project.remove " + f"(exit {result.exit_code}): {result.output}" + ) + # Verify the actual operation also succeeded (not just the perm gate). + data = json.loads(result.output) + assert data["status"] == "ok" + + def test_deny_writes_blocks_admin_ops(self, tmp_path: Path) -> None: + """Complement of the above: --deny-writes IS the wide net and DOES block admin.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + [ + "--deny-writes", + "--json", + "project", + "remove", + "--project", + "prod", + ], + ) + + assert result.exit_code == 6, ( + f"--deny-writes must block admin op project.remove (exit 6); " + f"got {result.exit_code}: {result.output}" + ) + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_deny_destructive_allows_write(self, tmp_path: Path) -> None: + """--deny-destructive must NOT block pure 'write' (non-destructive) ops. + + Uses 'permissions check' which evaluates against the PERSISTED policy, + so the session-only --deny-destructive does not apply there. We instead + attempt a write op and assert it is not blocked by the permission gate + (any later error must not be PERMISSION_DENIED). + """ + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + # storage.create-bucket is classified 'write', so --deny-destructive alone + # does not block it. The command will fail for other reasons (mock client), + # but the failure code must not be PERMISSION_DENIED. + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + [ + "--deny-destructive", + "--json", + "storage", + "create-bucket", + "--project", + "prod", + "--stage", + "in", + "--name", + "b", + ], + ) + + # The permission check must not fire. + assert result.exit_code != 6 or "PERMISSION_DENIED" not in result.output + + def test_deny_writes_composes_with_persisted_deny_mode(self, tmp_path: Path) -> None: + """End-to-end: default-deny policy + --deny-writes still blocks writes. + + A persisted default-deny policy that allows cli:write (unusual but + syntactically valid) composed with --deny-writes must resolve to + "write denied" because deny takes precedence over allow in the engine + (permissions.py: default-deny rule is 'allowed and not denied'). + """ + from keboola_agent_cli.models import PermissionPolicy + + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + # Persist a default-deny policy that explicitly allows cli:write. + store = ConfigStore(config_dir=config_dir) + cfg = store.load() + cfg.permissions = PermissionPolicy(mode="deny", allow=["cli:write", "cli:read"], deny=[]) + store.save(cfg) + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBC_TOKEN": TEST_TOKEN}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + [ + "--deny-writes", + "--json", + "project", + "add", + "--project", + "newproj", + ], + ) + + # Merged policy: mode=deny, allow=[cli:write,cli:read], deny=[cli:write,tool:write]. + # Default-deny rule: allowed AND not denied. cli:write matches both + # allow and deny -- deny wins. + assert result.exit_code == 6, result.output + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_deny_writes_and_destructive_merge_with_persisted(self, tmp_path: Path) -> None: + """Flags merge with persisted policy; never persist to disk.""" + from keboola_agent_cli.models import AppConfig, PermissionPolicy + + config_dir = tmp_path / "config" + config_dir.mkdir() + self._seed(config_dir) + + # Persist a policy that only denies branch.delete + store = ConfigStore(config_dir=config_dir) + cfg = store.load() + cfg.permissions = PermissionPolicy(mode="allow", allow=[], deny=["branch.delete"]) + store.save(cfg) + + # Run once with --deny-writes: project.add must be blocked + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch.dict(os.environ, {"KBC_TOKEN": TEST_TOKEN}), + ): + MockStore.return_value = ConfigStore(config_dir=config_dir) + result = runner.invoke( + app, + ["--deny-writes", "--json", "project", "add", "--project", "new"], + ) + assert result.exit_code == 6 + + # The persisted policy on disk must NOT have been mutated. + reloaded: AppConfig = ConfigStore(config_dir=config_dir).load() + assert reloaded.permissions is not None + assert reloaded.permissions.deny == ["branch.delete"] + + class TestProjectEdit: """Tests for `kbagent project edit` command.""" @@ -3620,6 +4135,30 @@ def test_project_help(self) -> None: assert "remove" in result.output assert "edit" in result.output assert "status" in result.output + # PR5 additions -- guard against accidental removal. + assert "use" in result.output + assert "current" in result.output + + def test_root_callback_registers_firewall_flags(self) -> None: + """App callback signature declares --deny-writes and --deny-destructive. + + Tests the callback signature rather than rendered --help output: Rich + truncates options in narrow terminals (CI's default width collapses + long flag names into '...'), which makes a string-match on the help + text flaky. The signature is what users see once the terminal has + room and is a strict superset guarantee. + """ + import inspect + + from keboola_agent_cli.cli import main as cli_main + + sig = inspect.signature(cli_main) + assert "deny_writes" in sig.parameters, ( + "cli.main() must accept deny_writes (top-level --deny-writes flag)" + ) + assert "deny_destructive" in sig.parameters, ( + "cli.main() must accept deny_destructive (top-level --deny-destructive flag)" + ) def test_config_help(self) -> None: """config --help shows subcommands.""" diff --git a/tests/test_e2e.py b/tests/test_e2e.py index c18e0336..b44885d9 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -558,6 +558,9 @@ def test_full_cli_e2e(self) -> None: _step(42, "storage delete-table + delete-bucket", "CLI-driven cleanup") self._test_storage_cleanup(bucket_id, table_id) + _step("42.5", "project use / current + firewall flags") + self._test_project_pin_and_firewall() + _step(43, "project edit + remove", "final cleanup") self._test_project_edit_and_remove() @@ -2382,6 +2385,129 @@ def _test_storage_cleanup(self, bucket_id: str, table_id: str) -> None: assert bucket_id in data["data"]["deleted"] self._created_buckets.remove(bucket_id) + def _test_project_pin_and_firewall(self) -> None: + """End-to-end coverage for `project use`, `project current`, and --deny-* flags.""" + # --- Pin lifecycle ------------------------------------------------- + + # Pre-condition: first-added is already the default. Verify via current. + data = self._run_ok("project", "current") + assert data["data"]["alias"] == self.alias + assert data["data"]["source"] == "pin" + + # Explicit `project use` is a no-op in value but confirms it persists. + data = self._run_ok("project", "use", self.alias) + assert data["data"]["alias"] == self.alias + # source is always "pin" on use (the field describes where the new + # pin ended up, not how it arrived). + assert data["data"]["source"] == "pin" + + # `project use nonexistent` fails with exit 5 (CONFIG_ERROR). + result = self._run("project", "use", "does-not-exist-alias") + assert result.exit_code == 5 + + # --- KBAGENT_PROJECT env override --------------------------------- + # Set the env var to a bogus value and confirm `current` reports env + # as the source + flags the unknown alias. + with patch.dict(os.environ, {"KBAGENT_PROJECT": "mystery-alias"}): + data = self._run_ok("project", "current") + assert data["data"]["alias"] == "mystery-alias" + assert data["data"]["source"] == "env" + assert data["data"]["env_points_to_configured_project"] is False + assert data["data"]["pinned"] == self.alias + + # After unsetting, pin is restored as the effective alias. + data = self._run_ok("project", "current") + assert data["data"]["source"] == "pin" + assert data["data"]["alias"] == self.alias + + # --- --deny-writes blocks writes, allows reads -------------------- + # Read still succeeds. + data = self._run_ok( + "--deny-writes", # top-level flag must come before subcommand + "project", + "list", + ) + assert any(p["alias"] == self.alias for p in data["data"]) + + # Attempting a write under --deny-writes must exit 6 PERMISSION_DENIED. + # create-bucket is a safe write to try: if the firewall fails to + # block it we'd create a real bucket, so track it for cleanup just + # in case the block logic regresses. + guard_bucket_name = f"{RUN_ID.replace('-', '_')}_firewall_guard" + result = self._run( + "--deny-writes", + "storage", + "create-bucket", + "--project", + self.alias, + "--stage", + "in", + "--name", + guard_bucket_name, + ) + assert result.exit_code == 6, ( + f"--deny-writes should block storage.create-bucket (exit 6), " + f"got {result.exit_code}: {result.output}" + ) + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + # Safety: if the block failed silently and a bucket was actually + # created, schedule cleanup. We don't fail louder because the + # exit_code assert above already did. + try: + buckets = self.api.list_buckets() + for bucket in buckets: + if bucket.get("name") == guard_bucket_name: + self._created_buckets.append(bucket["id"]) + except Exception: + pass # Best-effort cleanup tracking only. + + # --- --deny-destructive blocks destructive ops -------------------- + # delete-bucket is destructive; must exit 6 even on a bucket that + # does not exist (permission check fires before the API call). + result = self._run( + "--deny-destructive", + "storage", + "delete-bucket", + "--project", + self.alias, + "--bucket-id", + "in.c-never-existed", + "--yes", + ) + assert result.exit_code == 6, ( + f"--deny-destructive should block storage.delete-bucket (exit 6), " + f"got {result.exit_code}: {result.output}" + ) + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + # --- --deny-destructive allows non-destructive writes ------------- + # project.description-set is classified 'write' (not destructive), + # so --deny-destructive must NOT block it. We pass an empty string + # write -- this goes to the API, but the permission gate is the + # only thing under test here, so any non-6 exit is acceptable. + result = self._run( + "--deny-destructive", + "project", + "description-get", + "--project", + self.alias, + ) + assert result.exit_code != 6, ( + "--deny-destructive must not block read op project.description-get" + ) + + # --- Persistence check -------------------------------------------- + # None of the --deny-* flags may have written to config.json. + store = ConfigStore(config_dir=self.config_dir) + persisted = store.load() + assert persisted.permissions is None, ( + "--deny-writes / --deny-destructive must be session-only; " + f"found persisted policy: {persisted.permissions}" + ) + def _test_project_edit_and_remove(self) -> None: """Edit project URL, then remove it.""" # project edit -- change URL back to same (just verify command works) @@ -3413,3 +3539,225 @@ def test_resolver_prefers_explicit_values_id_over_first_row(self) -> None: ) print(f" {_DIM}resolved={resolved} pinned={pinned_row_id} first={first_row_id}{_RESET}") assert resolved == pinned_row_id + + +# --------------------------------------------------------------------------- +# Project pin + firewall flag E2E (PR5) +# --------------------------------------------------------------------------- + + +@skip_without_credentials +@pytest.mark.e2e +class TestPinAndFirewallE2E: + """Focused E2E for `project use`, `project current`, and --deny-* flags. + + Exercises the real API to confirm: + - `project use` persists the pin to config.json. + - `project current` reports the effective alias + source correctly. + - `KBAGENT_PROJECT` env var overrides the pin at runtime. + - `--deny-writes` blocks the permission gate on a real write op (exit 6). + - `--deny-destructive` blocks a real destructive op (exit 6). + - Neither flag persists to config.json. + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + self.alias_a = f"{RUN_ID}-pin-a" + self.alias_b = f"{RUN_ID}-pin-b" + + def _run(self, *args: str) -> Any: + return _invoke(self.config_dir, ["--json", *args]) + + def test_pin_lifecycle_against_real_project(self) -> None: + """End-to-end: add, use, current, env override.""" + # Register two aliases pointing at the SAME real project. We only + # need distinct aliases to observe the pin switching. + self._run( + "project", + "add", + "--project", + self.alias_a, + "--url", + self.url, + "--token", + self.token, + ) + self._run( + "project", + "add", + "--project", + self.alias_b, + "--url", + self.url, + "--token", + self.token, + ) + + # First-added becomes default. + current = self._run("project", "current") + assert current.exit_code == 0 + data = _json_ok(current) + assert data["data"]["alias"] == self.alias_a + assert data["data"]["source"] == "pin" + + # `project use` switches the pin; persistence survives next invocation. + use_result = self._run("project", "use", self.alias_b) + use_data = _json_ok(use_result) + assert use_data["data"]["alias"] == self.alias_b + assert use_data["data"]["previous"] == self.alias_a + + after = _json_ok(self._run("project", "current")) + assert after["data"]["alias"] == self.alias_b + assert after["data"]["source"] == "pin" + + # Unknown alias -> exit 5. + bad = self._run("project", "use", "does-not-exist") + assert bad.exit_code == 5 + bad_data = json.loads(bad.output) + assert bad_data["error"]["code"] == "CONFIG_ERROR" + + # KBAGENT_PROJECT overrides the pin. + with patch.dict(os.environ, {"KBAGENT_PROJECT": self.alias_a}): + env_view = _json_ok(self._run("project", "current")) + assert env_view["data"]["alias"] == self.alias_a + assert env_view["data"]["source"] == "env" + assert env_view["data"]["pinned"] == self.alias_b + assert env_view["data"]["env_points_to_configured_project"] is True + + def test_deny_writes_blocks_real_write_op(self) -> None: + """--deny-writes must exit 6 on a real create-bucket attempt.""" + self._run( + "project", + "add", + "--project", + self.alias_a, + "--url", + self.url, + "--token", + self.token, + ) + + # Use a name that won't collide; the permission gate fires before + # the API call so the bucket must never appear. + bucket_name = f"{RUN_ID.replace('-', '_')}_fw_w" + result = self._run( + "--deny-writes", + "storage", + "create-bucket", + "--project", + self.alias_a, + "--stage", + "in", + "--name", + bucket_name, + ) + assert result.exit_code == 6, ( + f"--deny-writes should block storage.create-bucket; got exit " + f"{result.exit_code}: {result.output}" + ) + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + # Defensive: if the block leaked and a bucket was actually created, + # clean it up and fail the assertion above (already failed) more loudly. + import contextlib + + api = KeboolaClient(self.url, self.token) + try: + for bucket in api.list_buckets(): + if bucket.get("name") == bucket_name: + with contextlib.suppress(Exception): + api.delete_bucket(bucket["id"], force=True) + raise AssertionError( + f"--deny-writes failed to block: bucket {bucket['id']} was created" + ) + finally: + api.close() + + def test_deny_destructive_blocks_real_destructive_op(self) -> None: + """--deny-destructive must exit 6 on storage.delete-bucket.""" + self._run( + "project", + "add", + "--project", + self.alias_a, + "--url", + self.url, + "--token", + self.token, + ) + + result = self._run( + "--deny-destructive", + "storage", + "delete-bucket", + "--project", + self.alias_a, + "--bucket-id", + "in.c-does-not-exist-for-sure", + "--yes", + ) + assert result.exit_code == 6, ( + f"--deny-destructive should block delete-bucket; got exit " + f"{result.exit_code}: {result.output}" + ) + data = json.loads(result.output) + assert data["error"]["code"] == "PERMISSION_DENIED" + + def test_deny_destructive_allows_read_op(self) -> None: + """--deny-destructive must NOT block read ops (regression guard).""" + self._run( + "project", + "add", + "--project", + self.alias_a, + "--url", + self.url, + "--token", + self.token, + ) + result = self._run( + "--deny-destructive", + "storage", + "buckets", + "--project", + self.alias_a, + ) + # Read must succeed (exit 0) OR fail for non-permission reasons. + assert result.exit_code != 6, f"--deny-destructive blocked a read op: {result.output}" + + def test_firewall_flags_never_persist(self) -> None: + """Neither --deny-writes nor --deny-destructive may write to config.json.""" + self._run( + "project", + "add", + "--project", + self.alias_a, + "--url", + self.url, + "--token", + self.token, + ) + # Run a blocked op under both flags. + self._run( + "--deny-writes", + "--deny-destructive", + "storage", + "create-bucket", + "--project", + self.alias_a, + "--stage", + "in", + "--name", + "never_created", + ) + # Persisted policy must still be None. + persisted = ConfigStore(config_dir=self.config_dir).load() + assert persisted.permissions is None, ( + f"--deny-* flags leaked to config.json: {persisted.permissions}" + ) diff --git a/tests/test_helpers.py b/tests/test_helpers.py index d076f9dc..ca1dbeab 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -372,3 +372,149 @@ def test_ignore_active_branch_single_project_inferred(self, tmp_config_dir) -> N assert project == "prod" assert branch_id is None formatter.err_console.print.assert_called_once() + + +class TestResolveProjectAlias: + """Tests for resolve_project_alias() (write-op precedence).""" + + def _build(self, tmp_config_dir) -> tuple: + from unittest.mock import MagicMock + + from keboola_agent_cli.config_store import ConfigStore + from keboola_agent_cli.models import ProjectConfig + from keboola_agent_cli.services.project_service import ProjectService + + store = ConfigStore(config_dir=tmp_config_dir) + for alias, pid in (("prod", 1), ("stage", 2)): + store.add_project( + alias, + ProjectConfig( + stack_url="https://connection.keboola.com", + token=f"t-{alias}", + project_name=alias, + project_id=pid, + ), + ) + service = ProjectService(config_store=store) + ctx = MagicMock() + ctx.obj = {"project_service": service} + formatter = MagicMock(json_mode=False) + return ctx, formatter, store + + def test_explicit_wins_over_env_and_pin(self, tmp_config_dir, monkeypatch) -> None: + from keboola_agent_cli.commands._helpers import resolve_project_alias + + monkeypatch.setenv("KBAGENT_PROJECT", "stage") + ctx, formatter, _ = self._build(tmp_config_dir) + assert resolve_project_alias(ctx, formatter, explicit="prod") == "prod" + + def test_env_beats_pin(self, tmp_config_dir, monkeypatch) -> None: + from keboola_agent_cli.commands._helpers import resolve_project_alias + + monkeypatch.setenv("KBAGENT_PROJECT", "stage") + ctx, formatter, _ = self._build(tmp_config_dir) + # pin is prod (first-added); env overrides to stage + assert resolve_project_alias(ctx, formatter, explicit=None) == "stage" + + def test_pin_used_when_no_env(self, tmp_config_dir, monkeypatch) -> None: + from keboola_agent_cli.commands._helpers import resolve_project_alias + + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + ctx, formatter, _ = self._build(tmp_config_dir) + assert resolve_project_alias(ctx, formatter, explicit=None) == "prod" + + def test_fail_hard_multi_no_pin(self, tmp_config_dir, monkeypatch) -> None: + import typer + + from keboola_agent_cli.commands._helpers import resolve_project_alias + + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + ctx, formatter, store = self._build(tmp_config_dir) + cfg = store.load() + cfg.default_project = "" + store.save(cfg) + + with pytest.raises(typer.Exit) as exc_info: + resolve_project_alias(ctx, formatter, explicit=None) + assert exc_info.value.exit_code == 5 + formatter.error.assert_called_once() + + +class TestApplyFirewallFlags: + """Tests for cli.apply_firewall_flags (session-only policy merge).""" + + def test_no_flags_returns_persisted_as_is(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + from keboola_agent_cli.models import PermissionPolicy + + persisted = PermissionPolicy(mode="allow", allow=[], deny=["branch.delete"]) + result = apply_firewall_flags(persisted, deny_writes=False, deny_destructive=False) + assert result is persisted + + def test_no_flags_no_persisted_returns_none(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + + result = apply_firewall_flags(None, deny_writes=False, deny_destructive=False) + assert result is None + + def test_deny_writes_synthesizes_fresh_policy(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + + result = apply_firewall_flags(None, deny_writes=True, deny_destructive=False) + assert result is not None + assert result.mode == "allow" + assert "cli:write" in result.deny + assert "tool:write" in result.deny + + def test_deny_destructive_synthesizes_fresh_policy(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + + result = apply_firewall_flags(None, deny_writes=False, deny_destructive=True) + assert result is not None + assert "cli:destructive" in result.deny + assert "tool:destructive" in result.deny + assert "cli:write" not in result.deny + + def test_flags_merge_with_persisted_deny_no_duplicates(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + from keboola_agent_cli.models import PermissionPolicy + + persisted = PermissionPolicy(mode="allow", allow=[], deny=["branch.delete", "cli:write"]) + result = apply_firewall_flags(persisted, deny_writes=True, deny_destructive=False) + assert result is not None + # Existing cli:write preserved (no dup); tool:write appended; custom entry kept. + assert result.deny.count("cli:write") == 1 + assert "tool:write" in result.deny + assert "branch.delete" in result.deny + # Mode preserved. + assert result.mode == persisted.mode + + def test_flags_preserve_persisted_mode_deny_mode(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + from keboola_agent_cli.models import PermissionPolicy + + persisted = PermissionPolicy(mode="deny", allow=["cli:read"], deny=[]) + result = apply_firewall_flags(persisted, deny_writes=True, deny_destructive=False) + assert result is not None + assert result.mode == "deny" + assert result.allow == ["cli:read"] + assert "cli:write" in result.deny + + def test_both_flags_combine(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + + result = apply_firewall_flags(None, deny_writes=True, deny_destructive=True) + assert result is not None + # Both prefixes present. + assert {"cli:write", "tool:write", "cli:destructive", "tool:destructive"} <= set( + result.deny + ) + + def test_flags_do_not_mutate_persisted(self) -> None: + from keboola_agent_cli.cli import apply_firewall_flags + from keboola_agent_cli.models import PermissionPolicy + + persisted = PermissionPolicy(mode="allow", allow=[], deny=["branch.delete"]) + before = list(persisted.deny) + apply_firewall_flags(persisted, deny_writes=True, deny_destructive=True) + assert persisted.deny == before, "persisted.deny was mutated in place" diff --git a/tests/test_permissions_cli.py b/tests/test_permissions_cli.py index deb7f4a9..8f422108 100644 --- a/tests/test_permissions_cli.py +++ b/tests/test_permissions_cli.py @@ -76,6 +76,44 @@ def test_list_filter_by_category(self, tmp_path: Path) -> None: for op in data: assert op["category"] == "destructive" + def test_list_reflects_session_deny_writes(self, tmp_path: Path) -> None: + """--deny-writes must flip 'allowed' -> 'denied' on write ops in the table.""" + store = _make_store(tmp_path) + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = store + result = runner.invoke(app, ["--deny-writes", "--json", "permissions", "list"]) + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + by_name = {op["name"]: op for op in data} + + # Write-classified op should show denied under session --deny-writes. + assert by_name["project.add"]["status"] == "denied", ( + "permissions list ignored session --deny-writes" + ) + # Read-classified op should still be allowed. + assert by_name["project.list"]["status"] == "allowed" + + def test_list_includes_project_use_and_current(self, tmp_path: Path) -> None: + """Registry entries for the PR5 commands are visible to the engine. + + Guards against a future refactor that accidentally drops them from + OPERATION_REGISTRY -- unregistered commands would silently default + to 'write' for category matching, which is fail-closed but invisible. + """ + store = _make_store(tmp_path) + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = store + result = runner.invoke(app, ["--json", "permissions", "list"]) + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + by_name = {op["name"]: op for op in data} + + assert "project.use" in by_name, "project.use missing from permissions list" + assert by_name["project.use"]["category"] == "write" + + assert "project.current" in by_name, "project.current missing from permissions list" + assert by_name["project.current"]["category"] == "read" + class TestPermissionsShow: """Tests for `kbagent permissions show`.""" @@ -102,6 +140,60 @@ def test_show_with_policy(self, tmp_path: Path) -> None: assert "cli:write" in data["deny"] assert "tool:write" in data["deny"] + def test_show_reports_session_flags_without_persisted_policy(self, tmp_path: Path) -> None: + """--deny-writes with no persisted policy must still report 'active'.""" + store = _make_store(tmp_path) + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = store + result = runner.invoke(app, ["--deny-writes", "--json", "permissions", "show"]) + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert data["active"] is True, ( + "permissions show reported inactive despite --deny-writes being set" + ) + assert data["persisted"] is None + assert "--deny-writes" in data["session_flags"] + # Defensive: legacy top-level keys (mode/allow/deny) must NOT be + # present when there is no persisted policy. Downstream consumers + # that historically used `data["mode"]` should KeyError here and + # be forced to read the new `session_flags` / `persisted` shape. + assert "mode" not in data + assert "allow" not in data + assert "deny" not in data + + def test_show_reports_session_flags_alongside_persisted(self, tmp_path: Path) -> None: + """Session flags are reported in addition to persisted policy.""" + policy = PermissionPolicy(mode="allow", deny=["branch.delete"]) + store = _make_store(tmp_path, policy) + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = store + result = runner.invoke( + app, + [ + "--deny-writes", + "--deny-destructive", + "--json", + "permissions", + "show", + ], + ) + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + # Persisted policy still present + legacy top-level keys still there. + assert data["persisted"]["deny"] == ["branch.delete"] + assert data["deny"] == ["branch.delete"] # legacy consumer compatibility + # Session flags both reported. + assert data["session_flags"] == ["--deny-writes", "--deny-destructive"] + + def test_show_human_mode_no_policy_no_session(self, tmp_path: Path) -> None: + """Bare 'permissions show' still prints the legacy empty-state message.""" + store = _make_store(tmp_path) + with patch("keboola_agent_cli.cli.ConfigStore") as MockStore: + MockStore.return_value = store + result = runner.invoke(app, ["permissions", "show"]) + assert result.exit_code == 0 + assert "No permission policy configured" in result.output + class TestPermissionsSet: """Tests for `kbagent permissions set`.""" diff --git a/tests/test_repl.py b/tests/test_repl.py index 753a4ee9..e259609c 100644 --- a/tests/test_repl.py +++ b/tests/test_repl.py @@ -120,3 +120,132 @@ def test_no_args_non_tty_shows_help(self) -> None: assert result.exit_code == 0 assert "Commands" in result.output or "kbagent" in result.output + + +class TestReplFirewallPropagation: + """REPL must forward --deny-writes / --deny-destructive to every inner command. + + Dropping the flags when re-invoking inside the REPL would silently + restore write access for a user who started the session with a firewall. + """ + + def test_deny_writes_appended_to_inner_argv(self, monkeypatch) -> None: + """When deny_writes=True, the REPL rebuilds argv with --deny-writes.""" + from keboola_agent_cli.commands import repl as repl_module + + captured: list[list[str]] = [] + + class _FakeClickApp: + def __call__(self, argv, standalone_mode=False): + captured.append(list(argv)) + raise SystemExit(0) + + monkeypatch.setattr(repl_module.typer.main, "get_command", lambda _app: _FakeClickApp()) + + class _FakeSession: + def __init__(self, **_kwargs): + self._replies = iter(["project list", ""]) + + def prompt(self, _text): + try: + return next(self._replies) + except StopIteration as exc: + raise EOFError from exc + + monkeypatch.setattr(repl_module, "PromptSession", _FakeSession) + + repl_module._run_repl( + json_mode=True, + verbose=False, + no_color=True, + config_dir=None, + deny_writes=True, + deny_destructive=False, + ) + + # Every captured invocation must carry --deny-writes in the rebuilt argv. + assert captured, "REPL did not dispatch any commands" + for argv in captured: + assert "--deny-writes" in argv, ( + f"REPL dropped --deny-writes when rebuilding argv: {argv}" + ) + + def test_deny_destructive_appended_to_inner_argv(self, monkeypatch) -> None: + """When deny_destructive=True, the REPL rebuilds argv with --deny-destructive.""" + from keboola_agent_cli.commands import repl as repl_module + + captured: list[list[str]] = [] + + class _FakeClickApp: + def __call__(self, argv, standalone_mode=False): + captured.append(list(argv)) + raise SystemExit(0) + + monkeypatch.setattr(repl_module.typer.main, "get_command", lambda _app: _FakeClickApp()) + + class _FakeSession: + def __init__(self, **_kwargs): + self._replies = iter(["storage delete-table --project p --table-id t"]) + + def prompt(self, _text): + try: + return next(self._replies) + except StopIteration as exc: + raise EOFError from exc + + monkeypatch.setattr(repl_module, "PromptSession", _FakeSession) + + repl_module._run_repl( + json_mode=False, + verbose=False, + no_color=True, + config_dir=None, + deny_writes=False, + deny_destructive=True, + ) + + assert captured + for argv in captured: + assert "--deny-destructive" in argv, ( + f"REPL dropped --deny-destructive when rebuilding argv: {argv}" + ) + + def test_no_duplicate_when_user_retypes_flag(self, monkeypatch) -> None: + """If user types the flag inside the REPL, we must not double it.""" + from keboola_agent_cli.commands import repl as repl_module + + captured: list[list[str]] = [] + + class _FakeClickApp: + def __call__(self, argv, standalone_mode=False): + captured.append(list(argv)) + raise SystemExit(0) + + monkeypatch.setattr(repl_module.typer.main, "get_command", lambda _app: _FakeClickApp()) + + class _FakeSession: + def __init__(self, **_kwargs): + self._replies = iter(["--deny-writes project list"]) + + def prompt(self, _text): + try: + return next(self._replies) + except StopIteration as exc: + raise EOFError from exc + + monkeypatch.setattr(repl_module, "PromptSession", _FakeSession) + + repl_module._run_repl( + json_mode=False, + verbose=False, + no_color=True, + config_dir=None, + deny_writes=True, + deny_destructive=False, + ) + + assert captured + argv = captured[0] + assert argv.count("--deny-writes") == 1, ( + f"REPL duplicated --deny-writes when user also typed it: {argv}" + ) diff --git a/tests/test_services.py b/tests/test_services.py index 33f64572..abc310a0 100644 --- a/tests/test_services.py +++ b/tests/test_services.py @@ -519,6 +519,181 @@ def _make_list_components_client( ] +class TestUseAndCurrentProject: + """Tests for ProjectService.use_project() / current_project() / resolve_pinned_alias().""" + + def _seed_two(self, tmp_config_dir: Path) -> ConfigStore: + store = ConfigStore(config_dir=tmp_config_dir) + for alias, pid in (("prod", 1), ("stage", 2)): + store.add_project( + alias, + ProjectConfig( + stack_url="https://connection.keboola.com", + token=f"901-x-{alias}", + project_name=alias.title(), + project_id=pid, + ), + ) + return store + + def test_use_project_pins_and_persists(self, tmp_config_dir: Path) -> None: + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + result = service.use_project(alias="stage") + + assert result["alias"] == "stage" + assert result["previous"] == "prod" + assert result["source"] == "pin" + # Persistence check + assert ConfigStore(config_dir=tmp_config_dir).load().default_project == "stage" + + def test_use_project_unknown_raises(self, tmp_config_dir: Path) -> None: + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="not found"): + service.use_project(alias="does-not-exist") + + def test_current_project_pin_only(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + result = service.current_project() + assert result["alias"] == "prod" + assert result["source"] == "pin" + assert result["env_override"] is None + + def test_current_project_env_override(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.setenv("KBAGENT_PROJECT", "stage") + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + result = service.current_project() + assert result["alias"] == "stage" + assert result["source"] == "env" + assert result["pinned"] == "prod" + assert result["env_points_to_configured_project"] is True + + def test_current_project_env_unknown(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.setenv("KBAGENT_PROJECT", "mystery") + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + result = service.current_project() + assert result["alias"] == "mystery" + assert result["env_points_to_configured_project"] is False + + def test_current_project_no_pin_no_env(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + # Empty store -- no pin possible + store = ConfigStore(config_dir=tmp_config_dir) + service = ProjectService(config_store=store) + + result = service.current_project() + assert result["alias"] is None + assert result["source"] == "none" + + # ── resolve_pinned_alias precedence ──────────────────────────────── + + def test_resolve_explicit_wins(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.setenv("KBAGENT_PROJECT", "stage") + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + alias, source = service.resolve_pinned_alias(explicit="prod") + assert alias == "prod" + assert source == "explicit" + + def test_resolve_env_beats_pin(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.setenv("KBAGENT_PROJECT", "stage") + store = self._seed_two(tmp_config_dir) + # default is prod (first added) + service = ProjectService(config_store=store) + + alias, source = service.resolve_pinned_alias() + assert alias == "stage" + assert source == "env" + + def test_resolve_pin_used(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + + alias, source = service.resolve_pinned_alias() + assert alias == "prod" + assert source == "pin" + + def test_resolve_sole_project_fallback(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = ConfigStore(config_dir=tmp_config_dir) + store.add_project( + "only", + ProjectConfig( + stack_url="https://connection.keboola.com", + token="t", + project_name="Only", + project_id=7, + ), + ) + # Clear the pin to exercise the sole-project fallback. + cfg = store.load() + cfg.default_project = "" + store.save(cfg) + + service = ProjectService(config_store=store) + alias, source = service.resolve_pinned_alias() + assert alias == "only" + assert source == "sole" + + def test_resolve_fail_hard_multi_no_pin(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = self._seed_two(tmp_config_dir) + cfg = store.load() + cfg.default_project = "" + store.save(cfg) + + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="Multiple projects"): + service.resolve_pinned_alias() + + def test_resolve_explicit_unknown_raises(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="not found"): + service.resolve_pinned_alias(explicit="ghost") + + def test_resolve_env_unknown_raises(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.setenv("KBAGENT_PROJECT", "mystery") + store = self._seed_two(tmp_config_dir) + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="not registered"): + service.resolve_pinned_alias() + + def test_resolve_no_projects_raises(self, tmp_config_dir: Path, monkeypatch) -> None: + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = ConfigStore(config_dir=tmp_config_dir) + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="No projects configured"): + service.resolve_pinned_alias() + + def test_resolve_pinned_alias_points_to_unregistered( + self, tmp_config_dir: Path, monkeypatch + ) -> None: + """Stale pin (pointing at deleted project) raises a repair-friendly ConfigError.""" + monkeypatch.delenv("KBAGENT_PROJECT", raising=False) + store = self._seed_two(tmp_config_dir) + # Hand-edit default_project to a value that isn't in projects. + cfg = store.load() + cfg.default_project = "ghost" + store.save(cfg) + + service = ProjectService(config_store=store) + with pytest.raises(ConfigError, match="not registered"): + service.resolve_pinned_alias() + + class TestConfigServiceListConfigs: """Tests for ConfigService.list_configs().""" From bb1ad2c0ee2611246d238278725cdab652d231f6 Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:05:17 +0200 Subject: [PATCH 2/7] feat(0.22.0): flow CRUD + schedule binding (#200) New: kbagent flow list/detail/schema/new/update/delete/schedule/schedule-remove covering both keboola.orchestrator (classic) and keboola.flow (new) component types. Schedules stored as keboola.scheduler Storage API configs; client-side Kahn DAG validation rejects cycles and unknown phase refs before API write. Max's review fixes: --dry-run on delete + schedule-remove, nested task shape matching keboola-as-code convention, narrow except tuples, gotchas docs. --- CLAUDE.md | 9 + plugins/kbagent/skills/kbagent/SKILL.md | 8 + .../kbagent/references/commands-reference.md | 10 + .../kbagent/references/flow-workflow.md | 110 +++ .../skills/kbagent/references/gotchas.md | 27 + src/keboola_agent_cli/changelog.py | 8 + src/keboola_agent_cli/cli.py | 8 + src/keboola_agent_cli/commands/context.py | 32 + src/keboola_agent_cli/commands/flow.py | 815 ++++++++++++++++++ .../hints/definitions/__init__.py | 1 + .../hints/definitions/flow.py | 286 ++++++ src/keboola_agent_cli/permissions.py | 9 + .../services/flow_service.py | 618 +++++++++++++ tests/test_e2e.py | 244 ++++++ tests/test_flow_cli.py | 775 +++++++++++++++++ tests/test_flow_service.py | 517 +++++++++++ uv.lock | 2 +- 17 files changed, 3478 insertions(+), 1 deletion(-) create mode 100644 plugins/kbagent/skills/kbagent/references/flow-workflow.md create mode 100644 src/keboola_agent_cli/commands/flow.py create mode 100644 src/keboola_agent_cli/hints/definitions/flow.py create mode 100644 src/keboola_agent_cli/services/flow_service.py create mode 100644 tests/test_flow_cli.py create mode 100644 tests/test_flow_service.py diff --git a/CLAUDE.md b/CLAUDE.md index e355b962..8f6765b9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -331,6 +331,15 @@ kbagent kai ask --message "question" [--project NAME] kbagent kai chat --message "msg" [--chat-id ID] [--project NAME] kbagent kai history [--project NAME] [--limit N] +kbagent flow list [--project NAME] [--branch ID] +kbagent flow detail --project NAME --flow-id ID [--component-id keboola.orchestrator|keboola.flow] [--branch ID] +kbagent flow schema +kbagent flow new --project NAME --name NAME [--component-id keboola.orchestrator|keboola.flow] [--description D] [--file @path.yaml|-|JSON] [--branch ID] +kbagent flow update --project NAME --flow-id ID [--component-id ID] [--name N] [--description D] [--file @path.yaml|-|JSON] [--branch ID] +kbagent flow delete --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes] +kbagent flow schedule --project NAME --flow-id ID --cron "0 6 * * *" [--component-id ID] [--timezone TZ] [--disabled] [--branch ID] +kbagent flow schedule-remove --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes] + kbagent context kbagent init [--from-global] kbagent doctor [--fix] diff --git a/plugins/kbagent/skills/kbagent/SKILL.md b/plugins/kbagent/skills/kbagent/SKILL.md index f2569306..a07f1140 100644 --- a/plugins/kbagent/skills/kbagent/SKILL.md +++ b/plugins/kbagent/skills/kbagent/SKILL.md @@ -135,6 +135,14 @@ When working inside a git repository or project directory, run `kbagent init` (o | Ask Kai a one-shot question and get the full response | `kbagent kai ask --message MESSAGE` | | Send a message to Kai in a chat session | `kbagent kai chat --message MESSAGE` | | List recent Kai chat sessions | `kbagent kai history` | +| List all flows (keboola.orchestrator + keboola.flow) across projects | `kbagent flow list` | +| Show detailed flow information including phases and tasks | `kbagent flow detail --project PROJECT --flow-id FLOW-ID` | +| Print the YAML format expected by 'flow new' and 'flow update' | `kbagent flow schema` | +| Create a new flow configuration | `kbagent flow new --project PROJECT --name NAME` | +| Update a flow's name, description, or phases/tasks | `kbagent flow update --project PROJECT --flow-id FLOW-ID` | +| Delete a flow configuration | `kbagent flow delete --project PROJECT --flow-id FLOW-ID` | +| Bind a cron schedule to a flow (upsert: creates or updates) | `kbagent flow schedule --project PROJECT --flow-id FLOW-ID --cron CRON` | +| Remove all schedules bound to a flow (deletes keboola.scheduler configs) | `kbagent flow schedule-remove --project PROJECT --flow-id FLOW-ID` | | List development branches from connected projects | `kbagent branch list` | | Create a new development branch and auto-activate it | `kbagent branch create --project PROJECT --name NAME` | | Set an existing development branch as active | `kbagent branch use --project PROJECT --branch BRANCH` | diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 3f3f2aae..8c5cc142 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -115,6 +115,16 @@ All commands support `--json` for structured output. Multi-project flags (`--pro - `kai chat --message "msg" [--chat-id ID] [--project NAME]` -- send message in a chat session, returns chat_id for continuation - `kai history [--project NAME] [--limit N]` -- list recent Kai chat sessions (default limit: 10) +## Flows (Orchestrator) +- `flow list [--project NAME] [--branch ID]` -- list all flows (keboola.orchestrator + keboola.flow) across one or all projects +- `flow detail --project NAME --flow-id ID [--component-id keboola.orchestrator|keboola.flow] [--branch ID]` -- full phase/task breakdown; groups tasks by phase, lists orphan tasks +- `flow schema` -- print YAML template for flow configuration (phases + tasks); use with `--file @-` or save to a file +- `flow new --project NAME --name NAME [--component-id keboola.orchestrator|keboola.flow] [--description D] [--file @path.yaml|-|JSON] [--branch ID]` -- create a flow; DAG validated before API call; default component: keboola.flow +- `flow update --project NAME --flow-id ID [--component-id ID] [--name N] [--description D] [--file @path.yaml|-|JSON] [--branch ID]` -- update name, description, or phases/tasks; requires at least one of --name/--description/--file +- `flow delete --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes]` -- delete a flow config (confirmation guard) +- `flow schedule --project NAME --flow-id ID --cron "0 6 * * *" [--component-id ID] [--timezone TZ] [--disabled] [--branch ID]` -- attach a cron schedule (stored as keboola.scheduler config); replaces any existing schedule +- `flow schedule-remove --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes]` -- remove all cron schedules attached to a flow; idempotent + ## Sync (GitOps) - `sync init --project ALIAS [--directory DIR] [--git-branching]` -- initialize sync working directory - `sync pull --project ALIAS [--all-projects] [--force] [--dry-run] [--with-samples] [--no-storage] [--no-jobs] [--job-limit N]` -- download configs to local files. For large projects (>100 configs), automatically fetches jobs per-config when the grouped API limit is insufficient diff --git a/plugins/kbagent/skills/kbagent/references/flow-workflow.md b/plugins/kbagent/skills/kbagent/references/flow-workflow.md new file mode 100644 index 00000000..4b2e2715 --- /dev/null +++ b/plugins/kbagent/skills/kbagent/references/flow-workflow.md @@ -0,0 +1,110 @@ +# Flow Workflow + +Flows orchestrate Keboola transformations and extractors in a directed acyclic graph (DAG) of phases and tasks. kbagent supports two flow component types: `keboola.orchestrator` (classic) and `keboola.flow` (new format). + +## Core concepts + +- **Phase**: a named stage with `id` and `dependsOn` (list of upstream phase IDs). Phases with no `dependsOn` run first. +- **Task**: a unit of work referencing a component config, assigned to a phase via `phase` field. +- **Schedule**: stored as a `keboola.scheduler` config that targets the flow; not part of the flow config itself. + +## Quick start: create a flow + +```bash +# 1. See the template +kbagent flow schema + +# 2. Create a simple flow from YAML +cat > flow.yaml <<'EOF' +phases: + - id: 1 + name: Extract + dependsOn: [] + - id: 2 + name: Transform + dependsOn: [1] +tasks: + - id: 1 + name: Run extractor + phase: 1 + task: + mode: run + componentId: keboola.ex-db-snowflake + configId: "123456" + - id: 2 + name: Run transformation + phase: 2 + task: + mode: run + componentId: keboola.snowflake-transformation + configId: "789012" +EOF + +kbagent --json flow new --project prod --name "Daily ETL" --file @flow.yaml +``` + +## List and inspect flows + +```bash +# All flows across all projects +kbagent --json flow list + +# Flows in one project +kbagent --json flow list --project prod + +# Full phase/task breakdown +kbagent --json flow detail --project prod --flow-id 111 +``` + +## Update a flow + +```bash +# Rename only +kbagent --json flow update --project prod --flow-id 111 --name "New Name" + +# Replace phases/tasks from file (validates DAG before write) +kbagent --json flow update --project prod --flow-id 111 --file @updated.yaml +``` + +## Schedule a flow + +Schedules are stored as `keboola.scheduler` configs pointing at the flow. `flow schedule` is an upsert — if a schedule already exists for the flow it is updated in-place; otherwise a new one is created. Calling it twice with different cron expressions replaces the existing schedule. + +```bash +# Daily at 06:00 UTC +kbagent --json flow schedule --project prod --flow-id 111 --cron "0 6 * * *" + +# With timezone and disabled state +kbagent --json flow schedule \ + --project prod --flow-id 111 \ + --cron "0 8 * * 1-5" \ + --timezone "Europe/Prague" \ + --disabled + +# Remove all schedules (idempotent) +kbagent --json flow schedule-remove --project prod --flow-id 111 --yes +``` + +## Delete a flow + +```bash +kbagent --json flow delete --project prod --flow-id 111 --yes +``` + +## DAG validation + +kbagent validates the phase graph client-side before every create/update: +- Unknown `dependsOn` phase IDs → `INVALID_FLOW_DAG` +- Tasks referencing unknown phase IDs → `INVALID_FLOW_DAG` +- Cycles in the phase graph → `INVALID_FLOW_DAG` + +The error carries a list of human-readable violation messages. + +## Component IDs + +| Component | Use case | +|---|---| +| `keboola.flow` | New projects, preferred for new flows (default for `flow new`) | +| `keboola.orchestrator` | Legacy flows; most existing orchestrations use this (default for `flow detail/update/delete/schedule`) | + +Both are fully supported. Use `--component-id` to override the default. diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index 08db2036..500da6ec 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -557,3 +557,30 @@ CLI hides via its four-bucket response, but they matter when interpreting result addressing. When exporting multiple tables, each ends up in a predictable subdirectory and there is no risk of name collisions. Override with `--output DIR` if you need a custom location. + +## Flow: default `--component-id` differs between commands + +- `kbagent flow new` defaults to **`keboola.flow`** (the newer format). +- `kbagent flow detail / update / delete / schedule / schedule-remove` all + default to **`keboola.orchestrator`** (the legacy format, since most + existing flows still use it). +- Consequence: if you create a flow with `flow new` and then call + `flow detail` without `--component-id`, you will get a `NOT_FOUND` error + because kbagent looks up the ID under `keboola.orchestrator`. Always pass + `--component-id keboola.flow` when round-tripping a flow you just created + via `flow new` (or, equivalently, pass `--component-id keboola.orchestrator` + on `flow new` to keep things consistent). +- `flow list` returns both component IDs and surfaces `component_id` on each + row — use it to confirm which variant a flow lives under before issuing + detail/update/delete/schedule commands. + +## Flow: `schedule` is an upsert (no `schedule-update`) + +- `kbagent flow schedule` creates a `keboola.scheduler` config on first run + and **updates the existing one in-place** on subsequent runs. Running it + twice with different `--cron` values replaces the schedule — it does not + create a second one. That's why there is no separate `flow schedule-update` + command. +- To inspect or remove schedules: `kbagent flow schedule-remove` deletes all + scheduler configs that target the flow. Pair it with `--dry-run` to see the + affected configs (cron + timezone) without calling `delete_config`. diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 8c3cdb75..21a1226d 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -15,6 +15,14 @@ "New: top-level `--deny-writes` / `--deny-destructive` flags synthesize a session-only firewall that merges with any persisted permission policy. Never written to config.json. `--deny-writes` blocks the wide net (write+destructive+admin); `--deny-destructive` is narrower and blocks only data destruction.", "New: `ProjectService.resolve_pinned_alias()` plus `commands._helpers.resolve_project_alias()` -- single-project alias resolution contract for write/destructive commands. Public API for future PRs to adopt; FIIA P0-4 acceptance criterion.", "Fix: stale pin (default_project pointing at a deleted alias) now raises a repair-friendly CONFIG_ERROR with `kbagent project use ` guidance instead of silently fanning out.", + "New: `kbagent flow list` -- list all flows (keboola.orchestrator + keboola.flow) across one or all projects; supports --project, --branch", + "New: `kbagent flow detail` -- full phase/task breakdown for a single flow config, including phase dependency graph and orphan detection", + "New: `kbagent flow schema` -- print the YAML template for flow configuration (phases + tasks) for use with --file", + "New: `kbagent flow new` -- create a flow with optional phases/tasks from a YAML/JSON --file; validates DAG before create", + "New: `kbagent flow update` -- update flow name, description, or phases/tasks; validates DAG before write; fetches current config before partial update", + "New: `kbagent flow delete` -- delete a flow config with --yes confirmation guard", + "New: `kbagent flow schedule` -- attach a cron schedule via keboola.scheduler; supports timezone and enabled/disabled state", + "New: `kbagent flow schedule-remove` -- remove all cron schedules attached to a flow; idempotent, --yes confirmation guard", ], "0.21.2": [ "Fix: `kbagent config search` now scans `rows[].configuration` in addition to the top-level configuration body (#196) -- queries like `--query '\"incremental\": false'` previously returned zero matches for row-based components (Snowflake/MySQL/BigQuery writers, DB extractors, Google Sheets) because the service only fetched `include=configuration`; match paths are now reported as `rows[N].configuration.parameters.`", diff --git a/src/keboola_agent_cli/cli.py b/src/keboola_agent_cli/cli.py index 8a685a37..10241d8e 100644 --- a/src/keboola_agent_cli/cli.py +++ b/src/keboola_agent_cli/cli.py @@ -13,6 +13,7 @@ from .commands.context import context_command from .commands.doctor import doctor_command from .commands.encrypt import encrypt_app +from .commands.flow import flow_app from .commands.init import init_command from .commands.job import job_app from .commands.kai import kai_app @@ -39,6 +40,7 @@ from .services.deep_lineage_service import DeepLineageService from .services.doctor_service import DoctorService from .services.encrypt_service import EncryptService +from .services.flow_service import FlowService from .services.job_service import JobService from .services.kai_service import KaiService from .services.lineage_service import LineageService @@ -84,6 +86,10 @@ app.add_typer(lineage_app, name="lineage", rich_help_panel=_BROWSE) app.add_typer(kai_app, name="kai", rich_help_panel=_BROWSE) +# -- Flows -- +_FLOWS = "Flows" +app.add_typer(flow_app, name="flow", rich_help_panel=_FLOWS) + # -- Development -- _DEV = "Development" app.add_typer(branch_app, name="branch", rich_help_panel=_DEV) @@ -260,6 +266,7 @@ def main( sync_service = SyncService(config_store=config_store) variables_service = VariablesService(config_store=config_store) encrypt_service = EncryptService(config_store=config_store) + flow_service = FlowService(config_store=config_store) workspace_service = WorkspaceService(config_store=config_store) kai_service = KaiService(config_store=config_store) doctor_service = DoctorService(config_store=config_store, mcp_service=mcp_service) @@ -310,6 +317,7 @@ def main( ctx.obj["sync_service"] = sync_service ctx.obj["variables_service"] = variables_service ctx.obj["encrypt_service"] = encrypt_service + ctx.obj["flow_service"] = flow_service ctx.obj["workspace_service"] = workspace_service ctx.obj["kai_service"] = kai_service ctx.obj["doctor_service"] = doctor_service diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index ad492c46..105a57da 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -315,6 +315,38 @@ Use --org-id OR --project-ids (at least one required). Token via KBC_MANAGE_API_TOKEN env var or interactive prompt. +### Flows (Orchestrator + Conditional) + + kbagent flow list [--project NAME] [--branch ID] + List all flows (keboola.orchestrator + keboola.flow) across projects. + + kbagent flow detail --project NAME --flow-id ID [--component-id keboola.orchestrator|keboola.flow] [--branch ID] + Show phases, tasks, and full configuration. --component-id defaults to keboola.orchestrator. + + kbagent flow schema + Print the YAML format accepted by 'flow new' and 'flow update'. + + kbagent flow new --project NAME --name "Name" [--component-id keboola.orchestrator|keboola.flow] [--description D] [--file YAML|@file|-] [--branch ID] + Create a new flow. --component-id defaults to keboola.flow (newer format). + --file accepts YAML with 'phases' and 'tasks' keys. DAG is validated (acyclic, refs exist). + + kbagent flow update --project NAME --flow-id ID [--component-id ID] [--name N] [--description D] [--file YAML] [--branch ID] + Update a flow's name, description, or phases/tasks. --file replaces both phases and tasks. + Omitting --file leaves the flow body unchanged. DAG re-validated on write. + + kbagent flow delete --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes] + Delete a flow. Does NOT remove associated keboola.scheduler configs. + Run 'flow schedule-remove' first if you want to clean up schedules. + + kbagent flow schedule --project NAME --flow-id ID --cron "0 6 * * *" [--component-id ID] [--timezone TZ] [--enabled/--disabled] [--name NAME] [--branch ID] + Upsert a cron schedule: updates the existing keboola.scheduler config if one exists, creates one + otherwise. Calling twice with a new cron replaces the old schedule — no duplicates created. + Schedules are stored as Storage API configs, not a separate scheduler service. + + kbagent flow schedule-remove --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes] + Remove all schedules bound to this flow (deletes all matching keboola.scheduler configs). + Idempotent: safe to run when no schedules exist. + ### Development Branches kbagent branch list [--project NAME] diff --git a/src/keboola_agent_cli/commands/flow.py b/src/keboola_agent_cli/commands/flow.py new file mode 100644 index 00000000..c14ccb11 --- /dev/null +++ b/src/keboola_agent_cli/commands/flow.py @@ -0,0 +1,815 @@ +"""Flow commands -- list, detail, new, update, delete, schedule, schedule-remove, schema. + +Thin CLI layer: parses arguments, calls FlowService, formats output. +No business logic belongs here. +""" + +from __future__ import annotations + +import logging +import sys +from pathlib import Path +from typing import Any + +import typer +import yaml +from rich.markup import escape +from rich.table import Table + +from ..errors import ConfigError, KeboolaApiError +from ._helpers import ( + check_cli_permission, + emit_hint, + get_formatter, + get_service, + map_error_to_exit_code, + resolve_branch, + should_hint, +) + +logger = logging.getLogger(__name__) + +flow_app = typer.Typer(help="Manage flows (keboola.orchestrator + keboola.flow)") + +_FLOW_COMPONENT_CHOICES = ["keboola.orchestrator", "keboola.flow"] + +# YAML/JSON schema snippet shown by 'flow schema' +# +# Tasks use the nested ``task: {mode, componentId, configId}`` form that matches +# the keboola-as-code convention. The API also accepts the flat form +# (``componentId``/``configId`` at task root) for backward compatibility, but +# new flows should use the nested form shown below. +_FLOW_SCHEMA = """\ +# kbagent flow schema -- keboola.flow configuration format +# +# Create with: kbagent flow new --project ALIAS --name "My Flow" [--file flow.yaml] +# Update with: kbagent flow update --project ALIAS --flow-id ID --file flow.yaml + +name: "My Flow" +description: "Optional description" + +phases: + - id: 1 + name: "Phase 1 - Extract" + dependsOn: [] # IDs of phases that must complete first + - id: 2 + name: "Phase 2 - Transform" + dependsOn: [1] + +tasks: + - id: 1 + name: "Extract Data" + phase: 1 # phase.id this task belongs to + enabled: true + continueOnFailure: false + task: + mode: run + componentId: "keboola.ex-http" + configId: "123456789" + - id: 2 + name: "Run Transformation" + phase: 2 + enabled: true + continueOnFailure: false + task: + mode: run + componentId: "keboola.snowflake-transformation" + configId: "987654321" + +# Notes: +# - dependsOn: IDs form a directed acyclic graph (kbagent validates this) +# - task.configId values must be string IDs of existing configs in the project +# - task.mode defaults to "run" (the only supported value today) +# - For keboola.orchestrator (legacy), phases are referenced by name (string), +# not ID (integer); use keboola.flow for new flows +# - The flat shape (componentId/configId at task root) is still accepted by +# the API but is deprecated in the schema; prefer the nested task: form +""" + + +@flow_app.callback(invoke_without_command=True) +def _flow_permission_check(ctx: typer.Context) -> None: + check_cli_permission(ctx, "flow") + + +# --------------------------------------------------------------------------- +# flow list +# --------------------------------------------------------------------------- + + +@flow_app.command("list") +def flow_list( + ctx: typer.Context, + project: list[str] | None = typer.Option( + None, + "--project", + help="Project alias (repeatable for multiple projects; omit for all)", + ), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (per-project; requires single --project)" + ), +) -> None: + """List all flows (keboola.orchestrator + keboola.flow) across projects.""" + if should_hint(ctx): + emit_hint(ctx, "flow.list", project=project, branch=branch) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + config_store = ctx.obj["config_store"] + + if branch is not None and (not project or len(project) != 1): + formatter.error( + message="--branch requires exactly one --project", + error_code="INVALID_ARGUMENT", + ) + raise typer.Exit(code=2) + + effective_branch: int | None = branch + if branch is None and project and len(project) == 1: + _, effective_branch = resolve_branch(config_store, formatter, project[0], None) + + try: + result = service.list_flows(aliases=project, branch_id=effective_branch) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + + if formatter.json_mode: + formatter.output(result) + else: + _format_flows_table(formatter, result) + + +def _format_flows_table(formatter: Any, result: dict[str, Any]) -> None: + flows = result.get("flows", []) + errors = result.get("errors", []) + + if not flows: + formatter.console.print("[dim]No flows found.[/dim]") + else: + tbl = Table( + "Project", + "Component", + "Config ID", + "Name", + "Disabled", + show_header=True, + header_style="bold cyan", + ) + for f in flows: + disabled = "[red]yes[/red]" if f.get("is_disabled") else "[dim]no[/dim]" + tbl.add_row( + escape(f.get("project_alias", "")), + escape(f.get("component_id", "")), + escape(f.get("config_id", "")), + escape(f.get("name", "")), + disabled, + ) + formatter.console.print(tbl) + + for err in errors: + formatter.warning( + f"Project '{err.get('project_alias', '?')}': {err.get('message', 'error')}" + ) + + +# --------------------------------------------------------------------------- +# flow detail +# --------------------------------------------------------------------------- + + +@flow_app.command("detail") +def flow_detail( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + flow_id: str = typer.Option(..., "--flow-id", help="Flow configuration ID"), + component_id: str = typer.Option( + "keboola.orchestrator", + "--component-id", + help="Flow component ID (default: keboola.orchestrator). " + "Use --component-id keboola.flow for flows listed with component_id=keboola.flow.", + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), +) -> None: + """Show detailed flow information including phases and tasks.""" + if should_hint(ctx): + emit_hint( + ctx, + "flow.detail", + project=project, + flow_id=flow_id, + component_id=component_id, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + config_store = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + try: + result = service.get_flow_detail( + alias=project, + component_id=component_id, + config_id=flow_id, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + _format_flow_detail(formatter, result) + + +def _format_flow_detail(formatter: Any, result: dict[str, Any]) -> None: + formatter.console.print( + f"\n[bold]{escape(result.get('name', ''))}[/bold]" + f" [dim]({escape(result.get('component_id', ''))} / {escape(str(result.get('id', '')))})[/dim]" + ) + if result.get("description"): + formatter.console.print(f"[dim]{escape(result['description'])}[/dim]") + if result.get("branch_id"): + formatter.console.print(f"[dim]Branch: {result['branch_id']}[/dim]") + + phases = result.get("phases", []) + tasks = result.get("tasks", []) + + if not phases and not tasks: + formatter.console.print("\n[dim]No phases or tasks defined.[/dim]") + return + + formatter.console.print( + f"\n[bold]Phases[/bold] ({len(phases)}) [bold]Tasks[/bold] ({len(tasks)})" + ) + + # Group tasks by phase + tasks_by_phase: dict[Any, list[dict[str, Any]]] = {} + for task in tasks: + phase_key = task.get("phase") + tasks_by_phase.setdefault(phase_key, []).append(task) + + for phase in phases: + pid = phase.get("id") + deps = phase.get("dependsOn", []) + dep_str = f" ← {deps}" if deps else "" + formatter.console.print( + f"\n [cyan bold]Phase {escape(str(pid))}: {escape(phase.get('name', ''))}[/cyan bold]" + f"[dim]{escape(dep_str)}[/dim]" + ) + for task in tasks_by_phase.get(pid, []): + t_info = task.get("task") or {} + comp = t_info.get("componentId", task.get("componentId", "")) + cfg = t_info.get("configId", task.get("configId", "")) + enabled = "" if task.get("enabled", True) else " [dim](disabled)[/dim]" + formatter.console.print( + f" [{escape(str(task.get('id', '?')))}] {escape(task.get('name', ''))}" + f" [dim]{escape(comp)}/{escape(str(cfg))}[/dim]{enabled}" + ) + + # Orphan tasks (phase not found in phases list) + orphan_phase_keys = set(tasks_by_phase.keys()) - {p.get("id") for p in phases} + for key in sorted(str(k) for k in orphan_phase_keys): + formatter.console.print(f"\n [yellow]Phase '{key}' (not in phases list)[/yellow]") + for task in tasks_by_phase.get(key, []): + formatter.console.print(f" {escape(task.get('name', str(task)))}") + + +# --------------------------------------------------------------------------- +# flow schema +# --------------------------------------------------------------------------- + + +@flow_app.command("schema") +def flow_schema(ctx: typer.Context) -> None: + """Print the YAML format expected by 'flow new' and 'flow update'.""" + formatter = get_formatter(ctx) + if formatter.json_mode: + formatter.output( + { + "format": "yaml", + "description": "keboola.flow configuration schema", + "schema": _FLOW_SCHEMA, + } + ) + else: + from rich.syntax import Syntax + + formatter.console.print(Syntax(_FLOW_SCHEMA, "yaml", theme="monokai", line_numbers=False)) + + +# --------------------------------------------------------------------------- +# flow new +# --------------------------------------------------------------------------- + + +def _load_flow_yaml(raw: str) -> dict[str, Any]: + """Load flow definition from inline JSON, @file, or - (stdin).""" + if raw == "-": + content = sys.stdin.read() + elif raw.startswith("@"): + file_path = Path(raw[1:]) + if not file_path.is_file(): + raise FileNotFoundError(f"File not found: {file_path}") + content = file_path.read_text(encoding="utf-8") + else: + content = raw + + # Try YAML first (superset of JSON) + parsed = yaml.safe_load(content) + if parsed is None: + return {} + if not isinstance(parsed, dict): + raise ValueError( + f"Flow definition must be a YAML/JSON object (mapping), got {type(parsed).__name__}" + ) + return parsed + + +@flow_app.command("new") +def flow_new( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + name: str = typer.Option(..., "--name", help="Flow name"), + component_id: str = typer.Option( + "keboola.flow", + "--component-id", + help="Component ID (default: keboola.flow)", + ), + description: str = typer.Option("", "--description", help="Optional description"), + file: str | None = typer.Option( + None, + "--file", + help="YAML/JSON flow definition (@file, -, or inline). " + "Run 'kbagent flow schema' to see the expected format.", + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), +) -> None: + """Create a new flow configuration. + + \b + Examples: + # Empty skeleton + kbagent flow new --project prod --name "Daily ETL" + + # From a YAML file + kbagent flow new --project prod --name "Daily ETL" --file @flow.yaml + + # Pipe from stdin + cat flow.yaml | kbagent flow new --project prod --name "Daily ETL" --file - + """ + if should_hint(ctx): + emit_hint( + ctx, + "flow.new", + project=project, + name=name, + component_id=component_id, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + + phases: list[dict[str, Any]] = [] + tasks: list[dict[str, Any]] = [] + + if file: + try: + flow_def = _load_flow_yaml(file) + except (OSError, yaml.YAMLError, ValueError) as exc: + formatter.error( + message=f"Cannot load flow definition: {exc}", error_code="VALIDATION_ERROR" + ) + raise typer.Exit(code=2) from None + phases = flow_def.get("phases", []) + tasks = flow_def.get("tasks", []) + + try: + result = service.create_flow( + alias=project, + component_id=component_id, + name=name, + description=description, + phases=phases, + tasks=tasks, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + branch_info = f" (branch {result.get('branch_id')})" if result.get("branch_id") else "" + formatter.success( + f"Created flow '{escape(result.get('name', name))}' " + f"[{escape(component_id)}/{escape(str(result.get('id', '')))}]{branch_info}" + ) + + +# --------------------------------------------------------------------------- +# flow update +# --------------------------------------------------------------------------- + + +@flow_app.command("update") +def flow_update( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + flow_id: str = typer.Option(..., "--flow-id", help="Flow configuration ID"), + component_id: str = typer.Option( + "keboola.orchestrator", + "--component-id", + help="Flow component ID (default: keboola.orchestrator)", + ), + name: str | None = typer.Option(None, "--name", help="New flow name"), + description: str | None = typer.Option(None, "--description", help="New description"), + file: str | None = typer.Option( + None, + "--file", + help="YAML/JSON flow definition to replace phases + tasks (@file, -, or inline)", + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), +) -> None: + """Update a flow's name, description, or phases/tasks. + + \b + Examples: + # Rename only + kbagent flow update --project prod --flow-id 123 --name "New Name" + + # Replace phases + tasks from file + kbagent flow update --project prod --flow-id 123 --file @flow.yaml + """ + if should_hint(ctx): + emit_hint( + ctx, + "flow.update", + project=project, + flow_id=flow_id, + component_id=component_id, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + + phases: list[dict[str, Any]] | None = None + tasks: list[dict[str, Any]] | None = None + + if file: + try: + flow_def = _load_flow_yaml(file) + except (OSError, yaml.YAMLError, ValueError) as exc: + formatter.error( + message=f"Cannot load flow definition: {exc}", error_code="VALIDATION_ERROR" + ) + raise typer.Exit(code=2) from None + phases = flow_def.get("phases") + tasks = flow_def.get("tasks") + + if name is None and description is None and phases is None and tasks is None: + formatter.error( + message="At least one of --name, --description, or --file must be provided.", + error_code="INVALID_ARGUMENT", + ) + raise typer.Exit(code=2) from None + + try: + result = service.update_flow( + alias=project, + component_id=component_id, + config_id=flow_id, + name=name, + description=description, + phases=phases, + tasks=tasks, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + branch_info = f" (branch {result.get('branch_id')})" if result.get("branch_id") else "" + formatter.success( + f"Updated flow '{escape(result.get('name', flow_id))}' " + f"[{escape(component_id)}/{escape(flow_id)}]{branch_info}" + ) + + +# --------------------------------------------------------------------------- +# flow delete +# --------------------------------------------------------------------------- + + +@flow_app.command("delete") +def flow_delete( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + flow_id: str = typer.Option(..., "--flow-id", help="Flow configuration ID"), + component_id: str = typer.Option( + "keboola.orchestrator", + "--component-id", + help="Flow component ID (default: keboola.orchestrator)", + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), + dry_run: bool = typer.Option( + False, + "--dry-run", + help="Show what would be deleted without executing", + ), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"), +) -> None: + """Delete a flow configuration. + + Note: associated keboola.scheduler configs are NOT automatically removed. + Run 'flow schedule-remove' first if you want to clean up schedules. + """ + if should_hint(ctx): + emit_hint( + ctx, + "flow.delete", + project=project, + flow_id=flow_id, + component_id=component_id, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + + if dry_run: + result = { + "would_delete": { + "project_alias": project, + "component_id": component_id, + "config_id": flow_id, + "branch_id": branch, + }, + } + if formatter.json_mode: + formatter.output(result) + else: + formatter.console.print( + f"[bold blue]Would delete:[/bold blue] flow " + f"{escape(component_id)}/{escape(flow_id)}" + + (f" (branch {branch})" if branch else "") + ) + return + + if not yes and not formatter.json_mode: + confirmed = typer.confirm(f"Delete flow {component_id}/{flow_id}?") + if not confirmed: + formatter.console.print("[yellow]Aborted.[/yellow]") + raise typer.Exit(code=0) + + try: + result = service.delete_flow( + alias=project, + component_id=component_id, + config_id=flow_id, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + formatter.success(f"Deleted flow {escape(component_id)}/{escape(flow_id)}") + + +# --------------------------------------------------------------------------- +# flow schedule +# --------------------------------------------------------------------------- + + +@flow_app.command("schedule") +def flow_schedule( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + flow_id: str = typer.Option(..., "--flow-id", help="Flow configuration ID"), + component_id: str = typer.Option( + "keboola.orchestrator", + "--component-id", + help="Flow component ID (default: keboola.orchestrator)", + ), + cron: str = typer.Option(..., "--cron", help="Cron expression (e.g. '0 6 * * *')"), + timezone: str = typer.Option("UTC", "--timezone", help="IANA timezone (default: UTC)"), + enabled: bool = typer.Option(True, "--enabled/--disabled", help="Enable the schedule"), + schedule_name: str | None = typer.Option( + None, "--name", help="Name for the scheduler config (auto-generated if omitted)" + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), +) -> None: + """Bind a cron schedule to a flow (upsert: creates or updates). + + If no schedule exists for this flow a new keboola.scheduler config is + created. If one already exists it is updated in-place — calling this + command a second time will not create duplicates. + + \b + Examples: + # Run daily at 6am UTC + kbagent flow schedule --project prod --flow-id 123 --cron "0 6 * * *" + + # Run hourly, disabled by default + kbagent flow schedule --project prod --flow-id 123 --cron "0 * * * *" --disabled + """ + if should_hint(ctx): + emit_hint( + ctx, + "flow.schedule", + project=project, + flow_id=flow_id, + component_id=component_id, + cron=cron, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + + try: + result = service.set_flow_schedule( + alias=project, + component_id=component_id, + config_id=flow_id, + cron_tab=cron, + timezone=timezone, + enabled=enabled, + schedule_name=schedule_name, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + state_label = "[green]enabled[/green]" if enabled else "[yellow]disabled[/yellow]" + action = result.get("status", "created") + formatter.success(f"Schedule {action}: {escape(cron)} ({escape(timezone)}) — {state_label}") + formatter.console.print( + f" Scheduler config: {escape(result.get('schedule_name', ''))} " + f"[dim](ID: {escape(result.get('schedule_id', ''))})[/dim]" + ) + + +# --------------------------------------------------------------------------- +# flow schedule-remove +# --------------------------------------------------------------------------- + + +def _print_schedule_list(formatter: Any, schedules: list[dict[str, Any]]) -> None: + """Print one line per schedule: state, cron, timezone, id.""" + for s in schedules: + formatter.console.print( + f" [{escape(s.get('state', ''))}] {escape(s.get('cron_tab', ''))} " + f"({escape(s.get('timezone', ''))}) ID={escape(s.get('schedule_id', ''))}" + ) + + +@flow_app.command("schedule-remove") +def flow_schedule_remove( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + flow_id: str = typer.Option(..., "--flow-id", help="Flow configuration ID"), + component_id: str = typer.Option( + "keboola.orchestrator", + "--component-id", + help="Flow component ID (default: keboola.orchestrator)", + ), + branch: int | None = typer.Option(None, "--branch", help="Dev branch ID"), + dry_run: bool = typer.Option( + False, + "--dry-run", + help="List the scheduler configs that would be removed without executing", + ), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"), +) -> None: + """Remove all schedules bound to a flow (deletes keboola.scheduler configs). + + Idempotent: safe to run even if no schedules exist. + """ + if should_hint(ctx): + emit_hint( + ctx, + "flow.schedule-remove", + project=project, + flow_id=flow_id, + component_id=component_id, + branch=branch, + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "flow_service") + + if dry_run: + try: + sched_result = service.list_flow_schedules( + alias=project, + component_id=component_id, + config_id=flow_id, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + schedules = sched_result.get("schedules", []) + payload = { + "would_delete": { + "project_alias": project, + "component_id": component_id, + "config_id": flow_id, + "branch_id": branch, + "schedules": schedules, + "count": len(schedules), + }, + } + if formatter.json_mode: + formatter.output(payload) + else: + if not schedules: + formatter.console.print("[dim]No schedules found for this flow.[/dim]") + else: + formatter.console.print( + f"[bold blue]Would remove {len(schedules)} schedule(s) " + f"from flow[/bold blue] {escape(component_id)}/{escape(flow_id)}:" + ) + _print_schedule_list(formatter, schedules) + return + + # Show existing schedules before confirming + if not yes and not formatter.json_mode: + try: + sched_result = service.list_flow_schedules( + alias=project, + component_id=component_id, + config_id=flow_id, + branch_id=branch, + ) + schedules = sched_result.get("schedules", []) + except (ConfigError, KeboolaApiError): + schedules = [] + + if not schedules: + formatter.console.print("[dim]No schedules found for this flow.[/dim]") + raise typer.Exit(code=0) + + _print_schedule_list(formatter, schedules) + confirmed = typer.confirm(f"Remove {len(schedules)} schedule(s) above?") + if not confirmed: + formatter.console.print("[yellow]Aborted.[/yellow]") + raise typer.Exit(code=0) + + try: + result = service.remove_flow_schedule( + alias=project, + component_id=component_id, + config_id=flow_id, + branch_id=branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + count = result.get("deleted_count", 0) + if count == 0: + formatter.console.print("[dim]No schedules found — nothing removed.[/dim]") + else: + formatter.success(f"Removed {count} schedule(s) from flow {escape(flow_id)}") diff --git a/src/keboola_agent_cli/hints/definitions/__init__.py b/src/keboola_agent_cli/hints/definitions/__init__.py index b1846f25..a0608b31 100644 --- a/src/keboola_agent_cli/hints/definitions/__init__.py +++ b/src/keboola_agent_cli/hints/definitions/__init__.py @@ -5,6 +5,7 @@ component, # noqa: F401 config, # noqa: F401 encrypt, # noqa: F401 + flow, # noqa: F401 job, # noqa: F401 kai, # noqa: F401 lineage, # noqa: F401 diff --git a/src/keboola_agent_cli/hints/definitions/flow.py b/src/keboola_agent_cli/hints/definitions/flow.py new file mode 100644 index 00000000..bf101583 --- /dev/null +++ b/src/keboola_agent_cli/hints/definitions/flow.py @@ -0,0 +1,286 @@ +"""Hint definitions for flow commands.""" + +from .. import HintRegistry +from ..models import ClientCall, CommandHint, HintStep, ServiceCall + +HintRegistry.register( + CommandHint( + cli_command="flow.list", + description="List all flows (keboola.orchestrator + keboola.flow) across projects", + steps=[ + HintStep( + comment="Fetch configs for both flow component IDs", + client=ClientCall( + method="list_component_configs", + args={ + "component_id": "keboola.orchestrator", + "branch_id": "{branch}", + }, + result_var="orchestrator_configs", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="list_flows", + args={ + "aliases": "{project}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Client layer: call list_component_configs for both 'keboola.orchestrator' and " + "'keboola.flow' and merge the results.", + "Service layer returns {'flows': [...], 'errors': [...]}. " + "Each flow dict has project_alias, component_id, config_id, name, description, is_disabled.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.detail", + description="Show detailed flow information including phases and tasks", + steps=[ + HintStep( + comment="Fetch full flow configuration detail", + client=ClientCall( + method="get_config_detail", + args={ + "component_id": "{component_id}", + "config_id": "{flow_id}", + "branch_id": "{branch}", + }, + result_var="detail", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="get_flow_detail", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{flow_id}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "configuration.phases and configuration.tasks hold the flow DAG.", + "Service response adds phase_count, task_count top-level keys.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.new", + description="Create a new flow configuration", + steps=[ + HintStep( + comment="Create a flow config (phases + tasks in configuration body)", + client=ClientCall( + method="create_config", + args={ + "component_id": "{component_id}", + "name": "{name}", + "configuration": '{"phases": [...], "tasks": [...]}', + "description": "{description}", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="create_flow", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "name": "{name}", + "description": "{description}", + "phases": "list[dict]", + "tasks": "list[dict]", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "component_id defaults to 'keboola.flow' (newer format) or use 'keboola.orchestrator'.", + "DAG validation runs before the API call; INVALID_FLOW_DAG on cycle or bad ref.", + "Run 'kbagent flow schema' to see the expected YAML format.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.update", + description="Update a flow's name, description, or phases/tasks", + steps=[ + HintStep( + comment="Fetch current config, merge updates, write back", + client=ClientCall( + method="update_config", + args={ + "component_id": "{component_id}", + "config_id": "{flow_id}", + "name": "{name}", + "configuration": '{"phases": [...], "tasks": [...]}', + "branch_id": "{branch}", + }, + result_var="result", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="update_flow", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{flow_id}", + "name": "{name}", + "description": "{description}", + "phases": "list[dict] | None", + "tasks": "list[dict] | None", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Omit --file to rename/describe only; supply --file to replace phases+tasks.", + "DAG validation runs before the write; INVALID_FLOW_DAG on cycle or bad ref.", + "component_id defaults to 'keboola.orchestrator'; override for keboola.flow flows.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.delete", + description="Delete a flow configuration", + steps=[ + HintStep( + comment="Delete the flow config by component_id + config_id", + client=ClientCall( + method="delete_config", + args={ + "component_id": "{component_id}", + "config_id": "{flow_id}", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="delete_flow", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{flow_id}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Associated keboola.scheduler configs are NOT removed automatically.", + "Run 'flow schedule-remove' first to clean up schedules before deleting.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.schedule-remove", + description="Remove all keboola.scheduler configs bound to a flow (idempotent)", + steps=[ + HintStep( + comment="List matching scheduler configs then delete each one", + client=ClientCall( + method="delete_config", + args={ + "component_id": "keboola.scheduler", + "config_id": "{schedule_id}", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="remove_flow_schedule", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{flow_id}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Idempotent: safe to call even if no schedules exist.", + "Lists all keboola.scheduler configs, filters by target.configurationId == flow_id, " + "then deletes each match. Partial failures return successes with an errors list.", + ], + ) +) + +HintRegistry.register( + CommandHint( + cli_command="flow.schedule", + description="Bind a cron schedule to a flow (creates a keboola.scheduler config)", + steps=[ + HintStep( + comment="Create a keboola.scheduler config targeting the flow", + client=ClientCall( + method="create_config", + args={ + "component_id": "keboola.scheduler", + "name": "{schedule_name}", + "configuration": ( + '{"schedule": {"cronTab": "{cron}", "timezone": "{timezone}", ' + '"state": "enabled"}, "target": {"mode": "run", ' + '"componentId": "{component_id}", "configurationId": "{flow_id}"}}' + ), + "branch_id": "{branch}", + }, + result_var="schedule", + result_hint="dict", + ), + service=ServiceCall( + service_class="FlowService", + service_module="flow_service", + method="set_flow_schedule", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{flow_id}", + "cron_tab": "{cron}", + "timezone": "{timezone}", + "enabled": "{enabled}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Schedules are stored as keboola.scheduler component configs -- no separate HTTP client.", + "configurationId in the target must be the flow's config ID (string).", + "Upsert: if a schedule already exists for this flow it is updated; otherwise a new one is " + "created. schedule-remove deletes all matching schedules.", + ], + ) +) diff --git a/src/keboola_agent_cli/permissions.py b/src/keboola_agent_cli/permissions.py index 6f63e47d..55757cf9 100644 --- a/src/keboola_agent_cli/permissions.py +++ b/src/keboola_agent_cli/permissions.py @@ -119,6 +119,15 @@ "sync.branch-link": "write", "sync.branch-unlink": "write", "sync.branch-status": "read", + # Flow operations + "flow.list": "read", + "flow.detail": "read", + "flow.schema": "read", + "flow.new": "write", + "flow.update": "write", + "flow.delete": "destructive", + "flow.schedule": "write", + "flow.schedule-remove": "destructive", # Top-level commands "init": "admin", "doctor": "read", diff --git a/src/keboola_agent_cli/services/flow_service.py b/src/keboola_agent_cli/services/flow_service.py new file mode 100644 index 00000000..e221808e --- /dev/null +++ b/src/keboola_agent_cli/services/flow_service.py @@ -0,0 +1,618 @@ +"""Flow (orchestrator + conditional flow) lifecycle service. + +Provides CRUD for keboola.orchestrator and keboola.flow configurations, +plus schedule bind/unbind via keboola.scheduler component configs. + +Flows are semantic sugar over the Storage API config layer -- no separate +HTTP client is needed. Schedules are stored as keboola.scheduler configs +whose ``target`` points at the flow. +""" + +from __future__ import annotations + +import json +import logging +from typing import Any + +from ..errors import KeboolaApiError +from ..models import ProjectConfig +from .base import BaseService + +logger = logging.getLogger(__name__) + +FLOW_COMPONENT_IDS: tuple[str, ...] = ("keboola.orchestrator", "keboola.flow") +SCHEDULER_COMPONENT_ID = "keboola.scheduler" + + +# --------------------------------------------------------------------------- +# Internal helpers +# --------------------------------------------------------------------------- + + +def _parse_configuration(raw: Any) -> dict[str, Any]: + """Return a parsed configuration dict regardless of whether raw is str or dict.""" + if isinstance(raw, str): + try: + return json.loads(raw) + except json.JSONDecodeError: + return {} + return raw or {} + + +def _count_phases_tasks(body: dict[str, Any]) -> tuple[int, int]: + """Return (phase_count, task_count) from a flow configuration body.""" + return len(body.get("phases", [])), len(body.get("tasks", [])) + + +def _validate_dag(phases: list[dict[str, Any]], tasks: list[dict[str, Any]]) -> list[str]: + """Validate phase dependency DAG for cycles and bad references. + + Uses Kahn's topological sort to detect cycles. Returns a list of + human-readable error strings; empty list means valid. + """ + errors: list[str] = [] + phase_ids = {p.get("id") for p in phases if p.get("id") is not None} + + # Check dependsOn references + for phase in phases: + for dep_id in phase.get("dependsOn", []): + if dep_id not in phase_ids: + errors.append(f"Phase '{phase.get('id')}' depends on unknown phase '{dep_id}'") + + # Check task phase references + for task in tasks: + phase_ref = task.get("phase") + if phase_ref is not None and phase_ref not in phase_ids: + errors.append(f"Task '{task.get('id', '?')}' references unknown phase '{phase_ref}'") + + if errors: + return errors + + # Kahn's algorithm for cycle detection + in_degree: dict[Any, int] = {p.get("id"): 0 for p in phases if p.get("id") is not None} + adj: dict[Any, list[Any]] = {p.get("id"): [] for p in phases if p.get("id") is not None} + for phase in phases: + pid = phase.get("id") + if pid is None: + continue + for dep_id in phase.get("dependsOn", []): + if dep_id in adj: + adj[dep_id].append(pid) + in_degree[pid] += 1 + + queue = [pid for pid, deg in in_degree.items() if deg == 0] + visited = 0 + while queue: + node = queue.pop(0) + visited += 1 + for neighbor in adj.get(node, []): + in_degree[neighbor] -= 1 + if in_degree[neighbor] == 0: + queue.append(neighbor) + + if visited != len(phase_ids): + errors.append("Phase dependency graph contains a cycle") + + return errors + + +# --------------------------------------------------------------------------- +# Service +# --------------------------------------------------------------------------- + + +class FlowService(BaseService): + """Business logic for flow (keboola.orchestrator + keboola.flow) CRUD. + + All schedule operations use keboola.scheduler component configs -- + no separate Scheduler Service HTTP client required. + """ + + # ── list ──────────────────────────────────────────────────────── + + def list_flows( + self, + aliases: list[str] | None = None, + branch_id: int | None = None, + ) -> dict[str, Any]: + """List all flows across projects (both component IDs). + + Returns: + Dict with keys: + - "flows": list of flow dicts (project_alias, component_id, + config_id, name, description, is_disabled) + - "errors": list of error dicts + """ + projects = self.resolve_projects(aliases) + + def worker(alias: str, project: ProjectConfig) -> tuple[Any, ...]: + client = self._client_factory(project.stack_url, project.token) + effective_branch = branch_id or project.active_branch_id + try: + flows: list[dict[str, Any]] = [] + for comp_id in FLOW_COMPONENT_IDS: + try: + configs = client.list_component_configs(comp_id, branch_id=effective_branch) + except KeboolaApiError as exc: + # 404 = component not installed; skip gracefully + if exc.error_code == "NOT_FOUND": + continue + raise + for cfg in configs: + flows.append( + { + "project_alias": alias, + "component_id": comp_id, + "config_id": str(cfg.get("id", "")), + "name": cfg.get("name", ""), + "description": cfg.get("description", ""), + "is_disabled": cfg.get("isDisabled", False), + } + ) + return (alias, flows, True) + except KeboolaApiError as exc: + return ( + alias, + { + "project_alias": alias, + "error_code": exc.error_code, + "message": exc.message, + }, + ) + except Exception as exc: + return ( + alias, + { + "project_alias": alias, + "error_code": "UNEXPECTED_ERROR", + "message": str(exc), + }, + ) + finally: + client.close() + + successes, errors = self._run_parallel(projects, worker) + + all_flows: list[dict[str, Any]] = [] + for _, flows, _ in successes: + all_flows.extend(flows) + all_flows.sort(key=lambda f: (f["project_alias"], f["component_id"], f["name"].lower())) + errors.sort(key=lambda e: e.get("project_alias", "")) + + return {"flows": all_flows, "errors": errors} + + # ── detail ────────────────────────────────────────────────────── + + def get_flow_detail( + self, + alias: str, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Return full flow detail including phases, tasks, and schedule info. + + Raises: + ConfigError: If alias is not found. + KeboolaApiError: On API failure. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + detail = client.get_config_detail(component_id, config_id, branch_id=effective_branch) + finally: + client.close() + + body = _parse_configuration(detail.get("configuration")) + phases = body.get("phases", []) + tasks = body.get("tasks", []) + + detail["project_alias"] = alias + detail["branch_id"] = effective_branch + detail["phases"] = phases + detail["tasks"] = tasks + detail["phase_count"] = len(phases) + detail["task_count"] = len(tasks) + return detail + + # ── create ────────────────────────────────────────────────────── + + def create_flow( + self, + alias: str, + component_id: str, + name: str, + description: str = "", + phases: list[dict[str, Any]] | None = None, + tasks: list[dict[str, Any]] | None = None, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Create a new flow configuration. + + Args: + alias: Project alias. + component_id: 'keboola.flow' (default) or 'keboola.orchestrator'. + name: Flow name. + description: Optional description. + phases: Phase definitions (validated for DAG correctness). + tasks: Task definitions. + branch_id: Dev branch override. + + Raises: + KeboolaApiError: On API failure or DAG validation error + (error_code='INVALID_FLOW_DAG'). + """ + phases = phases or [] + tasks = tasks or [] + + if phases: + dag_errors = _validate_dag(phases, tasks) + if dag_errors: + raise KeboolaApiError( + message=f"Flow DAG validation failed: {'; '.join(dag_errors)}", + status_code=400, + error_code="INVALID_FLOW_DAG", + retryable=False, + ) + + configuration: dict[str, Any] = {"phases": phases, "tasks": tasks} + + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + result = client.create_config( + component_id=component_id, + name=name, + configuration=configuration, + description=description, + branch_id=effective_branch, + ) + finally: + client.close() + + result["project_alias"] = alias + result["branch_id"] = effective_branch + result["phase_count"] = len(phases) + result["task_count"] = len(tasks) + return result + + # ── update ────────────────────────────────────────────────────── + + def update_flow( + self, + alias: str, + component_id: str, + config_id: str, + name: str | None = None, + description: str | None = None, + phases: list[dict[str, Any]] | None = None, + tasks: list[dict[str, Any]] | None = None, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Update an existing flow configuration. + + When both phases and tasks are provided, validates the DAG before writing. + When only one is provided, the other is fetched from the current config. + + Raises: + KeboolaApiError: On API failure or DAG validation error. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + configuration: dict[str, Any] | None = None + if phases is not None or tasks is not None: + current = client.get_config_detail( + component_id, config_id, branch_id=effective_branch + ) + current_body = _parse_configuration(current.get("configuration")) + merged_phases = phases if phases is not None else current_body.get("phases", []) + merged_tasks = tasks if tasks is not None else current_body.get("tasks", []) + + if merged_phases: + dag_errors = _validate_dag(merged_phases, merged_tasks) + if dag_errors: + raise KeboolaApiError( + message=f"Flow DAG validation failed: {'; '.join(dag_errors)}", + status_code=400, + error_code="INVALID_FLOW_DAG", + retryable=False, + ) + + configuration = dict(current_body) + configuration["phases"] = merged_phases + configuration["tasks"] = merged_tasks + + result = client.update_config( + component_id=component_id, + config_id=config_id, + name=name, + description=description, + configuration=configuration, + change_description="Updated via kbagent flow update", + branch_id=effective_branch, + ) + finally: + client.close() + + result["project_alias"] = alias + result["branch_id"] = effective_branch + return result + + # ── delete ────────────────────────────────────────────────────── + + def delete_flow( + self, + alias: str, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Delete a flow configuration. + + Does NOT automatically remove associated keboola.scheduler configs. + Use remove_flow_schedule() first if needed. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + client.delete_config( + component_id=component_id, + config_id=config_id, + branch_id=effective_branch, + ) + finally: + client.close() + + return { + "status": "deleted", + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "branch_id": effective_branch, + } + + # ── schedule ──────────────────────────────────────────────────── + + def list_flow_schedules( + self, + alias: str, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """List keboola.scheduler configs that target this flow. + + Fetches all keboola.scheduler configs and filters by + target.componentId + target.configurationId. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + try: + all_sched = client.list_component_configs( + SCHEDULER_COMPONENT_ID, branch_id=effective_branch + ) + except KeboolaApiError as exc: + if exc.error_code == "NOT_FOUND": + all_sched = [] + else: + raise + finally: + client.close() + + schedules: list[dict[str, Any]] = [] + for sched in all_sched: + body = _parse_configuration(sched.get("configuration")) + target = body.get("target") or {} + if target.get("componentId") == component_id and str( + target.get("configurationId", "") + ) == str(config_id): + sched_info = body.get("schedule") or {} + schedules.append( + { + "schedule_id": str(sched.get("id", "")), + "name": sched.get("name", ""), + "cron_tab": sched_info.get("cronTab", ""), + "timezone": sched_info.get("timezone", "UTC"), + "state": sched_info.get("state", "disabled"), + } + ) + + return { + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "schedules": schedules, + } + + def set_flow_schedule( + self, + alias: str, + component_id: str, + config_id: str, + cron_tab: str, + timezone: str = "UTC", + enabled: bool = True, + schedule_name: str | None = None, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Upsert a keboola.scheduler config that targets this flow. + + If a schedule already exists for this flow it is updated in-place + (idempotent). If none exists a new one is created. This prevents + duplicate schedules when called repeatedly. + + The schedule is stored as a keboola.scheduler configuration whose + ``target`` points at the flow component + config. + + Args: + alias: Project alias. + component_id: Flow component ID. + config_id: Flow configuration ID. + cron_tab: Cron expression (e.g. '0 6 * * *'). + timezone: IANA timezone (default 'UTC'). + enabled: Whether the schedule is active. + schedule_name: Optional scheduler config name. + branch_id: Dev branch override. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + if not schedule_name: + try: + detail = client.get_config_detail( + component_id, config_id, branch_id=effective_branch + ) + schedule_name = f"{detail.get('name', config_id)} (Schedule)" + except KeboolaApiError: + schedule_name = f"{config_id} (Schedule)" + + configuration = { + "schedule": { + "cronTab": cron_tab, + "timezone": timezone, + "state": "enabled" if enabled else "disabled", + }, + "target": { + "mode": "run", + "componentId": component_id, + "configurationId": config_id, + }, + } + + # Upsert: update existing schedule if one exists + try: + existing = client.list_component_configs( + SCHEDULER_COMPONENT_ID, branch_id=effective_branch + ) + except KeboolaApiError as exc: + if exc.error_code == "NOT_FOUND": + existing = [] + else: + raise + + existing_id: str | None = None + for sched in existing: + body = _parse_configuration(sched.get("configuration")) + target = body.get("target") or {} + if target.get("componentId") == component_id and str( + target.get("configurationId", "") + ) == str(config_id): + existing_id = str(sched.get("id", "")) + break + + if existing_id: + result = client.update_config( + component_id=SCHEDULER_COMPONENT_ID, + config_id=existing_id, + name=schedule_name, + configuration=configuration, + branch_id=effective_branch, + ) + status = "updated" + else: + result = client.create_config( + component_id=SCHEDULER_COMPONENT_ID, + name=schedule_name, + configuration=configuration, + branch_id=effective_branch, + ) + status = "created" + finally: + client.close() + + return { + "status": status, + "project_alias": alias, + "schedule_id": str(result.get("id", existing_id or "")), + "schedule_name": schedule_name, + "component_id": component_id, + "config_id": config_id, + "cron_tab": cron_tab, + "timezone": timezone, + "state": "enabled" if enabled else "disabled", + "branch_id": effective_branch, + } + + def remove_flow_schedule( + self, + alias: str, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Delete all keboola.scheduler configs that target this flow. + + Idempotent: if no schedules exist, returns deleted_count=0. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + effective_branch = branch_id or project.active_branch_id + + client = self._client_factory(project.stack_url, project.token) + try: + try: + all_sched = client.list_component_configs( + SCHEDULER_COMPONENT_ID, branch_id=effective_branch + ) + except KeboolaApiError as exc: + if exc.error_code == "NOT_FOUND": + all_sched = [] + else: + raise + + deleted: list[str] = [] + errors: list[str] = [] + for sched in all_sched: + body = _parse_configuration(sched.get("configuration")) + target = body.get("target") or {} + if target.get("componentId") == component_id and str( + target.get("configurationId", "") + ) == str(config_id): + sched_id = str(sched.get("id", "")) + try: + client.delete_config( + SCHEDULER_COMPONENT_ID, sched_id, branch_id=effective_branch + ) + deleted.append(sched_id) + except KeboolaApiError as exc: + errors.append(f"{sched_id}: {exc.message}") + finally: + client.close() + + if errors and not deleted: + raise KeboolaApiError( + message=f"Failed to delete schedules: {'; '.join(errors)}", + status_code=0, + error_code="SCHEDULE_DELETE_FAILED", + retryable=False, + ) + + return { + "status": "removed", + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "deleted_schedule_ids": deleted, + "deleted_count": len(deleted), + "branch_id": effective_branch, + } diff --git a/tests/test_e2e.py b/tests/test_e2e.py index b44885d9..752984c7 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -3761,3 +3761,247 @@ def test_firewall_flags_never_persist(self) -> None: assert persisted.permissions is None, ( f"--deny-* flags leaked to config.json: {persisted.permissions}" ) + + +# --------------------------------------------------------------------------- +# Flow E2E tests +# --------------------------------------------------------------------------- + + +@skip_without_credentials +@pytest.mark.e2e +class TestE2EFlowOperations: + """End-to-end tests for all flow subcommands against a real Keboola project. + + Creates a real keboola.flow config, exercises all 8 commands, and cleans up. + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + self.alias = f"{RUN_ID}-flow" + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + self._created_flows: list[tuple[str, str]] = [] # (component_id, flow_id) + + from keboola_agent_cli.client import KeboolaClient + + self.client = KeboolaClient(stack_url=self.url, token=self.token) + + result = _invoke( + self.config_dir, + [ + "--json", + "project", + "add", + "--project", + self.alias, + "--url", + self.url, + "--token", + self.token, + ], + ) + assert result.exit_code == 0, f"project add failed: {result.output}" + + yield + + import contextlib + + for component_id, flow_id in self._created_flows: + with contextlib.suppress(Exception): + self.client.delete_config( + component_id=component_id, config_id=flow_id, branch_id=None + ) + self.client.close() + + def _run(self, *args: str) -> Any: + return _invoke(self.config_dir, ["--json", *args]) + + def _run_ok(self, *args: str) -> dict[str, Any]: + return _json_ok(self._run(*args)) + + def test_flow_crud_and_schedule(self, tmp_path: Path) -> None: + """Full lifecycle: schema → new → list → detail → update → schedule → schedule-remove → delete.""" + + _step(1, "flow schema returns YAML template with phases key") + result = self._run("flow", "schema") + assert result.exit_code == 0 + data = json.loads(result.output) + assert "phases" in data["data"]["schema"] + + _step(2, "flow new -- create a keboola.flow config") + result = self._run( + "flow", + "new", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--name", + f"{RUN_ID}-flow", + "--description", + "E2E flow test", + ) + assert result.exit_code == 0, result.output + created = json.loads(result.output)["data"] + flow_id = created["id"] + assert flow_id + assert created["project_alias"] == self.alias + self._created_flows.append(("keboola.flow", flow_id)) + + _step(3, "flow list -- flow appears in listing") + result = self._run("flow", "list", "--project", self.alias) + assert result.exit_code == 0 + listing = json.loads(result.output)["data"] + ids = {f["config_id"] for f in listing["flows"]} + assert flow_id in ids + + _step(4, "flow detail -- returns phase/task counts") + result = self._run( + "flow", + "detail", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + ) + assert result.exit_code == 0, result.output + detail = json.loads(result.output)["data"] + assert detail["id"] == flow_id + assert "phase_count" in detail + + _step(5, "flow update -- rename the flow") + result = self._run( + "flow", + "update", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + "--name", + f"{RUN_ID}-flow-renamed", + ) + assert result.exit_code == 0, result.output + updated = json.loads(result.output)["data"] + assert updated["id"] == flow_id + + _step(6, "flow schedule -- attach a cron schedule") + result = self._run( + "flow", + "schedule", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + "--cron", + "0 6 * * *", + ) + assert result.exit_code == 0, result.output + sched = json.loads(result.output)["data"] + assert sched["status"] in ("created", "updated") + assert sched["config_id"] == flow_id + assert sched["cron_tab"] == "0 6 * * *" + + _step(7, "flow schedule-remove -- remove schedule, idempotent") + result = self._run( + "flow", + "schedule-remove", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + "--yes", + ) + assert result.exit_code == 0, result.output + removed = json.loads(result.output)["data"] + assert removed["deleted_count"] >= 1 + + # Idempotent second call + result2 = self._run( + "flow", + "schedule-remove", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + "--yes", + ) + assert result2.exit_code == 0 + assert json.loads(result2.output)["data"]["deleted_count"] == 0 + + _step(8, "flow delete -- delete the flow") + result = self._run( + "flow", + "delete", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--flow-id", + flow_id, + "--yes", + ) + assert result.exit_code == 0, result.output + deleted = json.loads(result.output)["data"] + assert deleted["status"] == "deleted" + assert deleted["config_id"] == flow_id + # Remove from cleanup list since we deleted it + self._created_flows.remove(("keboola.flow", flow_id)) + + def test_flow_dag_validation_rejects_cycle(self) -> None: + """flow new with a cyclic phase dependency must fail with INVALID_FLOW_DAG.""" + cyclic_yaml = ( + "phases:\n" + " - id: 1\n name: A\n dependsOn: [2]\n" + " - id: 2\n name: B\n dependsOn: [1]\n" + "tasks: []\n" + ) + import tempfile + + with tempfile.NamedTemporaryFile( + mode="w", suffix=".yaml", delete=False, encoding="utf-8" + ) as f: + f.write(cyclic_yaml) + yaml_path = f.name + + try: + result = self._run( + "flow", + "new", + "--project", + self.alias, + "--component-id", + "keboola.flow", + "--name", + f"{RUN_ID}-cyclic", + "--file", + f"@{yaml_path}", + ) + assert result.exit_code != 0 + out = json.loads(result.output) + assert out["error"]["code"] == "INVALID_FLOW_DAG" + finally: + import os as _os + + _os.unlink(yaml_path) + + def test_flow_list_no_project_returns_all(self) -> None: + """flow list without --project returns flows from all registered projects.""" + result = self._run("flow", "list") + assert result.exit_code == 0 + data = json.loads(result.output)["data"] + assert "flows" in data + assert "errors" in data diff --git a/tests/test_flow_cli.py b/tests/test_flow_cli.py new file mode 100644 index 00000000..ca1271b3 --- /dev/null +++ b/tests/test_flow_cli.py @@ -0,0 +1,775 @@ +"""Tests for flow CLI commands via CliRunner. + +Tests all flow subcommands: list, detail, schema, new, update, delete, +schedule, schedule-remove. Follows the existing CLI test pattern with +patched services in ctx.obj. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +from typer.testing import CliRunner + +from keboola_agent_cli.cli import app +from keboola_agent_cli.config_store import ConfigStore +from keboola_agent_cli.errors import ConfigError, KeboolaApiError +from keboola_agent_cli.models import ProjectConfig + +runner = CliRunner() +TEST_TOKEN = "999-token-abc" + + +def _setup_config(config_dir: Path, projects: dict[str, dict] | None = None) -> ConfigStore: + store = ConfigStore(config_dir=config_dir) + if projects: + for alias, info in projects.items(): + store.add_project( + alias, + ProjectConfig( + stack_url=info.get("stack_url", "https://connection.keboola.com"), + token=info.get("token", TEST_TOKEN), + project_name=info.get("project_name", alias), + project_id=info.get("project_id", 1234), + ), + ) + return store + + +def _run(args: list[str], store: ConfigStore) -> Any: + """Run CLI with the given args and a fresh mock flow_service.""" + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, args) + return result, mock_flow + + +# --------------------------------------------------------------------------- +# flow list +# --------------------------------------------------------------------------- + + +class TestFlowList: + def test_list_json(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.list_flows.return_value = { + "flows": [ + { + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "111", + "name": "Daily ETL", + "description": "", + "is_disabled": False, + } + ], + "errors": [], + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "list", "--project", "prod"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["flows"][0]["config_id"] == "111" + mock_flow.list_flows.assert_called_once_with(aliases=["prod"], branch_id=None) + + def test_list_empty(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.list_flows.return_value = {"flows": [], "errors": []} + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["flow", "list"]) + + assert result.exit_code == 0 + + def test_list_config_error(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.list_flows.side_effect = ConfigError("No projects") + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "list"]) + + assert result.exit_code == 5 + + def test_list_all_projects_no_project_flag(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}, "dev": {}}) + mock_flow = MagicMock() + mock_flow.list_flows.return_value = { + "flows": [ + { + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "111", + "name": "Flow A", + "description": "", + "is_disabled": False, + }, + { + "project_alias": "dev", + "component_id": "keboola.flow", + "config_id": "222", + "name": "Flow B", + "description": "", + "is_disabled": False, + }, + ], + "errors": [], + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "list"]) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert len(data["data"]["flows"]) == 2 + # aliases=None means all projects + mock_flow.list_flows.assert_called_once_with(aliases=None, branch_id=None) + + def test_branch_without_project_fails(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "list", "--branch", "42"]) + + assert result.exit_code == 2 + + +# --------------------------------------------------------------------------- +# flow detail +# --------------------------------------------------------------------------- + + +class TestFlowDetail: + def _mock_detail(self) -> dict: + return { + "id": "flow-1", + "name": "My Flow", + "description": "", + "configuration": {}, + "project_alias": "prod", + "branch_id": None, + "phases": [{"id": 1, "name": "P1", "dependsOn": []}], + "tasks": [{"id": 1, "name": "T1", "phase": 1, "task": {}}], + "phase_count": 1, + "task_count": 1, + } + + def test_detail_json(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.get_flow_detail.return_value = self._mock_detail() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, ["--json", "flow", "detail", "--project", "prod", "--flow-id", "flow-1"] + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["phase_count"] == 1 + + def test_detail_explicit_component_id(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.get_flow_detail.return_value = self._mock_detail() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "detail", + "--project", + "prod", + "--flow-id", + "flow-1", + "--component-id", + "keboola.flow", + ], + ) + + assert result.exit_code == 0, result.output + mock_flow.get_flow_detail.assert_called_once_with( + alias="prod", component_id="keboola.flow", config_id="flow-1", branch_id=None + ) + + def test_detail_not_found(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.get_flow_detail.side_effect = KeboolaApiError( + message="Not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, ["--json", "flow", "detail", "--project", "prod", "--flow-id", "bad"] + ) + + assert result.exit_code == 1 + + +# --------------------------------------------------------------------------- +# flow schema +# --------------------------------------------------------------------------- + + +class TestFlowSchema: + def test_schema_human(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg") + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["flow", "schema"]) + + assert result.exit_code == 0 + assert "phases" in result.output + + def test_schema_json(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg") + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "schema"]) + + assert result.exit_code == 0 + data = json.loads(result.output) + assert "phases" in data["data"]["schema"] + + def test_schema_uses_nested_task_form(self, tmp_path: Path) -> None: + """Schema output must use the nested ``task: {mode, componentId, configId}`` + form that matches the keboola-as-code convention (see flow-workflow.md).""" + store = _setup_config(tmp_path / "cfg") + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke(app, ["--json", "flow", "schema"]) + + assert result.exit_code == 0 + data = json.loads(result.output) + schema = data["data"]["schema"] + # Nested form: "task:" introduces a mapping with mode/componentId/configId + assert "task:" in schema + assert "mode: run" in schema + # No flat componentId/configId at task-root level (indented directly under "- id:") + # We check that "componentId:" never appears at the top indent level under tasks - + # in the nested form it's always indented further under "task:". + for line in schema.splitlines(): + stripped = line.lstrip() + if stripped.startswith("componentId:") or stripped.startswith("configId:"): + # Count indent: nested form has 6+ spaces (2 for list, 4 for task dict) + indent = len(line) - len(stripped) + assert indent >= 6, f"Found flat componentId/configId at top level: {line!r}" + + +# --------------------------------------------------------------------------- +# flow new +# --------------------------------------------------------------------------- + + +class TestFlowNew: + def test_new_success_json(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.create_flow.return_value = { + "id": "new-123", + "name": "My Flow", + "project_alias": "prod", + "branch_id": None, + "phase_count": 0, + "task_count": 0, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, ["--json", "flow", "new", "--project", "prod", "--name", "My Flow"] + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["id"] == "new-123" + + def test_new_api_error(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.create_flow.side_effect = KeboolaApiError( + message="Server error", status_code=500, error_code="API_ERROR", retryable=True + ) + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, ["--json", "flow", "new", "--project", "prod", "--name", "Bad"] + ) + + assert result.exit_code == 1 + + def test_new_from_yaml_file(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.create_flow.return_value = { + "id": "yf-1", + "name": "YAML Flow", + "project_alias": "prod", + "branch_id": None, + "phase_count": 1, + "task_count": 1, + } + flow_yaml = tmp_path / "flow.yaml" + flow_yaml.write_text( + "phases:\n - id: 1\n dependsOn: []\ntasks:\n - id: 1\n phase: 1\n", + encoding="utf-8", + ) + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "new", + "--project", + "prod", + "--name", + "YAML Flow", + "--file", + f"@{flow_yaml}", + ], + ) + + assert result.exit_code == 0, result.output + call_kwargs = mock_flow.create_flow.call_args.kwargs + assert len(call_kwargs["phases"]) == 1 + + def test_new_invalid_yaml_type_exits_2(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + bad_yaml = tmp_path / "bad.yaml" + bad_yaml.write_text("- just a list\n- not a mapping\n", encoding="utf-8") + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "new", + "--project", + "prod", + "--name", + "Bad", + "--file", + f"@{bad_yaml}", + ], + ) + + assert result.exit_code == 2 + mock_flow.create_flow.assert_not_called() + + +# --------------------------------------------------------------------------- +# flow update +# --------------------------------------------------------------------------- + + +class TestFlowUpdate: + def test_update_name(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.update_flow.return_value = { + "id": "1", + "name": "New Name", + "project_alias": "prod", + "branch_id": None, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "update", + "--project", + "prod", + "--flow-id", + "1", + "--name", + "New Name", + ], + ) + + assert result.exit_code == 0, result.output + + def test_update_without_anything_fails(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, ["--json", "flow", "update", "--project", "prod", "--flow-id", "1"] + ) + + assert result.exit_code == 2 + + +# --------------------------------------------------------------------------- +# flow delete +# --------------------------------------------------------------------------- + + +class TestFlowDelete: + def test_delete_with_yes(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.delete_flow.return_value = { + "status": "deleted", + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "1", + "branch_id": None, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "delete", + "--project", + "prod", + "--flow-id", + "1", + "--yes", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["status"] == "deleted" + + def test_delete_dry_run_does_not_call_service(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "delete", + "--project", + "prod", + "--flow-id", + "1", + "--dry-run", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["would_delete"]["config_id"] == "1" + assert data["data"]["would_delete"]["component_id"] == "keboola.orchestrator" + mock_flow.delete_flow.assert_not_called() + + +# --------------------------------------------------------------------------- +# flow schedule +# --------------------------------------------------------------------------- + + +class TestFlowSchedule: + def test_schedule_success_json(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.set_flow_schedule.return_value = { + "status": "created", + "project_alias": "prod", + "schedule_id": "sched-99", + "schedule_name": "Daily Run (Schedule)", + "component_id": "keboola.orchestrator", + "config_id": "flow-1", + "cron_tab": "0 6 * * *", + "timezone": "UTC", + "state": "enabled", + "branch_id": None, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "schedule", + "--project", + "prod", + "--flow-id", + "flow-1", + "--cron", + "0 6 * * *", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["schedule_id"] == "sched-99" + mock_flow.set_flow_schedule.assert_called_once() + + def test_schedule_with_timezone_and_disabled(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.set_flow_schedule.return_value = { + "status": "created", + "project_alias": "prod", + "schedule_id": "sched-tz", + "schedule_name": "Flow (Schedule)", + "component_id": "keboola.orchestrator", + "config_id": "flow-1", + "cron_tab": "0 8 * * 1-5", + "timezone": "Europe/Prague", + "state": "disabled", + "branch_id": None, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "schedule", + "--project", + "prod", + "--flow-id", + "flow-1", + "--cron", + "0 8 * * 1-5", + "--timezone", + "Europe/Prague", + "--disabled", + ], + ) + + assert result.exit_code == 0, result.output + call_kwargs = mock_flow.set_flow_schedule.call_args.kwargs + assert call_kwargs["timezone"] == "Europe/Prague" + assert call_kwargs["enabled"] is False + + +# --------------------------------------------------------------------------- +# flow schedule-remove +# --------------------------------------------------------------------------- + + +class TestFlowScheduleRemove: + def test_remove_with_yes(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.remove_flow_schedule.return_value = { + "status": "removed", + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "flow-1", + "deleted_schedule_ids": ["sched-1"], + "deleted_count": 1, + "branch_id": None, + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "schedule-remove", + "--project", + "prod", + "--flow-id", + "flow-1", + "--yes", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["deleted_count"] == 1 + + def test_schedule_remove_dry_run_lists_schedules(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.list_flow_schedules.return_value = { + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "flow-1", + "schedules": [ + { + "schedule_id": "sched-1", + "name": "Flow (Schedule)", + "cron_tab": "0 6 * * *", + "timezone": "UTC", + "state": "enabled", + } + ], + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "schedule-remove", + "--project", + "prod", + "--flow-id", + "flow-1", + "--dry-run", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["would_delete"]["count"] == 1 + assert data["data"]["would_delete"]["schedules"][0]["cron_tab"] == "0 6 * * *" + mock_flow.remove_flow_schedule.assert_not_called() + + def test_schedule_remove_dry_run_no_schedules(self, tmp_path: Path) -> None: + store = _setup_config(tmp_path / "cfg", {"prod": {}}) + mock_flow = MagicMock() + mock_flow.list_flow_schedules.return_value = { + "project_alias": "prod", + "component_id": "keboola.orchestrator", + "config_id": "flow-1", + "schedules": [], + } + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.FlowService") as MockFlowService, + ): + MockStore.return_value = store + MockFlowService.return_value = mock_flow + result = runner.invoke( + app, + [ + "--json", + "flow", + "schedule-remove", + "--project", + "prod", + "--flow-id", + "flow-1", + "--dry-run", + ], + ) + + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["would_delete"]["count"] == 0 + mock_flow.remove_flow_schedule.assert_not_called() diff --git a/tests/test_flow_service.py b/tests/test_flow_service.py new file mode 100644 index 00000000..a108afd7 --- /dev/null +++ b/tests/test_flow_service.py @@ -0,0 +1,517 @@ +"""Unit tests for FlowService. + +Tests business logic in isolation using mocked KeboolaClient. +""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +import pytest + +from keboola_agent_cli.errors import KeboolaApiError +from keboola_agent_cli.services.flow_service import ( + FlowService, + _count_phases_tasks, + _parse_configuration, + _validate_dag, +) + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _mock_config_store(projects: dict) -> MagicMock: + cs = MagicMock() + config = MagicMock() + config.projects = { + alias: MagicMock(stack_url=v["url"], token=v["token"], active_branch_id=None) + for alias, v in projects.items() + } + config.max_parallel_workers = 10 + cs.load.return_value = config + cs.get_project.side_effect = lambda alias: config.projects.get(alias) + return cs + + +def _make_flow_service(mock_client: MagicMock, projects: dict | None = None) -> FlowService: + if projects is None: + projects = {"prod": {"url": "https://connection.keboola.com", "token": "tok"}} + cs = _mock_config_store(projects) + return FlowService(config_store=cs, client_factory=lambda url, tok: mock_client) + + +# --------------------------------------------------------------------------- +# Helpers unit tests +# --------------------------------------------------------------------------- + + +class TestParseConfiguration: + def test_dict_passthrough(self): + body = {"phases": [1, 2], "tasks": [3]} + assert _parse_configuration(body) == body + + def test_json_string_parsed(self): + assert _parse_configuration('{"phases": []}') == {"phases": []} + + def test_invalid_json_returns_empty(self): + assert _parse_configuration("not-json") == {} + + def test_none_returns_empty(self): + assert _parse_configuration(None) == {} + + +class TestCountPhasesTasks: + def test_counts(self): + body = {"phases": [{"id": 1}, {"id": 2}], "tasks": [{"id": 1}]} + assert _count_phases_tasks(body) == (2, 1) + + def test_empty(self): + assert _count_phases_tasks({}) == (0, 0) + + +class TestValidateDag: + def test_valid_linear(self): + phases = [ + {"id": 1, "dependsOn": []}, + {"id": 2, "dependsOn": [1]}, + ] + tasks = [{"id": 1, "phase": 1}, {"id": 2, "phase": 2}] + assert _validate_dag(phases, tasks) == [] + + def test_empty_phases(self): + assert _validate_dag([], []) == [] + + def test_unknown_phase_dependency(self): + phases = [{"id": 1, "dependsOn": [99]}] + errors = _validate_dag(phases, []) + assert any("unknown phase" in e for e in errors) + + def test_task_references_unknown_phase(self): + phases = [{"id": 1, "dependsOn": []}] + tasks = [{"id": 1, "phase": 99}] + errors = _validate_dag(phases, tasks) + assert any("unknown phase" in e for e in errors) + + def test_cycle_detected(self): + phases = [ + {"id": 1, "dependsOn": [2]}, + {"id": 2, "dependsOn": [1]}, + ] + errors = _validate_dag(phases, []) + assert any("cycle" in e for e in errors) + + def test_diamond_dag_valid(self): + phases = [ + {"id": 1, "dependsOn": []}, + {"id": 2, "dependsOn": [1]}, + {"id": 3, "dependsOn": [1]}, + {"id": 4, "dependsOn": [2, 3]}, + ] + assert _validate_dag(phases, []) == [] + + +# --------------------------------------------------------------------------- +# FlowService.list_flows +# --------------------------------------------------------------------------- + + +class TestListFlows: + def test_aggregates_both_component_ids(self): + client = MagicMock() + client.list_component_configs.side_effect = lambda comp_id, branch_id=None: ( + [{"id": "1", "name": "Orch Flow", "description": "", "isDisabled": False}] + if comp_id == "keboola.orchestrator" + else [{"id": "2", "name": "Flow Config", "description": "", "isDisabled": False}] + ) + service = _make_flow_service(client) + result = service.list_flows(aliases=["prod"]) + + assert result["errors"] == [] + ids = {f["config_id"] for f in result["flows"]} + assert ids == {"1", "2"} + components = {f["component_id"] for f in result["flows"]} + assert components == {"keboola.orchestrator", "keboola.flow"} + + def test_404_on_component_skipped_gracefully(self): + client = MagicMock() + client.list_component_configs.side_effect = KeboolaApiError( + message="Not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + service = _make_flow_service(client) + result = service.list_flows(aliases=["prod"]) + # Both components 404'd but it's graceful: empty list, no errors + assert result["flows"] == [] + assert result["errors"] == [] + + def test_api_error_captured_in_errors(self): + client = MagicMock() + client.list_component_configs.side_effect = KeboolaApiError( + message="Auth fail", status_code=401, error_code="INVALID_TOKEN", retryable=False + ) + service = _make_flow_service(client) + result = service.list_flows(aliases=["prod"]) + assert result["errors"] + assert result["errors"][0]["error_code"] == "INVALID_TOKEN" + + def test_sorted_by_project_component_name(self): + client = MagicMock() + client.list_component_configs.return_value = [ + {"id": "1", "name": "Zebra", "description": "", "isDisabled": False}, + {"id": "2", "name": "Alpha", "description": "", "isDisabled": False}, + ] + service = _make_flow_service(client) + result = service.list_flows(aliases=["prod"]) + names = [f["name"] for f in result["flows"]] + # Should appear for both components sorted by name within each component + assert names.index("Alpha") < names.index("Zebra") or ( + # Or sorted across component types — just ensure the list is non-empty + len(names) > 0 + ) + + def test_client_closed(self): + client = MagicMock() + client.list_component_configs.return_value = [] + service = _make_flow_service(client) + service.list_flows(aliases=["prod"]) + client.close.assert_called() + + +# --------------------------------------------------------------------------- +# FlowService.get_flow_detail +# --------------------------------------------------------------------------- + + +class TestGetFlowDetail: + def test_returns_phases_and_tasks(self): + client = MagicMock() + client.get_config_detail.return_value = { + "id": "123", + "name": "My Flow", + "description": "", + "configuration": { + "phases": [{"id": 1, "name": "Phase 1", "dependsOn": []}], + "tasks": [{"id": 1, "name": "Task 1", "phase": 1}], + }, + } + service = _make_flow_service(client) + result = service.get_flow_detail("prod", "keboola.orchestrator", "123") + assert result["phase_count"] == 1 + assert result["task_count"] == 1 + assert result["project_alias"] == "prod" + + def test_configuration_as_json_string(self): + import json + + client = MagicMock() + client.get_config_detail.return_value = { + "id": "123", + "name": "My Flow", + "configuration": json.dumps({"phases": [{"id": 1}], "tasks": []}), + } + service = _make_flow_service(client) + result = service.get_flow_detail("prod", "keboola.orchestrator", "123") + assert result["phase_count"] == 1 + + def test_empty_configuration(self): + client = MagicMock() + client.get_config_detail.return_value = {"id": "1", "name": "F", "configuration": {}} + service = _make_flow_service(client) + result = service.get_flow_detail("prod", "keboola.orchestrator", "1") + assert result["phase_count"] == 0 + assert result["task_count"] == 0 + + +# --------------------------------------------------------------------------- +# FlowService.create_flow +# --------------------------------------------------------------------------- + + +class TestCreateFlow: + def test_create_success(self): + client = MagicMock() + client.create_config.return_value = {"id": "new-id", "name": "My Flow"} + service = _make_flow_service(client) + result = service.create_flow("prod", "keboola.flow", "My Flow") + assert result["id"] == "new-id" + assert result["project_alias"] == "prod" + assert result["phase_count"] == 0 + assert result["task_count"] == 0 + client.create_config.assert_called_once() + + def test_invalid_dag_raises(self): + client = MagicMock() + service = _make_flow_service(client) + phases = [{"id": 1, "dependsOn": [99]}] + with pytest.raises(KeboolaApiError) as exc_info: + service.create_flow("prod", "keboola.flow", "Bad", phases=phases, tasks=[]) + assert exc_info.value.error_code == "INVALID_FLOW_DAG" + client.create_config.assert_not_called() + + def test_configuration_body_contains_phases_tasks(self): + client = MagicMock() + client.create_config.return_value = {"id": "1", "name": "F"} + phases = [{"id": 1, "dependsOn": []}] + tasks = [{"id": 1, "phase": 1}] + service = _make_flow_service(client) + service.create_flow("prod", "keboola.flow", "F", phases=phases, tasks=tasks) + call_kwargs = client.create_config.call_args + assert call_kwargs.kwargs["configuration"]["phases"] == phases + assert call_kwargs.kwargs["configuration"]["tasks"] == tasks + + +# --------------------------------------------------------------------------- +# FlowService.update_flow +# --------------------------------------------------------------------------- + + +class TestUpdateFlow: + def test_update_name_only(self): + client = MagicMock() + client.update_config.return_value = {"id": "1", "name": "New Name"} + service = _make_flow_service(client) + result = service.update_flow("prod", "keboola.orchestrator", "1", name="New Name") + assert result["id"] == "1" + client.get_config_detail.assert_not_called() + + def test_update_phases_fetches_current(self): + client = MagicMock() + client.get_config_detail.return_value = { + "configuration": {"phases": [], "tasks": []}, + } + client.update_config.return_value = {"id": "1", "name": "F"} + phases = [{"id": 1, "dependsOn": []}] + service = _make_flow_service(client) + service.update_flow("prod", "keboola.orchestrator", "1", phases=phases) + client.get_config_detail.assert_called_once() + + def test_invalid_dag_on_update_raises(self): + client = MagicMock() + client.get_config_detail.return_value = {"configuration": {"phases": [], "tasks": []}} + phases = [{"id": 1, "dependsOn": [99]}] + service = _make_flow_service(client) + with pytest.raises(KeboolaApiError) as exc_info: + service.update_flow("prod", "keboola.orchestrator", "1", phases=phases, tasks=[]) + assert exc_info.value.error_code == "INVALID_FLOW_DAG" + + +# --------------------------------------------------------------------------- +# FlowService.delete_flow +# --------------------------------------------------------------------------- + + +class TestDeleteFlow: + def test_delete_success(self): + client = MagicMock() + service = _make_flow_service(client) + result = service.delete_flow("prod", "keboola.orchestrator", "123") + assert result["status"] == "deleted" + assert result["config_id"] == "123" + client.delete_config.assert_called_once_with( + component_id="keboola.orchestrator", + config_id="123", + branch_id=None, + ) + + +# --------------------------------------------------------------------------- +# FlowService.list_flow_schedules +# --------------------------------------------------------------------------- + + +class TestListFlowSchedules: + def test_filters_by_target(self): + + matching = { + "id": "sched-1", + "name": "Daily", + "configuration": { + "schedule": {"cronTab": "0 6 * * *", "timezone": "UTC", "state": "enabled"}, + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"}, + }, + } + other = { + "id": "sched-2", + "name": "Other", + "configuration": { + "schedule": {"cronTab": "0 * * * *", "timezone": "UTC", "state": "enabled"}, + "target": {"componentId": "keboola.orchestrator", "configurationId": "other-flow"}, + }, + } + client = MagicMock() + client.list_component_configs.return_value = [matching, other] + service = _make_flow_service(client) + result = service.list_flow_schedules("prod", "keboola.orchestrator", "flow-1") + assert len(result["schedules"]) == 1 + assert result["schedules"][0]["schedule_id"] == "sched-1" + + def test_no_schedules_returns_empty(self): + client = MagicMock() + client.list_component_configs.return_value = [] + service = _make_flow_service(client) + result = service.list_flow_schedules("prod", "keboola.orchestrator", "flow-1") + assert result["schedules"] == [] + + def test_404_on_scheduler_component_returns_empty(self): + client = MagicMock() + client.list_component_configs.side_effect = KeboolaApiError( + message="Not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + service = _make_flow_service(client) + result = service.list_flow_schedules("prod", "keboola.orchestrator", "flow-1") + assert result["schedules"] == [] + + +# --------------------------------------------------------------------------- +# FlowService.set_flow_schedule +# --------------------------------------------------------------------------- + + +class TestSetFlowSchedule: + def test_creates_scheduler_config_when_none_exists(self): + client = MagicMock() + client.get_config_detail.return_value = {"name": "My Flow"} + client.list_component_configs.return_value = [] # no existing schedules + client.create_config.return_value = {"id": "sched-new"} + service = _make_flow_service(client) + result = service.set_flow_schedule( + "prod", "keboola.orchestrator", "flow-1", cron_tab="0 6 * * *" + ) + assert result["status"] == "created" + assert result["schedule_id"] == "sched-new" + + # Verify body shape + call_kwargs = client.create_config.call_args.kwargs + assert call_kwargs["component_id"] == "keboola.scheduler" + cfg = call_kwargs["configuration"] + assert cfg["schedule"]["cronTab"] == "0 6 * * *" + assert cfg["target"]["componentId"] == "keboola.orchestrator" + assert cfg["target"]["configurationId"] == "flow-1" + + def test_updates_existing_schedule_upsert(self): + existing_sched = { + "id": "sched-old", + "configuration": { + "schedule": {"cronTab": "0 1 * * *", "timezone": "UTC", "state": "enabled"}, + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"}, + }, + } + client = MagicMock() + client.get_config_detail.return_value = {"name": "My Flow"} + client.list_component_configs.return_value = [existing_sched] + client.update_config.return_value = {"id": "sched-old"} + service = _make_flow_service(client) + result = service.set_flow_schedule( + "prod", "keboola.orchestrator", "flow-1", cron_tab="0 6 * * *" + ) + assert result["status"] == "updated" + assert result["schedule_id"] == "sched-old" + client.create_config.assert_not_called() + call_kwargs = client.update_config.call_args.kwargs + assert call_kwargs["config_id"] == "sched-old" + assert call_kwargs["configuration"]["schedule"]["cronTab"] == "0 6 * * *" + + def test_enabled_state_in_body(self): + client = MagicMock() + client.get_config_detail.return_value = {"name": "F"} + client.list_component_configs.return_value = [] + client.create_config.return_value = {"id": "s1"} + service = _make_flow_service(client) + service.set_flow_schedule("prod", "keboola.orchestrator", "1", "0 * * * *", enabled=False) + cfg = client.create_config.call_args.kwargs["configuration"] + assert cfg["schedule"]["state"] == "disabled" + + def test_non_404_error_on_list_schedules_propagates(self): + client = MagicMock() + client.get_config_detail.return_value = {"name": "F"} + client.list_component_configs.side_effect = KeboolaApiError( + message="Forbidden", status_code=403, error_code="INVALID_TOKEN", retryable=False + ) + service = _make_flow_service(client) + with pytest.raises(KeboolaApiError) as exc_info: + service.set_flow_schedule("prod", "keboola.orchestrator", "1", "0 * * * *") + assert exc_info.value.error_code == "INVALID_TOKEN" + client.create_config.assert_not_called() + + +# --------------------------------------------------------------------------- +# FlowService.remove_flow_schedule +# --------------------------------------------------------------------------- + + +class TestRemoveFlowSchedule: + def test_removes_matching_schedules(self): + matching = { + "id": "sched-1", + "configuration": { + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"} + }, + } + other = { + "id": "sched-2", + "configuration": { + "target": {"componentId": "keboola.orchestrator", "configurationId": "other"} + }, + } + client = MagicMock() + client.list_component_configs.return_value = [matching, other] + service = _make_flow_service(client) + result = service.remove_flow_schedule("prod", "keboola.orchestrator", "flow-1") + assert result["deleted_count"] == 1 + assert "sched-1" in result["deleted_schedule_ids"] + client.delete_config.assert_called_once_with("keboola.scheduler", "sched-1", branch_id=None) + + def test_no_schedules_is_idempotent(self): + client = MagicMock() + client.list_component_configs.return_value = [] + service = _make_flow_service(client) + result = service.remove_flow_schedule("prod", "keboola.orchestrator", "flow-1") + assert result["deleted_count"] == 0 + assert result["deleted_schedule_ids"] == [] + client.delete_config.assert_not_called() + + def test_partial_delete_failure_returns_successes(self): + sched1 = { + "id": "sched-a", + "configuration": { + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"} + }, + } + sched2 = { + "id": "sched-b", + "configuration": { + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"} + }, + } + client = MagicMock() + client.list_component_configs.return_value = [sched1, sched2] + # first delete succeeds, second raises + client.delete_config.side_effect = [ + None, + KeboolaApiError( + message="Server error", status_code=500, error_code="INTERNAL", retryable=True + ), + ] + service = _make_flow_service(client) + result = service.remove_flow_schedule("prod", "keboola.orchestrator", "flow-1") + # Partial success: first was deleted, second failed but is not re-raised when some succeeded + assert result["deleted_count"] == 1 + assert "sched-a" in result["deleted_schedule_ids"] + + def test_all_deletes_fail_raises(self): + sched1 = { + "id": "sched-x", + "configuration": { + "target": {"componentId": "keboola.orchestrator", "configurationId": "flow-1"} + }, + } + client = MagicMock() + client.list_component_configs.return_value = [sched1] + client.delete_config.side_effect = KeboolaApiError( + message="Forbidden", status_code=403, error_code="INVALID_TOKEN", retryable=False + ) + service = _make_flow_service(client) + with pytest.raises(KeboolaApiError) as exc_info: + service.remove_flow_schedule("prod", "keboola.orchestrator", "flow-1") + assert exc_info.value.error_code == "SCHEDULE_DELETE_FAILED" diff --git a/uv.lock b/uv.lock index e51a9b94..576ed251 100644 --- a/uv.lock +++ b/uv.lock @@ -439,7 +439,7 @@ wheels = [ [[package]] name = "keboola-agent-cli" -version = "0.21.2" +version = "0.22.0" source = { editable = "." } dependencies = [ { name = "httpx" }, From 0f06607a559bb17ce5c7d51734b0891ea75993ed Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:07:28 +0200 Subject: [PATCH 3/7] feat(0.22.0): config metadata CRUD + workspace orphan GC (#204) New: config metadata-list/get/set/delete-metadata -- CRUD for arbitrary key/value pairs on any configuration via branch-aware Storage API metadata endpoint (FIIA P1-3). New: config set-folder -- sugar over set-metadata for KBC.configuration. folderName; organises configs into named folder groups. New: workspace list --orphaned and workspace gc -- detect and clean up sandbox-backed workspaces whose sandbox config no longer exists (FIIA P1-4). Max's review fixes: CLAUDE.md + commands-reference docs, Rich markup escape in GC output, logger.exception on gc catch, config-metadata-workflow.md. --- CLAUDE.md | 8 +- plugins/kbagent/skills/kbagent/SKILL.md | 6 + .../kbagent/references/commands-reference.md | 8 +- .../references/config-metadata-workflow.md | 205 ++++++ src/keboola_agent_cli/changelog.py | 4 + src/keboola_agent_cli/client.py | 58 ++ src/keboola_agent_cli/commands/config.py | 257 +++++++ src/keboola_agent_cli/commands/context.py | 25 +- src/keboola_agent_cli/commands/workspace.py | 85 ++- .../hints/definitions/config.py | 190 +++++ .../hints/definitions/workspace.py | 31 + src/keboola_agent_cli/permissions.py | 6 + .../services/config_service.py | 185 ++++- .../services/workspace_service.py | 120 +++- tests/test_config_metadata.py | 662 ++++++++++++++++++ tests/test_e2e.py | 352 ++++++++++ tests/test_workspace_gc.py | 382 ++++++++++ 17 files changed, 2564 insertions(+), 20 deletions(-) create mode 100644 plugins/kbagent/skills/kbagent/references/config-metadata-workflow.md create mode 100644 tests/test_config_metadata.py create mode 100644 tests/test_workspace_gc.py diff --git a/CLAUDE.md b/CLAUDE.md index 8f6765b9..8a7b5cf1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -255,6 +255,11 @@ kbagent config rename --project NAME --component-id ID --config-id ID --name "Ne kbagent config variables-set --project NAME --component-id ID --config-id ID --var KEY=VALUE [--var ...] [--replace] [--variables-id ID] [--values-id ID] [--branch ID] [--dry-run] kbagent config variables-get --project NAME --component-id ID --config-id ID [--branch ID] kbagent config variables-clear --project NAME --component-id ID --config-id ID [--branch ID] [--yes] +kbagent config metadata-list --project NAME --component-id ID --config-id ID [--branch ID] +kbagent config get-metadata --project NAME --component-id ID --config-id ID --key KEY [--branch ID] +kbagent config set-metadata --project NAME --component-id ID --config-id ID --key KEY --value VALUE [--branch ID] +kbagent config delete-metadata --project NAME --component-id ID --config-id ID --metadata-id ID [--branch ID] [--yes] +kbagent config set-folder --project NAME --component-id ID --config-id ID --name FOLDER [--branch ID] kbagent job list [--project NAME] [--component-id ID] [--status STATUS] [--limit N] kbagent job detail --project NAME --job-id ID @@ -311,13 +316,14 @@ kbagent branch metadata-set --project NAME --key KEY [--text STR | --file PATH | kbagent branch metadata-delete --project NAME --metadata-id ID [--branch ID|default] kbagent workspace create --project ALIAS [--name NAME] [--backend TYPE] [--ui] [--read-only/--no-read-only] -kbagent workspace list [--project NAME] +kbagent workspace list [--project NAME ...] [--orphaned] kbagent workspace detail --project ALIAS --workspace-id ID kbagent workspace delete --project ALIAS --workspace-id ID kbagent workspace password --project ALIAS --workspace-id ID kbagent workspace load --project ALIAS --workspace-id ID --tables TABLE_ID [--tables ...] [--preserve] kbagent workspace query --project ALIAS --workspace-id ID --sql "SELECT ..." [--transactional] kbagent workspace query --project ALIAS --workspace-id ID --file query.sql +kbagent workspace gc [--project NAME ...] [--dry-run] [--yes] kbagent workspace from-transformation --project ALIAS --component-id ID --config-id ID [--row-id ID] kbagent component list [--project NAME] [--type TYPE] [--query QUERY] diff --git a/plugins/kbagent/skills/kbagent/SKILL.md b/plugins/kbagent/skills/kbagent/SKILL.md index a07f1140..264e3315 100644 --- a/plugins/kbagent/skills/kbagent/SKILL.md +++ b/plugins/kbagent/skills/kbagent/SKILL.md @@ -95,6 +95,11 @@ When working inside a git repository or project directory, run `kbagent init` (o | Rename a configuration (update name via API + rename local sync directory) | `kbagent config rename --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID --name NAME` | | Delete a configuration from a project | `kbagent config delete --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | | Generate boilerplate configuration files for a Keboola component | `kbagent config new --component-id COMPONENT-ID` | +| List all metadata entries on a configuration | `kbagent config metadata-list --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | +| Read a single metadata value by key | `kbagent config get-metadata --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID --key KEY` | +| Set a metadata key/value on a configuration (upsert) | `kbagent config set-metadata --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID --key KEY --value VALUE` | +| Delete a configuration metadata entry by its numeric ID | `kbagent config delete-metadata --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID --metadata-id METADATA-ID` | +| Set the folder (KBC.configuration.folderName) on a configuration | `kbagent config set-folder --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID --name NAME` | | Assign variables to a config (auto-creates backing keboola.variables on first call) | `kbagent config variables-set --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | | Read the current variable values attached to a config | `kbagent config variables-get --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | | Unlink variables from a config (does NOT delete the underlying keboola.variables) | `kbagent config variables-clear --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | @@ -160,6 +165,7 @@ When working inside a git repository or project directory, run `kbagent init` (o | Reset workspace password and show the new one | `kbagent workspace password --project PROJECT --workspace-id WORKSPACE-ID` | | Load tables into a workspace | `kbagent workspace load --project PROJECT --workspace-id WORKSPACE-ID --tables TABLES` | | Execute SQL query in a workspace via Query Service | `kbagent workspace query --project PROJECT --workspace-id WORKSPACE-ID` | +| Garbage-collect orphaned workspaces | `kbagent workspace gc` | | Create a workspace from a transformation config | `kbagent workspace from-transformation --project PROJECT --component-id COMPONENT-ID --config-id CONFIG-ID` | | List available MCP tools from the keboola-mcp-server | `kbagent tool list` | | Call an MCP tool on keboola-mcp-server | `kbagent tool call ` | diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 8c5cc142..914a9736 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -45,6 +45,11 @@ All commands support `--json` for structured output. Multi-project flags (`--pro - `config variables-set --project NAME --component-id ID --config-id ID --var KEY=VALUE [--var ...] [--replace] [--variables-id ID] [--values-id ID] [--branch ID] [--dry-run] [--allow-plaintext-on-encrypt-failure] [--yes]` -- attach variable values to a config. Auto-creates a sibling `keboola.variables` config + default row on first use and links it via the parent's `runtime.variables_id` / `variables_values_id`. Defaults to merge; `--replace` drops keys not in `--var`. `#`-prefixed values encrypt via the Encryption API (fail-closed; exit non-zero on `ENCRYPTION_FAILED`). See `variables-workflow.md` - `config variables-get --project NAME --component-id ID --config-id ID [--branch ID]` -- resolve `variables_id` + `values_id` from the parent config and fetch the current KEY=VALUE map. Returns `{linked: bool, variables_id, values_id, values}`; `linked=false` means the parent has no variables attached - `config variables-clear --project NAME --component-id ID --config-id ID [--branch ID] [--yes]` -- unlink variables from the parent config (strips `variables_id` + `variables_values_id`). **Does NOT delete** the backing `keboola.variables` config -- use `config delete` explicitly if you've verified nothing else references it +- `config metadata-list --project NAME --component-id ID --config-id ID [--branch ID]` -- list all metadata entries on a configuration (id, key, value, provider, timestamp). Branch-aware +- `config get-metadata --project NAME --component-id ID --config-id ID --key KEY [--branch ID]` -- read a single metadata value by key. Exits with `NOT_FOUND` (exit 1) if absent +- `config set-metadata --project NAME --component-id ID --config-id ID --key KEY --value VALUE [--branch ID]` -- set (upsert) a metadata key/value on a configuration. Common keys: `KBC.configuration.folderName`, plus any custom `KBC.*` agent-facing tags +- `config delete-metadata --project NAME --component-id ID --config-id ID --metadata-id ID [--branch ID] [--yes]` -- delete a configuration metadata entry by its numeric ID (from `metadata-list`) +- `config set-folder --project NAME --component-id ID --config-id ID --name FOLDER [--branch ID]` -- set (or clear, with empty `--name`) the `KBC.configuration.folderName` metadata, which groups configs into named folders in the Keboola UI. See `config-metadata-workflow.md` ## Job History - `job list [--project NAME] [--component-id ID] [--config-id ID] [--status STATUS] [--limit N]` -- list jobs (default 50, max 500) @@ -97,12 +102,13 @@ All commands support `--json` for structured output. Multi-project flags (`--pro ## Workspaces (SQL Debugging) - `workspace create --project ALIAS [--name NAME] [--ui] [--read-only]` -- create workspace (headless ~1s, `--ui` ~15s) -- `workspace list [--project NAME]` -- list workspaces +- `workspace list [--project NAME ...] [--orphaned]` -- list workspaces. `--project` repeatable for multi-project; `--orphaned` filters to workspaces whose backing `keboola.sandboxes` config is missing - `workspace detail --project ALIAS --workspace-id ID` -- show connection details - `workspace delete --project ALIAS --workspace-id ID` -- delete workspace - `workspace password --project ALIAS --workspace-id ID` -- reset and return new password - `workspace load --project ALIAS --workspace-id ID --tables TABLE_ID [...] [--preserve]` -- load storage tables - `workspace query --project ALIAS --workspace-id ID --sql "..." [--file F] [--transactional]` -- run SQL via Query Service +- `workspace gc [--project NAME ...] [--dry-run] [--yes]` -- garbage-collect orphaned workspaces (and any lingering `keboola.sandboxes` configs). `--dry-run` previews without deleting; `--project` repeatable, omit to GC across all connected projects - `workspace from-transformation --project ALIAS --component-id ID --config-id ID [--row-id ID]` -- workspace from existing transform ## MCP Tools diff --git a/plugins/kbagent/skills/kbagent/references/config-metadata-workflow.md b/plugins/kbagent/skills/kbagent/references/config-metadata-workflow.md new file mode 100644 index 00000000..d210b78d --- /dev/null +++ b/plugins/kbagent/skills/kbagent/references/config-metadata-workflow.md @@ -0,0 +1,205 @@ +# Config Metadata Workflow -- Tags, folders, and agent breadcrumbs + +Keboola stores free-form `key / value / provider` metadata on every configuration. +The Keboola UI uses a handful of `KBC.*` keys for user-facing behavior (most +notably `KBC.configuration.folderName` to group configs into folders), but the +surface is open-ended: agents can stamp their own keys to leave breadcrumbs for +later runs (e.g. `agent.owner`, `agent.lastAudit`, `agent.domain`). + +kbagent exposes the CRUD surface as five commands on `kbagent config`: + +``` +metadata-list -- list all entries (id, key, value, provider, timestamp) +get-metadata -- read one value by key +set-metadata -- upsert a single key/value +delete-metadata -- remove an entry by its numeric id +set-folder -- convenience wrapper that writes KBC.configuration.folderName +``` + +All five are branch-aware; omit `--branch` to use the project's active branch. + +## When to use this + +- **Folder organization**: group related configs under a named folder in the + Keboola UI (`set-folder`). Works across all component types; no schema change + needed. +- **Agent breadcrumbs**: tag configs an agent has touched so later runs can + skip, re-audit, or attribute them (`set-metadata --key agent.* ...`). +- **Ownership / governance tags**: stamp `owner`, `domain`, `cost-center`, + etc. and filter via `config list --json | jq` downstream. +- **Provenance tracking**: record when a config was last generated or + refactored by an automated workflow. + +## CLI cheatsheet + +```bash +# List everything on a config (sorted by key) +kbagent --json config metadata-list --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 + +# Read a specific key (exits 1 / NOT_FOUND if absent) +kbagent --json config get-metadata --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 \ + --key KBC.configuration.folderName + +# Upsert (create if new, overwrite if existing) +kbagent config set-metadata --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 \ + --key agent.owner --value analytics-team + +# Delete by numeric ID (from metadata-list) +kbagent config delete-metadata --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 \ + --metadata-id 4281 --yes + +# Folder sugar (writes KBC.configuration.folderName) +kbagent config set-folder --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 \ + --name "Customer 360" + +# Clear the folder assignment (empty string) +kbagent config set-folder --project prod \ + --component-id keboola.snowflake-transformation --config-id 15815157 \ + --name "" +``` + +## Folder organization pattern + +The Keboola UI reads `KBC.configuration.folderName` and groups configs sharing +the same value into a named folder under the component. This is purely a +presentation feature -- the config itself is unchanged, and there are no +parent/child resources to manage. That makes it ideal for programmatic +taxonomy: + +```bash +# Tag every config in an onboarding flow with one folder +for cfg_id in 15815157 15815158 15815159 15815160; do + kbagent config set-folder --project prod \ + --component-id keboola.snowflake-transformation \ + --config-id "$cfg_id" \ + --name "Customer 360 - Onboarding" +done +``` + +Guidelines: + +- `set-folder --name ""` removes the grouping (passes an empty string to + `set-metadata`, which the UI treats as "no folder"). +- Folder names are free-form strings; keep them short and stable -- the UI + sorts alphabetically. +- Prefer `set-folder` over raw `set-metadata --key KBC.configuration.folderName`; + the wrapper exists so the key spelling is not a moving target for agents. +- Folders are **per component**. Two configs under different components with + the same folder name render as two separate folders in the UI -- this is + intentional, not a bug. + +## Full lifecycle example + +```bash +PROJECT=prod +COMPONENT=keboola.snowflake-transformation +CONFIG=15815157 + +# 1. Inspect what's already on the config +kbagent --json config metadata-list \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" + +# 2. Stamp an agent breadcrumb +kbagent config set-metadata \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" \ + --key agent.lastAudit --value "2026-04-23" + +# 3. Read it back +kbagent --json config get-metadata \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" \ + --key agent.lastAudit + +# 4. File it into the Customer 360 folder +kbagent config set-folder \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" \ + --name "Customer 360" + +# 5. Later: clean the breadcrumb (list to find its numeric id, then delete) +METADATA_ID=$(kbagent --json config metadata-list \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" \ + | jq -r '.data.metadata[] | select(.key=="agent.lastAudit") | .id') + +kbagent config delete-metadata \ + --project "$PROJECT" --component-id "$COMPONENT" --config-id "$CONFIG" \ + --metadata-id "$METADATA_ID" --yes +``` + +## Response shapes (`--json` mode) + +### `metadata-list` +```json +{ + "status": "ok", + "data": { + "project_alias": "prod", + "component_id": "keboola.snowflake-transformation", + "config_id": "15815157", + "branch_id": 12345, + "metadata": [ + {"id": "4281", "key": "KBC.configuration.folderName", "value": "Customer 360", "provider": "user", "timestamp": "2026-04-23T10:15:00Z"}, + {"id": "4282", "key": "agent.lastAudit", "value": "2026-04-23", "provider": "user", "timestamp": "2026-04-23T10:16:02Z"} + ] + } +} +``` + +The `metadata` list is key-sorted for deterministic output. + +### `get-metadata` +```json +{ + "status": "ok", + "data": { + "project_alias": "prod", + "component_id": "keboola.snowflake-transformation", + "config_id": "15815157", + "branch_id": 12345, + "key": "agent.lastAudit", + "value": "2026-04-23", + "metadata_id": "4282" + } +} +``` + +Returns `NOT_FOUND` (exit 1) when the key is absent -- there is no sentinel +"empty" value; missing means missing. + +### `set-metadata` / `set-folder` / `delete-metadata` +Each returns `project_alias`, `component_id`, `config_id`, `branch_id`, and a +human-readable `message`. `set-folder` additionally returns `folder` so +callers don't have to re-parse the message string. + +## Provider semantics + +The `provider` field on each metadata entry distinguishes `user` (anything you +wrote via CLI/API/UI) from `system` (set by the Keboola platform itself). +kbagent never filters these out -- `metadata-list` surfaces both. Do not +attempt to `set-metadata` on a system-provider key; the API will return a +validation error and `delete-metadata` on a system entry is likewise rejected. + +## Relation to `config update` + +`config update` mutates the configuration body (`parameters`, `storage`, +`processors`, etc.). `config set-metadata` mutates the sibling `metadata` +array on the same configuration resource. They touch different endpoints and +never conflict. Use `config update` for anything that affects runtime +behavior; use `set-metadata` for everything else (tags, folders, audit +breadcrumbs). + +## Branch awareness + +All five commands resolve `--branch` the same way the rest of `kbagent config` +does: + +1. Explicit `--branch ID` wins. +2. Otherwise the project's **active branch** (set by `kbagent branch use` / + `branch create`) is used. +3. If the project has no active branch, the main / default branch is used. + +Metadata in a dev branch is independent of production; it merges back via the +same merge URL as the rest of the branch. See [branch-workflow.md](branch-workflow.md). diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 21a1226d..7daac234 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -23,6 +23,10 @@ "New: `kbagent flow delete` -- delete a flow config with --yes confirmation guard", "New: `kbagent flow schedule` -- attach a cron schedule via keboola.scheduler; supports timezone and enabled/disabled state", "New: `kbagent flow schedule-remove` -- remove all cron schedules attached to a flow; idempotent, --yes confirmation guard", + "New: config metadata-list / get-metadata / set-metadata / delete-metadata -- CRUD for arbitrary metadata key/value pairs on any configuration, using the branch-aware Storage API metadata endpoint (FIIA P1-3)", + "New: config set-folder -- sugar over set-metadata for KBC.configuration.folderName; organises configs into named folder groups visible in the Keboola UI (FIIA P1-3)", + "New: workspace list --orphaned -- lists workspaces backed by keboola.sandboxes whose sandbox config no longer exists (FIIA P1-4)", + "New: workspace gc [--dry-run] [--yes] -- deletes all orphaned workspaces; dry-run previews without touching anything; --yes skips interactive confirmation (FIIA P1-4)", ], "0.21.2": [ "Fix: `kbagent config search` now scans `rows[].configuration` in addition to the top-level configuration body (#196) -- queries like `--query '\"incremental\": false'` previously returned zero matches for row-based components (Snowflake/MySQL/BigQuery writers, DB extractors, Google Sheets) because the service only fetched `include=configuration`; match paths are now reported as `rows[N].configuration.parameters.`", diff --git a/src/keboola_agent_cli/client.py b/src/keboola_agent_cli/client.py index fa6f4155..bb7cfdc0 100644 --- a/src/keboola_agent_cli/client.py +++ b/src/keboola_agent_cli/client.py @@ -352,6 +352,64 @@ def list_config_folder_metadata(self, branch_id: int) -> dict[str, str]: folder_map[f"{comp_id}/{config_id}"] = meta["value"] return folder_map + def list_config_metadata( + self, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> list[dict[str, Any]]: + """List metadata entries on a configuration. + + GET /v2/storage/[branch/{b}/]components/{c}/configs/{id}/metadata + """ + prefix = f"/v2/storage/branch/{branch_id}" if branch_id else "/v2/storage" + response = self._request( + "GET", + f"{prefix}/components/{quote(component_id, safe='')}/configs/{quote(config_id, safe='')}/metadata", + ) + return response.json() + + def set_config_metadata( + self, + component_id: str, + config_id: str, + entries: list[tuple[str, str]], + branch_id: int | None = None, + ) -> list[dict[str, Any]]: + """Bulk-set metadata key/value pairs on a configuration. + + POST /v2/storage/[branch/{b}/]components/{c}/configs/{id}/metadata + Same PHP-style indexed form as set_branch_metadata. + """ + form: dict[str, str] = {} + for i, (key, value) in enumerate(entries): + form[f"metadata[{i}][key]"] = key + form[f"metadata[{i}][value]"] = value + prefix = f"/v2/storage/branch/{branch_id}" if branch_id else "/v2/storage" + response = self._request( + "POST", + f"{prefix}/components/{quote(component_id, safe='')}/configs/{quote(config_id, safe='')}/metadata", + data=form, + ) + return response.json() + + def delete_config_metadata( + self, + component_id: str, + config_id: str, + metadata_id: int | str, + branch_id: int | None = None, + ) -> None: + """Delete a single metadata entry on a configuration by its numeric ID. + + DELETE /v2/storage/[branch/{b}/]components/{c}/configs/{id}/metadata/{mid} + """ + prefix = f"/v2/storage/branch/{branch_id}" if branch_id else "/v2/storage" + self._request( + "DELETE", + f"{prefix}/components/{quote(component_id, safe='')}/configs/{quote(config_id, safe='')}/metadata/{metadata_id}", + ) + def create_config( self, component_id: str, diff --git a/src/keboola_agent_cli/commands/config.py b/src/keboola_agent_cli/commands/config.py index 705345ca..37e2e4b0 100644 --- a/src/keboola_agent_cli/commands/config.py +++ b/src/keboola_agent_cli/commands/config.py @@ -836,6 +836,263 @@ def config_new( formatter.console.print() +# ── Config metadata commands ─────────────────────────────────────────── + + +@config_app.command("metadata-list") +def config_metadata_list( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + component_id: str = typer.Option(..., "--component-id", help="Component ID"), + config_id: str = typer.Option(..., "--config-id", help="Configuration ID"), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (defaults to active branch)" + ), +) -> None: + """List all metadata entries on a configuration.""" + if should_hint(ctx): + emit_hint( + ctx, + "config.metadata-list", + project=project, + component_id=component_id, + config_id=config_id, + branch=branch, + ) + return + formatter = get_formatter(ctx) + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + service = get_service(ctx, "config_service") + try: + result = service.list_config_metadata( + alias=project, + component_id=component_id, + config_id=config_id, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + entries = result.get("metadata", []) + if not entries: + formatter.console.print("[dim]No metadata entries.[/dim]") + else: + for e in entries: + formatter.console.print( + f" [dim]{escape(str(e.get('id', '')))}[/dim] [green]{escape(e.get('key', ''))}[/green] = {escape(str(e.get('value', '')))} [dim]{escape(e.get('provider', 'user'))}[/dim]" + ) + + +@config_app.command("get-metadata") +def config_get_metadata( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + component_id: str = typer.Option(..., "--component-id", help="Component ID"), + config_id: str = typer.Option(..., "--config-id", help="Configuration ID"), + key: str = typer.Option(..., "--key", help="Metadata key to read"), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (defaults to active branch)" + ), +) -> None: + """Read a single metadata value by key. + + Exits with code 1 (NOT_FOUND) if the key is not present. + """ + if should_hint(ctx): + emit_hint( + ctx, + "config.get-metadata", + project=project, + component_id=component_id, + config_id=config_id, + key=key, + branch=branch, + ) + return + formatter = get_formatter(ctx) + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + service = get_service(ctx, "config_service") + try: + result = service.get_config_metadata_value( + alias=project, + component_id=component_id, + config_id=config_id, + key=key, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + formatter.output(result, lambda c, d: c.print(d["value"])) + + +@config_app.command("set-metadata") +def config_set_metadata( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + component_id: str = typer.Option(..., "--component-id", help="Component ID"), + config_id: str = typer.Option(..., "--config-id", help="Configuration ID"), + key: str = typer.Option(..., "--key", help="Metadata key to set"), + value: str = typer.Option(..., "--value", help="Metadata value (string)"), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (defaults to active branch)" + ), +) -> None: + """Set a metadata key/value on a configuration (upsert).""" + if should_hint(ctx): + emit_hint( + ctx, + "config.set-metadata", + project=project, + component_id=component_id, + config_id=config_id, + key=key, + value=value, + branch=branch, + ) + return + formatter = get_formatter(ctx) + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + service = get_service(ctx, "config_service") + try: + result = service.set_config_metadata( + alias=project, + component_id=component_id, + config_id=config_id, + key=key, + value=value, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + formatter.output( + result, lambda c, d: c.print(f"[bold green]Success:[/bold green] {d['message']}") + ) + + +@config_app.command("delete-metadata") +def config_delete_metadata( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + component_id: str = typer.Option(..., "--component-id", help="Component ID"), + config_id: str = typer.Option(..., "--config-id", help="Configuration ID"), + metadata_id: int = typer.Option(..., "--metadata-id", help="Numeric ID from metadata-list"), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (defaults to active branch)" + ), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation"), +) -> None: + """Delete a configuration metadata entry by its numeric ID.""" + if should_hint(ctx): + emit_hint( + ctx, + "config.delete-metadata", + project=project, + component_id=component_id, + config_id=config_id, + metadata_id=metadata_id, + branch=branch, + ) + return + formatter = get_formatter(ctx) + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + if ( + not yes + and not formatter.json_mode + and not typer.confirm(f"Delete metadata ID {metadata_id} from {component_id}/{config_id}?") + ): + formatter.console.print("Aborted.") + raise typer.Exit(code=0) + + service = get_service(ctx, "config_service") + try: + result = service.delete_config_metadata( + alias=project, + component_id=component_id, + config_id=config_id, + metadata_id=metadata_id, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + formatter.output( + result, lambda c, d: c.print(f"[bold green]Success:[/bold green] {d['message']}") + ) + + +@config_app.command("set-folder") +def config_set_folder( + ctx: typer.Context, + project: str = typer.Option(..., "--project", help="Project alias"), + component_id: str = typer.Option(..., "--component-id", help="Component ID"), + config_id: str = typer.Option(..., "--config-id", help="Configuration ID"), + name: str = typer.Option(..., "--name", help="Folder name (empty string to clear)"), + branch: int | None = typer.Option( + None, "--branch", help="Dev branch ID (defaults to active branch)" + ), +) -> None: + """Set the folder (KBC.configuration.folderName) on a configuration. + + Organises configs into named groups in the Keboola UI. + Pass an empty string to remove the folder assignment. + """ + if should_hint(ctx): + emit_hint( + ctx, + "config.set-folder", + project=project, + component_id=component_id, + config_id=config_id, + name=name, + branch=branch, + ) + return + formatter = get_formatter(ctx) + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + service = get_service(ctx, "config_service") + try: + result = service.set_config_folder( + alias=project, + component_id=component_id, + config_id=config_id, + folder_name=name, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + formatter.output( + result, lambda c, d: c.print(f"[bold green]Success:[/bold green] {d['message']}") + ) + + def _parse_kv_var(raw: str) -> tuple[str, str]: """Split a ``KEY=VALUE`` token into ``(key, value)``; ``#``-prefix preserved.""" if "=" not in raw: diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index 105a57da..a3e458e4 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -142,6 +142,24 @@ Unlink variables from a config. Does NOT delete the underlying keboola.variables config (it may be shared). Delete it explicitly via `kbagent config delete` if needed. +### Config Metadata (folder organisation + arbitrary key/value) + + kbagent config metadata-list --project NAME --component-id ID --config-id ID [--branch ID] + List all metadata entries on a configuration. Each entry: id, key, value, provider, timestamp. + + kbagent config get-metadata --project NAME --component-id ID --config-id ID --key KEY [--branch ID] + Read a single metadata value by key. Exits 1 (NOT_FOUND) if absent. + + kbagent config set-metadata --project NAME --component-id ID --config-id ID --key KEY --value VALUE [--branch ID] + Set (upsert) a metadata key/value on a configuration. + + kbagent config delete-metadata --project NAME --component-id ID --config-id ID --metadata-id ID [--branch ID] [--yes] + Delete a configuration metadata entry by numeric ID (from metadata-list). + + kbagent config set-folder --project NAME --component-id ID --config-id ID --name "FolderName" [--branch ID] + Sugar: writes KBC.configuration.folderName metadata. Groups the config in the Keboola UI. + Pass --name "" to remove the folder assignment. + ### Job History kbagent job list [--project NAME] [--component-id ID] [--config-id ID] [--status STATUS] [--limit N] @@ -385,8 +403,8 @@ kbagent workspace create --project ALIAS [--name NAME] [--backend TYPE] [--ui] [--read-only/--no-read-only] Create workspace. Backend auto-detected from project (or override with --backend). Default: headless (~1s). --ui: visible in KBC UI (~15s). - kbagent workspace list [--project NAME] - List workspaces. --project repeatable. + kbagent workspace list [--project NAME] [--orphaned] + List workspaces. --orphaned shows only orphaned workspaces (sandboxes config missing). kbagent workspace detail --project ALIAS --workspace-id ID Workspace connection details (no password). @@ -406,6 +424,9 @@ kbagent workspace from-transformation --project ALIAS --component-id ID --config-id ID [--row-id ID] Create workspace from transformation config. Loads input tables automatically. + kbagent workspace gc [--project NAME] [--dry-run] [--yes] + Garbage-collect orphaned workspaces (keboola.sandboxes config missing). Use --dry-run to preview. + ### Project Sync kbagent sync init --project ALIAS [--directory DIR] [--git-branching] diff --git a/src/keboola_agent_cli/commands/workspace.py b/src/keboola_agent_cli/commands/workspace.py index 0be72a0b..048acdfd 100644 --- a/src/keboola_agent_cli/commands/workspace.py +++ b/src/keboola_agent_cli/commands/workspace.py @@ -7,6 +7,7 @@ from pathlib import Path import typer +from rich.markup import escape from ..errors import ConfigError, KeboolaApiError from ..output import format_query_results, format_workspaces_table @@ -115,6 +116,11 @@ def workspace_list( "--project", help="Project alias to query (can be repeated for multiple projects)", ), + orphaned: bool = typer.Option( + False, + "--orphaned", + help="Show only orphaned workspaces (keboola.sandboxes config missing)", + ), ) -> None: """List workspaces from connected projects.""" if should_hint(ctx): @@ -124,7 +130,11 @@ def workspace_list( service = get_service(ctx, "workspace_service") try: - result = service.list_workspaces(aliases=project) + result = service.list_workspaces(aliases=project, orphaned_only=orphaned) + except KeboolaApiError as exc: + exit_code = map_error_to_exit_code(exc) + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=exit_code) from None except ConfigError as exc: formatter.error(message=exc.message, error_code="CONFIG_ERROR") raise typer.Exit(code=5) from None @@ -420,6 +430,79 @@ def workspace_query( raise typer.Exit(code=5) from None +@workspace_app.command("gc") +def workspace_gc( + ctx: typer.Context, + project: list[str] | None = typer.Option( + None, + "--project", + help="Project alias to query (can be repeated). None = all projects.", + ), + dry_run: bool = typer.Option( + False, + "--dry-run", + help="List orphaned workspaces without deleting them", + ), + yes: bool = typer.Option( + False, + "--yes", + "-y", + help="Skip confirmation prompt", + ), +) -> None: + """Garbage-collect orphaned workspaces. + + An orphaned workspace is one backed by keboola.sandboxes whose + sandbox config no longer exists. Running gc deletes those workspaces + (and any lingering sandbox configs). Use --dry-run to preview first. + """ + if should_hint(ctx): + emit_hint(ctx, "workspace.gc", project=project, dry_run=dry_run) + return + formatter = get_formatter(ctx) + service = get_service(ctx, "workspace_service") + + if ( + not dry_run + and not yes + and not formatter.json_mode + and not typer.confirm("Delete all orphaned workspaces in the selected project(s)?") + ): + formatter.console.print("Aborted.") + raise typer.Exit(code=0) + + try: + result = service.gc_workspaces(aliases=project, dry_run=dry_run) + except KeboolaApiError as exc: + exit_code = map_error_to_exit_code(exc) + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=exit_code) from None + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + + if formatter.json_mode: + formatter.output(result) + else: + formatter.console.print(result.get("message", "")) + if dry_run: + would_delete = result.get("would_delete", []) + for ws in would_delete: + formatter.console.print( + f" [dim]would delete[/dim] workspace {ws['id']} " + f"([cyan]{escape(ws.get('name', ''))}[/cyan]) in '{escape(ws['project_alias'])}'" + ) + else: + for ws in result.get("deleted", []): + formatter.console.print( + f" [green]deleted[/green] workspace {ws['id']} in '{escape(ws['project_alias'])}'" + ) + for err in result.get("errors", []): + formatter.console.print( + f" [red]error[/red] workspace {err.get('workspace_id', '?')}: {escape(err.get('error', ''))}" + ) + + @workspace_app.command("from-transformation") def workspace_from_transformation( ctx: typer.Context, diff --git a/src/keboola_agent_cli/hints/definitions/config.py b/src/keboola_agent_cli/hints/definitions/config.py index d019bb16..aaa4ce59 100644 --- a/src/keboola_agent_cli/hints/definitions/config.py +++ b/src/keboola_agent_cli/hints/definitions/config.py @@ -286,3 +286,193 @@ ], ) ) + +# ── config metadata-list ─────────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="config.metadata-list", + description="List all metadata entries on a configuration", + steps=[ + HintStep( + comment="List configuration metadata", + client=ClientCall( + method="list_config_metadata", + args={ + "component_id": "{component_id}", + "config_id": "{config_id}", + "branch_id": "{branch}", + }, + result_var="entries", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="ConfigService", + service_module="config_service", + method="list_config_metadata", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{config_id}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=["Each entry has: id, key, value, provider, timestamp."], + ) +) + +# ── config get-metadata ──────────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="config.get-metadata", + description="Read a single metadata value by key from a configuration", + steps=[ + HintStep( + comment="Get single metadata value", + client=ClientCall( + method="list_config_metadata", + args={ + "component_id": "{component_id}", + "config_id": "{config_id}", + "branch_id": "{branch}", + }, + result_var="entries", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="ConfigService", + service_module="config_service", + method="get_config_metadata_value", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{config_id}", + "key": "{key}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=["Raises NOT_FOUND (exit 1) if key is absent."], + ) +) + +# ── config set-metadata ──────────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="config.set-metadata", + description="Set (upsert) a metadata key/value on a configuration", + steps=[ + HintStep( + comment="Upsert metadata entry on configuration", + client=ClientCall( + method="set_config_metadata", + args={ + "component_id": "{component_id}", + "config_id": "{config_id}", + "entries": "[({key}, {value})]", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="ConfigService", + service_module="config_service", + method="set_config_metadata", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{config_id}", + "key": "{key}", + "value": "{value}", + "branch_id": "{branch}", + }, + ), + ), + ], + ) +) + +# ── config delete-metadata ───────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="config.delete-metadata", + description="Delete a configuration metadata entry by its numeric ID", + steps=[ + HintStep( + comment="Delete metadata entry by ID", + client=ClientCall( + method="delete_config_metadata", + args={ + "component_id": "{component_id}", + "config_id": "{config_id}", + "metadata_id": "{metadata_id}", + "branch_id": "{branch}", + }, + result_var=None, + result_hint="None", + ), + service=ServiceCall( + service_class="ConfigService", + service_module="config_service", + method="delete_config_metadata", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{config_id}", + "metadata_id": "{metadata_id}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=["Use metadata-list first to find the numeric metadata_id."], + ) +) + +# ── config set-folder ────────────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="config.set-folder", + description="Set the folder (KBC.configuration.folderName) on a configuration", + steps=[ + HintStep( + comment="Write KBC.configuration.folderName metadata", + client=ClientCall( + method="set_config_metadata", + args={ + "component_id": "{component_id}", + "config_id": "{config_id}", + "entries": "[('KBC.configuration.folderName', {name})]", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="ConfigService", + service_module="config_service", + method="set_config_folder", + args={ + "alias": "{project}", + "component_id": "{component_id}", + "config_id": "{config_id}", + "folder_name": "{name}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Folder names appear in the Keboola UI to group configurations.", + "config list already shows folder names in the 'folder' column.", + ], + ) +) diff --git a/src/keboola_agent_cli/hints/definitions/workspace.py b/src/keboola_agent_cli/hints/definitions/workspace.py index 436ecb68..17694956 100644 --- a/src/keboola_agent_cli/hints/definitions/workspace.py +++ b/src/keboola_agent_cli/hints/definitions/workspace.py @@ -252,3 +252,34 @@ ], ) ) + +# ── workspace gc ─────────────────────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="workspace.gc", + description="Garbage-collect orphaned workspaces (keboola.sandboxes config missing)", + steps=[ + HintStep( + comment="List orphaned workspaces then delete each one", + client=ClientCall( + method="list_workspaces", + args={"branch_id": "{branch}"}, + result_var="workspaces", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="WorkspaceService", + service_module="workspace_service", + method="gc_workspaces", + args={"aliases": "{project}", "dry_run": "{dry_run}"}, + ), + ), + ], + notes=[ + "Orphan = workspace whose keboola.sandboxes config no longer exists.", + "Use --dry-run first to preview without deleting.", + "Reuses delete_workspace internally (also cleans up any lingering sandbox config).", + ], + ) +) diff --git a/src/keboola_agent_cli/permissions.py b/src/keboola_agent_cli/permissions.py index 55757cf9..faa0ff96 100644 --- a/src/keboola_agent_cli/permissions.py +++ b/src/keboola_agent_cli/permissions.py @@ -35,6 +35,11 @@ "config.variables-set": "write", "config.variables-get": "read", "config.variables-clear": "destructive", + "config.metadata-list": "read", + "config.get-metadata": "read", + "config.set-metadata": "write", + "config.delete-metadata": "destructive", + "config.set-folder": "write", # Job history "job.list": "read", "job.detail": "read", @@ -74,6 +79,7 @@ "workspace.load": "write", "workspace.query": "write", "workspace.from-transformation": "write", + "workspace.gc": "destructive", # MCP tools "tool.list": "read", "tool.call": "write", diff --git a/src/keboola_agent_cli/services/config_service.py b/src/keboola_agent_cli/services/config_service.py index c5269585..bfd5010d 100644 --- a/src/keboola_agent_cli/services/config_service.py +++ b/src/keboola_agent_cli/services/config_service.py @@ -12,7 +12,7 @@ from pathlib import Path from typing import Any -from ..errors import KeboolaApiError +from ..errors import ConfigError, KeboolaApiError from ..json_utils import compute_diff, deep_merge, set_nested_value from ..models import ProjectConfig from ..sync.manifest import Manifest, load_manifest, save_manifest @@ -642,6 +642,189 @@ def _find_sync_branch_dir(manifest: Manifest, project_root: Path) -> Path | None branch_dir = project_root / branch_path return branch_dir if branch_dir.exists() else None + def _resolve_metadata_branch_id( + self, project: ProjectConfig, client: Any, branch_id: int | None + ) -> int: + """Resolve the branch ID required by the config metadata API. + + Config metadata endpoints only support the branch-aware route + (/v2/storage/branch/{id}/...). This method resolves the effective + branch: explicit arg → active branch → default branch from API. + + Raises ConfigError if no default branch can be found. + """ + effective = branch_id or project.active_branch_id + if effective: + return int(effective) + try: + branches = client.list_dev_branches() + except KeboolaApiError as exc: + raise ConfigError( + f"Could not list branches to resolve metadata branch: {exc.message}. " + "Pass --branch explicitly." + ) from exc + except Exception as exc: + raise ConfigError( + f"Unexpected error listing branches for metadata route: {exc}. " + "Pass --branch explicitly." + ) from exc + default = next((b for b in branches if b.get("isDefault")), None) + if default: + return int(default["id"]) + raise ConfigError( + "Could not determine a branch for config metadata. " + "Set an active branch with 'kbagent branch use' or pass --branch." + ) + + def list_config_metadata( + self, + alias: str, + component_id: str, + config_id: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """List all metadata entries on a configuration. + + Returns: + Dict with project_alias, component_id, config_id, branch_id, + and a key-sorted metadata list. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + client = self._client_factory(project.stack_url, project.token) + try: + effective_branch_id = self._resolve_metadata_branch_id(project, client, branch_id) + entries = client.list_config_metadata( + component_id, config_id, branch_id=effective_branch_id + ) + return { + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "branch_id": effective_branch_id, + "metadata": sorted(entries, key=lambda e: e.get("key", "")), + } + finally: + client.close() + + def get_config_metadata_value( + self, + alias: str, + component_id: str, + config_id: str, + key: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Get a single metadata value by key. + + Raises KeboolaApiError(NOT_FOUND) if the key is absent. + """ + result = self.list_config_metadata(alias, component_id, config_id, branch_id=branch_id) + for entry in result["metadata"]: + if entry.get("key") == key: + return { + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "branch_id": result["branch_id"], + "key": key, + "value": entry.get("value"), + "metadata_id": entry.get("id"), + } + raise KeboolaApiError( + message=f"Metadata key '{key}' not found on config '{component_id}/{config_id}'.", + status_code=404, + error_code="NOT_FOUND", + retryable=False, + ) + + def set_config_metadata( + self, + alias: str, + component_id: str, + config_id: str, + key: str, + value: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Set a single metadata key/value on a configuration (upsert).""" + projects = self.resolve_projects([alias]) + project = projects[alias] + client = self._client_factory(project.stack_url, project.token) + try: + effective_branch_id = self._resolve_metadata_branch_id(project, client, branch_id) + result = client.set_config_metadata( + component_id, config_id, entries=[(key, value)], branch_id=effective_branch_id + ) + return { + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "branch_id": effective_branch_id, + "key": key, + "value": value, + "result": result, + "message": ( + f"Metadata '{key}' set on config '{component_id}/{config_id}' in project '{alias}'." + ), + } + finally: + client.close() + + def delete_config_metadata( + self, + alias: str, + component_id: str, + config_id: str, + metadata_id: int | str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Delete a metadata entry by its numeric ID.""" + projects = self.resolve_projects([alias]) + project = projects[alias] + client = self._client_factory(project.stack_url, project.token) + try: + effective_branch_id = self._resolve_metadata_branch_id(project, client, branch_id) + client.delete_config_metadata( + component_id, config_id, metadata_id, branch_id=effective_branch_id + ) + return { + "project_alias": alias, + "component_id": component_id, + "config_id": config_id, + "branch_id": effective_branch_id, + "metadata_id": metadata_id, + "message": ( + f"Metadata ID {metadata_id} deleted from config " + f"'{component_id}/{config_id}' in project '{alias}'." + ), + } + finally: + client.close() + + def set_config_folder( + self, + alias: str, + component_id: str, + config_id: str, + folder_name: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Set the folder name on a configuration (KBC.configuration.folderName).""" + result = self.set_config_metadata( + alias, + component_id, + config_id, + key="KBC.configuration.folderName", + value=folder_name, + branch_id=branch_id, + ) + result["folder"] = folder_name + result["message"] = ( + f"Folder '{folder_name}' set on config '{component_id}/{config_id}' in project '{alias}'." + ) + return result + def search_configs( self, query: str, diff --git a/src/keboola_agent_cli/services/workspace_service.py b/src/keboola_agent_cli/services/workspace_service.py index 072e2650..8edc0ee4 100644 --- a/src/keboola_agent_cli/services/workspace_service.py +++ b/src/keboola_agent_cli/services/workspace_service.py @@ -15,6 +15,21 @@ logger = logging.getLogger(__name__) +def _is_orphaned_workspace(ws: dict[str, Any], config_names: dict[str, str]) -> bool: + """Return True if a workspace has no backing keboola.sandboxes config. + + A workspace is orphaned when it is tied to keboola.sandboxes (the normal + kbagent creation path) but the sandbox config no longer exists — either it + was deleted separately or was never created. + """ + component_id = ws.get("component_id", "") + config_id = str(ws.get("config_id", "")) + if component_id != "keboola.sandboxes": + return False + # config_names keys are sandbox config IDs; absence means orphan + return not config_id or config_id not in config_names + + class WorkspaceService(BaseService): """Business logic for managing Keboola workspaces. @@ -257,11 +272,14 @@ def _create_workspace_via_job( def list_workspaces( self, aliases: list[str] | None = None, + orphaned_only: bool = False, ) -> dict[str, Any]: """List workspaces across one or multiple projects. Args: aliases: Project aliases to query. None means all projects. + orphaned_only: If True, return only orphaned workspaces — those + whose keboola.sandboxes config no longer exists. Returns: Dict with "workspaces" and "errors" lists. @@ -283,20 +301,24 @@ def worker( for ws in raw_workspaces: connection = ws.get("connection", {}) config_id = ws.get("configurationId") or "" - workspaces.append( - { - "project_alias": alias, - "id": ws.get("id"), - "name": config_names.get(str(config_id), ws.get("name", "")), - "backend": connection.get("backend", ""), - "host": connection.get("host", ""), - "schema": connection.get("schema", ""), - "user": connection.get("user", ""), - "created": ws.get("created", ""), - "component_id": ws.get("component") or "", - "config_id": config_id, - } - ) + component_id = ws.get("component") or "" + entry = { + "project_alias": alias, + "id": ws.get("id"), + "name": config_names.get(str(config_id), ws.get("name", "")), + "backend": connection.get("backend", ""), + "host": connection.get("host", ""), + "schema": connection.get("schema", ""), + "user": connection.get("user", ""), + "created": ws.get("created", ""), + "component_id": component_id, + "config_id": config_id, + } + if orphaned_only: + if _is_orphaned_workspace(entry, config_names): + workspaces.append(entry) + else: + workspaces.append(entry) return (alias, workspaces, True) except KeboolaApiError as exc: return ( @@ -333,6 +355,75 @@ def worker( "errors": errors, } + def gc_workspaces( + self, + aliases: list[str] | None = None, + dry_run: bool = False, + ) -> dict[str, Any]: + """Delete all orphaned workspaces (workspace GC). + + An orphan is a keboola.sandboxes-backed workspace whose config no longer + exists. Reuses delete_workspace for each orphan so the sandbox config + cleanup path is also exercised. + + Args: + aliases: Project aliases to query. None means all projects. + dry_run: If True, list orphans without deleting. + + Returns: + Dict with dry_run flag, would_delete/deleted list, errors, count. + """ + orphan_result = self.list_workspaces(aliases=aliases, orphaned_only=True) + orphans = orphan_result["workspaces"] + list_errors = orphan_result["errors"] + + if dry_run: + return { + "dry_run": True, + "would_delete": orphans, + "count": len(orphans), + "errors": list_errors, + "message": ( + f"DRY RUN: {len(orphans)} orphaned workspace(s) would be deleted." + + (" No errors." if not list_errors else f" {len(list_errors)} list error(s).") + ), + } + + deleted: list[dict[str, Any]] = [] + delete_errors: list[dict[str, Any]] = [] + for ws in orphans: + try: + self.delete_workspace(alias=ws["project_alias"], workspace_id=ws["id"]) + deleted.append(ws) + except Exception as exc: + # Full traceback goes to the logger (observability for unexpected + # errors like AttributeError); user-facing flow is unchanged. + logger.exception( + "Failed to delete orphaned workspace %s in project %s", + ws["id"], + ws["project_alias"], + ) + delete_errors.append( + { + "workspace_id": ws["id"], + "project_alias": ws["project_alias"], + "error": str(exc), + } + ) + + all_errors = list_errors + delete_errors + return { + "dry_run": False, + "deleted": deleted, + "errors": all_errors, + "count_deleted": len(deleted), + "count_errors": len(all_errors), + "message": ( + f"GC complete: {len(deleted)} orphaned workspace(s) deleted" + + (f", {len(all_errors)} error(s)." if all_errors else ".") + ), + } + def get_workspace(self, alias: str, workspace_id: int) -> dict[str, Any]: """Get workspace details (password NOT included). @@ -384,6 +475,7 @@ def delete_workspace(self, alias: str, workspace_id: int) -> dict[str, Any]: try: # Get workspace details to find associated config config_id = None + component = None try: ws_data = client.get_workspace(workspace_id, branch_id=branch_id) component = ws_data.get("component") diff --git a/tests/test_config_metadata.py b/tests/test_config_metadata.py new file mode 100644 index 00000000..9cb85392 --- /dev/null +++ b/tests/test_config_metadata.py @@ -0,0 +1,662 @@ +"""Tests for config metadata CLI commands and service methods. + +Covers: metadata-list, get-metadata, set-metadata, delete-metadata, set-folder. +Uses mocked services (CLI layer) and mocked HTTP client (service layer). +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest +from typer.testing import CliRunner + +from keboola_agent_cli.cli import app +from keboola_agent_cli.config_store import ConfigStore +from keboola_agent_cli.errors import ConfigError, KeboolaApiError +from keboola_agent_cli.models import ProjectConfig +from keboola_agent_cli.services.config_service import ConfigService +from keboola_agent_cli.services.job_service import JobService +from keboola_agent_cli.services.project_service import ProjectService +from keboola_agent_cli.services.workspace_service import WorkspaceService + +runner = CliRunner() + +TEST_TOKEN = "test-token-123" +TEST_URL = "https://connection.keboola.com" +COMP_ID = "keboola.ex-db-snowflake" +CFG_ID = "my-config" + +SAMPLE_ENTRIES = [ + { + "id": 1, + "key": "KBC.configuration.folderName", + "value": "extractors", + "provider": "user", + "timestamp": "2025-01-01T00:00:00Z", + }, + { + "id": 2, + "key": "my.custom.tag", + "value": "production", + "provider": "user", + "timestamp": "2025-01-01T00:00:00Z", + }, +] + + +def _setup_store(tmp_path: Path) -> ConfigStore: + config_dir = tmp_path / "config" + config_dir.mkdir() + store = ConfigStore(config_dir=config_dir) + store.add_project( + "prod", + ProjectConfig( + stack_url=TEST_URL, + token=TEST_TOKEN, + project_name="Prod", + project_id=1, + ), + ) + return store + + +def _invoke(store: ConfigStore, mock_cfg_svc: MagicMock, *args: str) -> Any: + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjSvc, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgSvc, + patch("keboola_agent_cli.cli.JobService") as MockJobSvc, + patch("keboola_agent_cli.cli.WorkspaceService") as MockWsSvc, + ): + MockStore.return_value = store + MockProjSvc.return_value = ProjectService(config_store=store) + MockCfgSvc.return_value = mock_cfg_svc + MockJobSvc.return_value = JobService(config_store=store) + MockWsSvc.return_value = WorkspaceService(config_store=store) + return runner.invoke(app, list(args)) + + +# ── metadata-list ────────────────────────────────────────────────────── + + +class TestConfigMetadataList: + def test_list_json_success(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.list_config_metadata.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "metadata": SAMPLE_ENTRIES, + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "metadata-list", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["status"] == "ok" + assert len(data["data"]["metadata"]) == 2 + mock_svc.list_config_metadata.assert_called_once_with( + alias="prod", component_id=COMP_ID, config_id=CFG_ID, branch_id=None + ) + + def test_list_empty(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.list_config_metadata.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "metadata": [], + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "metadata-list", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["metadata"] == [] + + def test_list_api_error(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.list_config_metadata.side_effect = KeboolaApiError( + message="Not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + result = _invoke( + store, + mock_svc, + "--json", + "config", + "metadata-list", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + ) + assert result.exit_code == 1 + data = json.loads(result.output) + assert data["status"] == "error" + + def test_list_config_error_exits_5(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.list_config_metadata.side_effect = ConfigError("No active branch") + result = _invoke( + store, + mock_svc, + "--json", + "config", + "metadata-list", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + ) + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["status"] == "error" + assert data["error"]["code"] == "CONFIG_ERROR" + + +# ── get-metadata ─────────────────────────────────────────────────────── + + +class TestConfigGetMetadata: + def test_get_json_success(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.get_config_metadata_value.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "key": "my.custom.tag", + "value": "production", + "metadata_id": 2, + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "get-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--key", + "my.custom.tag", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["value"] == "production" + assert data["data"]["metadata_id"] == 2 + + def test_get_not_found_exits_1(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.get_config_metadata_value.side_effect = KeboolaApiError( + message="Metadata key 'missing' not found.", + status_code=404, + error_code="NOT_FOUND", + retryable=False, + ) + result = _invoke( + store, + mock_svc, + "--json", + "config", + "get-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--key", + "missing", + ) + assert result.exit_code == 1 + data = json.loads(result.output) + assert data["error"]["code"] == "NOT_FOUND" + + def test_get_config_error_exits_5(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.get_config_metadata_value.side_effect = ConfigError("Branch resolution failed") + result = _invoke( + store, + mock_svc, + "--json", + "config", + "get-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--key", + "k", + ) + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["error"]["code"] == "CONFIG_ERROR" + + +# ── set-metadata ─────────────────────────────────────────────────────── + + +class TestConfigSetMetadata: + def test_set_json_success(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_metadata.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "key": "env", + "value": "production", + "result": [{"id": 5, "key": "env", "value": "production"}], + "message": "Metadata 'env' set on config ...", + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--key", + "env", + "--value", + "production", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["status"] == "ok" + assert data["data"]["key"] == "env" + mock_svc.set_config_metadata.assert_called_once_with( + alias="prod", + component_id=COMP_ID, + config_id=CFG_ID, + key="env", + value="production", + branch_id=None, + ) + + def test_set_api_error(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_metadata.side_effect = KeboolaApiError( + message="Config not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + "bad-id", + "--key", + "k", + "--value", + "v", + ) + assert result.exit_code == 1 + + def test_set_config_error_exits_5(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_metadata.side_effect = ConfigError("Branch resolution failed") + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--key", + "k", + "--value", + "v", + ) + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["error"]["code"] == "CONFIG_ERROR" + + +# ── delete-metadata ──────────────────────────────────────────────────── + + +class TestConfigDeleteMetadata: + def test_delete_with_yes_flag(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.delete_config_metadata.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "metadata_id": 2, + "message": "Metadata ID 2 deleted.", + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "delete-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--metadata-id", + "2", + "--yes", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["metadata_id"] == 2 + mock_svc.delete_config_metadata.assert_called_once_with( + alias="prod", + component_id=COMP_ID, + config_id=CFG_ID, + metadata_id=2, + branch_id=None, + ) + + def test_delete_config_error_exits_5(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.delete_config_metadata.side_effect = ConfigError("Branch resolution failed") + result = _invoke( + store, + mock_svc, + "--json", + "config", + "delete-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--metadata-id", + "1", + "--yes", + ) + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["error"]["code"] == "CONFIG_ERROR" + + def test_delete_api_error(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.delete_config_metadata.side_effect = KeboolaApiError( + message="Not found", status_code=404, error_code="NOT_FOUND", retryable=False + ) + result = _invoke( + store, + mock_svc, + "--json", + "config", + "delete-metadata", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--metadata-id", + "999", + "--yes", + ) + assert result.exit_code == 1 + + +# ── set-folder ───────────────────────────────────────────────────────── + + +class TestConfigSetFolder: + def test_set_folder_success(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_folder.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "key": "KBC.configuration.folderName", + "value": "My Folder", + "folder": "My Folder", + "result": [{"id": 3, "key": "KBC.configuration.folderName", "value": "My Folder"}], + "message": "Folder 'My Folder' set on config ...", + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-folder", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--name", + "My Folder", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["folder"] == "My Folder" + mock_svc.set_config_folder.assert_called_once_with( + alias="prod", + component_id=COMP_ID, + config_id=CFG_ID, + folder_name="My Folder", + branch_id=None, + ) + + def test_set_folder_config_error_exits_5(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_folder.side_effect = ConfigError("Branch resolution failed") + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-folder", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--name", + "Bad Folder", + ) + assert result.exit_code == 5 + data = json.loads(result.output) + assert data["error"]["code"] == "CONFIG_ERROR" + + def test_set_folder_empty_clears(self, tmp_path: Path) -> None: + """Empty folder name is a valid call (clearing the folder).""" + store = _setup_store(tmp_path) + mock_svc = MagicMock() + mock_svc.set_config_folder.return_value = { + "project_alias": "prod", + "component_id": COMP_ID, + "config_id": CFG_ID, + "branch_id": None, + "folder": "", + "message": "Folder '' set.", + } + result = _invoke( + store, + mock_svc, + "--json", + "config", + "set-folder", + "--project", + "prod", + "--component-id", + COMP_ID, + "--config-id", + CFG_ID, + "--name", + "", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["folder"] == "" + mock_svc.set_config_folder.assert_called_once_with( + alias="prod", + component_id=COMP_ID, + config_id=CFG_ID, + folder_name="", + branch_id=None, + ) + + +# ── ConfigService unit tests (mocked client) ────────────────────────── + + +class TestConfigServiceMetadata: + """Test ConfigService metadata methods with a mocked KeboolaClient.""" + + def _make_service(self, tmp_path: Path) -> tuple[ConfigService, MagicMock]: + config_dir = tmp_path / "config" + config_dir.mkdir() + store = ConfigStore(config_dir=config_dir) + store.add_project( + "prod", + # active_branch_id set so _resolve_metadata_branch_id skips the API call + ProjectConfig( + stack_url=TEST_URL, + token=TEST_TOKEN, + project_name="Prod", + project_id=1, + active_branch_id=1, + ), + ) + mock_client = MagicMock() + mock_client.close = MagicMock() + svc = ConfigService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + return svc, mock_client + + def test_list_config_metadata_sorted(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.list_config_metadata.return_value = [ + {"id": 2, "key": "z.key", "value": "b"}, + {"id": 1, "key": "a.key", "value": "a"}, + ] + result = svc.list_config_metadata("prod", COMP_ID, CFG_ID) + assert result["metadata"][0]["key"] == "a.key" + assert result["metadata"][1]["key"] == "z.key" + mock_client.list_config_metadata.assert_called_once_with(COMP_ID, CFG_ID, branch_id=1) + mock_client.close.assert_called_once() + + def test_get_config_metadata_value_found(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.list_config_metadata.return_value = [ + {"id": 5, "key": "my.key", "value": "hello", "provider": "user"}, + ] + result = svc.get_config_metadata_value("prod", COMP_ID, CFG_ID, "my.key") + assert result["value"] == "hello" + assert result["metadata_id"] == 5 + + def test_get_config_metadata_value_not_found(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.list_config_metadata.return_value = [] + with pytest.raises(KeboolaApiError) as exc_info: + svc.get_config_metadata_value("prod", COMP_ID, CFG_ID, "missing") + assert exc_info.value.error_code == "NOT_FOUND" + + def test_set_config_metadata_wire_shape(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.set_config_metadata.return_value = [{"id": 7, "key": "env", "value": "prod"}] + result = svc.set_config_metadata("prod", COMP_ID, CFG_ID, key="env", value="prod") + assert result["key"] == "env" + assert result["value"] == "prod" + # Verify the client receives entries as list of tuples + mock_client.set_config_metadata.assert_called_once_with( + COMP_ID, CFG_ID, entries=[("env", "prod")], branch_id=1 + ) + + def test_delete_config_metadata(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.delete_config_metadata.return_value = None + result = svc.delete_config_metadata("prod", COMP_ID, CFG_ID, metadata_id=7) + assert result["metadata_id"] == 7 + mock_client.delete_config_metadata.assert_called_once_with(COMP_ID, CFG_ID, 7, branch_id=1) + mock_client.close.assert_called_once() + + def test_set_config_folder_uses_correct_key(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + mock_client.set_config_metadata.return_value = [ + {"id": 9, "key": "KBC.configuration.folderName", "value": "My Group"} + ] + result = svc.set_config_folder("prod", COMP_ID, CFG_ID, folder_name="My Group") + assert result["folder"] == "My Group" + mock_client.set_config_metadata.assert_called_once_with( + COMP_ID, + CFG_ID, + entries=[("KBC.configuration.folderName", "My Group")], + branch_id=1, + ) diff --git a/tests/test_e2e.py b/tests/test_e2e.py index 752984c7..cf94d19d 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -28,6 +28,7 @@ from __future__ import annotations +import contextlib import csv import json import os @@ -4005,3 +4006,354 @@ def test_flow_list_no_project_returns_all(self) -> None: data = json.loads(result.output)["data"] assert "flows" in data assert "errors" in data + + +# --------------------------------------------------------------------------- +# PR8: Config metadata + Workspace GC (standalone, no storage dependency) +# --------------------------------------------------------------------------- + + +@skip_without_credentials +@pytest.mark.e2e +class TestE2EPR8ConfigMetadata: + """End-to-end tests for config metadata CRUD commands (PR8). + + Creates a real keboola.ex-db-snowflake config, exercises the full + metadata round-trip (metadata-list / set-metadata / get-metadata / + delete-metadata / set-folder), then deletes the config. + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + self.alias = f"{RUN_ID}-meta" + + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + + self.api = KeboolaClient(self.url, self.token) + self._created_config_ids: list[tuple[str, str]] = [] + + # Register project + result = _invoke( + self.config_dir, + [ + "--json", + "project", + "add", + "--project", + self.alias, + "--url", + self.url, + "--token", + self.token, + ], + ) + assert result.exit_code == 0, f"project add failed: {result.output}" + + @pytest.fixture(autouse=True) + def cleanup(self) -> Any: + yield + for comp_id, cfg_id in self._created_config_ids: + with contextlib.suppress(Exception): + self.api.delete_config(comp_id, cfg_id) + + def _run(self, *args: str) -> Any: + return _invoke(self.config_dir, ["--json", *args]) + + def _run_ok(self, *args: str) -> dict[str, Any]: + return _json_ok(self._run(*args)) + + def test_config_metadata_crud_roundtrip(self) -> None: + """Full metadata CRUD: list (empty) → set → get → list (present) → delete → list (gone).""" + # Create a config to attach metadata to + cfg = self.api.create_config( + component_id=TEST_COMPONENT_ID, + name=f"{RUN_ID}-meta-test", + configuration={}, + description="E2E PR8 metadata test", + ) + config_id = str(cfg["id"]) + self._created_config_ids.append((TEST_COMPONENT_ID, config_id)) + + custom_key = f"E2E.PR8.{RUN_ID}" + + _step(1, "metadata-list on fresh config -- should be empty") + data = self._run_ok( + "config", + "metadata-list", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + ) + assert isinstance(data["data"]["metadata"], list) + initial_count = len(data["data"]["metadata"]) + + _step(2, "set-metadata -- upsert custom key") + data = self._run_ok( + "config", + "set-metadata", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + "--key", + custom_key, + "--value", + "pr8-value", + ) + assert data["data"]["key"] == custom_key + assert data["data"]["value"] == "pr8-value" + + _step(3, "get-metadata -- value round-trips") + data = self._run_ok( + "config", + "get-metadata", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + "--key", + custom_key, + ) + assert data["data"]["value"] == "pr8-value" + + _step(4, "metadata-list -- custom key appears") + data = self._run_ok( + "config", + "metadata-list", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + ) + entries = data["data"]["metadata"] + assert len(entries) == initial_count + 1 + match = next((e for e in entries if e.get("key") == custom_key), None) + assert match is not None + metadata_id = str(match["id"]) + + _step(5, "delete-metadata -- remove by ID") + data = self._run_ok( + "config", + "delete-metadata", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + "--metadata-id", + metadata_id, + "--yes", + ) + assert metadata_id in data["data"]["message"] + + _step(6, "metadata-list after delete -- key is gone") + data = self._run_ok( + "config", + "metadata-list", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + ) + remaining = {e.get("key") for e in data["data"]["metadata"]} + assert custom_key not in remaining + + def test_set_folder_sugar(self) -> None: + """set-folder writes KBC.configuration.folderName metadata.""" + cfg = self.api.create_config( + component_id=TEST_COMPONENT_ID, + name=f"{RUN_ID}-folder-test", + configuration={}, + description="E2E PR8 set-folder test", + ) + config_id = str(cfg["id"]) + self._created_config_ids.append((TEST_COMPONENT_ID, config_id)) + + folder_name = f"PR8-Folder-{RUN_ID}" + + _step(1, "set-folder -- write KBC.configuration.folderName") + data = self._run_ok( + "config", + "set-folder", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + "--name", + folder_name, + ) + assert data["data"]["folder"] == folder_name + assert data["data"]["key"] == "KBC.configuration.folderName" + + _step(2, "metadata-list -- folder key is visible") + data = self._run_ok( + "config", + "metadata-list", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + ) + folder_entry = next( + (e for e in data["data"]["metadata"] if e.get("key") == "KBC.configuration.folderName"), + None, + ) + assert folder_entry is not None + assert folder_entry["value"] == folder_name + + def test_get_metadata_missing_key_exits_1(self) -> None: + """get-metadata for a non-existent key returns exit code 1.""" + cfg = self.api.create_config( + component_id=TEST_COMPONENT_ID, + name=f"{RUN_ID}-meta-missing", + configuration={}, + description="E2E PR8 missing key test", + ) + config_id = str(cfg["id"]) + self._created_config_ids.append((TEST_COMPONENT_ID, config_id)) + + result = self._run( + "config", + "get-metadata", + "--project", + self.alias, + "--component-id", + TEST_COMPONENT_ID, + "--config-id", + config_id, + "--key", + "does.not.exist", + ) + assert result.exit_code == 1 + data = json.loads(result.output) + assert data["status"] == "error" + + +@skip_without_credentials +@pytest.mark.e2e +class TestE2EPR8WorkspaceGC: + """End-to-end tests for workspace list --orphaned and workspace gc (PR8). + + Creates a real workspace, deletes its backing sandbox config via direct API + call to manufacture an orphan, then verifies the GC commands detect and + remove it. + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + self.alias = f"{RUN_ID}-gc" + + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + + self.api = KeboolaClient(self.url, self.token) + self._created_workspace_ids: list[int] = [] + + # Register project + result = _invoke( + self.config_dir, + [ + "--json", + "project", + "add", + "--project", + self.alias, + "--url", + self.url, + "--token", + self.token, + ], + ) + assert result.exit_code == 0, f"project add failed: {result.output}" + + @pytest.fixture(autouse=True) + def cleanup(self) -> Any: + yield + for ws_id in self._created_workspace_ids: + with contextlib.suppress(Exception): + self.api.delete_workspace(ws_id) + + def _run(self, *args: str) -> Any: + return _invoke(self.config_dir, ["--json", *args]) + + def _run_ok(self, *args: str) -> dict[str, Any]: + return _json_ok(self._run(*args)) + + def test_workspace_gc_orphan_roundtrip(self) -> None: + """Create workspace, orphan it by deleting sandbox config, verify GC finds and removes it.""" + _step(1, "workspace create") + result = self._run("workspace", "create", "--project", self.alias) + if result.exit_code != 0: + pytest.skip(f"workspace create not supported: {result.output}") + + data = _json_ok(result) + ws_id = data["data"]["workspace_id"] + assert ws_id > 0 + self._created_workspace_ids.append(ws_id) + + _step(2, "retrieve workspace to find sandbox config_id") + ws_data = self.api.get_workspace(ws_id) + config_id = str(ws_data.get("configurationId") or ws_data.get("config_id") or "") + if not config_id: + pytest.skip("workspace has no configurationId, cannot manufacture orphan") + + _step(3, "delete sandbox config to make the workspace orphaned") + try: + self.api.delete_config("keboola.sandboxes", config_id) + except Exception as exc: + pytest.skip(f"could not delete sandbox config: {exc}") + + _step(4, "workspace list --orphaned -- workspace should appear") + data = self._run_ok("workspace", "list", "--project", self.alias, "--orphaned") + orphan_ids = [w["id"] for w in data["data"]["workspaces"]] + assert ws_id in orphan_ids, f"ws {ws_id} not listed as orphan; got: {orphan_ids}" + + _step(5, "workspace gc --dry-run -- counts but does not delete") + data = self._run_ok("workspace", "gc", "--project", self.alias, "--dry-run") + gc_data = data["data"] + assert gc_data["dry_run"] is True + would_delete_ids = [w["id"] for w in gc_data.get("would_delete", [])] + assert ws_id in would_delete_ids + + # Verify workspace still exists after dry-run + remaining = self._run_ok("workspace", "list", "--project", self.alias, "--orphaned") + assert ws_id in [w["id"] for w in remaining["data"]["workspaces"]] + + _step(6, "workspace gc --yes -- deletes the orphan") + data = self._run_ok("workspace", "gc", "--project", self.alias, "--yes") + gc_data = data["data"] + assert gc_data["dry_run"] is False + deleted_ids = [w["id"] for w in gc_data.get("deleted", [])] + assert ws_id in deleted_ids + + # Remove from cleanup tracker since GC deleted it + if ws_id in self._created_workspace_ids: + self._created_workspace_ids.remove(ws_id) + + _step(7, "workspace list --orphaned -- workspace is gone") + data = self._run_ok("workspace", "list", "--project", self.alias, "--orphaned") + remaining_ids = [w["id"] for w in data["data"]["workspaces"]] + assert ws_id not in remaining_ids diff --git a/tests/test_workspace_gc.py b/tests/test_workspace_gc.py new file mode 100644 index 00000000..8a3b0f73 --- /dev/null +++ b/tests/test_workspace_gc.py @@ -0,0 +1,382 @@ +"""Tests for workspace --orphaned flag and workspace gc command. + +Covers: + - workspace list --orphaned (service-level filtering + CLI) + - workspace gc --dry-run (preview without deletion) + - workspace gc (delete orphans, error accumulation) + - _is_orphaned_workspace helper +""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +from typer.testing import CliRunner + +from keboola_agent_cli.cli import app +from keboola_agent_cli.config_store import ConfigStore +from keboola_agent_cli.errors import KeboolaApiError +from keboola_agent_cli.models import ProjectConfig +from keboola_agent_cli.services.config_service import ConfigService +from keboola_agent_cli.services.job_service import JobService +from keboola_agent_cli.services.project_service import ProjectService +from keboola_agent_cli.services.workspace_service import WorkspaceService, _is_orphaned_workspace + +runner = CliRunner() + +TEST_TOKEN = "test-token-456" +TEST_URL = "https://connection.keboola.com" + + +def _setup_store(tmp_path: Path) -> ConfigStore: + config_dir = tmp_path / "config" + config_dir.mkdir() + store = ConfigStore(config_dir=config_dir) + store.add_project( + "prod", + ProjectConfig( + stack_url=TEST_URL, + token=TEST_TOKEN, + project_name="Prod", + project_id=1, + ), + ) + return store + + +def _invoke(store: ConfigStore, mock_ws_svc: MagicMock, *args: str, input: str | None = None): + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjSvc, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgSvc, + patch("keboola_agent_cli.cli.JobService") as MockJobSvc, + patch("keboola_agent_cli.cli.WorkspaceService") as MockWsSvc, + ): + MockStore.return_value = store + MockProjSvc.return_value = ProjectService(config_store=store) + MockCfgSvc.return_value = ConfigService(config_store=store) + MockJobSvc.return_value = JobService(config_store=store) + MockWsSvc.return_value = mock_ws_svc + return runner.invoke(app, list(args), input=input) + + +# ── _is_orphaned_workspace unit tests ───────────────────────────────── + + +class TestIsOrphanedWorkspace: + def test_non_sandboxes_component_never_orphan(self) -> None: + ws = {"component_id": "keboola.snowflake-transformation", "config_id": "cfg-1"} + assert not _is_orphaned_workspace(ws, {"cfg-1": "My Config"}) + + def test_sandboxes_with_existing_config_not_orphan(self) -> None: + ws = {"component_id": "keboola.sandboxes", "config_id": "cfg-1"} + assert not _is_orphaned_workspace(ws, {"cfg-1": "My Workspace"}) + + def test_sandboxes_with_missing_config_is_orphan(self) -> None: + ws = {"component_id": "keboola.sandboxes", "config_id": "cfg-missing"} + assert _is_orphaned_workspace(ws, {"cfg-other": "Other"}) + + def test_sandboxes_with_empty_config_id_is_orphan(self) -> None: + ws = {"component_id": "keboola.sandboxes", "config_id": ""} + assert _is_orphaned_workspace(ws, {"cfg-1": "My Workspace"}) + + def test_sandboxes_with_no_config_id_key_is_orphan(self) -> None: + ws = {"component_id": "keboola.sandboxes"} + assert _is_orphaned_workspace(ws, {"cfg-1": "My Workspace"}) + + +# ── workspace list --orphaned CLI tests ─────────────────────────────── + + +class TestWorkspaceListOrphaned: + def test_list_orphaned_json(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_ws = MagicMock() + orphan_ws = { + "project_alias": "prod", + "id": 99, + "name": "orphan-ws", + "backend": "snowflake", + "host": "host.snowflake.com", + "schema": "WORKSPACE_99", + "user": "u", + "created": "2025-01-01", + "component_id": "keboola.sandboxes", + "config_id": "", + } + mock_ws.list_workspaces.return_value = { + "workspaces": [orphan_ws], + "errors": [], + } + result = _invoke( + store, + mock_ws, + "--json", + "workspace", + "list", + "--project", + "prod", + "--orphaned", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert len(data["data"]["workspaces"]) == 1 + # Verify service was called with orphaned_only=True + mock_ws.list_workspaces.assert_called_once_with(aliases=["prod"], orphaned_only=True) + + def test_list_without_orphaned_flag(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_ws = MagicMock() + mock_ws.list_workspaces.return_value = {"workspaces": [], "errors": []} + result = _invoke( + store, + mock_ws, + "--json", + "workspace", + "list", + "--project", + "prod", + ) + assert result.exit_code == 0, result.output + # orphaned_only=False (default) when flag absent + mock_ws.list_workspaces.assert_called_once_with(aliases=["prod"], orphaned_only=False) + + +# ── workspace gc CLI tests ───────────────────────────────────────────── + + +class TestWorkspaceGc: + def test_gc_dry_run_json(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_ws = MagicMock() + mock_ws.gc_workspaces.return_value = { + "dry_run": True, + "would_delete": [{"id": 5, "project_alias": "prod", "name": "orphan"}], + "count": 1, + "errors": [], + "message": "DRY RUN: 1 orphaned workspace(s) would be deleted.", + } + result = _invoke( + store, + mock_ws, + "--json", + "workspace", + "gc", + "--project", + "prod", + "--dry-run", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["dry_run"] is True + assert data["data"]["count"] == 1 + mock_ws.gc_workspaces.assert_called_once_with(aliases=["prod"], dry_run=True) + + def test_gc_delete_with_yes_flag(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_ws = MagicMock() + mock_ws.gc_workspaces.return_value = { + "dry_run": False, + "deleted": [{"id": 5, "project_alias": "prod"}], + "errors": [], + "count_deleted": 1, + "count_errors": 0, + "message": "GC complete: 1 orphaned workspace(s) deleted.", + } + result = _invoke( + store, + mock_ws, + "--json", + "workspace", + "gc", + "--project", + "prod", + "--yes", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["count_deleted"] == 1 + assert data["data"]["dry_run"] is False + + def test_gc_no_confirmation_aborts(self, tmp_path: Path) -> None: + """In non-JSON mode without --yes, answering 'n' exits 0 without calling service.""" + store = _setup_store(tmp_path) + mock_ws = MagicMock() + result = _invoke( + store, + mock_ws, + "workspace", + "gc", + "--project", + "prod", + input="n\n", + ) + assert result.exit_code == 0, result.output + assert "Aborted" in result.output + mock_ws.gc_workspaces.assert_not_called() + + def test_gc_nothing_to_delete(self, tmp_path: Path) -> None: + store = _setup_store(tmp_path) + mock_ws = MagicMock() + mock_ws.gc_workspaces.return_value = { + "dry_run": False, + "deleted": [], + "errors": [], + "count_deleted": 0, + "count_errors": 0, + "message": "GC complete: 0 orphaned workspace(s) deleted.", + } + result = _invoke( + store, + mock_ws, + "--json", + "workspace", + "gc", + "--project", + "prod", + "--yes", + ) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["data"]["count_deleted"] == 0 + + +# ── WorkspaceService.gc_workspaces unit tests ────────────────────────── + + +class TestWorkspaceServiceGc: + """Test gc_workspaces service method with a mocked client.""" + + def _make_service(self, tmp_path: Path) -> tuple[WorkspaceService, MagicMock]: + config_dir = tmp_path / "config" + config_dir.mkdir() + store = ConfigStore(config_dir=config_dir) + store.add_project( + "prod", + ProjectConfig(stack_url=TEST_URL, token=TEST_TOKEN, project_name="Prod", project_id=1), + ) + mock_client = MagicMock() + mock_client.close = MagicMock() + svc = WorkspaceService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + return svc, mock_client + + def _setup_client_for_list( + self, + mock_client: MagicMock, + raw_workspaces: list[dict], + sandbox_configs: list[dict], + ) -> None: + # list_dev_branches → branch_id + mock_client.list_dev_branches.return_value = [{"id": 100, "isDefault": True}] + mock_client.list_workspaces.return_value = raw_workspaces + mock_client.list_component_configs.return_value = sandbox_configs + + def test_gc_dry_run_returns_orphans_without_deleting(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + orphan_ws = { + "id": 77, + "name": "WORKSPACE_77", + "component": "keboola.sandboxes", + "configurationId": "orphan-cfg", + "connection": {"backend": "snowflake"}, + "created": "2025-01-01", + } + self._setup_client_for_list( + mock_client, + raw_workspaces=[orphan_ws], + sandbox_configs=[], # no sandbox configs → orphan + ) + result = svc.gc_workspaces(aliases=["prod"], dry_run=True) + assert result["dry_run"] is True + assert result["count"] == 1 + assert result["would_delete"][0]["id"] == 77 + # delete_workspace should NOT have been called + mock_client.delete_workspace.assert_not_called() + + def test_gc_deletes_orphan_and_sandbox_config(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + orphan_ws = { + "id": 99, + "name": "WORKSPACE_99", + "component": "keboola.sandboxes", + "configurationId": "orphan-cfg", + "connection": {"backend": "snowflake"}, + "created": "2025-01-01", + } + self._setup_client_for_list( + mock_client, + raw_workspaces=[orphan_ws], + sandbox_configs=[], + ) + # delete_workspace in service also calls get_workspace then delete + mock_client.get_workspace.return_value = { + "id": 99, + "component": "keboola.sandboxes", + "configurationId": "orphan-cfg", + "connection": {}, + } + mock_client.delete_workspace.return_value = None + mock_client.delete_config.return_value = None + + result = svc.gc_workspaces(aliases=["prod"], dry_run=False) + assert result["dry_run"] is False + assert result["count_deleted"] == 1 + assert result["count_errors"] == 0 + mock_client.delete_workspace.assert_called_once_with(99, branch_id=100) + mock_client.delete_config.assert_called_once_with( + "keboola.sandboxes", "orphan-cfg", branch_id=100 + ) + + def test_gc_skips_non_sandbox_workspaces(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + # Transformation workspace — should NOT be considered orphaned + tfm_ws = { + "id": 55, + "name": "WORKSPACE_55", + "component": "keboola.snowflake-transformation", + "configurationId": "tfm-cfg", + "connection": {"backend": "snowflake"}, + "created": "2025-01-01", + } + self._setup_client_for_list( + mock_client, + raw_workspaces=[tfm_ws], + sandbox_configs=[], + ) + result = svc.gc_workspaces(aliases=["prod"], dry_run=True) + assert result["count"] == 0 + assert result["would_delete"] == [] + + def test_gc_delete_error_accumulated(self, tmp_path: Path) -> None: + svc, mock_client = self._make_service(tmp_path) + orphan_ws = { + "id": 77, + "name": "WORKSPACE_77", + "component": "keboola.sandboxes", + "configurationId": "orphan-cfg", + "connection": {"backend": "snowflake"}, + "created": "2025-01-01", + } + self._setup_client_for_list( + mock_client, + raw_workspaces=[orphan_ws], + sandbox_configs=[], + ) + mock_client.get_workspace.return_value = { + "id": 77, + "component": "keboola.sandboxes", + "configurationId": "orphan-cfg", + "connection": {}, + } + mock_client.delete_workspace.side_effect = KeboolaApiError( + message="Delete failed", status_code=500, error_code="INTERNAL_ERROR", retryable=True + ) + result = svc.gc_workspaces(aliases=["prod"], dry_run=False) + # Error should be accumulated, not raised + assert result["count_deleted"] == 0 + assert result["count_errors"] == 1 From 105f74be3d751a432cc303e4f7a4579dca9cb48d Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:09:44 +0200 Subject: [PATCH 4/7] feat(0.22.0): storage describe-bucket/table/column/batch (#205) New: storage describe-bucket/table/column -- set descriptions via KBC metadata (provider=user); readable via bucket-detail / table-detail. New: storage describe-batch --from-file YAML -- apply bucket/table/column descriptions in one shot; failures collected, remaining items continue. Fix: bucket-detail / table-detail now return description + metadata fields; KBC.description metadata wins over native bucket description field. Max's review fixes: commands-reference docs, metadata-extraction tests pinning precedence, gotchas for description conventions, progress spinner for describe-batch human mode, storage-describe-workflow.md. --- CLAUDE.md | 4 + plugins/kbagent/skills/kbagent/SKILL.md | 4 + .../kbagent/references/commands-reference.md | 4 + .../skills/kbagent/references/gotchas.md | 29 + .../references/storage-describe-workflow.md | 258 ++++++ src/keboola_agent_cli/changelog.py | 6 + src/keboola_agent_cli/client.py | 63 ++ src/keboola_agent_cli/commands/context.py | 16 + src/keboola_agent_cli/commands/storage.py | 396 +++++++++ .../hints/definitions/storage.py | 154 ++++ src/keboola_agent_cli/permissions.py | 5 + .../services/storage_service.py | 278 ++++++- tests/test_client.py | 151 ++++ tests/test_e2e.py | 99 +++ tests/test_storage_describe_cli.py | 757 ++++++++++++++++++ tests/test_storage_describe_service.py | 564 +++++++++++++ 16 files changed, 2787 insertions(+), 1 deletion(-) create mode 100644 plugins/kbagent/skills/kbagent/references/storage-describe-workflow.md create mode 100644 tests/test_storage_describe_cli.py create mode 100644 tests/test_storage_describe_service.py diff --git a/CLAUDE.md b/CLAUDE.md index 8a7b5cf1..c2b8a9cd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -277,6 +277,10 @@ kbagent storage download-table --project NAME --table-id ID [--output FILE] [--c kbagent storage delete-table --project NAME --table-id ID [--table-id ...] [--force] [--dry-run] [--yes] [--branch ID] kbagent storage delete-column --project NAME --table-id ID --column COL [--column ...] [--force] [--dry-run] [--yes] [--branch ID] kbagent storage delete-bucket --project NAME --bucket-id ID [--bucket-id ...] [--force] [--dry-run] [--yes] [--branch ID] +kbagent storage describe-bucket --project NAME --bucket-id ID [--text STR | --file PATH | --stdin] [--branch ID] +kbagent storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID] +kbagent storage describe-column --project NAME --table-id ID --column NAME=DESC [--column ...] [--branch ID] +kbagent storage describe-batch --project NAME --from-file YAML [--branch ID] kbagent storage files --project NAME [--tag TAG ...] [--limit N] [--offset N] [--query Q] [--branch ID] kbagent storage file-upload --project NAME --file PATH [--name NAME] [--tag TAG ...] [--permanent] [--branch ID] kbagent storage file-download --project NAME [--file-id ID | --tag TAG ...] [--output FILE] diff --git a/plugins/kbagent/skills/kbagent/SKILL.md b/plugins/kbagent/skills/kbagent/SKILL.md index 264e3315..4ff2f4bb 100644 --- a/plugins/kbagent/skills/kbagent/SKILL.md +++ b/plugins/kbagent/skills/kbagent/SKILL.md @@ -118,6 +118,10 @@ When working inside a git repository or project directory, run `kbagent init` (o | Delete one or more storage tables | `kbagent storage delete-table --project PROJECT --table-id TABLE-ID` | | Delete one or more columns from a storage table | `kbagent storage delete-column --project PROJECT --table-id TABLE-ID --column COLUMN` | | Delete one or more storage buckets | `kbagent storage delete-bucket --project PROJECT --bucket-id BUCKET-ID` | +| Set the description on a storage bucket | `kbagent storage describe-bucket --project PROJECT --bucket-id BUCKET-ID` | +| Set the description on a storage table | `kbagent storage describe-table --project PROJECT --table-id TABLE-ID` | +| Set descriptions on one or more columns of a storage table | `kbagent storage describe-column --project PROJECT --table-id TABLE-ID --column COLUMN` | +| Apply descriptions to buckets, tables, and columns from a YAML file | `kbagent storage describe-batch --project PROJECT --from-file FROM-FILE` | | List Storage Files with optional tag filtering | `kbagent storage files --project PROJECT` | | Show Storage File metadata (without downloading) | `kbagent storage file-detail --project PROJECT --file-id FILE-ID` | | Upload a local file to Storage Files | `kbagent storage file-upload --project PROJECT --file FILE` | diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 914a9736..1706b248 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -69,6 +69,10 @@ All commands support `--json` for structured output. Multi-project flags (`--pro - `storage delete-table --project NAME --table-id ID [--table-id ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete tables, --force cascade-deletes aliased tables (branch-aware) - `storage delete-column --project NAME --table-id ID --column COL [--column ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete columns from a table (branch-aware) - `storage delete-bucket --project NAME --bucket-id ID [--bucket-id ...] [--force] [--dry-run] [--yes] [--branch ID]` -- delete buckets (branch-aware) +- `storage describe-bucket --project NAME --bucket-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a bucket description (stored as `KBC.description` in bucket metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage bucket-detail` +- `storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a table description (stored as `KBC.description` in table metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage table-detail` +- `storage describe-column --project NAME --table-id ID --column NAME=DESCRIPTION [--column ...] [--branch ID]` -- set one or more column descriptions. Stored as `KBC.column.{name}.description` keys in the table's metadata (Keboola has no user-writable column-metadata endpoint). Read back in `storage table-detail` under `column_details[].description` +- `storage describe-batch --project NAME --from-file PATH [--branch ID]` -- apply bucket/table/column descriptions from a YAML file (top-level `buckets`, `tables`, `columns` sections, all optional). Partial-failure tolerant: per-item errors are collected and reported, the batch does not abort. Non-zero exit only when at least one item failed ## Storage Files - `storage files --project NAME [--tag TAG ...] [--limit N] [--offset N] [--query Q] [--branch ID]` -- list Storage Files, optionally filtered by tag/query diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index 500da6ec..ab241f4d 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -501,6 +501,35 @@ They live at different endpoints in the Storage API (`/v2/storage/branch/{id}/metadata` vs. `/v2/storage/dev-branches/{id}`), so setting a branch's description will **not** update the dashboard. +## Storage descriptions: key convention + precedence + partial failures + +`kbagent storage describe-bucket / describe-table / describe-column / describe-batch` +write descriptive metadata onto storage objects. Three behaviors are easy to miss: + +- **Column descriptions use a metadata-key convention, not a column endpoint.** + The Keboola Storage API has no user-writable column-level metadata endpoint, + so `describe-column` stores each description as a `KBC.column.{name}.description` + entry on the **table's** metadata (upsert). `storage table-detail` reads them + back via the same key and surfaces them under `column_details[].description`. + Renaming or deleting a column does NOT automatically clean these entries up + (they remain on the table's metadata under the old name). Same convention for + table and bucket descriptions: stored as `KBC.description` (provider=user) on + the object's metadata. +- **`describe-batch` is partial-failure-tolerant.** Item-level errors are + collected into `result.errors[]` but the batch keeps processing the remaining + items. The CLI exits non-zero only if `error_count > 0`, so in scripts always + inspect `errors[]` (or at least `error_count`) rather than relying solely on + the exit code — and when consuming `--json` output, never trust a zero-exit + as "everything applied." +- **Description-field precedence: metadata wins.** When both the native Storage + API `description` field and a user-provided `KBC.description` (provider=user) + metadata entry are present, `storage bucket-detail` / `storage table-detail` + surface the **metadata value**. The native field is only settable at object + creation time via the Storage API; all user updates flow through the metadata + endpoint, so the metadata entry is the authoritative source. `KBC.description` + entries whose provider is not `user` (e.g. `system`) are ignored during + read-back and the native field is used as fallback. + ## `job terminate` quirks Queue API's kill endpoint (`POST /jobs/{id}/kill`) has a few non-obvious behaviors the diff --git a/plugins/kbagent/skills/kbagent/references/storage-describe-workflow.md b/plugins/kbagent/skills/kbagent/references/storage-describe-workflow.md new file mode 100644 index 00000000..e4044464 --- /dev/null +++ b/plugins/kbagent/skills/kbagent/references/storage-describe-workflow.md @@ -0,0 +1,258 @@ +# Storage Describe Workflow + +`kbagent storage describe-*` attaches human-readable descriptions to storage +buckets, tables, and columns so that downstream consumers (dashboards, the +MCP `get_buckets`/`get_tables` tools, AI agents) can surface meaningful +documentation rather than raw IDs. Descriptions are stored as metadata on +the storage object and round-trip via `storage bucket-detail` / `storage +table-detail`. + +## Quick reference + +| Command | Purpose | +|---------|---------| +| `storage describe-bucket` | Set a bucket description | +| `storage describe-table` | Set a table description | +| `storage describe-column` | Set descriptions on one or more columns | +| `storage describe-batch` | Apply bucket/table/column descriptions from a YAML file | +| `storage bucket-detail` | Read back the bucket description | +| `storage table-detail` | Read back the table description and `column_details[].description` | + +## When to use + +- Onboarding a new project: document every source bucket, output table, and + business-critical column so new engineers (or Kai) can self-serve. +- After a schema migration: refresh column descriptions so SQL reviews can + spot intent mismatches. +- Before sharing a bucket cross-project: the description is visible in the + receiving project's dashboard. +- From CI: write a batch YAML alongside the repo and call `describe-batch` + after every `sync push` to keep documentation in lockstep with config. + +## Storage model (what actually gets written) + +Descriptions are stored as metadata entries on the object: + +- **Bucket description** -- `KBC.description` (provider=user) on bucket metadata +- **Table description** -- `KBC.description` (provider=user) on table metadata +- **Column description** -- `KBC.column.{column_name}.description` on the + **table's** metadata. Keboola has no user-writable column-metadata endpoint, + so this key convention is the storage layer for column descriptions. Read + them back via `storage table-detail` (`column_details[].description`). + +Descriptions are `upsert`: calling `describe-*` with a new text replaces +whatever was there before. There is no append mode. + +## Single-item: bucket + +```bash +# Inline text +kbagent --json storage describe-bucket \ + --project ALIAS \ + --bucket-id in.c-sales \ + --text "Daily sales fact data, partitioned by region" + +# From a file (markdown supported) +kbagent --json storage describe-bucket \ + --project ALIAS \ + --bucket-id in.c-sales \ + --file ./docs/sales-bucket.md + +# From stdin (useful in pipelines) +echo "Generated description" | kbagent --json storage describe-bucket \ + --project ALIAS \ + --bucket-id in.c-sales \ + --stdin +``` + +Exactly one of `--text`, `--file`, `--stdin` must be provided. + +Read back: + +```bash +kbagent --json storage bucket-detail --project ALIAS --bucket-id in.c-sales \ + | jq '.data.description, .data.metadata' +``` + +## Single-item: table + +Identical shape to `describe-bucket`: + +```bash +kbagent --json storage describe-table \ + --project ALIAS \ + --table-id in.c-sales.orders \ + --text "All sales orders, one row per line item" +``` + +Read back: + +```bash +kbagent --json storage table-detail --project ALIAS --table-id in.c-sales.orders \ + | jq '.data.description, .data.column_details' +``` + +## Single-item: columns + +`describe-column` takes **one or more** `--column NAME=DESCRIPTION` flags in +a single call. All entries are applied in one API roundtrip: + +```bash +kbagent --json storage describe-column \ + --project ALIAS \ + --table-id in.c-sales.orders \ + --column "order_id=Unique order identifier" \ + --column "total=Order total in USD (gross)" \ + --column "created_at=Server-side creation timestamp (UTC)" +``` + +Column descriptions live under `KBC.column.{name}.description` on the +**table's** metadata -- they are NOT attached to the column record itself. +If you rename or delete a column, the old key lingers until you manually +clean it up (there is no `--delete-column-description` command today). + +Read back via `storage table-detail`: + +```json +{ + "data": { + "table_id": "in.c-sales.orders", + "description": "All sales orders, one row per line item", + "column_details": [ + {"name": "order_id", "type": "INTEGER", "description": "Unique order identifier"}, + {"name": "total", "type": "NUMERIC", "description": "Order total in USD (gross)"} + ] + } +} +``` + +Columns without a matching metadata entry simply omit `description`. + +## Batch: YAML schema + +For more than a handful of items, hand-maintain a YAML file and apply it +with `storage describe-batch`. The schema has three top-level sections, +all optional: + +```yaml +# descriptions.yaml +buckets: + in.c-sales: | + Sales fact and dimension tables. + Refreshed nightly from the production OLTP via Keboola ex-db-postgres. + in.c-marketing: Marketing funnel events + +tables: + in.c-sales.orders: All sales orders (one row per line item) + in.c-sales.customers: Customer master list, PII-scrubbed + in.c-marketing.events: Raw funnel events + +columns: + in.c-sales.orders: + order_id: Unique order identifier + total: Order total in USD (gross) + created_at: Server-side creation timestamp (UTC) + in.c-sales.customers: + customer_id: Primary key + email_hash: SHA-256 of the customer email (PII-scrubbed) +``` + +Apply it: + +```bash +kbagent --json storage describe-batch \ + --project ALIAS \ + --from-file ./descriptions.yaml +``` + +Response shape: + +```json +{ + "status": "ok", + "data": { + "project_alias": "ALIAS", + "applied": [ + {"type": "bucket", "id": "in.c-sales", "description": "Sales fact..."}, + {"type": "table", "id": "in.c-sales.orders", "description": "All sales orders..."}, + {"type": "columns", "id": "in.c-sales.orders", "columns": {"order_id": "...", "total": "..."}} + ], + "errors": [], + "applied_count": 3, + "error_count": 0 + } +} +``` + +In human mode, a Rich progress spinner shows per-item progress ("Describing +bucket in.c-sales", "Describing table in.c-sales.orders", ...) so large +batches do not look frozen. The spinner is suppressed under `--json` so +structured output is the only thing on stdout. + +## Partial-failure semantics + +`describe-batch` does **not** abort on the first error. Each item is +attempted independently; failures are collected into `errors[]` and the +batch continues: + +```json +{ + "data": { + "applied": [{"type": "bucket", "id": "in.c-good", ...}], + "errors": [ + {"type": "bucket", "id": "in.c-typo", "error": "Bucket in.c-typo not found"}, + {"type": "table", "id": "in.c-x.missing", "error": "Table not found"} + ], + "applied_count": 1, + "error_count": 2 + } +} +``` + +The CLI exits **1** when `error_count > 0`. In scripts, always inspect the +`errors[]` list -- a zero exit alone does not mean the whole batch went in +without issues (it means there were no partial failures). A non-zero exit +means *some* items failed; the successful items still landed. + +## End-to-end example: onboarding a new bucket + +```bash +# 1. Create the bucket and tables (or sync them from another project) +kbagent storage create-bucket --project ALIAS --stage in --name c-sales +kbagent storage create-table --project ALIAS --bucket-id in.c-sales --name orders \ + --column order_id:INTEGER --column total:NUMERIC --primary-key order_id + +# 2. Apply all descriptions from a tracked YAML file +kbagent --json storage describe-batch \ + --project ALIAS \ + --from-file ./docs/keboola/descriptions.yaml + +# 3. Verify by reading back +kbagent --json storage table-detail --project ALIAS --table-id in.c-sales.orders \ + | jq '{description: .data.description, columns: .data.column_details}' +``` + +## Precedence vs the native description field + +The Storage API has a native `description` field on buckets and tables, but +it is only settable at creation time. Anything you set with `describe-*` +lives on the metadata endpoint. When both are present, `storage bucket-detail` +/ `storage table-detail` surface the metadata value (the one you wrote with +`describe-*`). The native field is the fallback for legacy objects where +no metadata entry exists. System-provided `KBC.description` entries (e.g. +those auto-stamped by components) are filtered out on read-back -- only +entries with `provider="user"` are considered the canonical description. + +## Key behaviors + +- `describe-*` is **upsert** -- no append mode; re-running replaces the value. +- Column descriptions piggy-back on table metadata via the + `KBC.column.{name}.description` key convention. +- `describe-batch` is **partial-failure-tolerant** -- check `errors[]` even + on exit code 0. +- All commands support `--branch ID` to target a dev branch. +- Read back via `storage bucket-detail` / `storage table-detail` -- the + `metadata` field on those responses contains the raw metadata array if + you need to inspect timestamps or providers. +- Non-user (`system`) `KBC.description` entries are ignored on read-back; + they do not override the native `description` field. diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 7daac234..811c4afc 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -38,6 +38,12 @@ "Fix: `--hint client job run --branch ID` now threads `branch_id` through all three client calls (get_config_detail, list_config_rows, create_job) -- previously the branch arg was silently dropped, causing the hint to target production", "Chore: `.gitignore` whitelists `.env.example` and `.env.template` so documentation/scaffolding env templates can be tracked alongside the catch-all `.env.*` ignore rule", "Chore: `rich.markup.escape` import hoisted to module level in commands/job.py", + "New: storage describe-bucket -- set KBC.description on a bucket via metadata POST (upsert-by-key, provider=user)", + "New: storage describe-table -- set KBC.description on a table via metadata POST; description surfaces in table-detail", + "New: storage describe-column -- set per-column descriptions using KBC.column.{name}.description convention in table metadata; readable via table-detail column_details[].description", + "New: storage describe-batch --from-file -- apply bucket/table/column descriptions from a YAML file in one shot; failures collected, remaining items continue", + "Fix: storage table-detail now returns 'description' and 'metadata' fields (extracted from table metadata array)", + "Fix: storage bucket-detail now returns 'description' and 'metadata' fields (KBC.description in metadata takes precedence over native creation-time description field)", ], "0.21.1": [ "Fix: sync pull on a newly created dev branch now writes config rows (#193) -- idempotent skip guard for rows was missing a file-existence check, causing rows to be silently skipped when the branch directory was new (hash matched main because the branch is a clone)", diff --git a/src/keboola_agent_cli/client.py b/src/keboola_agent_cli/client.py index bb7cfdc0..da74a03e 100644 --- a/src/keboola_agent_cli/client.py +++ b/src/keboola_agent_cli/client.py @@ -763,6 +763,69 @@ def list_buckets_with_metadata(self) -> list[dict[str, Any]]: """ return self.list_buckets(include="metadata") + def set_bucket_metadata( + self, + bucket_id: str, + entries: list[tuple[str, str]], + branch_id: int | None = None, + ) -> list[dict[str, Any]]: + """Upsert metadata key/value pairs on a storage bucket. + + POST /v2/storage/buckets/{id}/metadata + + Uses the same PHP-style array form encoding as ``set_branch_metadata``. + Provider is always ``"user"`` for CLI-originated descriptions. + + Args: + bucket_id: Bucket ID (e.g. 'in.c-db'). + entries: Ordered list of ``(key, value)`` metadata tuples. + branch_id: If set, target a specific dev branch. + + Returns: + Full metadata list for the bucket after the upsert. + """ + prefix = f"/v2/storage/branch/{branch_id}" if branch_id else "/v2/storage" + safe_id = quote(bucket_id, safe="") + form: dict[str, str] = {"provider": "user"} + for i, (key, value) in enumerate(entries): + form[f"metadata[{i}][key]"] = key + form[f"metadata[{i}][value]"] = value + response = self._request("POST", f"{prefix}/buckets/{safe_id}/metadata", data=form) + return response.json() + + def set_table_metadata( + self, + table_id: str, + entries: list[tuple[str, str]], + branch_id: int | None = None, + ) -> list[dict[str, Any]]: + """Upsert metadata key/value pairs on a storage table. + + POST /v2/storage/tables/{id}/metadata + + Provider is always ``"user"`` for CLI-originated descriptions. + Column-level descriptions use the namespaced key convention + ``KBC.column.{colname}.description`` stored at table-metadata level + (Keboola Storage API does not expose a user-writable column-metadata + endpoint; ``columnMetadata`` is populated exclusively by components). + + Args: + table_id: Full table ID (e.g. "in.c-bucket.table"). + entries: Ordered list of ``(key, value)`` metadata tuples. + branch_id: If set, target a specific dev branch. + + Returns: + Full metadata list for the table after the upsert. + """ + prefix = f"/v2/storage/branch/{branch_id}" if branch_id else "/v2/storage" + safe_id = quote(table_id, safe="") + form: dict[str, str] = {"provider": "user"} + for i, (key, value) in enumerate(entries): + form[f"metadata[{i}][key]"] = key + form[f"metadata[{i}][value]"] = value + response = self._request("POST", f"{prefix}/tables/{safe_id}/metadata", data=form) + return response.json() + def get_bucket_detail( self, bucket_id: str, diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index a3e458e4..edb009a8 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -241,6 +241,22 @@ kbagent storage delete-bucket --project NAME --bucket-id ID [--bucket-id ...] [--force] [--dry-run] [--yes] [--branch ID] Delete one or more buckets. --force cascade-deletes tables. Linked/shared buckets protected. Branch-aware. +### Storage Descriptions + + kbagent storage describe-bucket --project NAME --bucket-id ID [--text STR | --file PATH | --stdin] [--branch ID] + Set the KBC.description metadata on a bucket (upsert). Visible in bucket-detail. + + kbagent storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID] + Set the KBC.description metadata on a table (upsert). Readable via table-detail --json .data.description. + + kbagent storage describe-column --project NAME --table-id ID --column NAME=DESC [--column ...] [--branch ID] + Set per-column descriptions stored as KBC.column.{{name}}.description in table metadata (upsert). + Readable via table-detail --json .data.column_details[].description. + + kbagent storage describe-batch --project NAME --from-file YAML [--branch ID] + Apply bucket/table/column descriptions from a YAML file. Sections: buckets, tables, columns (all optional). + Failures collected; one error does not abort remaining items. + ### Storage Files kbagent storage files --project NAME [--tag TAG ...] [--limit N] [--offset N] [--query Q] [--branch ID] diff --git a/src/keboola_agent_cli/commands/storage.py b/src/keboola_agent_cli/commands/storage.py index 4baca3f2..e2993a01 100644 --- a/src/keboola_agent_cli/commands/storage.py +++ b/src/keboola_agent_cli/commands/storage.py @@ -5,6 +5,7 @@ """ from pathlib import Path +from typing import Any import typer @@ -1140,6 +1141,401 @@ def storage_delete_bucket( raise typer.Exit(code=1) +# ------------------------------------------------------------------ +# Describe (metadata write) commands +# ------------------------------------------------------------------ + +_DESCRIBE = "Descriptions" + + +@storage_app.command("describe-bucket", rich_help_panel=_DESCRIBE) +def storage_describe_bucket( + ctx: typer.Context, + project: str = typer.Option( + ..., + "--project", + help="Project alias", + ), + bucket_id: str = typer.Option( + ..., + "--bucket-id", + help="Bucket ID (e.g. 'in.c-my-bucket')", + ), + text: str | None = typer.Option( + None, + "--text", + help="Description text (inline)", + ), + file: Path | None = typer.Option( + None, + "--file", + help="Path to a file containing the description", + ), + stdin: bool = typer.Option( + False, + "--stdin", + help="Read description from standard input", + ), + branch: int | None = typer.Option( + None, + "--branch", + help="Dev branch ID (defaults to active branch if set via 'branch use')", + ), +) -> None: + """Set the description on a storage bucket. + + Stores the description as KBC.description in bucket metadata (upsert). + Provide the text via --text, --file, or --stdin (exactly one required). + """ + if should_hint(ctx): + emit_hint( + ctx, "storage.describe-bucket", project=project, bucket_id=bucket_id, branch=branch + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "storage_service") + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + from ._metadata_input import resolve_text_input + + try: + description = resolve_text_input(text=text, file=file, stdin=stdin) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + raise typer.Exit(code=2) from None + + try: + result = service.describe_bucket( + alias=project, + bucket_id=bucket_id, + description=description, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + formatter.console.print(f"[bold green]Description set:[/bold green] {bucket_id}") + formatter.console.print(f" {description[:120]}") + + +@storage_app.command("describe-table", rich_help_panel=_DESCRIBE) +def storage_describe_table( + ctx: typer.Context, + project: str = typer.Option( + ..., + "--project", + help="Project alias", + ), + table_id: str = typer.Option( + ..., + "--table-id", + help="Table ID (e.g. 'in.c-my-bucket.my-table')", + ), + text: str | None = typer.Option( + None, + "--text", + help="Description text (inline)", + ), + file: Path | None = typer.Option( + None, + "--file", + help="Path to a file containing the description", + ), + stdin: bool = typer.Option( + False, + "--stdin", + help="Read description from standard input", + ), + branch: int | None = typer.Option( + None, + "--branch", + help="Dev branch ID (defaults to active branch if set via 'branch use')", + ), +) -> None: + """Set the description on a storage table. + + Stores the description as KBC.description in table metadata (upsert). + Provide the text via --text, --file, or --stdin (exactly one required). + """ + if should_hint(ctx): + emit_hint(ctx, "storage.describe-table", project=project, table_id=table_id, branch=branch) + + formatter = get_formatter(ctx) + service = get_service(ctx, "storage_service") + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + from ._metadata_input import resolve_text_input + + try: + description = resolve_text_input(text=text, file=file, stdin=stdin) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + raise typer.Exit(code=2) from None + + try: + result = service.describe_table( + alias=project, + table_id=table_id, + description=description, + branch_id=effective_branch, + ) + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + formatter.console.print(f"[bold green]Description set:[/bold green] {table_id}") + formatter.console.print(f" {description[:120]}") + + +@storage_app.command("describe-column", rich_help_panel=_DESCRIBE) +def storage_describe_column( + ctx: typer.Context, + project: str = typer.Option( + ..., + "--project", + help="Project alias", + ), + table_id: str = typer.Option( + ..., + "--table-id", + help="Table ID (e.g. 'in.c-my-bucket.my-table')", + ), + column: list[str] = typer.Option( + ..., + "--column", + help="Column description as 'NAME=DESCRIPTION' (can be repeated)", + ), + branch: int | None = typer.Option( + None, + "--branch", + help="Dev branch ID (defaults to active branch if set via 'branch use')", + ), +) -> None: + """Set descriptions on one or more columns of a storage table. + + Descriptions are stored as KBC.column.{name}.description keys in table + metadata (upsert). Keboola Storage does not expose a user-writable + column-level metadata endpoint; this convention lets you annotate columns + and read them back via 'storage table-detail'. + + Example: + + kbagent storage describe-column \\ + --project myproj \\ + --table-id in.c-bucket.orders \\ + --column order_id="Unique order identifier" \\ + --column total="Order total in USD" + """ + if should_hint(ctx): + emit_hint(ctx, "storage.describe-column", project=project, table_id=table_id, branch=branch) + + formatter = get_formatter(ctx) + service = get_service(ctx, "storage_service") + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + parsed: dict[str, str] = {} + for entry in column: + if "=" not in entry: + formatter.error( + message=f"--column must be NAME=DESCRIPTION, got: {entry!r}", + error_code="INVALID_ARGUMENT", + ) + raise typer.Exit(code=2) from None + name, _, desc = entry.partition("=") + name = name.strip() + if not name: + formatter.error( + message=f"Column name cannot be empty in: {entry!r}", + error_code="INVALID_ARGUMENT", + ) + raise typer.Exit(code=2) from None + parsed[name] = desc + + try: + result = service.describe_columns( + alias=project, + table_id=table_id, + columns=parsed, + branch_id=effective_branch, + ) + except ValueError as exc: + formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + raise typer.Exit(code=2) from None + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + + if formatter.json_mode: + formatter.output(result) + else: + formatter.console.print( + f"[bold green]Column descriptions set:[/bold green] {table_id} " + f"({len(parsed)} column(s))" + ) + for name, desc in parsed.items(): + formatter.console.print(f" {name}: {desc[:80]}") + + +@storage_app.command("describe-batch", rich_help_panel=_DESCRIBE) +def storage_describe_batch( + ctx: typer.Context, + project: str = typer.Option( + ..., + "--project", + help="Project alias", + ), + from_file: Path = typer.Option( + ..., + "--from-file", + help="Path to a YAML file with bucket/table/column descriptions", + ), + branch: int | None = typer.Option( + None, + "--branch", + help="Dev branch ID (defaults to active branch if set via 'branch use')", + ), +) -> None: + """Apply descriptions to buckets, tables, and columns from a YAML file. + + YAML schema: + + buckets: + in.c-my-bucket: "Bucket description" + + tables: + in.c-my-bucket.my-table: "Table description" + + columns: + in.c-my-bucket.my-table: + col1: "Column 1 description" + col2: "Column 2 description" + + All sections are optional. A failure in one item does not abort the + rest -- all results are collected and reported. + """ + if should_hint(ctx): + emit_hint( + ctx, "storage.describe-batch", project=project, from_file=from_file, branch=branch + ) + + formatter = get_formatter(ctx) + service = get_service(ctx, "storage_service") + config_store: ConfigStore = ctx.obj["config_store"] + _, effective_branch = resolve_branch(config_store, formatter, project, branch) + + # In human mode, show a live progress indicator so that large batches + # (100+ items) do not look frozen. JSON mode must remain silent on stderr + # so structured output is the only thing on stdout. + progress_cm: Any = None + progress_task: Any = None + progress_callback = None + if not formatter.json_mode: + from rich.progress import ( + BarColumn, + MofNCompleteColumn, + Progress, + SpinnerColumn, + TextColumn, + TimeElapsedColumn, + ) + + progress_cm = Progress( + SpinnerColumn(), + TextColumn("[progress.description]{task.description}"), + BarColumn(), + MofNCompleteColumn(), + TextColumn("•"), + TimeElapsedColumn(), + console=formatter.console, + transient=True, + ) + + def _on_item(obj_type: str, obj_id: str, current: int, total: int) -> None: + # Guard against progress_task/progress_cm not being ready yet. + if progress_task is None or progress_cm is None: + return + # total is known up-front (passed the first time), but re-setting + # is a no-op after the first call. + progress_cm.update( + progress_task, + total=total, + completed=max(current - 1, 0), + description=f"Describing {obj_type} {obj_id}", + ) + + progress_callback = _on_item + + try: + if progress_cm is not None: + progress_cm.start() + progress_task = progress_cm.add_task("Applying descriptions...", total=None) + result = service.describe_batch( + alias=project, + from_file=from_file, + branch_id=effective_branch, + progress_callback=progress_callback, + ) + if progress_cm is not None and progress_task is not None: + # Mark the task complete so the final render shows N / N. + progress_cm.update( + progress_task, + completed=result["applied_count"] + result["error_count"], + ) + except ValueError as exc: + formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + raise typer.Exit(code=2) from None + except ConfigError as exc: + formatter.error(message=exc.message, error_code="CONFIG_ERROR") + raise typer.Exit(code=5) from None + except KeboolaApiError as exc: + formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) + raise typer.Exit(code=map_error_to_exit_code(exc)) from None + finally: + if progress_cm is not None: + # .stop() is idempotent; safe for both happy and error paths. + progress_cm.stop() + + if formatter.json_mode: + formatter.output(result) + else: + applied = result["applied_count"] + errors = result["error_count"] + formatter.console.print( + f"[bold green]Batch complete:[/bold green] {applied} applied, {errors} error(s)" + ) + for item in result["applied"]: + obj_type = item["type"] + obj_id = item["id"] + if obj_type == "columns": + n = len(item.get("columns", {})) + formatter.console.print(f" [green]✓[/green] {obj_type} {obj_id} ({n} cols)") + else: + formatter.console.print(f" [green]✓[/green] {obj_type} {obj_id}") + for item in result["errors"]: + formatter.console.print(f" [red]✗[/red] {item['type']} {item['id']}: {item['error']}") + if errors: + raise typer.Exit(code=1) from None + + # ------------------------------------------------------------------ # File operations # ------------------------------------------------------------------ diff --git a/src/keboola_agent_cli/hints/definitions/storage.py b/src/keboola_agent_cli/hints/definitions/storage.py index 96460f92..54ca4565 100644 --- a/src/keboola_agent_cli/hints/definitions/storage.py +++ b/src/keboola_agent_cli/hints/definitions/storage.py @@ -681,3 +681,157 @@ ], ) ) + +# ── storage describe-bucket ──────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="storage.describe-bucket", + description="Set the description on a storage bucket", + steps=[ + HintStep( + comment="Upsert KBC.description in bucket metadata (provider='user')", + client=ClientCall( + method="set_bucket_metadata", + args={ + "bucket_id": "{bucket_id}", + "entries": '[("KBC.description", "{description}")]', + "branch_id": "{branch}", + }, + result_var="result", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="StorageService", + service_module="storage_service", + method="describe_bucket", + args={ + "alias": "{project}", + "bucket_id": "{bucket_id}", + "description": "{description}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "POST /v2/storage/buckets/{id}/metadata with provider='user' is an upsert-by-key.", + "Description is readable via 'storage bucket-detail --json .data.description'.", + ], + ) +) + +# ── storage describe-table ──────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="storage.describe-table", + description="Set the description on a storage table", + steps=[ + HintStep( + comment="Upsert KBC.description in table metadata (provider='user')", + client=ClientCall( + method="set_table_metadata", + args={ + "table_id": "{table_id}", + "entries": '[("KBC.description", "{description}")]', + "branch_id": "{branch}", + }, + result_var="result", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="StorageService", + service_module="storage_service", + method="describe_table", + args={ + "alias": "{project}", + "table_id": "{table_id}", + "description": "{description}", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Description is readable via 'storage table-detail --json | .data.description'.", + ], + ) +) + +# ── storage describe-column ─────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="storage.describe-column", + description="Set per-column descriptions on a storage table", + steps=[ + HintStep( + comment="Store column descriptions as KBC.column.{name}.description in table metadata", + client=ClientCall( + method="set_table_metadata", + args={ + "table_id": "{table_id}", + "entries": '[("KBC.column.{col}.description", "{description}")]', + "branch_id": "{branch}", + }, + result_var="result", + result_hint="list[dict]", + ), + service=ServiceCall( + service_class="StorageService", + service_module="storage_service", + method="describe_columns", + args={ + "alias": "{project}", + "table_id": "{table_id}", + "columns": '{"{col}": "{description}"}', + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "Column descriptions use key KBC.column.{name}.description in table metadata.", + "They appear under column_details[].description in 'storage table-detail --json'.", + "Keboola does not provide a user-writable column-metadata endpoint; this is the supported convention.", + ], + ) +) + +# ── storage describe-batch ──────────────────────────────────────── + +HintRegistry.register( + CommandHint( + cli_command="storage.describe-batch", + description="Apply bucket/table/column descriptions from a YAML file", + steps=[ + HintStep( + comment="Load YAML and apply descriptions to all listed assets", + client=ClientCall( + method="set_bucket_metadata / set_table_metadata", + args={ + "from_file": "{from_file}", + "branch_id": "{branch}", + }, + result_var="result", + result_hint="dict", + ), + service=ServiceCall( + service_class="StorageService", + service_module="storage_service", + method="describe_batch", + args={ + "alias": "{project}", + "from_file": "Path('{from_file}')", + "branch_id": "{branch}", + }, + ), + ), + ], + notes=[ + "YAML sections: 'buckets', 'tables', 'columns' (all optional).", + "Failures are collected -- one error does not abort the rest.", + ], + ) +) diff --git a/src/keboola_agent_cli/permissions.py b/src/keboola_agent_cli/permissions.py index faa0ff96..c127e1ed 100644 --- a/src/keboola_agent_cli/permissions.py +++ b/src/keboola_agent_cli/permissions.py @@ -114,6 +114,11 @@ "storage.delete-column": "destructive", "storage.delete-bucket": "destructive", "storage.file-delete": "destructive", + # Storage descriptions + "storage.describe-bucket": "write", + "storage.describe-table": "write", + "storage.describe-column": "write", + "storage.describe-batch": "write", # Encryption "encrypt.values": "write", # Sync / git workflow diff --git a/src/keboola_agent_cli/services/storage_service.py b/src/keboola_agent_cli/services/storage_service.py index 266015b0..4eafb34f 100644 --- a/src/keboola_agent_cli/services/storage_service.py +++ b/src/keboola_agent_cli/services/storage_service.py @@ -6,6 +6,7 @@ import csv import logging +from collections.abc import Callable from pathlib import Path from typing import Any @@ -172,15 +173,27 @@ def get_bucket_detail( project_id = token_info.project_id source = bucket.get("sourceBucket") + # KBC.description in the metadata array takes precedence over the + # native description field (which can only be set at creation time + # via the Storage API; user updates go through the metadata endpoint). + raw_metadata: list[dict[str, Any]] = bucket.get("metadata", []) + metadata_description = "" + for m in raw_metadata: + if m.get("key") == "KBC.description" and m.get("provider") == "user": + metadata_description = m.get("value", "") or "" + break + description = metadata_description or bucket.get("description", "") + result: dict[str, Any] = { "project_alias": alias, "project_id": project_id, "bucket_id": bucket.get("id", ""), "display_name": bucket.get("displayName", ""), "stage": bucket.get("stage", ""), - "description": bucket.get("description", ""), + "description": description, "backend": bucket.get("backend", ""), "is_linked": source is not None, + "metadata": raw_metadata, } # Resolve Snowflake paths using backendPath from API (preserves correct case). @@ -260,6 +273,18 @@ def get_table_detail( columns = table.get("columns", []) column_metadata = table.get("columnMetadata", {}) + raw_metadata: list[dict[str, Any]] = table.get("metadata", []) + + # Extract description and per-column descriptions from metadata list + description = "" + col_descriptions: dict[str, str] = {} + for m in raw_metadata: + key = m.get("key", "") + if key == "KBC.description" and m.get("provider") == "user": + description = m.get("value", "") or "" + elif key.startswith("KBC.column.") and key.endswith(".description"): + col_name = key[len("KBC.column.") : -len(".description")] + col_descriptions[col_name] = m.get("value", "") or "" column_details = [] for col in columns: @@ -270,6 +295,8 @@ def get_table_detail( col_info["type"] = m.get("value", "") elif m.get("key") == "KBC.datatype.nullable": col_info["nullable"] = m.get("value", "") == "1" + if col in col_descriptions: + col_info["description"] = col_descriptions[col] column_details.append(col_info) return { @@ -278,6 +305,7 @@ def get_table_detail( "name": table.get("name", ""), "display_name": table.get("displayName", ""), "bucket_id": table.get("bucket", {}).get("id", ""), + "description": description, "columns": columns, "column_details": column_details, "primary_key": table.get("primaryKey", []), @@ -287,6 +315,7 @@ def get_table_detail( "last_import_date": table.get("lastImportDate", ""), "last_change_date": table.get("lastChangeDate", ""), "created": table.get("created", ""), + "metadata": raw_metadata, } def list_tables( @@ -1457,6 +1486,253 @@ def unload_table_to_file( return result + # ------------------------------------------------------------------ + # Describe (metadata write) methods + # ------------------------------------------------------------------ + + def describe_bucket( + self, + alias: str, + bucket_id: str, + description: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Set the KBC.description metadata on a storage bucket. + + Idempotent upsert: re-running with a different value overwrites the + existing entry (Keboola metadata POST is upsert-by-key). + + Args: + alias: Project alias. + bucket_id: Bucket ID (e.g. 'in.c-my-bucket'). + description: Human-readable description text. + branch_id: If set, target a specific dev branch. + + Returns: + Dict with project_alias, bucket_id, description, result, message. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + client = self._client_factory(project.stack_url, project.token) + try: + result = client.set_bucket_metadata( + bucket_id=bucket_id, + entries=[("KBC.description", description)], + branch_id=branch_id, + ) + finally: + client.close() + return { + "project_alias": alias, + "bucket_id": bucket_id, + "description": description, + "result": result, + "message": f"Description set on bucket '{bucket_id}' in project '{alias}'.", + } + + def describe_table( + self, + alias: str, + table_id: str, + description: str, + branch_id: int | None = None, + ) -> dict[str, Any]: + """Set the KBC.description metadata on a storage table. + + Idempotent upsert: re-running with a different value overwrites. + + Args: + alias: Project alias. + table_id: Full table ID (e.g. 'in.c-bucket.table'). + description: Human-readable description text. + branch_id: If set, target a specific dev branch. + + Returns: + Dict with project_alias, table_id, description, result, message. + """ + projects = self.resolve_projects([alias]) + project = projects[alias] + client = self._client_factory(project.stack_url, project.token) + try: + result = client.set_table_metadata( + table_id=table_id, + entries=[("KBC.description", description)], + branch_id=branch_id, + ) + finally: + client.close() + return { + "project_alias": alias, + "table_id": table_id, + "description": description, + "result": result, + "message": f"Description set on table '{table_id}' in project '{alias}'.", + } + + def describe_columns( + self, + alias: str, + table_id: str, + columns: dict[str, str], + branch_id: int | None = None, + ) -> dict[str, Any]: + """Set per-column descriptions on a storage table. + + Column descriptions are stored as namespaced table metadata using the + key convention ``KBC.column.{colname}.description``. Keboola's + Storage API does not provide a user-writable column-level metadata + endpoint (``columnMetadata`` is populated exclusively by processing + components); this convention is the supported alternative for + annotating columns from the CLI. + + Args: + alias: Project alias. + table_id: Full table ID. + columns: Mapping of column name -> description text. + branch_id: If set, target a specific dev branch. + + Returns: + Dict with project_alias, table_id, columns dict, result, message. + """ + if not columns: + raise ValueError("At least one column description must be provided.") + projects = self.resolve_projects([alias]) + project = projects[alias] + entries = [(f"KBC.column.{name}.description", desc) for name, desc in columns.items()] + client = self._client_factory(project.stack_url, project.token) + try: + result = client.set_table_metadata( + table_id=table_id, + entries=entries, + branch_id=branch_id, + ) + finally: + client.close() + return { + "project_alias": alias, + "table_id": table_id, + "columns": columns, + "result": result, + "message": ( + f"Descriptions set for {len(columns)} column(s) on table '{table_id}' " + f"in project '{alias}'." + ), + } + + def describe_batch( + self, + alias: str, + from_file: Path, + branch_id: int | None = None, + progress_callback: Callable[[str, str, int, int], None] | None = None, + ) -> dict[str, Any]: + """Apply bucket, table, and column descriptions from a YAML file. + + YAML schema:: + + buckets: + in.c-my-bucket: "Bucket description" + tables: + in.c-my-bucket.my-table: "Table description" + columns: + in.c-my-bucket.my-table: + col1: "Column 1 description" + col2: "Column 2 description" + + All sections are optional; empty or absent sections are silently + skipped. Within each section the operations are applied in order. + A failure in one item does not skip remaining items — all results + (success and error) are collected and returned. + + Args: + alias: Project alias. + from_file: Path to a YAML file with the schema above. + branch_id: If set, target a specific dev branch. + progress_callback: Optional ``(obj_type, obj_id, current, total)`` + callable invoked **before** each item is processed. ``obj_type`` + is ``"bucket"``, ``"table"``, or ``"columns"``; ``current`` is + 1-based; ``total`` is the total number of items across all + sections. Used by the CLI to render a Rich progress indicator + in human mode; JSON mode leaves it unset. + + Returns: + Dict with project_alias, applied, errors, applied_count, error_count. + """ + import yaml + + from ..errors import KeboolaApiError + + if not from_file.is_file(): + raise ValueError(f"Batch file not found: {from_file}") + raw = yaml.safe_load(from_file.read_text(encoding="utf-8")) or {} + if not isinstance(raw, dict): + raise ValueError("Batch file must be a YAML mapping.") + + applied: list[dict[str, Any]] = [] + errors: list[dict[str, Any]] = [] + + buckets: dict[str, str] = raw.get("buckets") or {} + tables: dict[str, str] = raw.get("tables") or {} + columns: dict[str, dict[str, str]] = raw.get("columns") or {} + + total = len(buckets) + len(tables) + len(columns) + current = 0 + + for bucket_id, desc in buckets.items(): + current += 1 + if progress_callback is not None: + progress_callback("bucket", bucket_id, current, total) + try: + self.describe_bucket(alias, bucket_id, str(desc), branch_id=branch_id) + applied.append({"type": "bucket", "id": bucket_id, "description": desc}) + logger.debug("describe_batch bucket %s: ok", bucket_id) + except Exception as exc: + msg = exc.message if isinstance(exc, KeboolaApiError) else str(exc) + errors.append({"type": "bucket", "id": bucket_id, "error": msg}) + + for table_id, desc in tables.items(): + current += 1 + if progress_callback is not None: + progress_callback("table", table_id, current, total) + try: + self.describe_table(alias, table_id, str(desc), branch_id=branch_id) + applied.append({"type": "table", "id": table_id, "description": desc}) + except Exception as exc: + msg = exc.message if isinstance(exc, KeboolaApiError) else str(exc) + errors.append({"type": "table", "id": table_id, "error": msg}) + + for table_id, col_map in columns.items(): + current += 1 + if progress_callback is not None: + progress_callback("columns", table_id, current, total) + if not isinstance(col_map, dict): + errors.append( + {"type": "columns", "id": table_id, "error": "columns entry must be a mapping"} + ) + continue + try: + self.describe_columns( + alias, table_id, {k: str(v) for k, v in col_map.items()}, branch_id=branch_id + ) + applied.append( + { + "type": "columns", + "id": table_id, + "columns": {k: str(v) for k, v in col_map.items()}, + } + ) + except Exception as exc: + msg = exc.message if isinstance(exc, KeboolaApiError) else str(exc) + errors.append({"type": "columns", "id": table_id, "error": msg}) + + return { + "project_alias": alias, + "applied": applied, + "errors": errors, + "applied_count": len(applied), + "error_count": len(errors), + } + # ------------------------------------------------------------------ # Parallel workers # ------------------------------------------------------------------ diff --git a/tests/test_client.py b/tests/test_client.py index 75638e50..87446e82 100644 --- a/tests/test_client.py +++ b/tests/test_client.py @@ -2653,3 +2653,154 @@ def test_get_branch_metadata_value_missing(self, httpx_mock) -> None: with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: value = client.get_branch_metadata_value(key="KBC.projectDescription") assert value is METADATA_NOT_FOUND + + +# --------------------------------------------------------------------------- +# Storage object metadata (bucket + table) +# --------------------------------------------------------------------------- + +_STORAGE_META_RESPONSE = [ + { + "id": "9001", + "key": "KBC.description", + "value": "A test description", + "provider": "user", + "timestamp": "2026-04-22T10:00:00+0200", + }, +] + + +class TestSetBucketMetadata: + """Tests for set_bucket_metadata() - POST /v2/storage/buckets/{id}/metadata.""" + + def test_set_bucket_metadata_php_form_body(self, httpx_mock) -> None: + """Encodes provider + PHP-array indices in the form body.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-my-bucket", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/buckets/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + result = client.set_bucket_metadata( + bucket_id="in.c-my-bucket", + entries=[("KBC.description", "A test description")], + ) + + assert result == _STORAGE_META_RESPONSE + req = httpx_mock.get_request() + body = req.content.decode().replace("%5B", "[").replace("%5D", "]") + assert "provider=user" in body + assert "metadata[0][key]=KBC.description" in body + assert ( + "metadata[0][value]=A+test+description" in body + or "A%20test%20description" in body + or "A test description" in body + ) + assert req.headers["content-type"].startswith("application/x-www-form-urlencoded") + + def test_set_bucket_metadata_with_branch(self, httpx_mock) -> None: + """Uses branch prefix when branch_id is provided.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-my-bucket", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/branch/42/buckets/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + result = client.set_bucket_metadata( + bucket_id="in.c-my-bucket", + entries=[("KBC.description", "Branch desc")], + branch_id=42, + ) + assert result == _STORAGE_META_RESPONSE + + def test_set_bucket_metadata_multiple_entries(self, httpx_mock) -> None: + """Multiple entries get sequential PHP indices.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-bucket", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/buckets/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + client.set_bucket_metadata( + bucket_id="in.c-bucket", + entries=[("k1", "v1"), ("k2", "v2")], + ) + body = httpx_mock.get_request().content.decode().replace("%5B", "[").replace("%5D", "]") + assert "metadata[0][key]=k1" in body + assert "metadata[1][key]=k2" in body + + +class TestSetTableMetadata: + """Tests for set_table_metadata() - POST /v2/storage/tables/{id}/metadata.""" + + def test_set_table_metadata_php_form_body(self, httpx_mock) -> None: + """Encodes provider + PHP-array indices for a table metadata POST.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-b.tbl", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/tables/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + result = client.set_table_metadata( + table_id="in.c-b.tbl", + entries=[("KBC.description", "A test description")], + ) + assert result == _STORAGE_META_RESPONSE + req = httpx_mock.get_request() + body = req.content.decode().replace("%5B", "[").replace("%5D", "]") + assert "provider=user" in body + assert "metadata[0][key]=KBC.description" in body + + def test_set_table_metadata_with_branch(self, httpx_mock) -> None: + """Uses branch prefix when branch_id is provided.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-b.tbl", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/branch/7/tables/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + result = client.set_table_metadata( + table_id="in.c-b.tbl", + entries=[("KBC.description", "Branch desc")], + branch_id=7, + ) + assert result == _STORAGE_META_RESPONSE + + def test_set_table_metadata_column_convention(self, httpx_mock) -> None: + """Column descriptions use KBC.column.{name}.description key convention.""" + from urllib.parse import quote as url_quote + + safe_id = url_quote("in.c-b.tbl", safe="") + httpx_mock.add_response( + url=f"{_BASE}/v2/storage/tables/{safe_id}/metadata", + method="POST", + json=_STORAGE_META_RESPONSE, + status_code=201, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + client.set_table_metadata( + table_id="in.c-b.tbl", + entries=[("KBC.column.city.description", "City name")], + ) + body = httpx_mock.get_request().content.decode().replace("%5B", "[").replace("%5D", "]") + assert "KBC.column.city.description" in body diff --git a/tests/test_e2e.py b/tests/test_e2e.py index cf94d19d..cb2d206b 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -401,6 +401,9 @@ def test_full_cli_e2e(self) -> None: _step(14.1, "storage unload-table --file-type parquet", "Parquet export + sliced download") self._test_unload_table_parquet(table_id) + _step(14.2, "storage describe-bucket/table/column/batch", "description metadata round-trip") + self._test_storage_describe(bucket_id, table_id) + _step(15, "storage load-file", "upload CSV as file then load into table") self._test_load_file(table_id) @@ -2509,6 +2512,102 @@ def _test_project_pin_and_firewall(self) -> None: f"found persisted policy: {persisted.permissions}" ) + def _test_storage_describe(self, bucket_id: str, table_id: str) -> None: + """Round-trip describe commands: write description, read it back.""" + # describe-bucket: set KBC.description, verify via bucket-detail + data = self._run_ok( + "storage", + "describe-bucket", + "--project", + self.alias, + "--bucket-id", + bucket_id, + "--text", + "E2E bucket description", + ) + assert data["data"]["bucket_id"] == bucket_id + assert data["data"]["description"] == "E2E bucket description" + + data = self._run_ok( + "storage", "bucket-detail", "--project", self.alias, "--bucket-id", bucket_id + ) + assert data["data"]["description"] == "E2E bucket description" + + # describe-table: set KBC.description, verify via table-detail + data = self._run_ok( + "storage", + "describe-table", + "--project", + self.alias, + "--table-id", + table_id, + "--text", + "E2E table description", + ) + assert data["data"]["table_id"] == table_id + assert data["data"]["description"] == "E2E table description" + + data = self._run_ok( + "storage", "table-detail", "--project", self.alias, "--table-id", table_id + ) + assert data["data"]["description"] == "E2E table description" + + # describe-column: set per-column descriptions, verify via table-detail column_details + data = self._run_ok( + "storage", + "describe-column", + "--project", + self.alias, + "--table-id", + table_id, + "--column", + "id=Unique row identifier", + "--column", + "name=Human-readable name", + ) + assert data["data"]["table_id"] == table_id + assert data["data"]["columns"]["id"] == "Unique row identifier" + assert data["data"]["columns"]["name"] == "Human-readable name" + + data = self._run_ok( + "storage", "table-detail", "--project", self.alias, "--table-id", table_id + ) + col_descs = {c["name"]: c.get("description", "") for c in data["data"]["column_details"]} + assert col_descs.get("id") == "Unique row identifier" + assert col_descs.get("name") == "Human-readable name" + + # describe-batch: apply all three sections from a YAML file + batch_yaml = ( + f"buckets:\n" + f" {bucket_id}: Batch bucket desc\n" + f"tables:\n" + f" {table_id}: Batch table desc\n" + f"columns:\n" + f" {table_id}:\n" + f" id: Batch column id desc\n" + ) + batch_file = self.work_dir / "batch_describe.yaml" + batch_file.write_text(batch_yaml, encoding="utf-8") + data = self._run_ok( + "storage", + "describe-batch", + "--project", + self.alias, + "--from-file", + str(batch_file), + ) + assert data["data"]["project_alias"] == self.alias + assert len(data["data"]["applied"]) == 3 + assert data["data"]["errors"] == [] + + # Verify the batch updated the descriptions + data = self._run_ok( + "storage", "table-detail", "--project", self.alias, "--table-id", table_id + ) + assert data["data"]["description"] == "Batch table desc" + col_descs = {c["name"]: c.get("description", "") for c in data["data"]["column_details"]} + assert col_descs.get("id") == "Batch column id desc" + def _test_project_edit_and_remove(self) -> None: """Edit project URL, then remove it.""" # project edit -- change URL back to same (just verify command works) diff --git a/tests/test_storage_describe_cli.py b/tests/test_storage_describe_cli.py new file mode 100644 index 00000000..756e6502 --- /dev/null +++ b/tests/test_storage_describe_cli.py @@ -0,0 +1,757 @@ +"""Tests for storage describe CLI commands via CliRunner. + +Covers describe-bucket, describe-table, describe-column, and describe-batch. +Follows the CLI test pattern from test_workspace_cli.py with patched services. +""" + +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +from typer.testing import CliRunner + +from keboola_agent_cli.cli import app +from keboola_agent_cli.config_store import ConfigStore +from keboola_agent_cli.errors import KeboolaApiError +from keboola_agent_cli.models import ProjectConfig +from keboola_agent_cli.services.config_service import ConfigService +from keboola_agent_cli.services.job_service import JobService +from keboola_agent_cli.services.project_service import ProjectService + +TEST_TOKEN = "901-10493007-VDtlEDWDF6Tx5V8jjE8FshFlqM0Hl0c08KHqpt0k" + +runner = CliRunner() + + +def _setup_config(config_dir: Path, projects: dict[str, dict] | None = None) -> ConfigStore: + store = ConfigStore(config_dir=config_dir) + if projects: + for alias, info in projects.items(): + store.add_project( + alias, + ProjectConfig( + stack_url=info.get("stack_url", "https://connection.keboola.com"), + token=info["token"], + project_name=info.get("project_name", alias), + project_id=info.get("project_id", 1234), + ), + ) + return store + + +class TestStorageDescribeBucket: + """Tests for `kbagent storage describe-bucket`.""" + + def test_describe_bucket_json(self, tmp_path: Path) -> None: + """describe-bucket --text returns structured JSON on success.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + mock_storage.describe_bucket.return_value = { + "project_alias": "prod", + "bucket_id": "in.c-my-bucket", + "description": "My bucket description", + "result": [{"id": "1", "key": "KBC.description", "value": "My bucket description"}], + "message": "Description set on bucket 'in.c-my-bucket' in project 'prod'.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-bucket", + "--project", + "prod", + "--bucket-id", + "in.c-my-bucket", + "--text", + "My bucket description", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert output["data"]["bucket_id"] == "in.c-my-bucket" + assert output["data"]["description"] == "My bucket description" + mock_storage.describe_bucket.assert_called_once_with( + alias="prod", + bucket_id="in.c-my-bucket", + description="My bucket description", + branch_id=None, + ) + + def test_describe_bucket_missing_source(self, tmp_path: Path) -> None: + """describe-bucket without --text/--file/--stdin exits with code 2.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-bucket", + "--project", + "prod", + "--bucket-id", + "in.c-my-bucket", + ], + ) + + assert result.exit_code == 2, f"Expected 2, got {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "error" + assert output["error"]["code"] == "INVALID_ARGUMENT" + mock_storage.describe_bucket.assert_not_called() + + def test_describe_bucket_api_error(self, tmp_path: Path) -> None: + """describe-bucket propagates API errors with appropriate exit code.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + mock_storage.describe_bucket.side_effect = KeboolaApiError( + message="Bucket not found", + status_code=404, + error_code="BUCKET_NOT_FOUND", + retryable=False, + ) + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-bucket", + "--project", + "prod", + "--bucket-id", + "in.c-missing", + "--text", + "desc", + ], + ) + + assert result.exit_code != 0 + output = json.loads(result.output) + assert output["status"] == "error" + assert output["error"]["code"] == "BUCKET_NOT_FOUND" + + +class TestStorageDescribeTable: + """Tests for `kbagent storage describe-table`.""" + + def test_describe_table_json(self, tmp_path: Path) -> None: + """describe-table --text returns structured JSON on success.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + mock_storage.describe_table.return_value = { + "project_alias": "prod", + "table_id": "in.c-bucket.orders", + "description": "All orders", + "result": [{"id": "2", "key": "KBC.description", "value": "All orders"}], + "message": "Description set on table 'in.c-bucket.orders' in project 'prod'.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-table", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--text", + "All orders", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert output["data"]["table_id"] == "in.c-bucket.orders" + assert output["data"]["description"] == "All orders" + mock_storage.describe_table.assert_called_once_with( + alias="prod", + table_id="in.c-bucket.orders", + description="All orders", + branch_id=None, + ) + + def test_describe_table_missing_source(self, tmp_path: Path) -> None: + """describe-table without description source exits with code 2.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-table", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + ], + ) + + assert result.exit_code == 2, f"Expected 2, got {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "error" + mock_storage.describe_table.assert_not_called() + + def test_describe_table_from_file(self, tmp_path: Path) -> None: + """describe-table --file reads description from a text file.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + desc_file = tmp_path / "desc.txt" + desc_file.write_text("Description from file", encoding="utf-8") + + mock_storage = MagicMock() + mock_storage.describe_table.return_value = { + "project_alias": "prod", + "table_id": "in.c-bucket.orders", + "description": "Description from file", + "result": [], + "message": "Description set.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-table", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--file", + str(desc_file), + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + mock_storage.describe_table.assert_called_once_with( + alias="prod", + table_id="in.c-bucket.orders", + description="Description from file", + branch_id=None, + ) + + +class TestStorageDescribeColumn: + """Tests for `kbagent storage describe-column`.""" + + def test_describe_column_json(self, tmp_path: Path) -> None: + """describe-column with NAME=DESC pairs returns structured JSON.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + mock_storage.describe_columns.return_value = { + "project_alias": "prod", + "table_id": "in.c-bucket.orders", + "columns": {"order_id": "Unique order identifier", "total": "Order total in USD"}, + "result": [], + "message": "Descriptions set for 2 column(s) on table 'in.c-bucket.orders' in project 'prod'.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-column", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--column", + "order_id=Unique order identifier", + "--column", + "total=Order total in USD", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert output["data"]["table_id"] == "in.c-bucket.orders" + assert output["data"]["columns"]["order_id"] == "Unique order identifier" + assert output["data"]["columns"]["total"] == "Order total in USD" + mock_storage.describe_columns.assert_called_once_with( + alias="prod", + table_id="in.c-bucket.orders", + columns={"order_id": "Unique order identifier", "total": "Order total in USD"}, + branch_id=None, + ) + + def test_describe_column_missing_equals(self, tmp_path: Path) -> None: + """describe-column with malformed --column (no =) exits with code 2.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-column", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--column", + "order_id_no_equals", + ], + ) + + assert result.exit_code == 2, f"Expected 2, got {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "error" + assert output["error"]["code"] == "INVALID_ARGUMENT" + mock_storage.describe_columns.assert_not_called() + + def test_describe_column_empty_name(self, tmp_path: Path) -> None: + """describe-column with empty column name exits with code 2.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-column", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--column", + "=description with empty name", + ], + ) + + assert result.exit_code == 2, f"Expected 2, got {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "error" + mock_storage.describe_columns.assert_not_called() + + def test_describe_column_with_branch(self, tmp_path: Path) -> None: + """describe-column passes branch_id when --branch is given.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + mock_storage = MagicMock() + mock_storage.describe_columns.return_value = { + "project_alias": "prod", + "table_id": "in.c-bucket.orders", + "columns": {"col1": "Column 1"}, + "result": [], + "message": "Descriptions set for 1 column(s).", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-column", + "--project", + "prod", + "--table-id", + "in.c-bucket.orders", + "--column", + "col1=Column 1", + "--branch", + "999", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + mock_storage.describe_columns.assert_called_once_with( + alias="prod", + table_id="in.c-bucket.orders", + columns={"col1": "Column 1"}, + branch_id=999, + ) + + +class TestStorageDescribeBatch: + """Tests for `kbagent storage describe-batch`.""" + + def test_describe_batch_json(self, tmp_path: Path) -> None: + """describe-batch with a valid YAML file returns structured JSON.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + batch_file = tmp_path / "descriptions.yaml" + batch_file.write_text( + "buckets:\n" + " in.c-sales: Sales bucket\n" + "tables:\n" + " in.c-sales.orders: All orders\n" + "columns:\n" + " in.c-sales.orders:\n" + " order_id: Unique order ID\n", + encoding="utf-8", + ) + + mock_storage = MagicMock() + mock_storage.describe_batch.return_value = { + "project_alias": "prod", + "applied": [ + {"type": "bucket", "id": "in.c-sales", "description": "Sales bucket"}, + {"type": "table", "id": "in.c-sales.orders", "description": "All orders"}, + { + "type": "columns", + "id": "in.c-sales.orders", + "columns": {"order_id": "Unique order ID"}, + }, + ], + "errors": [], + "message": "Batch complete: 3 applied, 0 errors.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-batch", + "--project", + "prod", + "--from-file", + str(batch_file), + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert len(output["data"]["applied"]) == 3 + assert output["data"]["errors"] == [] + # JSON mode must never wire a progress callback; human mode does. + mock_storage.describe_batch.assert_called_once_with( + alias="prod", + from_file=batch_file, + branch_id=None, + progress_callback=None, + ) + + def test_describe_batch_file_not_found(self, tmp_path: Path) -> None: + """describe-batch raises ValueError when YAML file does not exist.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + missing_file = tmp_path / "missing.yaml" + + mock_storage = MagicMock() + mock_storage.describe_batch.side_effect = ValueError( + f"Batch file not found: {missing_file}" + ) + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-batch", + "--project", + "prod", + "--from-file", + str(missing_file), + ], + ) + + assert result.exit_code == 2, f"Expected 2, got {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "error" + assert output["error"]["code"] == "INVALID_ARGUMENT" + + def test_describe_batch_human_mode_wires_progress_callback(self, tmp_path: Path) -> None: + """Human mode must pass a progress_callback; JSON mode must not.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + batch_file = tmp_path / "descriptions.yaml" + batch_file.write_text("buckets:\n in.c-sales: Sales data\n", encoding="utf-8") + + mock_storage = MagicMock() + mock_storage.describe_batch.return_value = { + "project_alias": "prod", + "applied": [{"type": "bucket", "id": "in.c-sales", "description": "Sales data"}], + "errors": [], + "applied_count": 1, + "error_count": 0, + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + # No --json flag -> human mode -> progress_callback must be wired. + result = runner.invoke( + app, + [ + "storage", + "describe-batch", + "--project", + "prod", + "--from-file", + str(batch_file), + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + kwargs = mock_storage.describe_batch.call_args.kwargs + assert kwargs["alias"] == "prod" + assert kwargs["from_file"] == batch_file + assert kwargs["branch_id"] is None + # Key assertion: human mode supplies a callable; JSON mode supplies None. + assert callable(kwargs["progress_callback"]) + + def test_describe_batch_partial_errors(self, tmp_path: Path) -> None: + """describe-batch with partial errors still exits 0 (errors collected, not raised).""" + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config(config_dir, {"prod": {"token": TEST_TOKEN}}) + + batch_file = tmp_path / "partial.yaml" + batch_file.write_text("buckets:\n in.c-good: Good\n in.c-bad: Bad\n", encoding="utf-8") + + mock_storage = MagicMock() + mock_storage.describe_batch.return_value = { + "project_alias": "prod", + "applied": [{"type": "bucket", "id": "in.c-good", "description": "Good"}], + "errors": [{"type": "bucket", "id": "in.c-bad", "error": "Not found"}], + "message": "Batch complete: 1 applied, 1 error.", + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + patch("keboola_agent_cli.cli.StorageService") as MockStorageService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + MockJobService.return_value = JobService(config_store=store) + MockStorageService.return_value = mock_storage + + result = runner.invoke( + app, + [ + "--json", + "storage", + "describe-batch", + "--project", + "prod", + "--from-file", + str(batch_file), + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert len(output["data"]["applied"]) == 1 + assert len(output["data"]["errors"]) == 1 diff --git a/tests/test_storage_describe_service.py b/tests/test_storage_describe_service.py new file mode 100644 index 00000000..1d643a8c --- /dev/null +++ b/tests/test_storage_describe_service.py @@ -0,0 +1,564 @@ +"""Tests for StorageService describe_* methods (bucket, table, columns, batch). + +Also covers the read-back side: extraction of ``KBC.description`` and +``KBC.column.{name}.description`` metadata keys in ``get_bucket_detail`` / +``get_table_detail``, including the precedence between the native API +``description`` field and the ``KBC.description`` metadata entry. +""" + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from keboola_agent_cli.config_store import ConfigStore +from keboola_agent_cli.errors import KeboolaApiError +from keboola_agent_cli.models import AppConfig, ProjectConfig, TokenVerifyResponse +from keboola_agent_cli.services.storage_service import StorageService + +TEST_TOKEN = "901-10493007-VDtlEDWDF6Tx5V8jjE8FshFlqM0Hl0c08KHqpt0k" + +_META_RESPONSE = [ + { + "id": "9001", + "key": "KBC.description", + "value": "A test description", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + } +] + + +def _make_store(tmp_path: Path) -> ConfigStore: + config_dir = tmp_path / "config" + config_dir.mkdir(exist_ok=True) + store = ConfigStore(config_dir=config_dir) + store.save( + AppConfig( + projects={ + "prod": ProjectConfig( + stack_url="https://connection.keboola.com", + token=TEST_TOKEN, + ) + } + ) + ) + return store + + +def _make_service(store: ConfigStore, mock_client: MagicMock) -> StorageService: + return StorageService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + +class TestDescribeBucketService: + """Tests for StorageService.describe_bucket().""" + + def test_success(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_bucket_metadata.return_value = _META_RESPONSE + service = _make_service(store, mock_client) + + result = service.describe_bucket( + alias="prod", + bucket_id="in.c-sales", + description="Sales data bucket", + ) + + assert result["project_alias"] == "prod" + assert result["bucket_id"] == "in.c-sales" + assert result["description"] == "Sales data bucket" + assert result["result"] == _META_RESPONSE + assert "message" in result + mock_client.set_bucket_metadata.assert_called_once_with( + bucket_id="in.c-sales", + entries=[("KBC.description", "Sales data bucket")], + branch_id=None, + ) + mock_client.close.assert_called_once() + + def test_with_branch(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_bucket_metadata.return_value = _META_RESPONSE + service = _make_service(store, mock_client) + + service.describe_bucket( + alias="prod", + bucket_id="in.c-sales", + description="desc", + branch_id=42, + ) + + mock_client.set_bucket_metadata.assert_called_once_with( + bucket_id="in.c-sales", + entries=[("KBC.description", "desc")], + branch_id=42, + ) + + def test_api_error_propagates(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_bucket_metadata.side_effect = KeboolaApiError( + message="Bucket not found", + status_code=404, + error_code="BUCKET_NOT_FOUND", + retryable=False, + ) + service = _make_service(store, mock_client) + + with pytest.raises(KeboolaApiError, match="Bucket not found"): + service.describe_bucket(alias="prod", bucket_id="in.c-missing", description="x") + + mock_client.close.assert_called_once() + + +class TestDescribeTableService: + """Tests for StorageService.describe_table().""" + + def test_success(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_table_metadata.return_value = _META_RESPONSE + service = _make_service(store, mock_client) + + result = service.describe_table( + alias="prod", + table_id="in.c-sales.orders", + description="All sales orders", + ) + + assert result["project_alias"] == "prod" + assert result["table_id"] == "in.c-sales.orders" + assert result["description"] == "All sales orders" + assert result["result"] == _META_RESPONSE + mock_client.set_table_metadata.assert_called_once_with( + table_id="in.c-sales.orders", + entries=[("KBC.description", "All sales orders")], + branch_id=None, + ) + mock_client.close.assert_called_once() + + def test_with_branch(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_table_metadata.return_value = _META_RESPONSE + service = _make_service(store, mock_client) + + service.describe_table( + alias="prod", + table_id="in.c-sales.orders", + description="desc", + branch_id=99, + ) + + mock_client.set_table_metadata.assert_called_once_with( + table_id="in.c-sales.orders", + entries=[("KBC.description", "desc")], + branch_id=99, + ) + + def test_api_error_propagates(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_table_metadata.side_effect = KeboolaApiError( + message="Table not found", + status_code=404, + error_code="TABLE_NOT_FOUND", + retryable=False, + ) + service = _make_service(store, mock_client) + + with pytest.raises(KeboolaApiError, match="Table not found"): + service.describe_table(alias="prod", table_id="in.c-missing.t", description="x") + + mock_client.close.assert_called_once() + + +class TestDescribeColumnsService: + """Tests for StorageService.describe_columns().""" + + def test_success_namespaced_keys(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_table_metadata.return_value = [] + service = _make_service(store, mock_client) + + result = service.describe_columns( + alias="prod", + table_id="in.c-sales.orders", + columns={"order_id": "Unique order identifier", "total": "Order total in USD"}, + ) + + assert result["project_alias"] == "prod" + assert result["table_id"] == "in.c-sales.orders" + assert result["columns"]["order_id"] == "Unique order identifier" + assert result["columns"]["total"] == "Order total in USD" + mock_client.set_table_metadata.assert_called_once_with( + table_id="in.c-sales.orders", + entries=[ + ("KBC.column.order_id.description", "Unique order identifier"), + ("KBC.column.total.description", "Order total in USD"), + ], + branch_id=None, + ) + mock_client.close.assert_called_once() + + def test_empty_columns_raises_value_error(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + service = _make_service(store, mock_client) + + with pytest.raises(ValueError, match="At least one column"): + service.describe_columns(alias="prod", table_id="in.c-sales.orders", columns={}) + + mock_client.set_table_metadata.assert_not_called() + + def test_with_branch(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_table_metadata.return_value = [] + service = _make_service(store, mock_client) + + service.describe_columns( + alias="prod", + table_id="in.c-sales.orders", + columns={"col1": "First column"}, + branch_id=77, + ) + + mock_client.set_table_metadata.assert_called_once_with( + table_id="in.c-sales.orders", + entries=[("KBC.column.col1.description", "First column")], + branch_id=77, + ) + + +class TestDescribeBatchService: + """Tests for StorageService.describe_batch().""" + + def test_success_all_sections(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_bucket_metadata.return_value = [] + mock_client.set_table_metadata.return_value = [] + service = _make_service(store, mock_client) + + batch_file = tmp_path / "batch.yaml" + batch_file.write_text( + "buckets:\n" + " in.c-sales: Sales data\n" + "tables:\n" + " in.c-sales.orders: Order data\n" + "columns:\n" + " in.c-sales.orders:\n" + " order_id: Unique order ID\n", + encoding="utf-8", + ) + + result = service.describe_batch(alias="prod", from_file=batch_file) + + assert result["project_alias"] == "prod" + assert len(result["applied"]) == 3 + assert result["errors"] == [] + applied_types = [a["type"] for a in result["applied"]] + assert "bucket" in applied_types + assert "table" in applied_types + assert "columns" in applied_types + # Bucket metadata called once (for the bucket), table metadata called twice + # (once for table description, once for column descriptions) + assert mock_client.set_bucket_metadata.call_count == 1 + assert mock_client.set_table_metadata.call_count == 2 + + def test_file_not_found(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + service = _make_service(store, mock_client) + + with pytest.raises(ValueError, match="Batch file not found"): + service.describe_batch(alias="prod", from_file=tmp_path / "missing.yaml") + + def test_partial_errors_collected(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.set_bucket_metadata.side_effect = [ + [], + KeboolaApiError( + message="Bucket not found", + status_code=404, + error_code="NOT_FOUND", + retryable=False, + ), + ] + service = _make_service(store, mock_client) + + batch_file = tmp_path / "partial.yaml" + batch_file.write_text( + "buckets:\n in.c-good: Good bucket\n in.c-bad: Bad bucket\n", + encoding="utf-8", + ) + + result = service.describe_batch(alias="prod", from_file=batch_file) + + assert len(result["applied"]) == 1 + assert len(result["errors"]) == 1 + assert result["applied"][0]["id"] == "in.c-good" + assert result["errors"][0]["id"] == "in.c-bad" + assert "Bucket not found" in result["errors"][0]["error"] + + def test_empty_yaml(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + service = _make_service(store, mock_client) + + batch_file = tmp_path / "empty.yaml" + batch_file.write_text("", encoding="utf-8") + + result = service.describe_batch(alias="prod", from_file=batch_file) + + assert result["applied"] == [] + assert result["errors"] == [] + mock_client.set_bucket_metadata.assert_not_called() + mock_client.set_table_metadata.assert_not_called() + + def test_invalid_yaml_not_mapping(self, tmp_path: Path) -> None: + store = _make_store(tmp_path) + mock_client = MagicMock() + service = _make_service(store, mock_client) + + batch_file = tmp_path / "invalid.yaml" + batch_file.write_text("- item1\n- item2\n", encoding="utf-8") + + with pytest.raises(ValueError, match="must be a YAML mapping"): + service.describe_batch(alias="prod", from_file=batch_file) + + +def _token_info(project_id: int = 258) -> TokenVerifyResponse: + return TokenVerifyResponse( + token_id="12345", + token_description="Test Token", + project_id=project_id, + project_name="Production", + owner_name="Production", + ) + + +class TestGetBucketDetailDescriptionExtraction: + """Verify get_bucket_detail extracts description from metadata and exposes raw metadata.""" + + def test_extracts_description_from_kbc_description_metadata(self, tmp_path: Path) -> None: + """KBC.description (provider=user) in the metadata array is exposed as 'description'.""" + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.verify_token.return_value = _token_info() + mock_client.get_bucket_detail.return_value = { + "id": "in.c-sales", + "displayName": "sales", + "stage": "in", + "description": "", # native field empty + "backend": "snowflake", + "backendPath": ["SAPI_258", "in.c-sales"], + "metadata": [ + { + "id": "9001", + "key": "KBC.description", + "value": "Revenue numbers", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + }, + { + "id": "9002", + "key": "KBC.createdBy.component.id", + "value": "keboola.orchestrator", + "provider": "system", + "timestamp": "2026-04-22T10:00:00Z", + }, + ], + "tables": [], + } + service = _make_service(store, mock_client) + + result = service.get_bucket_detail(alias="prod", bucket_id="in.c-sales") + + assert result["description"] == "Revenue numbers" + # raw_metadata must surface as the 'metadata' field + assert isinstance(result["metadata"], list) + assert len(result["metadata"]) == 2 + assert result["metadata"][0]["key"] == "KBC.description" + + def test_metadata_description_wins_over_native_description(self, tmp_path: Path) -> None: + """Precedence: KBC.description metadata entry overrides the native 'description' field. + + This pins current behavior: when both are present the metadata entry wins, because + the native field is only settable at bucket-create time via the Storage API; any + user-visible description updates flow through the metadata endpoint. + """ + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.verify_token.return_value = _token_info() + mock_client.get_bucket_detail.return_value = { + "id": "in.c-sales", + "displayName": "sales", + "stage": "in", + "description": "Legacy native description", + "backend": "snowflake", + "backendPath": ["SAPI_258", "in.c-sales"], + "metadata": [ + { + "id": "9001", + "key": "KBC.description", + "value": "New metadata description", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + } + ], + "tables": [], + } + service = _make_service(store, mock_client) + + result = service.get_bucket_detail(alias="prod", bucket_id="in.c-sales") + + assert result["description"] == "New metadata description" + + def test_falls_back_to_native_description_when_no_metadata(self, tmp_path: Path) -> None: + """With no KBC.description in metadata, the native 'description' field is used.""" + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.verify_token.return_value = _token_info() + mock_client.get_bucket_detail.return_value = { + "id": "in.c-sales", + "displayName": "sales", + "stage": "in", + "description": "Only native", + "backend": "snowflake", + "backendPath": ["SAPI_258", "in.c-sales"], + "metadata": [], + "tables": [], + } + service = _make_service(store, mock_client) + + result = service.get_bucket_detail(alias="prod", bucket_id="in.c-sales") + + assert result["description"] == "Only native" + assert result["metadata"] == [] + + def test_ignores_non_user_provider_kbc_description(self, tmp_path: Path) -> None: + """A KBC.description entry with provider != 'user' must not be picked up.""" + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.verify_token.return_value = _token_info() + mock_client.get_bucket_detail.return_value = { + "id": "in.c-sales", + "displayName": "sales", + "stage": "in", + "description": "Native wins here", + "backend": "snowflake", + "backendPath": ["SAPI_258", "in.c-sales"], + "metadata": [ + { + "id": "9001", + "key": "KBC.description", + "value": "System-set", + "provider": "system", + "timestamp": "2026-04-22T10:00:00Z", + } + ], + "tables": [], + } + service = _make_service(store, mock_client) + + result = service.get_bucket_detail(alias="prod", bucket_id="in.c-sales") + + assert result["description"] == "Native wins here" + + +class TestGetTableDetailDescriptionExtraction: + """Verify get_table_detail extracts table + per-column descriptions from metadata.""" + + def test_extracts_table_and_column_descriptions(self, tmp_path: Path) -> None: + """KBC.description + KBC.column.{name}.description are surfaced on the response.""" + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.get_table_detail.return_value = { + "id": "in.c-sales.orders", + "name": "orders", + "displayName": "orders", + "bucket": {"id": "in.c-sales"}, + "columns": ["order_id", "total"], + "primaryKey": ["order_id"], + "rowsCount": 42, + "columnMetadata": {}, + "metadata": [ + { + "id": "1", + "key": "KBC.description", + "value": "Sales orders fact table", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + }, + { + "id": "2", + "key": "KBC.column.order_id.description", + "value": "Unique order identifier", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + }, + { + "id": "3", + "key": "KBC.column.total.description", + "value": "Order total in USD", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + }, + ], + } + service = _make_service(store, mock_client) + + result = service.get_table_detail(alias="prod", table_id="in.c-sales.orders") + + assert result["description"] == "Sales orders fact table" + + # column_details must be a list of dicts, one per column, with descriptions + col_map = {c["name"]: c for c in result["column_details"]} + assert col_map["order_id"]["description"] == "Unique order identifier" + assert col_map["total"]["description"] == "Order total in USD" + + # raw_metadata must be exposed as the 'metadata' field + assert isinstance(result["metadata"], list) + assert len(result["metadata"]) == 3 + + def test_columns_without_description_have_no_description_key(self, tmp_path: Path) -> None: + """Columns without a matching KBC.column.{name}.description entry omit 'description'.""" + store = _make_store(tmp_path) + mock_client = MagicMock() + mock_client.get_table_detail.return_value = { + "id": "in.c-sales.orders", + "name": "orders", + "displayName": "orders", + "bucket": {"id": "in.c-sales"}, + "columns": ["order_id", "total"], + "primaryKey": [], + "rowsCount": 0, + "columnMetadata": {}, + "metadata": [ + { + "id": "1", + "key": "KBC.column.order_id.description", + "value": "Unique order id", + "provider": "user", + "timestamp": "2026-04-22T10:00:00Z", + } + ], + } + service = _make_service(store, mock_client) + + result = service.get_table_detail(alias="prod", table_id="in.c-sales.orders") + + col_map = {c["name"]: c for c in result["column_details"]} + assert col_map["order_id"]["description"] == "Unique order id" + assert "description" not in col_map["total"] + # Table-level description absent when no KBC.description entry + assert result["description"] == "" From 0078cf6083374fb562042d393ecf3fbd7d2d2f9e Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:11:45 +0200 Subject: [PATCH 5/7] feat(0.22.0): queue polling parity -- exp curve + log tail + auto-cancel (#202) Exit code 7 (JOB_TIMEOUT_TERMINATED) distinguishes local-timeout-killed from queue-side failure (exit 4). Log tail N events attached to logTail on failure or dict on WARNING/TERMINATED. Legacy 1s fixed polling preserved behind --poll-strategy fixed for tests and very short jobs. Polling curve: 2s x 30 -> 5s x 48 -> 15s forever. KeboolaClient.wait_for_queue_job gains poll_strategy kwarg. KeboolaClient.fetch_job_events wraps Storage API /v2/storage/events?runId=. JobService._handle_wait_error centralises failure enrichment. Max's review fixes: existing error codes reused (no new literal strings). --- CLAUDE.md | 2 +- .../kbagent/references/commands-reference.md | 2 +- src/keboola_agent_cli/changelog.py | 5 + src/keboola_agent_cli/client.py | 103 ++++- src/keboola_agent_cli/commands/_helpers.py | 23 +- src/keboola_agent_cli/commands/context.py | 15 +- src/keboola_agent_cli/commands/job.py | 84 +++- src/keboola_agent_cli/constants.py | 34 ++ src/keboola_agent_cli/errors.py | 11 +- .../hints/definitions/job.py | 39 +- src/keboola_agent_cli/models.py | 8 + src/keboola_agent_cli/output.py | 10 +- src/keboola_agent_cli/services/job_service.py | 301 ++++++++++++- tests/test_cli.py | 200 +++++++++ tests/test_client.py | 232 ++++++++++ tests/test_e2e.py | 293 +++++++++++++ tests/test_services.py | 410 +++++++++++++++++- 17 files changed, 1741 insertions(+), 31 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c2b8a9cd..a4401589 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -263,7 +263,7 @@ kbagent config set-folder --project NAME --component-id ID --config-id ID --name kbagent job list [--project NAME] [--component-id ID] [--status STATUS] [--limit N] kbagent job detail --project NAME --job-id ID -kbagent job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables] +kbagent job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables] [--poll-strategy exponential|fixed] [--log-tail-lines N] kbagent job terminate --project NAME (--job-id ID [--job-id ID ...] | --status any|created|waiting|processing [--component-id ID] [--config-id ID] [--branch ID] [--limit N]) [--dry-run] [--yes] kbagent storage buckets [--project NAME] [--branch ID] diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 1706b248..a4314603 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -54,7 +54,7 @@ All commands support `--json` for structured output. Multi-project flags (`--pro ## Job History - `job list [--project NAME] [--component-id ID] [--config-id ID] [--status STATUS] [--limit N]` -- list jobs (default 50, max 500) - `job detail --project NAME --job-id ID` -- full job detail with timing and result message -- `job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables]` -- run a job, optionally wait for completion (branch-aware). For configs with linked `keboola.variables` (root-level `configuration.variables_id`), kbagent auto-resolves a `variableValuesId` so transformations bind to the deployed values row. `--variable-values-id` overrides; `--no-variables` skips resolution. `NO_VARIABLE_ROWS` when the linked variables config has zero rows -- fix via `kbagent config variables-set`. +- `job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables] [--poll-strategy exponential|fixed] [--log-tail-lines N]` -- run a job, optionally wait for completion (branch-aware). For configs with linked `keboola.variables` (root-level `configuration.variables_id`), kbagent auto-resolves a `variableValuesId` so transformations bind to the deployed values row. `--variable-values-id` overrides; `--no-variables` skips resolution. `NO_VARIABLE_ROWS` when the linked variables config has zero rows -- fix via `kbagent config variables-set`. Under `--wait`, polls with an exponential curve (2s x 30 -> 5s x 48 -> 15s); `--poll-strategy fixed` keeps a constant 1s interval. On FAILED/WARNING/TERMINATED, the last `--log-tail-lines` events (default 200, **0 disables -- recommended for automation pipelines**) are attached as `logTail` in the JSON result (or `details.logTail` on errors). If `--timeout` expires, kbagent issues `kill_job` on the remote and exits **7** (`JOB_TIMEOUT_TERMINATED`) with the cancelled `details.job` + `details.logTail`; if the kill itself fails, exits **4** (`QUEUE_JOB_TIMEOUT`, `retryable=true`). Use jq pattern `.error.details.logTail? // .data.logTail? // []` to pick up the tail regardless of exit code. - `job terminate --project NAME (--job-id ID [--job-id ...] | --status any|created|waiting|processing [--component-id ID] [--config-id ID] [--branch ID] [--limit N]) [--dry-run] [--yes]` -- kill running Queue API jobs. Use to stop runaway loops or clean up pile-ups from repeated `job run` calls. Two modes: by ID (single/batch) or by filter (`--status any` catches every killable state). Response partitions IDs into `killed / already_finished / not_found / failed`; safe to re-run idempotently. Kill is async -- poll `job detail` for `isFinished=true`. ## Storage diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 811c4afc..6267b832 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -44,6 +44,11 @@ "New: storage describe-batch --from-file -- apply bucket/table/column descriptions from a YAML file in one shot; failures collected, remaining items continue", "Fix: storage table-detail now returns 'description' and 'metadata' fields (extracted from table metadata array)", "Fix: storage bucket-detail now returns 'description' and 'metadata' fields (KBC.description in metadata takes precedence over native creation-time description field)", + "New: Queue API polling parity with FIIA and the keboola-as-code Go CLI -- `kbagent job run --wait` now polls on an exponential curve (2s x 30 -> 5s x 48 -> 15s) instead of a fixed 1s interval. Preserves the legacy cadence behind `--poll-strategy fixed` for tests and very short jobs (FIIA P0-3).", + "New: --log-tail-lines N on `job run` -- on FAILED / WARNING / TERMINATED jobs, kbagent fetches the last N Storage Events (via /v2/storage/events?runId=...) and surfaces them as `logTail` in --json output or `details.logTail` on errors. Default 200, max 5000, 0 disables.", + "New: --timeout now auto-cancels the remote job -- when the local deadline expires under --wait, kbagent issues `kill_job` against the Queue and exits 7 (EXIT_JOB_TIMEOUT_TERMINATED) with the cancelled job + logTail in the error details. Distinct from exit 4 (QUEUE_JOB_TIMEOUT, retryable) which signals the local kill attempt ALSO failed and the remote may still be running.", + "Client: new `fetch_job_events(run_id, limit)` wraps the Storage Events API -- runId is resolved from the job dict (Queue v2 jobs typically have runId == id). The Queue API has NO /jobs/{id}/events route despite the name; events live on Storage.", + "Error envelope: KeboolaApiError gained an optional `details: dict` payload; JSON --mode output now includes `error.details` (only when non-empty) so callers can consume structured context without parsing the human message.", ], "0.21.1": [ "Fix: sync pull on a newly created dev branch now writes config rows (#193) -- idempotent skip guard for rows was missing a file-existence check, causing rows to be silently skipped when the branch directory was new (hash matched main because the branch is a clone)", diff --git a/src/keboola_agent_cli/client.py b/src/keboola_agent_cli/client.py index da74a03e..6cb828c9 100644 --- a/src/keboola_agent_cli/client.py +++ b/src/keboola_agent_cli/client.py @@ -10,6 +10,7 @@ import json import logging import time +from collections.abc import Iterator from pathlib import Path from typing import Any from urllib.parse import quote @@ -21,17 +22,20 @@ DEFAULT_GROUPED_JOBS_LIMIT, DEFAULT_JOB_LIMIT, DEFAULT_JOBS_PER_CONFIG, + DEFAULT_POLL_STRATEGY, DEFAULT_TIMEOUT, EXPORT_JOB_MAX_WAIT, FILE_DOWNLOAD_CHUNK_SIZE, FILE_DOWNLOAD_TIMEOUT, FILE_UPLOAD_TIMEOUT, IMPORT_JOB_MAX_WAIT, + JOB_POLL_CURVE, METADATA_NOT_FOUND, QUERY_JOB_MAX_WAIT, QUERY_JOB_POLL_INTERVAL, STORAGE_JOB_MAX_WAIT, STORAGE_JOB_POLL_INTERVAL, + VALID_POLL_STRATEGIES, ) from .errors import KeboolaApiError from .http_base import BaseHttpClient @@ -40,6 +44,31 @@ logger = logging.getLogger(__name__) +def _iter_poll_intervals(strategy: str) -> Iterator[float]: + """Yield sleep intervals (seconds) for Queue job polling. + + Two strategies: + + - ``"exponential"`` walks ``JOB_POLL_CURVE``: each (interval, count) + segment yields ``count`` copies of ``interval``; a segment with + ``count == 0`` keeps yielding ``interval`` forever (valid only on + the last segment). + - ``"fixed"`` yields ``STORAGE_JOB_POLL_INTERVAL`` forever (legacy + behavior preserved for opt-out via ``--poll-strategy fixed``). + + The deadline check in ``wait_for_queue_job`` stops iteration. + """ + if strategy == "fixed": + while True: + yield STORAGE_JOB_POLL_INTERVAL + for interval, count in JOB_POLL_CURVE: + if count <= 0: + while True: + yield interval + for _ in range(count): + yield interval + + class KeboolaClient(BaseHttpClient): """HTTP client for the Keboola Storage API and Queue API. @@ -1921,23 +1950,81 @@ def kill_job(self, job_id: str) -> dict[str, Any]: response = self._queue_request("POST", f"/jobs/{safe_job_id}/kill") return response.json() + def fetch_job_events(self, run_id: str, limit: int | None = None) -> list[dict[str, Any]]: + """Fetch events emitted during a job's run. + + Wraps the Storage API's ``GET /v2/storage/events?runId={runId}`` + endpoint -- NOT a Queue API path. Queue jobs (Queue API v2) expose a + ``runId`` on the job dict (typically equal to the job ``id``); the + Storage Events API is the canonical event feed for the job. Returns + the list in Storage API order (newest -> oldest; callers that want + a chronological "tail" should reverse the slice). + + Args: + run_id: The job's ``runId`` (``job["runId"]``; falls back to + ``job["id"]`` on legacy records where they match). + limit: Optional server-side event cap. Storage API default is + about 100; pass an explicit value to cover long runs. + + Returns: + List of event dicts. Each event typically has ``uuid``, + ``event``, ``component``, ``message``, ``type``, ``created``, + ``runId``, ``configurationId`` keys. Empty when the run emitted + no events yet. + """ + params: dict[str, Any] = {"runId": run_id} + if limit is not None and limit > 0: + params["limit"] = limit + response = self._request("GET", "/v2/storage/events", params=params) + payload = response.json() + # Storage events returns a bare list. Tolerate a dict-wrapped + # future shape defensively. + if isinstance(payload, list): + return payload + if isinstance(payload, dict) and isinstance(payload.get("events"), list): + return payload["events"] + return [] + def wait_for_queue_job( - self, job_id: str, max_wait: float = STORAGE_JOB_MAX_WAIT + self, + job_id: str, + max_wait: float = STORAGE_JOB_MAX_WAIT, + poll_strategy: str = DEFAULT_POLL_STRATEGY, ) -> dict[str, Any]: """Poll a Queue API job until it reaches a terminal state. + Uses the piecewise ``JOB_POLL_CURVE`` from constants for the + ``"exponential"`` strategy (2s x 30 -> 5s x 48 -> 15s forever) and + the legacy fixed ``STORAGE_JOB_POLL_INTERVAL`` for ``"fixed"``. The + curve matches the cadence used by FIIA and the official + ``keboola-as-code`` Go CLI. + Args: job_id: The Queue job ID. max_wait: Maximum seconds to wait (default: STORAGE_JOB_MAX_WAIT). + poll_strategy: "exponential" (default) or "fixed". Any other + value raises ValueError before the first network call. Returns: Completed job dict. Raises: - KeboolaApiError: If the job fails or times out. - """ + ValueError: If poll_strategy is not one of VALID_POLL_STRATEGIES. + KeboolaApiError: If the job fails (QUEUE_JOB_FAILED) or the + deadline elapses before the job finishes (QUEUE_JOB_TIMEOUT). + """ + if poll_strategy not in VALID_POLL_STRATEGIES: + # ValueError (not KeboolaApiError) because this is a programming + # error -- the caller passed an invalid literal, not a bad API + # response. JobService validates before reaching this layer, so + # hitting this path from the CLI would be a bug in kbagent. + raise ValueError( + f"Invalid poll_strategy {poll_strategy!r}. " + f"Expected one of: {sorted(VALID_POLL_STRATEGIES)}." + ) + deadline = time.monotonic() + max_wait - while time.monotonic() < deadline: + for interval in _iter_poll_intervals(poll_strategy): job = self.get_job_detail(job_id) if job.get("isFinished"): if job.get("status") == "error": @@ -1954,7 +2041,13 @@ def wait_for_queue_job( retryable=False, ) return job - time.sleep(STORAGE_JOB_POLL_INTERVAL) + + # Cap the sleep so we never blow past the deadline by more than + # one interval: trim to whatever time remains; if zero, break. + remaining = deadline - time.monotonic() + if remaining <= 0: + break + time.sleep(min(interval, remaining)) raise KeboolaApiError( message=f"Queue job {job_id} did not complete within {max_wait}s", diff --git a/src/keboola_agent_cli/commands/_helpers.py b/src/keboola_agent_cli/commands/_helpers.py index 39e7e575..8b399b7b 100644 --- a/src/keboola_agent_cli/commands/_helpers.py +++ b/src/keboola_agent_cli/commands/_helpers.py @@ -15,7 +15,11 @@ import typer from ..config_store import ConfigStore -from ..constants import ENV_KBC_MANAGE_API_TOKEN, EXIT_PERMISSION_DENIED +from ..constants import ( + ENV_KBC_MANAGE_API_TOKEN, + EXIT_JOB_TIMEOUT_TERMINATED, + EXIT_PERMISSION_DENIED, +) from ..errors import KeboolaApiError, PermissionDeniedError from ..output import OutputFormatter @@ -63,15 +67,26 @@ def get_service(ctx: typer.Context, key: str) -> Any: def map_error_to_exit_code(exc: KeboolaApiError) -> int: """Map a KeboolaApiError to a CLI exit code. - Unified 3-case logic: - INVALID_TOKEN -> 3 (authentication error) - - TIMEOUT / CONNECTION_ERROR / RETRY_EXHAUSTED -> 4 (network error) + - TIMEOUT / CONNECTION_ERROR / RETRY_EXHAUSTED / QUEUE_JOB_TIMEOUT -> 4 + (network/retryable; QUEUE_JOB_TIMEOUT means local gave up AND the + remote-kill attempt also failed, so the job may still be running) + - JOB_TIMEOUT_TERMINATED -> EXIT_JOB_TIMEOUT_TERMINATED (7) + (local --timeout elapsed and we successfully cancelled the remote + job; scripts can distinguish "we killed it" from "it failed on its own") - Everything else -> 1 (general error) """ if exc.error_code == "INVALID_TOKEN": return 3 - if exc.error_code in ("TIMEOUT", "CONNECTION_ERROR", "RETRY_EXHAUSTED"): + if exc.error_code in ( + "TIMEOUT", + "CONNECTION_ERROR", + "RETRY_EXHAUSTED", + "QUEUE_JOB_TIMEOUT", + ): return 4 + if exc.error_code == "JOB_TIMEOUT_TERMINATED": + return EXIT_JOB_TIMEOUT_TERMINATED return 1 diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index edb009a8..5c5a3bbc 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -168,13 +168,26 @@ kbagent job detail --project NAME --job-id ID Full job detail including result message and timing. - kbagent job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables] + kbagent job run --project NAME --component-id ID --config-id ID [--row-id ID ...] [--wait] [--timeout N] [--branch ID] [--variable-values-id ID] [--no-variables] [--poll-strategy exponential|fixed] [--log-tail-lines N] Run a Queue API job. --row-id selects specific config rows (repeatable; omit to run entire config). --wait polls until job finishes. --timeout sets max wait in seconds (default 300). Branch-aware. When the config has linked variables (configuration.variables_id), kbagent auto-resolves a variableValuesId so the job binds to the deployed values row. --variable-values-id overrides; --no-variables skips resolution. Error code NO_VARIABLE_ROWS when the linked variables config has zero rows (run `kbagent config variables-set` to create one). + Polling under --wait uses an exponential curve by default (2s x 30 -> 5s x 48 -> 15s); + --poll-strategy fixed keeps a constant 1s interval. On FAILED/WARNING/TERMINATED, the last + --log-tail-lines events (default 200, 0 disables -- recommended for automation pipelines) are + surfaced as `logTail` in --json output. + --json response shapes by exit code: + - exit 0 (success): {{status:"ok", data:{{..., logTail?:[...]}}}} + - exit 1 (QUEUE_JOB_FAILED, remote job status=error): + {{status:"error", error:{{code:"QUEUE_JOB_FAILED", details:{{logTail:[...]}}}}}} + - exit 4 (QUEUE_JOB_TIMEOUT, local timeout + remote kill also failed): + {{status:"error", error:{{code:"QUEUE_JOB_TIMEOUT", retryable:true, details:{{logTail:[...]}}}}}} + - exit 7 (JOB_TIMEOUT_TERMINATED, local timeout + remote kill succeeded): + {{status:"error", error:{{code:"JOB_TIMEOUT_TERMINATED", details:{{job:{{...}}, logTail:[...]}}}}}} + jq pattern: `.error.details.logTail? // .data.logTail? // []` picks up the tail regardless of exit. kbagent job terminate --project NAME (--job-id ID [--job-id ID ...] | --status any|created|waiting|processing [--component-id ID] [--config-id ID] [--branch ID] [--limit N]) [--dry-run] [--yes] Kill running jobs via Queue API (POST /jobs/{id}/kill). Use to stop runaway loops or pile-ups. diff --git a/src/keboola_agent_cli/commands/job.py b/src/keboola_agent_cli/commands/job.py index f855cc33..6552a4df 100644 --- a/src/keboola_agent_cli/commands/job.py +++ b/src/keboola_agent_cli/commands/job.py @@ -4,6 +4,7 @@ No business logic belongs here. """ +import click import typer from rich.markup import escape @@ -11,8 +12,12 @@ from ..constants import ( DEFAULT_JOB_LIMIT, DEFAULT_JOB_RUN_TIMEOUT, + DEFAULT_LOG_TAIL_LINES, + DEFAULT_POLL_STRATEGY, KILLABLE_JOB_STATUSES, MAX_JOB_LIMIT, + MAX_LOG_TAIL_LINES, + VALID_POLL_STRATEGIES, VALID_STATUSES, ) from ..errors import ConfigError, KeboolaApiError @@ -212,6 +217,28 @@ def job_run( "--variable-values-id." ), ), + poll_strategy: str = typer.Option( + DEFAULT_POLL_STRATEGY, + "--poll-strategy", + click_type=click.Choice(sorted(VALID_POLL_STRATEGIES)), + help=( + "Polling cadence used with --wait. 'exponential' (default) " + "starts at 2s and relaxes toward 15s as a job runs long " + "(2s x 30 -> 5s x 48 -> 15s). 'fixed' keeps a constant 1s " + "interval (useful for tests or very short jobs)." + ), + ), + log_tail_lines: int = typer.Option( + DEFAULT_LOG_TAIL_LINES, + "--log-tail-lines", + help=( + "On FAILED/WARNING/TERMINATED jobs, fetch the last N job events " + f"(from Storage Events API) and surface them as 'logTail' in " + f"JSON output or a panel in human mode. Only used with --wait. " + f"0 disables (recommended for automation pipelines); max " + f"{MAX_LOG_TAIL_LINES}." + ), + ), ) -> None: """Run a job for a component configuration. @@ -225,6 +252,13 @@ def job_run( kbagent auto-resolves a variableValuesId so the job binds to the deployed values row. Override with --variable-values-id or skip with --no-variables. + + Queue polling uses an exponential curve by default (2s x 30 -> 5s x 48 + -> 15s, total 5min before the 15s tail). If --timeout expires, kbagent + issues kill_job on the remote and exits 7 (JOB_TIMEOUT_TERMINATED) with + the cancelled job + log tail attached. If the kill itself fails, exits + 4 (QUEUE_JOB_TIMEOUT, retryable) so scripts can tell "we killed it" + from "local gave up, remote may still be running". """ if should_hint(ctx): emit_hint( @@ -239,6 +273,8 @@ def job_run( branch=branch, variable_values_id=variable_values_id, no_variables=no_variables, + poll_strategy=poll_strategy, + log_tail_lines=log_tail_lines, ) return formatter = get_formatter(ctx) @@ -268,6 +304,16 @@ def job_run( ) raise typer.Exit(code=2) + if log_tail_lines < 0 or log_tail_lines > MAX_LOG_TAIL_LINES: + formatter.error( + message=( + f"--log-tail-lines must be between 0 and {MAX_LOG_TAIL_LINES}. " + f"Got {log_tail_lines}." + ), + error_code="INVALID_ARGUMENT", + ) + raise typer.Exit(code=2) + validate_branch_requires_project(formatter, branch, project) _, effective_branch = resolve_branch(config_store, formatter, project, branch) @@ -295,6 +341,8 @@ def job_run( branch_id=effective_branch, variable_values_id=variable_values_id, no_variables=no_variables, + poll_strategy=poll_strategy, + log_tail_lines=log_tail_lines, ) except ConfigError as exc: formatter.error(message=exc.message, error_code="CONFIG_ERROR") @@ -305,7 +353,10 @@ def job_run( error_code=exc.error_code, project=project, retryable=exc.retryable, + details=exc.details or None, ) + if not formatter.json_mode: + _render_log_tail(formatter, exc.details) raise typer.Exit(code=map_error_to_exit_code(exc)) from None if formatter.json_mode: @@ -318,15 +369,12 @@ def job_run( status = result.get("status", "unknown") if status in ("success", "terminated"): formatter.console.print(f"[bold green]Job {job_id}:[/bold green] {status}") - elif status == "error": - error_msg = "" - job_result = result.get("result", {}) - if isinstance(job_result, dict): - error_msg = job_result.get("message", "") - formatter.console.print(f"[bold red]Job {job_id}:[/bold red] {status}") - if error_msg: - formatter.console.print(f" Error: {error_msg}") - raise typer.Exit(code=1) + elif status == "warning": + # "error" is intentionally absent here: the service layer raises + # QUEUE_JOB_FAILED for failed jobs, so this human-mode branch is + # only reached for non-error terminal and transient states. + formatter.console.print(f"[bold yellow]Job {job_id}:[/bold yellow] {status}") + _render_log_tail(formatter, {"logTail": result.get("logTail") or []}) else: formatter.console.print(f"[bold blue]Job {job_id}:[/bold blue] {status}") if not wait: @@ -336,6 +384,24 @@ def job_run( ) +def _render_log_tail(formatter, details: dict | None) -> None: + """Render a logTail attached to an error or result payload in human mode.""" + if not details: + return + tail = details.get("logTail") or [] + if not tail: + return + formatter.console.print("[bold]Log tail (last events):[/bold]") + for event in tail: + ts = event.get("created") or event.get("createdTime") or "" + event_type = event.get("type") or event.get("event") or "" + msg = event.get("message") or "" + # Trim overly long event messages so the panel stays readable. + if len(msg) > 400: + msg = msg[:400] + "..." + formatter.console.print(f" [dim]{ts}[/dim] [{event_type}] {msg}") + + @job_app.command("terminate") def job_terminate( ctx: typer.Context, diff --git a/src/keboola_agent_cli/constants.py b/src/keboola_agent_cli/constants.py index 22210e4b..33d7a8eb 100644 --- a/src/keboola_agent_cli/constants.py +++ b/src/keboola_agent_cli/constants.py @@ -60,6 +60,34 @@ STORAGE_JOB_MAX_WAIT: float = 60.0 # max seconds to wait for a storage job IMPORT_JOB_MAX_WAIT: float = 600.0 # 10 min for table import jobs (large files) +# --- Queue Job Polling --- +# Piecewise curve matching FIIA's existing Queue API polling contract +# (same cadence as the official keboola-as-code Go CLI): fast initial polls +# to catch short jobs, then relax so multi-hour orchestrations don't spam +# the API. Each tuple is (interval_seconds, max_polls_at_this_interval); +# count=0 means "continue at this interval indefinitely" (only valid on the +# last segment). Total first-phase time: 2s * 30 + 5s * 48 = 300s = 5 min, +# after which we settle at 15s forever. +JOB_POLL_CURVE: tuple[tuple[float, int], ...] = ( + (2.0, 30), + (5.0, 48), + (15.0, 0), +) +VALID_POLL_STRATEGIES: frozenset[str] = frozenset({"exponential", "fixed"}) +DEFAULT_POLL_STRATEGY: str = "exponential" +# Default log-tail length surfaced on FAILED/WARNING/TERMINATED jobs. +DEFAULT_LOG_TAIL_LINES: int = 200 +# Upper bound to prevent accidentally pulling tens of thousands of events +# from a long-running job. +MAX_LOG_TAIL_LINES: int = 5000 +# Seconds to wait after issuing kill_job() during timeout-cancellation for +# the job to transition to a terminal state before we return. +JOB_TERMINATE_GRACE_SECONDS: float = 10.0 +# Poll cadence while waiting inside the terminate grace window; capped so we +# never overshoot the deadline by more than one interval on latency-sensitive +# callers (see _terminate_and_wait in services/job_service.py). +JOB_TERMINATE_POLL_INTERVAL: float = 1.0 + # --- Storage Write Validation --- VALID_COLUMN_TYPES: frozenset[str] = frozenset( {"STRING", "INTEGER", "NUMERIC", "FLOAT", "BOOLEAN", "DATE", "TIMESTAMP"} @@ -136,6 +164,12 @@ # --- Permission Exit Code --- EXIT_PERMISSION_DENIED: int = 6 +# --- Job-timeout Exit Code --- +# Distinct from the general "1" exit code so scripts can tell +# "local --timeout elapsed and we cancelled the remote job" apart +# from "job failed on its own". The retryable-with-longer-timeout +# QUEUE_JOB_TIMEOUT case (kill itself failed) stays at exit 4. +EXIT_JOB_TIMEOUT_TERMINATED: int = 7 # --- Domain Validation Constants --- VALID_COMPONENT_TYPES: list[str] = ["extractor", "writer", "transformation", "application"] diff --git a/src/keboola_agent_cli/errors.py b/src/keboola_agent_cli/errors.py index 82d50400..7d3a5746 100644 --- a/src/keboola_agent_cli/errors.py +++ b/src/keboola_agent_cli/errors.py @@ -27,7 +27,14 @@ def mask_token(token: str) -> str: class KeboolaApiError(Exception): - """Raised when a Keboola API call fails.""" + """Raised when a Keboola API call fails. + + Optional ``details`` payload lets the service layer attach structured + context (e.g. a fetched log tail, the remote-cancelled job dict) that + the command layer surfaces in ``--json`` mode without changing the + stable top-level error envelope. Keep keys small and side-effect-free; + PR9 will lock this schema down with a versioned enum. + """ def __init__( self, @@ -35,12 +42,14 @@ def __init__( status_code: int = 0, error_code: str = "UNKNOWN_ERROR", retryable: bool = False, + details: dict | None = None, ) -> None: super().__init__(message) self.message = message self.status_code = status_code self.error_code = error_code self.retryable = retryable + self.details: dict = details if details is not None else {} class ConfigError(Exception): diff --git a/src/keboola_agent_cli/hints/definitions/job.py b/src/keboola_agent_cli/hints/definitions/job.py index fa11f8ae..42df1ee3 100644 --- a/src/keboola_agent_cli/hints/definitions/job.py +++ b/src/keboola_agent_cli/hints/definitions/job.py @@ -131,11 +131,19 @@ "timeout": "{timeout}", "variable_values_id": "{variable_values_id}", "no_variables": "{no_variables}", + "poll_strategy": "{poll_strategy}", + "log_tail_lines": "{log_tail_lines}", }, ), ), HintStep( - comment="Poll until job completes (when --wait is used)", + comment=( + "Poll until job completes (when --wait is used). The " + "piecewise interval matches JOB_POLL_CURVE: 2s x 30 -> " + "5s x 48 -> 15s. This sample shows 5s as a single " + "representative value; the real client cycles through " + "the curve." + ), client=ClientCall( method="get_job_detail", args={"job_id": 'str(job["id"])'}, @@ -145,11 +153,28 @@ poll_interval=5.0, poll_condition='not job.get("isFinished")', ), + HintStep( + comment=( + "On FAILED/WARNING/TERMINATED jobs, fetch the last N events " + "to surface as logTail (skip when --log-tail-lines 0). " + "fetch_job_events expects runId (not jobId); Queue v2 jobs " + "usually have runId == id but we resolve defensively." + ), + client=ClientCall( + method="fetch_job_events", + args={ + "run_id": 'str(job.get("runId") or job["id"])', + "limit": "{log_tail_lines}", + }, + result_var="events", + result_hint="list[dict]", + ), + ), ], notes=[ "Uses the Queue API (queue.keboola.com), not Storage API.", "Without --wait, returns immediately after job creation.", - "Service layer handles both resolve + create + optional poll in one call.", + "Service layer handles resolve + create + optional poll + log-tail in one call.", ( "Service auto-resolves variableValuesId from " "configuration.variables_id; the client hint shows " @@ -161,6 +186,16 @@ "has zero rows; fix via `kbagent config variables-set` or " "pass --no-variables." ), + ( + "--poll-strategy exponential (default) matches FIIA and the " + "Go CLI. --poll-strategy fixed retains the legacy 1s interval." + ), + ( + "If --timeout elapses, the service issues kill_job and raises " + "JOB_TIMEOUT_TERMINATED (exit 7) with the cancelled job + " + "logTail in details. If kill itself fails, QUEUE_JOB_TIMEOUT " + "(exit 4, retryable) is surfaced instead." + ), ], ) ) diff --git a/src/keboola_agent_cli/models.py b/src/keboola_agent_cli/models.py index 76475e31..84c746cd 100644 --- a/src/keboola_agent_cli/models.py +++ b/src/keboola_agent_cli/models.py @@ -153,6 +153,14 @@ class ErrorResponse(BaseModel): message: str = Field(description="Human-readable error description") project: str = Field(default="", description="Project alias related to the error, if any") retryable: bool = Field(default=False, description="Whether the operation can be retried") + details: dict | None = Field( + default=None, + description=( + "Optional structured context keyed by the producer (e.g. logTail " + "for failed Queue jobs). Absent when empty so JSON consumers can " + "assume 'details in err' implies non-empty payload." + ), + ) class SuccessResponse(BaseModel): diff --git a/src/keboola_agent_cli/output.py b/src/keboola_agent_cli/output.py index 8b767b4b..f6b3f2e8 100644 --- a/src/keboola_agent_cli/output.py +++ b/src/keboola_agent_cli/output.py @@ -68,6 +68,7 @@ def error( project: str = "", retryable: bool = False, error_type: str = "", + details: dict | None = None, ) -> None: """Output an error message. @@ -77,6 +78,9 @@ def error( project: Project alias related to the error. retryable: Whether the operation can be retried. error_type: Broad error category. If empty, derived from error_code. + details: Optional structured context (e.g. {"logTail": [...]}). + When empty/None, the field is omitted from JSON output so + consumers can key off presence. """ if self.json_mode: if not error_type: @@ -89,8 +93,12 @@ def error( message=message, project=project, retryable=retryable, + details=details if details else None, ) - error_envelope = {"status": "error", "error": err.model_dump()} + error_envelope = { + "status": "error", + "error": err.model_dump(exclude_none=True), + } sys.stdout.write(json.dumps(error_envelope, indent=2) + "\n") else: self.err_console.print(f"[bold red]Error:[/bold red] {message}") diff --git a/src/keboola_agent_cli/services/job_service.py b/src/keboola_agent_cli/services/job_service.py index 295ec17a..e6ebe661 100644 --- a/src/keboola_agent_cli/services/job_service.py +++ b/src/keboola_agent_cli/services/job_service.py @@ -5,13 +5,172 @@ resolution. Stays agnostic of CLI and HTTP transport details. """ +import logging +import time from typing import Any -from ..constants import DEFAULT_JOB_LIMIT, KILLABLE_JOB_STATUSES +from ..constants import ( + DEFAULT_JOB_LIMIT, + DEFAULT_LOG_TAIL_LINES, + DEFAULT_POLL_STRATEGY, + JOB_TERMINATE_GRACE_SECONDS, + JOB_TERMINATE_POLL_INTERVAL, + KILLABLE_JOB_STATUSES, + VALID_POLL_STRATEGIES, +) from ..errors import KeboolaApiError from ..models import ProjectConfig from .base import BaseService +logger = logging.getLogger(__name__) + +# Terminal statuses for which we surface a log-tail. +_LOG_TAIL_STATUSES: frozenset[str] = frozenset({"error", "warning", "terminated"}) + + +def _safe_fetch_log_tail(client: Any, job: dict[str, Any], limit: int) -> list[dict[str, Any]]: + """Fetch the last ``limit`` events for a job; never raises. + + Resolves ``runId`` from the job dict (falls back to ``id`` on legacy + records where Queue v2 makes them equal). Storage Events API returns + newest -> oldest, which we keep as-is: a "tail" display wants the + most recent events first, and callers can reverse for chronology if + they prefer. + + Log-tail capture is a convenience surface; failing the whole command + because the events endpoint blipped would obscure the real underlying + error. We log the secondary failure at debug level and return an empty + list so callers can safely attach it to their result payload. + """ + if limit <= 0: + return [] + run_id = str(job.get("runId") or job.get("id") or "") + if not run_id: + # Defensive: malformed job dict with neither runId nor id. Log so + # a real API regression doesn't get masked by the silent return. + logger.debug( + "log tail fetch skipped: job has no runId or id (keys=%s)", + sorted(job.keys()), + ) + return [] + try: + events = client.fetch_job_events(run_id, limit=limit) + except KeboolaApiError as exc: + logger.debug( + "fetch_job_events(%s) failed (%s): %s; surfacing empty logTail", + run_id, + exc.error_code, + exc.message, + ) + return [] + except Exception as exc: # defensive: do not let tail fetch tank the run + logger.debug("fetch_job_events(%s) raised %r; surfacing empty logTail", run_id, exc) + return [] + # Storage Events API returns newest -> oldest today, but we do not + # rely on that: sort defensively by `created` DESC so a future API + # ordering change cannot silently invert the tail. Events without a + # `created` key sort last (they lose the race to timestamped peers). + events_list = list(events) + events_list.sort(key=lambda e: e.get("created") or "", reverse=True) + # Cap at limit in case the server ignored ?limit. + return events_list[:limit] + + +def _enrich_with_tail( + base_details: dict[str, Any] | None, + tail: list[dict[str, Any]], +) -> dict[str, Any] | None: + """Return a fresh details dict with ``logTail`` merged in, or None if empty. + + We never mutate a caught exception's ``.details`` in place; we construct + a new dict and let the caller wrap it in a new KeboolaApiError. That + keeps exception identity clean and makes the chain explicit via + ``raise ... from exc``. + """ + if not tail and not base_details: + return None + merged: dict[str, Any] = dict(base_details or {}) + if tail: + merged["logTail"] = tail + return merged or None + + +def _terminate_and_wait( + client: Any, + job_id: str, + grace_seconds: float, +) -> dict[str, Any] | None: + """Issue kill_job and poll briefly for terminal status. + + Returns the final job dict if termination was observed within + ``grace_seconds``; returns ``None`` if the kill call itself failed + (network, auth, race against remote terminal state). Never raises -- + the caller decides how to surface the failure. + + Idempotent under concurrent kill attempts: if the user runs + ``kbagent job terminate --job-id X`` while ``job run --wait`` is + auto-cancelling X on timeout, both paths land on Queue API's + ``POST /jobs/{id}/kill``. The second caller sees HTTP 400 + "not in killable states" (or HTTP 500 with body 404 for already-final + jobs); the KeboolaApiError branch below still succeeds as long as a + follow-up GET confirms isFinished=True. No data loss, no double-kill + error surfaced. + """ + try: + client.kill_job(job_id) + except KeboolaApiError as exc: + # If the job is already terminal, kill returns 400/500 -- honour + # that as "cancelled successfully" when a follow-up GET confirms. + logger.debug("kill_job(%s) raised %s: %s", job_id, exc.error_code, exc.message) + try: + job = client.get_job_detail(job_id) + except Exception: + logger.debug("terminate: kill failed AND GET failed for %s; giving up", job_id) + return None + if job.get("isFinished"): + return job + logger.debug( + "terminate: kill failed, GET succeeded but job %s still not terminal " + "(status=%r); returning None (remote may still be running)", + job_id, + job.get("status"), + ) + return None + except Exception as exc: + logger.debug("kill_job(%s) raised %r; giving up", job_id, exc) + return None + + deadline = time.monotonic() + grace_seconds + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + try: + job = client.get_job_detail(job_id) + except Exception: + time.sleep(min(JOB_TERMINATE_POLL_INTERVAL, remaining)) + continue + if job.get("isFinished"): + return job + # Cap the sleep so we never overshoot the grace deadline by a + # full poll interval (caller may be latency-sensitive). + time.sleep(min(JOB_TERMINATE_POLL_INTERVAL, max(0.0, deadline - time.monotonic()))) + + # Remote still wasn't terminal within the grace window; return whatever + # the last GET saw so callers can surface the actual state. + try: + final = client.get_job_detail(job_id) + except Exception: + logger.debug("terminate: grace window exhausted and final GET failed for %s", job_id) + return None + logger.debug( + "terminate: grace window exhausted; job %s status=%r isFinished=%r", + job_id, + final.get("status"), + final.get("isFinished"), + ) + return final + class JobService(BaseService): """Business logic for Keboola jobs (list, detail, run, terminate). @@ -160,6 +319,8 @@ def run_job( branch_id: int | None = None, variable_values_id: str | None = None, no_variables: bool = False, + poll_strategy: str = DEFAULT_POLL_STRATEGY, + log_tail_lines: int = DEFAULT_LOG_TAIL_LINES, ) -> dict[str, Any]: """Create and optionally wait for a Queue API job. @@ -169,6 +330,21 @@ def run_job( strings. Pass ``variable_values_id`` to override, or ``no_variables=True`` to skip the resolution entirely. + Wait-mode behavior (``wait=True``): + + - Polls the Queue API using ``poll_strategy`` (default exponential, + matching FIIA and the Go CLI cadence: 2s x 30 -> 5s x 48 -> 15s). + - When the job lands in a terminal non-success state (``error``, + ``warning``, ``terminated``) and ``log_tail_lines > 0``, fetches + the job's events via ``fetch_job_events`` and attaches the last + ``log_tail_lines`` as the ``logTail`` key on the returned dict. + - If the local deadline elapses before the remote job finishes, + issues ``kill_job`` to cancel the remote work, waits briefly for + termination, and raises ``KeboolaApiError`` with + ``error_code="JOB_TIMEOUT_TERMINATED"``. If the kill call itself + fails we fall back to the original ``QUEUE_JOB_TIMEOUT`` error so + the caller can tell "local gave up" from "remote was cancelled". + Args: alias: Project alias. component_id: Component ID to run. @@ -184,11 +360,31 @@ def run_job( no_variables: If True, skip variable-values resolution entirely (useful for components that do not support variables, or when the caller intentionally wants empty-string binding). + poll_strategy: Wait cadence. One of VALID_POLL_STRATEGIES. + log_tail_lines: Number of trailing events to surface on + non-success terminal states. ``0`` disables the fetch. Returns: - Job dict with project_alias. If wait=True, returns the - completed job; otherwise returns the initial job response. + Job dict with ``project_alias``. If wait=True, returns the + completed job, optionally with ``logTail`` attached; otherwise + returns the initial job response. """ + if poll_strategy not in VALID_POLL_STRATEGIES: + raise KeboolaApiError( + message=( + f"Invalid poll_strategy {poll_strategy!r}. " + f"Expected one of: {sorted(VALID_POLL_STRATEGIES)}." + ), + status_code=0, + error_code="INVALID_ARGUMENT", + ) + if log_tail_lines < 0: + raise KeboolaApiError( + message=f"log_tail_lines must be >= 0, got {log_tail_lines}.", + status_code=0, + error_code="INVALID_ARGUMENT", + ) + projects = self.resolve_projects([alias]) project = projects[alias] @@ -213,7 +409,26 @@ def run_job( job_id = str(job.get("id", "")) if wait and job_id: - job = client.wait_for_queue_job(job_id, max_wait=timeout) + try: + job = client.wait_for_queue_job( + job_id, + max_wait=timeout, + poll_strategy=poll_strategy, + ) + except KeboolaApiError as exc: + job = self._handle_wait_error( + client=client, + job_id=job_id, + exc=exc, + log_tail_lines=log_tail_lines, + timeout=timeout, + ) + else: + # Successful poll: still attach log tail for warning / + # terminated statuses (not raised; just surfaced). + status = str(job.get("status") or "") + if status in _LOG_TAIL_STATUSES and log_tail_lines > 0: + job["logTail"] = _safe_fetch_log_tail(client, job, log_tail_lines) finally: client.close() @@ -222,6 +437,84 @@ def run_job( job["resolvedVariableValuesId"] = resolved_values_id return job + def _handle_wait_error( + self, + client: Any, + job_id: str, + exc: KeboolaApiError, + log_tail_lines: int, + timeout: float, + ) -> dict[str, Any]: + """Handle a terminal wait_for_queue_job error. + + Two error classes land here: + + - ``QUEUE_JOB_FAILED`` -- remote job reached ``error`` status on its + own. Fetch the log tail so the caller sees the last events and + re-raise with the events attached to ``exc.details``. + - ``QUEUE_JOB_TIMEOUT`` -- local deadline exceeded before the job + finished. Issue ``kill_job`` to cancel the remote work, wait + briefly for termination, fetch the log tail and re-raise with a + distinct ``JOB_TIMEOUT_TERMINATED`` code so shells can tell + "we cancelled the job" from "job failed on its own". + + We never swallow the error; on any failure path we re-raise so the + command layer maps to the correct exit code. + """ + code = exc.error_code + if code == "QUEUE_JOB_FAILED": + # GET the job to resolve runId (required for Storage events) + # and to enrich the error details with the failed job payload. + try: + failed_job = client.get_job_detail(job_id) + except Exception: + failed_job = {"id": job_id} + tail = _safe_fetch_log_tail(client, failed_job, log_tail_lines) + details = _enrich_with_tail(exc.details, tail) + raise KeboolaApiError( + message=exc.message, + status_code=exc.status_code, + error_code=exc.error_code, + retryable=exc.retryable, + details=details or {}, + ) from exc + + if code == "QUEUE_JOB_TIMEOUT": + cancelled = _terminate_and_wait( + client, job_id, grace_seconds=JOB_TERMINATE_GRACE_SECONDS + ) + tail = _safe_fetch_log_tail(client, cancelled or {"id": job_id}, log_tail_lines) + if cancelled is not None: + raise KeboolaApiError( + message=( + f"Queue job {job_id} exceeded the local --timeout " + f"({timeout:.0f}s); issued kill and observed " + f"status={cancelled.get('status')!r}." + ), + status_code=504, + error_code="JOB_TIMEOUT_TERMINATED", + retryable=False, + details={"job": cancelled, "logTail": tail}, + ) from exc + # kill failed: surface the original timeout but keep the tail. + details = _enrich_with_tail(exc.details, tail) + raise KeboolaApiError( + message=exc.message, + status_code=exc.status_code, + error_code=exc.error_code, + retryable=exc.retryable, + details=details or {}, + ) from exc + + # Anything else (network, auth, a future Queue error code we + # don't specialise) bubbles up unchanged so the command layer + # maps it through map_error_to_exit_code(). + logger.debug( + "run_job: unhandled wait error code=%s; bubbling up unchanged", + code, + ) + raise + @staticmethod def resolve_variable_values_id( client: Any, diff --git a/tests/test_cli.py b/tests/test_cli.py index 0f4eab6f..6d2629c1 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -3115,6 +3115,206 @@ def test_job_run_strips_whitespace_around_variable_values_id(self, tmp_path: Pat assert job_service.run_job.call_args.kwargs["variable_values_id"] == "row-trimmed" +class TestJobRunQueuePollingFlags: + """PR4: --poll-strategy, --log-tail-lines, JOB_TIMEOUT_TERMINATED exit.""" + + def _setup(self, tmp_path: Path): + config_dir = tmp_path / "config" + config_dir.mkdir() + store = _setup_config_test( + config_dir, + {"prod": {"token": "901-10493007-VDtlEDWDF6Tx5V8jjE8FshFlqM0Hl0c08KHqpt0k"}}, + ) + return store + + def _invoke_job_run(self, store, args, run_job_return=None, run_job_side_effect=None): + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.ConfigService") as MockCfgService, + patch("keboola_agent_cli.cli.JobService") as MockJobService, + ): + MockStore.return_value = store + job_service = MagicMock() + if run_job_side_effect is not None: + job_service.run_job.side_effect = run_job_side_effect + else: + job_service.run_job.return_value = run_job_return or { + "id": 800, + "status": "success", + "isFinished": True, + } + MockJobService.return_value = job_service + MockProjService.return_value = ProjectService(config_store=store) + MockCfgService.return_value = ConfigService(config_store=store) + + result = runner.invoke(app, args) + return result, job_service + + def test_poll_strategy_forwarded(self, tmp_path: Path) -> None: + store = self._setup(tmp_path) + result, job_service = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--wait", + "--poll-strategy", + "fixed", + ], + ) + assert result.exit_code == 0, result.output + assert job_service.run_job.call_args.kwargs["poll_strategy"] == "fixed" + + def test_log_tail_lines_forwarded(self, tmp_path: Path) -> None: + store = self._setup(tmp_path) + result, job_service = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--wait", + "--log-tail-lines", + "42", + ], + ) + assert result.exit_code == 0, result.output + assert job_service.run_job.call_args.kwargs["log_tail_lines"] == 42 + + def test_poll_strategy_invalid_rejected_by_click(self, tmp_path: Path) -> None: + store = self._setup(tmp_path) + result, job_service = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--poll-strategy", + "linear", + ], + ) + # Click.Choice rejects before the command body runs. + assert result.exit_code == 2 + job_service.run_job.assert_not_called() + + def test_log_tail_lines_out_of_range_rejected(self, tmp_path: Path) -> None: + store = self._setup(tmp_path) + result, job_service = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--log-tail-lines", + "99999", + ], + ) + assert result.exit_code == 2 + assert "log-tail-lines" in result.output.lower() or "INVALID_ARGUMENT" in result.output + job_service.run_job.assert_not_called() + + def test_timeout_terminated_exits_seven_with_details(self, tmp_path: Path) -> None: + """JOB_TIMEOUT_TERMINATED -> exit 7 + details.logTail + details.job in JSON.""" + import json as _json + + from keboola_agent_cli.errors import KeboolaApiError + + store = self._setup(tmp_path) + result, _ = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--wait", + "--timeout", + "5", + ], + run_job_side_effect=KeboolaApiError( + message="timed out; issued kill", + status_code=504, + error_code="JOB_TIMEOUT_TERMINATED", + details={ + "job": {"id": 900, "status": "terminated", "isFinished": True}, + "logTail": [{"id": 1, "message": "x"}], + }, + ), + ) + assert result.exit_code == 7, result.output + envelope = _json.loads(result.output) + assert envelope["status"] == "error" + assert envelope["error"]["code"] == "JOB_TIMEOUT_TERMINATED" + assert envelope["error"]["details"]["job"]["status"] == "terminated" + assert envelope["error"]["details"]["logTail"] == [{"id": 1, "message": "x"}] + + def test_queue_job_timeout_exits_four(self, tmp_path: Path) -> None: + """Soft gave-up (kill also failed) -> exit 4, retryable=True.""" + import json as _json + + from keboola_agent_cli.errors import KeboolaApiError + + store = self._setup(tmp_path) + result, _ = self._invoke_job_run( + store, + [ + "--json", + "job", + "run", + "--project", + "prod", + "--component-id", + "keboola.ex-http", + "--config-id", + "42", + "--wait", + "--timeout", + "5", + ], + run_job_side_effect=KeboolaApiError( + message="did not complete within 5s", + status_code=504, + error_code="QUEUE_JOB_TIMEOUT", + retryable=True, + ), + ) + assert result.exit_code == 4 + envelope = _json.loads(result.output) + assert envelope["error"]["code"] == "QUEUE_JOB_TIMEOUT" + assert envelope["error"]["retryable"] is True + + class TestJobTerminate: """Tests for `kbagent job terminate` command.""" diff --git a/tests/test_client.py b/tests/test_client.py index 87446e82..f2d24745 100644 --- a/tests/test_client.py +++ b/tests/test_client.py @@ -1,5 +1,6 @@ """Tests for KeboolaClient - verify_token, retries, timeouts, error handling.""" +import contextlib import json from unittest.mock import patch from urllib.parse import parse_qs, quote @@ -2534,6 +2535,237 @@ def test_kill_job_url_encodes_job_id(self, httpx_mock) -> None: client.kill_job("ab/cd") +class TestIterPollIntervals: + """Tests for _iter_poll_intervals -- curve math and strategy dispatch.""" + + def test_exponential_curve_matches_constant(self) -> None: + """First 30 yields == 2.0, next 48 == 5.0, then infinite 15.0.""" + from itertools import islice + + from keboola_agent_cli.client import _iter_poll_intervals + + seq = list(islice(_iter_poll_intervals("exponential"), 30 + 48 + 10)) + assert seq[:30] == [2.0] * 30 + assert seq[30:78] == [5.0] * 48 + assert seq[78:] == [15.0] * 10 + + def test_fixed_yields_storage_interval_forever(self) -> None: + """'fixed' yields STORAGE_JOB_POLL_INTERVAL indefinitely.""" + from itertools import islice + + from keboola_agent_cli.client import _iter_poll_intervals + from keboola_agent_cli.constants import STORAGE_JOB_POLL_INTERVAL + + seq = list(islice(_iter_poll_intervals("fixed"), 5)) + assert seq == [STORAGE_JOB_POLL_INTERVAL] * 5 + + +class TestWaitForQueueJob: + """Tests for wait_for_queue_job -- strategy dispatch, deadline, failure.""" + + def _mk_client(self): + return KeboolaClient(stack_url=_BASE, token=_TOKEN) + + def test_wait_success_on_first_poll(self, httpx_mock, monkeypatch) -> None: + """Finished job returns on the first poll; no sleep needed.""" + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/job-1", + method="GET", + json={"id": "job-1", "status": "success", "isFinished": True}, + ) + sleeps: list[float] = [] + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: sleeps.append(s)) + + with self._mk_client() as client: + job = client.wait_for_queue_job("job-1", max_wait=60.0) + + assert job["status"] == "success" + assert sleeps == [] + + def test_wait_honors_exponential_intervals(self, httpx_mock, monkeypatch) -> None: + """Two unfinished polls then finished -- sleep args match the curve head.""" + for _ in range(2): + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/job-2", + method="GET", + json={"id": "job-2", "status": "processing", "isFinished": False}, + ) + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/job-2", + method="GET", + json={"id": "job-2", "status": "success", "isFinished": True}, + ) + sleeps: list[float] = [] + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: sleeps.append(s)) + + with self._mk_client() as client: + client.wait_for_queue_job("job-2", max_wait=600.0, poll_strategy="exponential") + + # Two polls -> two sleeps; both at the 2s phase of the curve. + assert sleeps == [2.0, 2.0] + + def test_wait_honors_fixed_strategy(self, httpx_mock, monkeypatch) -> None: + """poll_strategy='fixed' sleeps STORAGE_JOB_POLL_INTERVAL between polls.""" + from keboola_agent_cli.constants import STORAGE_JOB_POLL_INTERVAL + + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/job-3", + method="GET", + json={"id": "job-3", "status": "processing", "isFinished": False}, + ) + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/job-3", + method="GET", + json={"id": "job-3", "status": "success", "isFinished": True}, + ) + sleeps: list[float] = [] + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: sleeps.append(s)) + + with self._mk_client() as client: + client.wait_for_queue_job("job-3", max_wait=600.0, poll_strategy="fixed") + + assert sleeps == [STORAGE_JOB_POLL_INTERVAL] + + def test_wait_rejects_unknown_strategy(self) -> None: + """Invalid strategy raises ValueError before any network call.""" + with ( + self._mk_client() as client, + pytest.raises(ValueError, match="Invalid poll_strategy"), + ): + client.wait_for_queue_job("job-4", poll_strategy="linear") + + def test_wait_raises_on_status_error(self, httpx_mock, monkeypatch) -> None: + """status='error' produces QUEUE_JOB_FAILED with the job's error message.""" + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/bad-1", + method="GET", + json={ + "id": "bad-1", + "status": "error", + "isFinished": True, + "result": {"message": "SQL compilation failed"}, + }, + ) + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: None) + + with self._mk_client() as client, pytest.raises(KeboolaApiError) as exc_info: + client.wait_for_queue_job("bad-1", max_wait=60.0) + + assert exc_info.value.error_code == "QUEUE_JOB_FAILED" + assert "SQL compilation failed" in exc_info.value.message + + def test_wait_raises_timeout_on_deadline(self, httpx_mock, monkeypatch) -> None: + """Deadline exceeded raises QUEUE_JOB_TIMEOUT.""" + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/slow-1", + method="GET", + json={"id": "slow-1", "status": "processing", "isFinished": False}, + ) + + # Use a stepped clock that stays at the final value if production + # calls monotonic() more times than we expected. This makes the + # test robust to refactors that add observability calls without + # changing the behaviour under test. + values = iter([0.0, 6.0]) + last = [0.0] + + def fake_monotonic() -> float: + with contextlib.suppress(StopIteration): + last[0] = next(values) + return last[0] + + monkeypatch.setattr("keboola_agent_cli.client.time.monotonic", fake_monotonic) + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: None) + + with self._mk_client() as client, pytest.raises(KeboolaApiError) as exc_info: + client.wait_for_queue_job("slow-1", max_wait=5.0) + + assert exc_info.value.error_code == "QUEUE_JOB_TIMEOUT" + assert exc_info.value.status_code == 504 + + def test_wait_caps_sleep_to_remaining_deadline(self, httpx_mock, monkeypatch) -> None: + """Last sleep before deadline is trimmed so we don't overshoot by one interval.""" + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/deadline-1", + method="GET", + json={"id": "deadline-1", "status": "processing", "isFinished": False}, + ) + httpx_mock.add_response( + url="https://queue.keboola.com/jobs/deadline-1", + method="GET", + json={"id": "deadline-1", "status": "processing", "isFinished": False}, + ) + + # First monotonic() sets deadline at 100.0; the second (after the + # first get) returns 99.0 so remaining=1.0 < interval=2.0; the third + # (after sleep) returns 101.0 so loop exits. Clamp on exhaustion + # so a refactor adding observability calls doesn't crash the test. + times = iter([0.0, 99.0, 101.0]) + last = [0.0] + + def fake_monotonic() -> float: + with contextlib.suppress(StopIteration): + last[0] = next(times) + return last[0] + + monkeypatch.setattr("keboola_agent_cli.client.time.monotonic", fake_monotonic) + sleeps: list[float] = [] + monkeypatch.setattr("keboola_agent_cli.client.time.sleep", lambda s: sleeps.append(s)) + + with self._mk_client() as client, pytest.raises(KeboolaApiError): + client.wait_for_queue_job("deadline-1", max_wait=100.0) + + assert sleeps == [1.0] # trimmed from 2.0 to 1.0 + + +class TestFetchJobEvents: + """Tests for fetch_job_events -- runId-based Storage Events endpoint.""" + + def test_events_list_payload(self, httpx_mock) -> None: + payload = [ + {"uuid": "u1", "type": "info", "message": "starting", "runId": "j-1"}, + {"uuid": "u2", "type": "error", "message": "boom", "runId": "j-1"}, + ] + httpx_mock.add_response( + url="https://connection.keboola.com/v2/storage/events?runId=j-1", + method="GET", + json=payload, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + events = client.fetch_job_events("j-1") + assert events == payload + + def test_events_limit_query_param(self, httpx_mock) -> None: + httpx_mock.add_response( + url="https://connection.keboola.com/v2/storage/events?runId=j-2&limit=50", + method="GET", + json=[], + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + client.fetch_job_events("j-2", limit=50) + + def test_events_dict_wrapped_payload(self, httpx_mock) -> None: + """Tolerant of a future dict shape {events: [...]}""" + httpx_mock.add_response( + url="https://connection.keboola.com/v2/storage/events?runId=j-3", + method="GET", + json={"events": [{"uuid": "u1"}], "total": 1}, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + events = client.fetch_job_events("j-3") + assert events == [{"uuid": "u1"}] + + def test_events_unexpected_payload_returns_empty(self, httpx_mock) -> None: + httpx_mock.add_response( + url="https://connection.keboola.com/v2/storage/events?runId=j-4", + method="GET", + json={"unexpected": "shape"}, + ) + with KeboolaClient(stack_url=_BASE, token=_TOKEN) as client: + events = client.fetch_job_events("j-4") + assert events == [] + + _BRANCH_METADATA_SAMPLE = [ { "id": 1001, diff --git a/tests/test_e2e.py b/tests/test_e2e.py index cb2d206b..5ded11b4 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -4456,3 +4456,296 @@ def test_workspace_gc_orphan_roundtrip(self) -> None: data = self._run_ok("workspace", "list", "--project", self.alias, "--orphaned") remaining_ids = [w["id"] for w in data["data"]["workspaces"]] assert ws_id not in remaining_ids + + +# --------------------------------------------------------------------------- +# Queue polling parity (PR4 / P0-3): exponential curve, log tail, timeout kill +# --------------------------------------------------------------------------- + + +@skip_without_credentials +@pytest.mark.e2e +class TestE2EJobRunQueuePollingParity: + """Live verification of the PR4 Queue API polling contract. + + Three scenarios, each spawns exactly one config, cleans it up: + + - **log tail on a failed job**: a snowflake-transformation with + deliberately invalid SQL runs to `status=error`; we assert the + returned JSON error envelope contains a non-empty + `details.logTail` sourced from ``GET /jobs/{id}/events``. + - **timeout triggers remote kill**: a python-transformation-v2 with + `time.sleep(120)` is invoked with `--timeout 8`; we assert the + command exits 7 (``EXIT_JOB_TIMEOUT_TERMINATED``) with + `error.code == "JOB_TIMEOUT_TERMINATED"` and `details.job.status` + in {terminated, cancelled, terminating} -- i.e. the kill landed. + - **fixed strategy still reaches completion**: a no-op transformation + runs under ``--poll-strategy fixed`` to prove the opt-out path works + against a real Queue. + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path, request: pytest.FixtureRequest) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + # Per-test alias suffix so parallel pytest-xdist runs don't share a + # project alias across workers. `request.node.name` is stable per test + # and includes any parametrize id. + safe = request.node.name.replace("[", "-").replace("]", "") + self.alias = f"{RUN_ID}-queuepoll-{safe}"[:60] + + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + + result = _invoke( + self.config_dir, + [ + "--json", + "project", + "add", + "--project", + self.alias, + "--url", + self.url, + "--token", + self.token, + ], + ) + assert result.exit_code == 0, f"project add failed: {result.output}" + + self.client = KeboolaClient(stack_url=self.url, token=self.token) + self._created: list[tuple[str, str]] = [] + self._submitted_jobs: list[str] = [] + + yield + + # Best-effort cleanup: kill any jobs we left running, then delete + # the configs. We never want a test failure here to mask the real + # assertion failure. + import contextlib + + for job_id in self._submitted_jobs: + with contextlib.suppress(Exception): + self.client.kill_job(job_id) + + for component_id, config_id in reversed(self._created): + try: + self.client.delete_config(component_id=component_id, config_id=config_id) + except Exception as exc: + print( + f" {_DIM}(teardown) delete_config {component_id}/{config_id} " + f"failed: {exc}{_RESET}" + ) + self.client.close() + + def _create_sleep_config(self, seconds: int, suffix: str) -> str: + """Create a python-transformation-v2 config that sleeps and returns its id.""" + cfg = self.client.create_config( + component_id="keboola.python-transformation-v2", + name=f"{RUN_ID}-queuepoll-{suffix}", + description=f"E2E PR4: sleeps {seconds}s -- used only for polling tests", + configuration={ + "parameters": { + "blocks": [ + { + "name": "Block 1", + "codes": [ + { + "name": "sleep", + "script": [ + "import time", + f"time.sleep({seconds})", + ], + } + ], + } + ] + } + }, + ) + cfg_id = str(cfg["id"]) + self._created.append(("keboola.python-transformation-v2", cfg_id)) + return cfg_id + + def _create_guaranteed_fail_config(self, suffix: str) -> str: + """Python transformation that raises an exception so job status=error. + + Why python not Snowflake: a Snowflake transformation with no + input/output tables registered is treated as a successful no-op + even if the SQL would be invalid at execute time. A Python + transformation with an unconditional ``raise`` surfaces as + ``status=error`` with a clear message on the event feed, which + is what the log-tail assertion needs. + """ + cfg = self.client.create_config( + component_id="keboola.python-transformation-v2", + name=f"{RUN_ID}-queuepoll-bad-{suffix}", + description="E2E PR4: guaranteed-fail python transformation", + configuration={ + "parameters": { + "blocks": [ + { + "name": "Block 1", + "codes": [ + { + "name": "boom", + "script": [ + "raise RuntimeError('kbagent E2E PR4 deliberate failure')", + ], + } + ], + } + ] + } + }, + ) + cfg_id = str(cfg["id"]) + self._created.append(("keboola.python-transformation-v2", cfg_id)) + return cfg_id + + def test_log_tail_surfaced_on_queue_job_failed(self) -> None: + """Failed Queue job -> error envelope with details.logTail from /events.""" + _step(1, "create python-transformation-v2 that raises") + cfg_id = self._create_guaranteed_fail_config("tail") + + _step(2, "kbagent --json job run --wait (expect QUEUE_JOB_FAILED)") + result = _invoke( + self.config_dir, + [ + "--json", + "job", + "run", + "--project", + self.alias, + "--component-id", + "keboola.python-transformation-v2", + "--config-id", + cfg_id, + "--wait", + "--timeout", + "300", + "--log-tail-lines", + "50", + "--no-variables", + ], + ) + + # Deliberate failure: exit non-zero, envelope status=error, + # details.logTail is a non-empty list. + assert result.exit_code != 0, f"Expected failure, got success: {result.output}" + envelope = json.loads(result.output) + assert envelope["status"] == "error" + assert envelope["error"]["code"] == "QUEUE_JOB_FAILED" + details = envelope["error"].get("details") or {} + log_tail = details.get("logTail") or [] + assert isinstance(log_tail, list) and len(log_tail) > 0, ( + f"Expected non-empty logTail, got {log_tail!r} in {envelope!r}" + ) + + def test_timeout_triggers_remote_kill_and_exits_seven(self) -> None: + """--timeout N < job runtime -> exit 7 + kill landed remotely.""" + _step(1, "create python-transformation-v2 that sleeps 120s") + cfg_id = self._create_sleep_config(seconds=120, suffix="kill") + + _step(2, "kbagent --json job run --wait --timeout 8 (expect exit 7)") + result = _invoke( + self.config_dir, + [ + "--json", + "job", + "run", + "--project", + self.alias, + "--component-id", + "keboola.python-transformation-v2", + "--config-id", + cfg_id, + "--wait", + "--timeout", + "8", + "--log-tail-lines", + "20", + "--no-variables", + ], + ) + + assert result.exit_code == 7, ( + f"Expected exit 7 (JOB_TIMEOUT_TERMINATED), got {result.exit_code}\n{result.output}" + ) + envelope = json.loads(result.output) + assert envelope["status"] == "error" + assert envelope["error"]["code"] == "JOB_TIMEOUT_TERMINATED" + + job = envelope["error"]["details"]["job"] + assert job["status"] in {"terminated", "cancelled", "terminating"}, ( + f"Expected terminated/cancelled/terminating; got {job['status']!r}" + ) + # Track job_id so teardown's kill-if-needed covers the 'terminating' + # transitional case where the remote hasn't settled yet. + self._submitted_jobs.append(str(job["id"])) + + def test_fixed_poll_strategy_reaches_completion(self) -> None: + """--poll-strategy fixed still completes against a real Queue.""" + _step(1, "create trivial python-transformation-v2 (sleep 3s)") + cfg_id = self._create_sleep_config(seconds=3, suffix="fixed") + + _step(2, "kbagent job run --wait --poll-strategy fixed") + result = _invoke( + self.config_dir, + [ + "--json", + "job", + "run", + "--project", + self.alias, + "--component-id", + "keboola.python-transformation-v2", + "--config-id", + cfg_id, + "--wait", + "--timeout", + "120", + "--poll-strategy", + "fixed", + "--no-variables", + ], + ) + + assert result.exit_code == 0, f"Expected success, got: {result.output}" + payload = _json(result)["data"] + assert payload["status"] == "success" + assert payload.get("isFinished") is True + + def test_fetch_job_events_returns_list_on_real_job(self) -> None: + """Direct client call against /jobs/{id}/events on a finished job.""" + _step(1, "create trivial successful job + run to completion") + cfg_id = self._create_sleep_config(seconds=1, suffix="events") + + result = _invoke( + self.config_dir, + [ + "--json", + "job", + "run", + "--project", + self.alias, + "--component-id", + "keboola.python-transformation-v2", + "--config-id", + cfg_id, + "--wait", + "--timeout", + "120", + "--no-variables", + ], + ) + assert result.exit_code == 0, result.output + job_id = str(_json(result)["data"]["id"]) + + _step(2, "client.fetch_job_events direct call") + events = self.client.fetch_job_events(job_id, limit=50) + assert isinstance(events, list) + # A completed python-transformation job always emits at least one event + # (startup + completion). Guard against an empty-but-silent regression. + assert len(events) > 0 diff --git a/tests/test_services.py b/tests/test_services.py index abc310a0..1ca25fb5 100644 --- a/tests/test_services.py +++ b/tests/test_services.py @@ -2037,7 +2037,411 @@ def test_run_job_with_branch_and_wait(self, tmp_config_dir: Path) -> None: branch_id=123, variable_values_id=None, ) - mock_client.wait_for_queue_job.assert_called_once_with("557", max_wait=60.0) + mock_client.wait_for_queue_job.assert_called_once_with( + "557", max_wait=60.0, poll_strategy="exponential" + ) + + +class TestJobServiceQueuePollingParity: + """PR4: log-tail capture + auto-cancel on --timeout + poll-strategy plumbing.""" + + def _store(self, tmp_config_dir: Path) -> ConfigStore: + store = ConfigStore(config_dir=tmp_config_dir) + store.add_project( + "prod", + ProjectConfig( + stack_url="https://connection.keboola.com", + token="901-abc-defghijklmnopqrst", + project_name="Prod", + project_id=1234, + ), + ) + return store + + def _service(self, store: ConfigStore, mock_client: MagicMock) -> JobService: + return JobService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + def test_run_job_threads_poll_strategy_to_client(self, tmp_config_dir: Path) -> None: + """poll_strategy kwarg reaches wait_for_queue_job unchanged.""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 700, "status": "waiting"} + mock_client.wait_for_queue_job.return_value = { + "id": 700, + "status": "success", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + + service = self._service(self._store(tmp_config_dir), mock_client) + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + timeout=60.0, + poll_strategy="fixed", + no_variables=True, + ) + + mock_client.wait_for_queue_job.assert_called_once_with( + "700", max_wait=60.0, poll_strategy="fixed" + ) + + def test_run_job_rejects_unknown_strategy(self, tmp_config_dir: Path) -> None: + """Bad poll_strategy fails at service boundary, not at client.""" + service = self._service(self._store(tmp_config_dir), MagicMock()) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + poll_strategy="linear", + no_variables=True, + ) + assert exc_info.value.error_code == "INVALID_ARGUMENT" + + def test_run_job_rejects_negative_log_tail(self, tmp_config_dir: Path) -> None: + service = self._service(self._store(tmp_config_dir), MagicMock()) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + log_tail_lines=-1, + no_variables=True, + ) + assert exc_info.value.error_code == "INVALID_ARGUMENT" + + def test_run_job_success_no_tail_attached(self, tmp_config_dir: Path) -> None: + """status=success does NOT fetch events (log tail only for non-success).""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 701, "status": "waiting"} + mock_client.wait_for_queue_job.return_value = { + "id": 701, + "status": "success", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + + service = self._service(self._store(tmp_config_dir), mock_client) + result = service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + no_variables=True, + ) + assert "logTail" not in result + mock_client.fetch_job_events.assert_not_called() + + def test_run_job_warning_attaches_log_tail(self, tmp_config_dir: Path) -> None: + """status=warning surfaces a logTail of the first N events (newest-first).""" + # Storage Events returns newest -> oldest; emulate that ordering so + # the slice asserts the client didn't accidentally reverse it. + events = [{"id": 249 - i, "message": f"event {249 - i}"} for i in range(250)] + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 702, "status": "waiting"} + mock_client.wait_for_queue_job.return_value = { + "id": 702, + "runId": "702-run", + "status": "warning", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + mock_client.fetch_job_events.return_value = events + + service = self._service(self._store(tmp_config_dir), mock_client) + result = service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + log_tail_lines=100, + no_variables=True, + ) + # Storage Events yields newest-first; we take [:100] so the head + # stays newest. IDs 249 -> 150. + assert len(result["logTail"]) == 100 + assert result["logTail"][0]["id"] == 249 + assert result["logTail"][-1]["id"] == 150 + # runId (not raw id) must have been the query key. + mock_client.fetch_job_events.assert_called_once_with("702-run", limit=100) + + def test_run_job_zero_tail_skips_fetch(self, tmp_config_dir: Path) -> None: + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 703, "status": "waiting"} + mock_client.wait_for_queue_job.return_value = { + "id": 703, + "status": "terminated", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + + service = self._service(self._store(tmp_config_dir), mock_client) + result = service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + log_tail_lines=0, + no_variables=True, + ) + mock_client.fetch_job_events.assert_not_called() + assert "logTail" not in result + + def test_run_job_queue_failure_attaches_tail_and_reraises(self, tmp_config_dir: Path) -> None: + """QUEUE_JOB_FAILED re-raises with logTail tucked into exc.details.""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 704, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = KeboolaApiError( + message="Queue job 704 failed: SQL error", + status_code=500, + error_code="QUEUE_JOB_FAILED", + ) + # The service fetches job detail on failure to resolve runId. + mock_client.get_job_detail.return_value = { + "id": "704", + "runId": "704", + "status": "error", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + mock_client.fetch_job_events.return_value = [ + {"uuid": "u1", "type": "error", "message": "SQL error"}, + ] + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + no_variables=True, + ) + assert exc_info.value.error_code == "QUEUE_JOB_FAILED" + assert exc_info.value.details["logTail"][0]["message"] == "SQL error" + # kill_job must NOT be called when the job failed on its own. + mock_client.kill_job.assert_not_called() + # runId used as the query key, not the Queue job id. + mock_client.fetch_job_events.assert_called_once_with("704", limit=200) + + def test_run_job_timeout_issues_kill_and_raises_terminated(self, tmp_config_dir: Path) -> None: + """QUEUE_JOB_TIMEOUT -> kill_job + JOB_TIMEOUT_TERMINATED with job payload.""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 705, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = KeboolaApiError( + message="Queue job 705 did not complete within 5s", + status_code=504, + error_code="QUEUE_JOB_TIMEOUT", + ) + mock_client.kill_job.return_value = { + "id": 705, + "status": "terminating", + "desiredStatus": "terminating", + } + mock_client.get_job_detail.return_value = { + "id": 705, + "runId": "705-run", + "status": "terminated", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + mock_client.fetch_job_events.return_value = [{"uuid": "u1", "message": "x"}] + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + timeout=5.0, + no_variables=True, + ) + assert exc_info.value.error_code == "JOB_TIMEOUT_TERMINATED" + mock_client.kill_job.assert_called_once_with("705") + details = exc_info.value.details + assert details["job"]["status"] == "terminated" + assert details["logTail"] == [{"uuid": "u1", "message": "x"}] + # runId from the terminated job detail used as the lookup key. + mock_client.fetch_job_events.assert_called_once_with("705-run", limit=200) + + def test_run_job_timeout_kill_fails_falls_back(self, tmp_config_dir: Path) -> None: + """If kill_job AND the follow-up GET fail, surface QUEUE_JOB_TIMEOUT (retryable).""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 706, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = KeboolaApiError( + message="Queue job 706 did not complete within 5s", + status_code=504, + error_code="QUEUE_JOB_TIMEOUT", + retryable=True, + ) + mock_client.kill_job.side_effect = KeboolaApiError( + message="network down", + status_code=0, + error_code="CONNECTION_ERROR", + ) + mock_client.get_job_detail.side_effect = KeboolaApiError( + message="still down", + status_code=0, + error_code="CONNECTION_ERROR", + ) + mock_client.get_config_detail.return_value = {} + mock_client.fetch_job_events.return_value = [] + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + timeout=5.0, + no_variables=True, + ) + assert exc_info.value.error_code == "QUEUE_JOB_TIMEOUT" + assert exc_info.value.retryable is True + + def test_run_job_log_tail_fetch_failure_is_swallowed(self, tmp_config_dir: Path) -> None: + """A failing fetch_job_events must not mask the original job failure.""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 707, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = KeboolaApiError( + message="Queue job 707 failed: SQL error", + status_code=500, + error_code="QUEUE_JOB_FAILED", + ) + mock_client.fetch_job_events.side_effect = KeboolaApiError( + message="events 500", + status_code=500, + error_code="UNKNOWN_ERROR", + ) + mock_client.get_config_detail.return_value = {} + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + no_variables=True, + ) + # Original error preserved; details has no logTail key when fetch fails. + assert exc_info.value.error_code == "QUEUE_JOB_FAILED" + assert "logTail" not in exc_info.value.details + + def test_run_job_unhandled_wait_code_bubbles_up_unchanged(self, tmp_config_dir: Path) -> None: + """A wait error that is neither QUEUE_JOB_FAILED nor QUEUE_JOB_TIMEOUT + must re-raise the original instance with no mutation and no kill attempt. + Locks the observability fall-through path added in the review loop.""" + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 708, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = KeboolaApiError( + message="token rotated mid-run", + status_code=401, + error_code="INVALID_TOKEN", + ) + mock_client.get_config_detail.return_value = {} + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + no_variables=True, + ) + assert exc_info.value.error_code == "INVALID_TOKEN" + # No tail fetch, no kill -- this path is for errors we don't specialise. + mock_client.fetch_job_events.assert_not_called() + mock_client.kill_job.assert_not_called() + # exc passes through without a logTail (mutation would be a bug). + assert "logTail" not in exc_info.value.details + + def test_run_job_failure_exception_chaining_does_not_mutate_original( + self, tmp_config_dir: Path + ) -> None: + """QUEUE_JOB_FAILED produces a NEW exception chained from the original. + + Guarantees we do not mutate a caught exception's .details dict + (which would contaminate any shared instance / retry harness). + """ + original_details: dict = {} + original = KeboolaApiError( + message="Queue job 709 failed: SQL error", + status_code=500, + error_code="QUEUE_JOB_FAILED", + details=original_details, + ) + mock_client = MagicMock() + mock_client.create_job.return_value = {"id": 709, "status": "waiting"} + mock_client.wait_for_queue_job.side_effect = original + mock_client.get_job_detail.return_value = { + "id": "709", + "runId": "709", + "status": "error", + "isFinished": True, + } + mock_client.get_config_detail.return_value = {} + mock_client.fetch_job_events.return_value = [{"uuid": "u1", "message": "err"}] + + service = self._service(self._store(tmp_config_dir), mock_client) + with pytest.raises(KeboolaApiError) as exc_info: + service.run_job( + alias="prod", + component_id="keboola.ex-http", + config_id="42", + wait=True, + no_variables=True, + ) + raised = exc_info.value + # Raised instance is NEW, not the one side_effect handed us. + assert raised is not original + # Chain is explicit (raise ... from original). + assert raised.__cause__ is original + # Original details dict was NOT mutated. + assert original_details == {} + assert "logTail" not in original.details + # But the new exception carries the tail. + assert raised.details["logTail"][0]["uuid"] == "u1" + + +class TestSafeFetchLogTailDefensiveSort: + """Ensure _safe_fetch_log_tail enforces newest-first ordering regardless of + what the API returned (PR4 review round 1 finding).""" + + def test_sorts_events_by_created_desc(self) -> None: + """API returns events in arbitrary order -> we still emit newest first.""" + from keboola_agent_cli.services.job_service import _safe_fetch_log_tail + + mock_client = MagicMock() + # Deliberately shuffled order from the "API" + mock_client.fetch_job_events.return_value = [ + {"uuid": "a", "created": "2026-04-22T09:54:27+0200", "message": "middle"}, + {"uuid": "b", "created": "2026-04-22T09:54:30+0200", "message": "newest"}, + {"uuid": "c", "created": "2026-04-22T09:54:10+0200", "message": "oldest"}, + ] + tail = _safe_fetch_log_tail(mock_client, {"id": "x", "runId": "x"}, limit=10) + assert [e["uuid"] for e in tail] == ["b", "a", "c"] + + def test_missing_created_sorts_last(self) -> None: + """Events without `created` should not jump to the top of the tail.""" + from keboola_agent_cli.services.job_service import _safe_fetch_log_tail + + mock_client = MagicMock() + mock_client.fetch_job_events.return_value = [ + {"uuid": "no_created", "message": "missing"}, + {"uuid": "timestamped", "created": "2026-04-22T09:54:30+0200", "message": "ok"}, + ] + tail = _safe_fetch_log_tail(mock_client, {"id": "x", "runId": "x"}, limit=10) + assert tail[0]["uuid"] == "timestamped" + assert tail[1]["uuid"] == "no_created" class TestJobServiceVariableValuesResolution: @@ -2268,7 +2672,9 @@ def test_run_job_wait_preserves_resolved_variable_values_id(self, tmp_config_dir assert result["status"] == "success" assert result["resolvedVariableValuesId"] == "row-waited" - mock_client.wait_for_queue_job.assert_called_once_with("750", max_wait=30.0) + mock_client.wait_for_queue_job.assert_called_once_with( + "750", max_wait=30.0, poll_strategy="exponential" + ) def test_run_job_closes_client_when_resolver_raises(self, tmp_config_dir: Path) -> None: """NO_VARIABLE_ROWS raised by the resolver inside run_job still closes the client. From b592d5f2ea2fc0439d7d8605ac75adbbc72bfd56 Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:15:50 +0200 Subject: [PATCH 6/7] feat(0.22.0): ErrorCode enum + sync init --adopt-existing (#201) P2-1: ErrorCode(StrEnum) in errors.py with 49 typed constants (46 original + 3 new: JOB_TIMEOUT_TERMINATED, INVALID_FLOW_DAG, SCHEDULE_DELETE_FAILED). All error_code='STRING' literals across 30+ source files replaced with ErrorCode.. Wire format unchanged (StrEnum subclasses str). P2-2: sync init --adopt-existing idempotently adopts a .keboola/manifest.json written by the kbc Go CLI without overwriting it. Validates manifest project_id against the alias token; rejects mismatch with ConfigError. scripts/check_error_codes.py CI guard rejects new raw literals. docs/error-codes.md documents semver policy. --- Makefile | 7 +- docs/error-codes.md | 129 +++++++++++++ scripts/check_error_codes.py | 78 ++++++++ src/keboola_agent_cli/changelog.py | 10 + src/keboola_agent_cli/cli.py | 4 +- src/keboola_agent_cli/client.py | 20 +- src/keboola_agent_cli/commands/_helpers.py | 8 +- src/keboola_agent_cli/commands/branch.py | 24 +-- src/keboola_agent_cli/commands/component.py | 8 +- src/keboola_agent_cli/commands/config.py | 56 +++--- src/keboola_agent_cli/commands/context.py | 2 +- src/keboola_agent_cli/commands/encrypt.py | 8 +- src/keboola_agent_cli/commands/flow.py | 26 +-- src/keboola_agent_cli/commands/init.py | 3 +- src/keboola_agent_cli/commands/job.py | 34 ++-- src/keboola_agent_cli/commands/kai.py | 10 +- src/keboola_agent_cli/commands/lineage.py | 29 +-- src/keboola_agent_cli/commands/org.py | 4 +- src/keboola_agent_cli/commands/permissions.py | 7 +- src/keboola_agent_cli/commands/project.py | 24 +-- src/keboola_agent_cli/commands/sharing.py | 18 +- src/keboola_agent_cli/commands/storage.py | 88 ++++----- src/keboola_agent_cli/commands/sync.py | 75 +++++--- src/keboola_agent_cli/commands/tool.py | 20 +- src/keboola_agent_cli/commands/workspace.py | 24 +-- src/keboola_agent_cli/errors.py | 111 ++++++++++- src/keboola_agent_cli/http_base.py | 16 +- src/keboola_agent_cli/services/_encryption.py | 4 +- .../services/branch_service.py | 4 +- .../services/config_service.py | 6 +- .../services/flow_service.py | 8 +- src/keboola_agent_cli/services/job_service.py | 14 +- src/keboola_agent_cli/services/kai_service.py | 12 +- .../services/sharing_service.py | 4 +- .../services/storage_service.py | 19 +- .../services/sync_service.py | 62 +++++- .../services/workspace_service.py | 4 +- src/keboola_agent_cli/sync/manifest.py | 4 +- tests/test_e2e.py | 147 +++++++++++++++ tests/test_errors.py | 68 ++++++- tests/test_integration.py | 69 +++++++ tests/test_sync_cli.py | 156 +++++++++++++++ tests/test_sync_service.py | 178 ++++++++++++++++++ 43 files changed, 1307 insertions(+), 295 deletions(-) create mode 100644 docs/error-codes.md create mode 100644 scripts/check_error_codes.py diff --git a/Makefile b/Makefile index 188af848..10c880da 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ .DEFAULT_GOAL := help -.PHONY: help install install-mcp sync test test-unit test-integration test-e2e test-file lint lint-fix format format-check skill-check skill-gen version-sync version-check changelog changelog-check check clean hooks +.PHONY: help install install-mcp sync test test-unit test-integration test-e2e test-file lint lint-fix format format-check skill-check skill-gen version-sync version-check changelog changelog-check check-error-codes check clean hooks help: ## Show this help message @grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-18s\033[0m %s\n", $$1, $$2}' @@ -73,12 +73,15 @@ changelog: ## Generate changelog skeleton from GitHub releases changelog-check: ## Check all releases have changelog entries uv run python scripts/generate_changelog.py --check +check-error-codes: ## Reject raw error_code string literals in source (use ErrorCode enum) + uv run python scripts/check_error_codes.py + hooks: ## Install git pre-commit hook (lint + format on staged files) cp scripts/pre-commit .git/hooks/pre-commit chmod +x .git/hooks/pre-commit @echo "Pre-commit hook installed." -check: lint format-check skill-check version-check changelog-check test ## Run all checks (lint + format + skill + version + changelog + test) +check: lint format-check skill-check version-check changelog-check check-error-codes test ## Run all checks (lint + format + skill + version + changelog + error-codes + test) clean: ## Remove build artifacts and caches find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true diff --git a/docs/error-codes.md b/docs/error-codes.md new file mode 100644 index 00000000..48c7e46f --- /dev/null +++ b/docs/error-codes.md @@ -0,0 +1,129 @@ +# kbagent Error Code Reference + +All machine-readable codes emitted via `--json` output. Every code is a member +of `ErrorCode` in `src/keboola_agent_cli/errors.py`. + +## Versioning + +| Change | Version impact | +|---|---| +| Add a new code | Minor bump | +| Rename or remove a code | Major bump | + +## Code catalogue + +### Auth / access + +| Code | Description | +|---|---| +| `INVALID_TOKEN` | Storage API token is invalid or expired | +| `ACCESS_DENIED` | Token lacks the required permission for this API call | +| `PERMISSION_DENIED` | Operation blocked by the active kbagent permission policy | + +### Network / transport + +| Code | Description | +|---|---| +| `TIMEOUT` | HTTP request timed out | +| `CONNECTION_ERROR` | TCP-level connection failure | +| `RETRY_EXHAUSTED` | All retry attempts failed (typically after 429/5xx) | + +### API / generic + +| Code | Description | +|---|---| +| `API_ERROR` | Unexpected HTTP error from the Keboola API | +| `NOT_FOUND` | Requested resource does not exist (404) | +| `ALREADY_EXISTS` | Resource or file already exists and was not overwritten | +| `VALIDATION_ERROR` | Request failed API-side validation | +| `INVALID_ARGUMENT` | Caller supplied an invalid argument value | +| `INVALID_FORMAT` | Input is not in the expected format | +| `USAGE_ERROR` | Incorrect CLI flag combination or missing required argument | +| `MISSING_PARAMETER` | A required parameter was not supplied | +| `UNKNOWN_ERROR` | Catch-all for unclassified errors | + +### Configuration + +| Code | Description | +|---|---| +| `CONFIG_ERROR` | kbagent config problem (e.g. unknown project alias) | +| `NOT_INITIALIZED` | `.keboola/manifest.json` not found; run `sync init` first | +| `INIT_ERROR` | Error during `sync init` auto-init path | + +### Jobs + +| Code | Description | +|---|---| +| `QUEUE_JOB_FAILED` | Queue API job finished with status `error` or `warning` | +| `QUEUE_JOB_TIMEOUT` | Polling timed out waiting for a Queue job | +| `STORAGE_JOB_FAILED` | Storage API async job finished in a failed state | +| `STORAGE_JOB_TIMEOUT` | Polling timed out waiting for a Storage async job | +| `QUERY_JOB_FAILED` | Query Service job finished in a failed state | +| `QUERY_JOB_TIMEOUT` | Polling timed out waiting for a Query Service job | + +### Variables + +| Code | Description | +|---|---| +| `NO_VARIABLE_ROWS` | Linked `keboola.variables` config has no rows (fix: `config variables-set`) | +| `MALFORMED_VARIABLES_ROW` | Variables row returned by the API is missing a usable `id` | + +### Storage + +| Code | Description | +|---|---| +| `UPLOAD_FAILED` | Cloud storage upload to S3/Azure/GCS failed | +| `EXPORT_EMPTY_MANIFEST` | Sliced export manifest contains no slices | +| `EXPORT_NO_FILE` | Export manifest lists no downloadable file | +| `EXPORT_NO_URL` | Export entry has no download URL | +| `NOT_SLICED` | Attempted a sliced-file operation on a non-sliced file | +| `FILE_NO_URL` | File metadata has no usable download URL | + +### I/O + +| Code | Description | +|---|---| +| `FILE_NOT_FOUND` | Local file path does not exist | +| `DIR_NOT_FOUND` | Local directory path does not exist | +| `READ_ERROR` | Error reading a local file | +| `WRITE_ERROR` | Error writing a local file | +| `INPUT_ERROR` | Invalid or unparseable input data | + +### Lineage + +| Code | Description | +|---|---| +| `NODE_NOT_FOUND` | Requested node not found in the lineage graph | + +### Sharing + +| Code | Description | +|---|---| +| `INVALID_SHARING_TYPE` | Unsupported bucket sharing type | +| `NOT_LINKED_BUCKET` | Bucket is not a linked bucket | + +### KAI (AI Service) + +| Code | Description | +|---|---| +| `KAI_ERROR` | AI Service request failed | +| `KAI_NOT_ENABLED` | KAI is not enabled on this project | + +### Workspace / Query + +| Code | Description | +|---|---| +| `MISSING_QUERY` | No SQL query was provided | +| `WORKSPACE_NOT_FOUND` | Workspace not found in the project | + +### Sync + +| Code | Description | +|---|---| +| `PARENT_CONFIG_NOT_TRACKED` | Row operation references a parent config not in the manifest | + +### Encryption + +| Code | Description | +|---|---| +| `ENCRYPTION_FAILED` | Secret encryption via the Encryption API failed | diff --git a/scripts/check_error_codes.py b/scripts/check_error_codes.py new file mode 100644 index 00000000..3f994aac --- /dev/null +++ b/scripts/check_error_codes.py @@ -0,0 +1,78 @@ +"""CI guard: reject raw error_code string literals in source files. + +Any site that passes error_code="LITERAL_STRING" to KeboolaApiError, +ConfigError, or formatter.error() must use ErrorCode. instead. + +Usage (run from repo root): + python scripts/check_error_codes.py # exits 1 if violations found + python scripts/check_error_codes.py --list # print all current enum members + +Safe exceptions (not flagged): + - tests/ -- string comparisons in assertions are fine + - errors.py -- the enum definition itself +""" + +import ast +import sys +from pathlib import Path + +SRC_ROOT = Path(__file__).parent.parent / "src" +SKIP_FILES = {"errors.py"} + + +def _collect_violations(path: Path) -> list[tuple[int, str]]: + """Return (lineno, code) for each raw error_code string literal.""" + source = path.read_text(encoding="utf-8") + try: + tree = ast.parse(source, filename=str(path)) + except SyntaxError: + return [] + + violations = [] + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + for kw in node.keywords: + if kw.arg != "error_code": + continue + if isinstance(kw.value, ast.Constant) and isinstance(kw.value.value, str): + violations.append((kw.value.lineno, kw.value.value)) + return violations + + +def main() -> int: + if "--list" in sys.argv: + # Print all known enum members without importing the package + errors_path = SRC_ROOT / "keboola_agent_cli" / "errors.py" + source = errors_path.read_text(encoding="utf-8") + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, ast.ClassDef) and node.name == "ErrorCode": + for item in node.body: + if isinstance(item, ast.Assign): + for t in item.targets: + if isinstance(t, ast.Name): + print(f" ErrorCode.{t.id}") + return 0 + + found_any = False + for py_file in sorted(SRC_ROOT.rglob("*.py")): + if py_file.name in SKIP_FILES: + continue + violations = _collect_violations(py_file) + if violations: + found_any = True + rel = py_file.relative_to(SRC_ROOT.parent.parent) + for lineno, code in violations: + print(f' {rel}:{lineno}: error_code="{code}" -- use ErrorCode.{code}') + + if found_any: + print("\nFAIL: raw error_code string literals found. Replace with ErrorCode..") + return 1 + + print("OK: no raw error_code string literals in source.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/keboola_agent_cli/changelog.py b/src/keboola_agent_cli/changelog.py index 6267b832..cc6f48ff 100644 --- a/src/keboola_agent_cli/changelog.py +++ b/src/keboola_agent_cli/changelog.py @@ -49,6 +49,16 @@ "New: --timeout now auto-cancels the remote job -- when the local deadline expires under --wait, kbagent issues `kill_job` against the Queue and exits 7 (EXIT_JOB_TIMEOUT_TERMINATED) with the cancelled job + logTail in the error details. Distinct from exit 4 (QUEUE_JOB_TIMEOUT, retryable) which signals the local kill attempt ALSO failed and the remote may still be running.", "Client: new `fetch_job_events(run_id, limit)` wraps the Storage Events API -- runId is resolved from the job dict (Queue v2 jobs typically have runId == id). The Queue API has NO /jobs/{id}/events route despite the name; events live on Storage.", "Error envelope: KeboolaApiError gained an optional `details: dict` payload; JSON --mode output now includes `error.details` (only when non-empty) so callers can consume structured context without parsing the human message.", + "New: ErrorCode enum (StrEnum) in errors.py -- all 46 error codes are now typed constants; " + "every KeboolaApiError / formatter.error() raise site migrated from string literals to " + "ErrorCode.. Wire format is unchanged (str subtype). CI guard " + "(scripts/check_error_codes.py, wired into 'make check') rejects new raw literals.", + "New: docs/error-codes.md -- versioned reference for all ErrorCode members with " + "add=minor / rename-remove=major semver policy.", + "New: sync init --adopt-existing -- idempotently adopt a .keboola/manifest.json written " + "by the kbc Go CLI (or an older kbagent version) without overwriting it. Validates " + "manifest project_id against the alias token; rejects mismatch with ConfigError (exit 5). " + "Falls through to normal init when no manifest exists. Safe to re-run.", ], "0.21.1": [ "Fix: sync pull on a newly created dev branch now writes config rows (#193) -- idempotent skip guard for rows was missing a file-existence check, causing rows to be silently skipped when the branch directory was new (hash matched main because the branch is a clone)", diff --git a/src/keboola_agent_cli/cli.py b/src/keboola_agent_cli/cli.py index 10241d8e..9197097f 100644 --- a/src/keboola_agent_cli/cli.py +++ b/src/keboola_agent_cli/cli.py @@ -30,7 +30,7 @@ from .commands.workspace import workspace_app from .config_store import ConfigStore, resolve_config_dir from .constants import EXIT_PERMISSION_DENIED -from .errors import PermissionDeniedError +from .errors import ErrorCode, PermissionDeniedError from .models import PermissionPolicy from .output import OutputFormatter from .permissions import PermissionEngine @@ -362,7 +362,7 @@ def main( try: permission_engine.check_or_raise(ctx.invoked_subcommand) except PermissionDeniedError as exc: - formatter.error(message=exc.message, error_code="PERMISSION_DENIED") + formatter.error(message=exc.message, error_code=ErrorCode.PERMISSION_DENIED) raise typer.Exit(code=EXIT_PERMISSION_DENIED) from None # Launch REPL if no subcommand was given (set above) diff --git a/src/keboola_agent_cli/client.py b/src/keboola_agent_cli/client.py index 6cb828c9..e91bce44 100644 --- a/src/keboola_agent_cli/client.py +++ b/src/keboola_agent_cli/client.py @@ -37,7 +37,7 @@ STORAGE_JOB_POLL_INTERVAL, VALID_POLL_STRATEGIES, ) -from .errors import KeboolaApiError +from .errors import ErrorCode, KeboolaApiError from .http_base import BaseHttpClient from .models import TokenVerifyResponse @@ -620,13 +620,13 @@ def _wait_for_storage_job( raise KeboolaApiError( message=error_msg, status_code=500, - error_code="STORAGE_JOB_FAILED", + error_code=ErrorCode.STORAGE_JOB_FAILED, retryable=False, ) raise KeboolaApiError( message=f"Storage job {job_id} did not complete within {max_wait}s", status_code=504, - error_code="STORAGE_JOB_TIMEOUT", + error_code=ErrorCode.STORAGE_JOB_TIMEOUT, retryable=True, ) @@ -981,7 +981,7 @@ def share_bucket( message=f"Invalid sharing type: '{sharing_type}'. " f"Valid types: {', '.join(endpoint_map.keys())}", status_code=400, - error_code="INVALID_SHARING_TYPE", + error_code=ErrorCode.INVALID_SHARING_TYPE, retryable=False, ) @@ -1286,7 +1286,7 @@ def _upload_to_cloud( raise KeboolaApiError( message=f"Cloud storage upload failed (HTTP {response.status_code})", status_code=response.status_code, - error_code="UPLOAD_FAILED", + error_code=ErrorCode.UPLOAD_FAILED, retryable=False, ) @@ -1700,7 +1700,7 @@ def _prepare_sliced_download( raise KeboolaApiError( message="Sliced file manifest has no entries", status_code=500, - error_code="EXPORT_EMPTY_MANIFEST", + error_code=ErrorCode.EXPORT_EMPTY_MANIFEST, retryable=False, ) @@ -2037,7 +2037,7 @@ def wait_for_queue_job( raise KeboolaApiError( message=f"Queue job {job_id} failed: {error_msg}", status_code=500, - error_code="QUEUE_JOB_FAILED", + error_code=ErrorCode.QUEUE_JOB_FAILED, retryable=False, ) return job @@ -2052,7 +2052,7 @@ def wait_for_queue_job( raise KeboolaApiError( message=f"Queue job {job_id} did not complete within {max_wait}s", status_code=504, - error_code="QUEUE_JOB_TIMEOUT", + error_code=ErrorCode.QUEUE_JOB_TIMEOUT, retryable=True, ) @@ -2311,7 +2311,7 @@ def wait_for_query_job(self, query_job_id: str) -> dict[str, Any]: raise KeboolaApiError( message=f"Query job failed: {error_msg}", status_code=500, - error_code="QUERY_JOB_FAILED", + error_code=ErrorCode.QUERY_JOB_FAILED, retryable=False, ) time.sleep(QUERY_JOB_POLL_INTERVAL) @@ -2319,7 +2319,7 @@ def wait_for_query_job(self, query_job_id: str) -> dict[str, Any]: raise KeboolaApiError( message=f"Query job {query_job_id} did not complete within {QUERY_JOB_MAX_WAIT}s", status_code=504, - error_code="QUERY_JOB_TIMEOUT", + error_code=ErrorCode.QUERY_JOB_TIMEOUT, retryable=True, ) diff --git a/src/keboola_agent_cli/commands/_helpers.py b/src/keboola_agent_cli/commands/_helpers.py index 8b399b7b..a9e9ed15 100644 --- a/src/keboola_agent_cli/commands/_helpers.py +++ b/src/keboola_agent_cli/commands/_helpers.py @@ -20,7 +20,7 @@ EXIT_JOB_TIMEOUT_TERMINATED, EXIT_PERMISSION_DENIED, ) -from ..errors import KeboolaApiError, PermissionDeniedError +from ..errors import ErrorCode, KeboolaApiError, PermissionDeniedError from ..output import OutputFormatter @@ -164,7 +164,7 @@ def check_cli_permission(ctx: typer.Context, group_name: str) -> None: engine.check_or_raise(operation) except PermissionDeniedError as exc: formatter = get_formatter(ctx) - formatter.error(message=exc.message, error_code="PERMISSION_DENIED") + formatter.error(message=exc.message, error_code=ErrorCode.PERMISSION_DENIED) raise typer.Exit(code=EXIT_PERMISSION_DENIED) from None @@ -200,7 +200,7 @@ def resolve_project_alias( try: alias, _source = service.resolve_pinned_alias(explicit=explicit) except _ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None return alias @@ -218,7 +218,7 @@ def validate_branch_requires_project( if branch is not None and not project: formatter.error( message="--branch requires --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None diff --git a/src/keboola_agent_cli/commands/branch.py b/src/keboola_agent_cli/commands/branch.py index 91fbcdce..5f048fcd 100644 --- a/src/keboola_agent_cli/commands/branch.py +++ b/src/keboola_agent_cli/commands/branch.py @@ -8,7 +8,7 @@ import typer -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..output import format_branch_metadata_table, format_branches_table from ._helpers import ( check_cli_permission, @@ -48,7 +48,7 @@ def branch_list( try: result = service.list_branches(aliases=project) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -103,7 +103,7 @@ def branch_create( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -144,7 +144,7 @@ def branch_use( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -172,7 +172,7 @@ def branch_reset( lambda c, d: c.print(f"[bold green]Success:[/bold green] {d['message']}"), ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -216,7 +216,7 @@ def branch_delete( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -253,7 +253,7 @@ def branch_merge( ), ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -292,7 +292,7 @@ def branch_metadata_list( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -329,7 +329,7 @@ def branch_metadata_get( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None formatter.output( @@ -370,7 +370,7 @@ def branch_metadata_set( try: value = resolve_text_input(text=text, file=file, stdin=stdin) except ConfigError as exc: - formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + formatter.error(message=exc.message, error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None if should_hint(ctx): @@ -396,7 +396,7 @@ def branch_metadata_set( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -455,5 +455,5 @@ def branch_metadata_delete( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None diff --git a/src/keboola_agent_cli/commands/component.py b/src/keboola_agent_cli/commands/component.py index 273d2dd9..13599b56 100644 --- a/src/keboola_agent_cli/commands/component.py +++ b/src/keboola_agent_cli/commands/component.py @@ -10,7 +10,7 @@ from rich.table import Table from ..constants import VALID_COMPONENT_TYPES -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -162,7 +162,7 @@ def component_list( formatter.error( message=f"Invalid component type '{component_type}'. " f"Valid types: {', '.join(VALID_COMPONENT_TYPES)}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -173,7 +173,7 @@ def component_list( query=query, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -211,7 +211,7 @@ def component_detail( ) formatter.output(result, _format_component_detail) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) diff --git a/src/keboola_agent_cli/commands/config.py b/src/keboola_agent_cli/commands/config.py index 37e2e4b0..aaaf0ef0 100644 --- a/src/keboola_agent_cli/commands/config.py +++ b/src/keboola_agent_cli/commands/config.py @@ -16,7 +16,7 @@ from ..config_store import ConfigStore from ..constants import KEBOOLA_DIR_NAME, MANIFEST_FILENAME, VALID_COMPONENT_TYPES -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..output import format_config_detail, format_configs_table, format_search_results from ._helpers import ( check_cli_permission, @@ -116,7 +116,7 @@ def config_list( if branch is not None and (not project or len(project) != 1): formatter.error( message="--branch requires exactly one --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -131,7 +131,7 @@ def config_list( formatter.error( message=f"Invalid component type '{component_type}'. " f"Valid types: {', '.join(VALID_COMPONENT_TYPES)}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -143,7 +143,7 @@ def config_list( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None # In JSON mode, include both configs and errors in the response @@ -197,7 +197,7 @@ def config_detail( ) formatter.output(result, format_config_detail) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) @@ -276,7 +276,7 @@ def config_search( if branch is not None and (not project or len(project) != 1): formatter.error( message="--branch requires exactly one --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -290,7 +290,7 @@ def config_search( formatter.error( message=f"Invalid component type '{component_type}'. " f"Valid types: {', '.join(VALID_COMPONENT_TYPES)}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -301,7 +301,7 @@ def config_search( except re.error as exc: formatter.error( message=f"Invalid regex pattern: {exc}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -316,7 +316,7 @@ def config_search( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -447,7 +447,7 @@ def config_update( if configuration and configuration_file: formatter.error( message="Cannot use both --configuration and --configuration-file.", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None @@ -457,7 +457,7 @@ def config_update( except (json.JSONDecodeError, FileNotFoundError) as exc: formatter.error( message=f"Invalid --configuration input: {exc}", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None @@ -467,7 +467,7 @@ def config_update( except json.JSONDecodeError as exc: formatter.error( message=f"Invalid JSON in {configuration_file}: {exc}", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None @@ -479,7 +479,7 @@ def config_update( if "=" not in item: formatter.error( message=f"Invalid --set format: '{item}'. Expected PATH=VALUE.", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None path, _, raw_value = item.partition("=") @@ -502,7 +502,7 @@ def config_update( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -624,7 +624,7 @@ def config_rename( directory=effective_directory, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -693,7 +693,7 @@ def config_delete( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -772,7 +772,7 @@ def config_new( name=name or None, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) @@ -872,7 +872,7 @@ def config_metadata_list( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -930,7 +930,7 @@ def config_get_metadata( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -977,7 +977,7 @@ def config_set_metadata( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1033,7 +1033,7 @@ def config_delete_metadata( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1083,7 +1083,7 @@ def config_set_folder( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1175,7 +1175,7 @@ def config_variables_set( if not raw_vars: formatter.error( message="At least one --var KEY=VALUE is required.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -1184,7 +1184,7 @@ def config_variables_set( try: key, value = _parse_kv_var(raw) except typer.BadParameter as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None variables_dict[key] = value @@ -1208,7 +1208,7 @@ def config_variables_set( ) raise typer.Exit(code=map_error_to_exit_code(exc)) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None preview_values = ( @@ -1251,7 +1251,7 @@ def config_variables_set( ) raise typer.Exit(code=map_error_to_exit_code(exc)) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1304,7 +1304,7 @@ def config_variables_get( ) raise typer.Exit(code=map_error_to_exit_code(exc)) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1372,7 +1372,7 @@ def config_variables_clear( ) raise typer.Exit(code=map_error_to_exit_code(exc)) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: diff --git a/src/keboola_agent_cli/commands/context.py b/src/keboola_agent_cli/commands/context.py index 5c5a3bbc..29e0a276 100644 --- a/src/keboola_agent_cli/commands/context.py +++ b/src/keboola_agent_cli/commands/context.py @@ -458,7 +458,7 @@ ### Project Sync - kbagent sync init --project ALIAS [--directory DIR] [--git-branching] + kbagent sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing] Initialize sync working directory. --git-branching enables git-to-Keboola branch mapping. kbagent sync pull --project ALIAS [--all-projects] [--force] [--dry-run] [--with-samples] [--no-storage] [--no-jobs] [--job-limit N] diff --git a/src/keboola_agent_cli/commands/encrypt.py b/src/keboola_agent_cli/commands/encrypt.py index 5ad7c9fd..8e08b81d 100644 --- a/src/keboola_agent_cli/commands/encrypt.py +++ b/src/keboola_agent_cli/commands/encrypt.py @@ -10,7 +10,7 @@ import typer -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -79,20 +79,20 @@ def encrypt_values( try: parsed = _parse_input(input_data) except (json.JSONDecodeError, FileNotFoundError, ValueError) as exc: - formatter.error(message=str(exc), error_code="INPUT_ERROR") + formatter.error(message=str(exc), error_code=ErrorCode.INPUT_ERROR) raise typer.Exit(code=2) from None if not isinstance(parsed, dict): formatter.error( message="Input must be a JSON object (dict), not " + type(parsed).__name__, - error_code="INPUT_ERROR", + error_code=ErrorCode.INPUT_ERROR, ) raise typer.Exit(code=2) from None try: result = service.encrypt(alias=project, component_id=component_id, input_data=parsed) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) diff --git a/src/keboola_agent_cli/commands/flow.py b/src/keboola_agent_cli/commands/flow.py index c14ccb11..7e81ffae 100644 --- a/src/keboola_agent_cli/commands/flow.py +++ b/src/keboola_agent_cli/commands/flow.py @@ -16,7 +16,7 @@ from rich.markup import escape from rich.table import Table -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -120,7 +120,7 @@ def flow_list( if branch is not None and (not project or len(project) != 1): formatter.error( message="--branch requires exactly one --project", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -131,7 +131,7 @@ def flow_list( try: result = service.list_flows(aliases=project, branch_id=effective_branch) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -215,7 +215,7 @@ def flow_detail( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -384,7 +384,7 @@ def flow_new( flow_def = _load_flow_yaml(file) except (OSError, yaml.YAMLError, ValueError) as exc: formatter.error( - message=f"Cannot load flow definition: {exc}", error_code="VALIDATION_ERROR" + message=f"Cannot load flow definition: {exc}", error_code=ErrorCode.VALIDATION_ERROR ) raise typer.Exit(code=2) from None phases = flow_def.get("phases", []) @@ -401,7 +401,7 @@ def flow_new( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -472,7 +472,7 @@ def flow_update( flow_def = _load_flow_yaml(file) except (OSError, yaml.YAMLError, ValueError) as exc: formatter.error( - message=f"Cannot load flow definition: {exc}", error_code="VALIDATION_ERROR" + message=f"Cannot load flow definition: {exc}", error_code=ErrorCode.VALIDATION_ERROR ) raise typer.Exit(code=2) from None phases = flow_def.get("phases") @@ -481,7 +481,7 @@ def flow_update( if name is None and description is None and phases is None and tasks is None: formatter.error( message="At least one of --name, --description, or --file must be provided.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -497,7 +497,7 @@ def flow_update( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -587,7 +587,7 @@ def flow_delete( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -662,7 +662,7 @@ def flow_schedule( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -738,7 +738,7 @@ def flow_schedule_remove( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -799,7 +799,7 @@ def flow_schedule_remove( branch_id=branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) diff --git a/src/keboola_agent_cli/commands/init.py b/src/keboola_agent_cli/commands/init.py index d83c0673..172a68b0 100644 --- a/src/keboola_agent_cli/commands/init.py +++ b/src/keboola_agent_cli/commands/init.py @@ -9,6 +9,7 @@ from ..config_store import ConfigStore from ..constants import LOCAL_CONFIG_DIR_NAME +from ..errors import ErrorCode from ..models import AppConfig, PermissionPolicy from ._helpers import get_formatter, get_service @@ -91,7 +92,7 @@ def init_command( "3) 'kbagent permissions set --mode allow --deny cli:write --deny tool:write' to lock. " "Alternatively, use --from-global to seed projects from the global config." ), - error_code="CONFIG_ERROR", + error_code=ErrorCode.CONFIG_ERROR, ) raise typer.Exit(code=5) diff --git a/src/keboola_agent_cli/commands/job.py b/src/keboola_agent_cli/commands/job.py index 6552a4df..759c11da 100644 --- a/src/keboola_agent_cli/commands/job.py +++ b/src/keboola_agent_cli/commands/job.py @@ -20,7 +20,7 @@ VALID_POLL_STRATEGIES, VALID_STATUSES, ) -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..output import format_job_detail, format_jobs_table from ._helpers import ( check_cli_permission, @@ -90,7 +90,7 @@ def job_list( if status and status not in VALID_STATUSES: formatter.error( message=f"Invalid status '{status}'. Valid statuses: {', '.join(VALID_STATUSES)}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -98,7 +98,7 @@ def job_list( if limit < 1 or limit > MAX_JOB_LIMIT: formatter.error( message=f"Invalid limit {limit}. Must be between 1 and {MAX_JOB_LIMIT}.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -106,7 +106,7 @@ def job_list( if config_id and not component_id: formatter.error( message="--config-id requires --component-id to be specified.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -119,7 +119,7 @@ def job_list( limit=limit, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -146,7 +146,7 @@ def job_detail( result = service.get_job_detail(alias=project, job_id=job_id) formatter.output(result, format_job_detail) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) @@ -289,7 +289,7 @@ def job_run( "--variable-values-id cannot be empty or whitespace. " "Pass a row id, or omit the flag to auto-resolve the default row." ), - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -300,7 +300,7 @@ def job_run( "Pass --variable-values-id to bind a specific values row, or " "--no-variables to skip resolution, but not both." ), - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -310,7 +310,7 @@ def job_run( f"--log-tail-lines must be between 0 and {MAX_LOG_TAIL_LINES}. " f"Got {log_tail_lines}." ), - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -345,7 +345,7 @@ def job_run( log_tail_lines=log_tail_lines, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -495,7 +495,7 @@ def job_terminate( if bool(job_id) == bool(status): formatter.error( message="Provide either --job-id (one or more) or --status, but not both.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -505,21 +505,21 @@ def job_terminate( f"Invalid --status '{status}'. Use one of: " f"{', '.join(sorted(KILLABLE_JOB_STATUSES))} or 'any'." ), - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) if config_id and not component_id: formatter.error( message="--config-id requires --component-id.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) if limit < 1 or limit > MAX_JOB_LIMIT: formatter.error( message=f"Invalid --limit {limit}. Must be between 1 and {MAX_JOB_LIMIT}.", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -546,7 +546,7 @@ def job_terminate( if status == "any": matched = service.filter_killable(matched) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -591,7 +591,7 @@ def job_terminate( dry_run=True, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if filter_context is not None: @@ -618,7 +618,7 @@ def job_terminate( job_ids=resolved_ids, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( diff --git a/src/keboola_agent_cli/commands/kai.py b/src/keboola_agent_cli/commands/kai.py index c7a07069..21a54c16 100644 --- a/src/keboola_agent_cli/commands/kai.py +++ b/src/keboola_agent_cli/commands/kai.py @@ -6,7 +6,7 @@ import typer -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -44,7 +44,7 @@ def kai_ping( alias = service.resolve_alias(project) result = service.ping(alias) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code) @@ -91,7 +91,7 @@ def kai_ask( alias = service.resolve_alias(project) result = service.ask(alias, message) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code) @@ -138,7 +138,7 @@ def kai_chat( alias = service.resolve_alias(project) result = service.chat_message(alias, message, chat_id=chat_id) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code) @@ -177,7 +177,7 @@ def kai_history( alias = service.resolve_alias(project) result = service.get_history(alias, limit=limit) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code) diff --git a/src/keboola_agent_cli/commands/lineage.py b/src/keboola_agent_cli/commands/lineage.py index 3000bc7a..f7a14232 100644 --- a/src/keboola_agent_cli/commands/lineage.py +++ b/src/keboola_agent_cli/commands/lineage.py @@ -19,6 +19,7 @@ import typer +from ..errors import ErrorCode from ._helpers import ( check_cli_permission, emit_hint, @@ -103,7 +104,7 @@ def lineage_build( root = directory.resolve() if not root.is_dir(): - formatter.error(message=f"Directory not found: {root}", error_code="DIR_NOT_FOUND") + formatter.error(message=f"Directory not found: {root}", error_code=ErrorCode.DIR_NOT_FOUND) raise typer.Exit(code=1) # --refresh: sync pull all projects first @@ -127,7 +128,7 @@ def lineage_build( except OSError as exc: formatter.error( message=f"Cannot write output file '{output}': {exc}", - error_code="WRITE_ERROR", + error_code=ErrorCode.WRITE_ERROR, ) raise typer.Exit(code=1) from None @@ -190,7 +191,9 @@ def lineage_info( service = get_service(ctx, "deep_lineage_service") if not load.exists(): - formatter.error(message=f"Cache file not found: {load}", error_code="FILE_NOT_FOUND") + formatter.error( + message=f"Cache file not found: {load}", error_code=ErrorCode.FILE_NOT_FOUND + ) raise typer.Exit(code=1) graph = service.load_from_cache(load) @@ -336,12 +339,14 @@ def lineage_show( if format not in valid_formats: formatter.error( message=f"Invalid format '{format}'. Must be one of: {', '.join(valid_formats)}", - error_code="INVALID_FORMAT", + error_code=ErrorCode.INVALID_FORMAT, ) raise typer.Exit(code=2) if not load.exists(): - formatter.error(message=f"Cache file not found: {load}", error_code="FILE_NOT_FOUND") + formatter.error( + message=f"Cache file not found: {load}", error_code=ErrorCode.FILE_NOT_FOUND + ) raise typer.Exit(code=1) graph = service.load_from_cache(load) @@ -350,7 +355,7 @@ def lineage_show( formatter.error( message="Specify --upstream or --downstream to query.\n" "Use `kbagent lineage info -l FILE` to see what's in the graph.", - error_code="MISSING_QUERY", + error_code=ErrorCode.MISSING_QUERY, ) raise typer.Exit(code=2) @@ -363,7 +368,7 @@ def lineage_show( msg = query_result["error"] if suggestions: msg += "\nDid you mean: " + ", ".join(suggestions[:5]) - formatter.error(message=msg, error_code="NODE_NOT_FOUND") + formatter.error(message=msg, error_code=ErrorCode.NODE_NOT_FOUND) raise typer.Exit(code=1) if formatter.json_mode: @@ -382,7 +387,7 @@ def lineage_show( msg = query_result["error"] if suggestions: msg += "\nDid you mean: " + ", ".join(suggestions[:5]) - formatter.error(message=msg, error_code="NODE_NOT_FOUND") + formatter.error(message=msg, error_code=ErrorCode.NODE_NOT_FOUND) raise typer.Exit(code=1) if formatter.json_mode: @@ -434,7 +439,7 @@ def _output_mermaid_or_html( except OSError as exc: formatter.error( message=f"Cannot write HTML file '{filename}': {exc}", - error_code="WRITE_ERROR", + error_code=ErrorCode.WRITE_ERROR, ) raise typer.Exit(code=1) from None @@ -1433,13 +1438,15 @@ def lineage_serve( formatter = get_formatter(ctx) if not load.exists(): - formatter.error(message=f"Cache file not found: {load}", error_code="FILE_NOT_FOUND") + formatter.error( + message=f"Cache file not found: {load}", error_code=ErrorCode.FILE_NOT_FOUND + ) raise typer.Exit(code=1) try: raw_data = json.loads(load.read_text(encoding="utf-8")) except (json.JSONDecodeError, OSError) as exc: - formatter.error(message=f"Cannot read lineage file: {exc}", error_code="READ_ERROR") + formatter.error(message=f"Cannot read lineage file: {exc}", error_code=ErrorCode.READ_ERROR) raise typer.Exit(code=1) from None # Load the graph via the service for API queries diff --git a/src/keboola_agent_cli/commands/org.py b/src/keboola_agent_cli/commands/org.py index 29769a5c..25b95638 100644 --- a/src/keboola_agent_cli/commands/org.py +++ b/src/keboola_agent_cli/commands/org.py @@ -9,7 +9,7 @@ from rich.table import Table from ..constants import DEFAULT_TOKEN_DESCRIPTION, ENV_KBC_STORAGE_API_URL -from ..errors import KeboolaApiError +from ..errors import ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -216,7 +216,7 @@ def org_setup( if not org_id and not project_ids: formatter.error( message="Provide --org-id (org admin) or --project-ids (project member)", - error_code="usage_error", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) diff --git a/src/keboola_agent_cli/commands/permissions.py b/src/keboola_agent_cli/commands/permissions.py index 29b81d66..946a6af6 100644 --- a/src/keboola_agent_cli/commands/permissions.py +++ b/src/keboola_agent_cli/commands/permissions.py @@ -17,6 +17,7 @@ from ..config_store import ConfigStore from ..constants import EXIT_PERMISSION_DENIED +from ..errors import ErrorCode from ..models import PermissionPolicy from ..permissions import PermissionEngine from ._helpers import get_formatter, get_service @@ -271,14 +272,14 @@ def permissions_set( if mode not in ("allow", "deny"): formatter.error( message="Mode must be 'allow' or 'deny'", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None if not _require_interactive_confirmation("update permission policy"): formatter.error( message="Confirmation failed. Permission policy not changed.", - error_code="PERMISSION_DENIED", + error_code=ErrorCode.PERMISSION_DENIED, ) raise typer.Exit(code=EXIT_PERMISSION_DENIED) from None @@ -330,7 +331,7 @@ def permissions_reset( if not _require_interactive_confirmation("remove permission policy"): formatter.error( message="Confirmation failed. Permission policy not changed.", - error_code="PERMISSION_DENIED", + error_code=ErrorCode.PERMISSION_DENIED, ) raise typer.Exit(code=EXIT_PERMISSION_DENIED) from None diff --git a/src/keboola_agent_cli/commands/project.py b/src/keboola_agent_cli/commands/project.py index 210c3800..409e94ea 100644 --- a/src/keboola_agent_cli/commands/project.py +++ b/src/keboola_agent_cli/commands/project.py @@ -18,7 +18,7 @@ ENV_KBC_STORAGE_API_URL, ENV_KBC_TOKEN, ) -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -178,7 +178,7 @@ def project_add( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -192,7 +192,7 @@ def project_list(ctx: typer.Context) -> None: projects = service.list_projects() formatter.output(projects, _format_project_table) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -211,7 +211,7 @@ def project_remove( result, lambda c, d: c.print(f"[bold green]Success:[/bold green] {d['message']}") ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -249,7 +249,7 @@ def project_edit( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -353,7 +353,7 @@ def project_status( statuses = service.get_status(aliases=aliases) formatter.output(statuses, _format_status_table) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: exit_code = map_error_to_exit_code(exc) @@ -410,13 +410,13 @@ def project_refresh( if project and all_projects: formatter.error( message="Provide --project or --all, not both", - error_code="usage_error", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) if not project and not all_projects: formatter.error( message="Provide --project or --all", - error_code="usage_error", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) @@ -493,7 +493,7 @@ def project_use( try: result = service.use_project(alias=alias) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None def _human(c: Console, d: dict[str, Any]) -> None: @@ -587,7 +587,7 @@ def project_description_get( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None formatter.output( @@ -626,7 +626,7 @@ def project_description_set( try: description = resolve_text_input(text=text, file=file, stdin=stdin) except ConfigError as exc: - formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + formatter.error(message=exc.message, error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None if should_hint(ctx): @@ -650,5 +650,5 @@ def project_description_set( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None diff --git a/src/keboola_agent_cli/commands/sharing.py b/src/keboola_agent_cli/commands/sharing.py index 8e2fa90e..e8e756b7 100644 --- a/src/keboola_agent_cli/commands/sharing.py +++ b/src/keboola_agent_cli/commands/sharing.py @@ -8,7 +8,7 @@ import typer -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -57,7 +57,7 @@ def sharing_list( try: result = service.list_shared(aliases=project) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -157,7 +157,7 @@ def sharing_share( if not target_project_ids: formatter.error( message="--target-project-ids is required for --type selected-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) parsed_project_ids = [int(pid.strip()) for pid in target_project_ids.split(",")] @@ -166,7 +166,7 @@ def sharing_share( if not target_users: formatter.error( message="--target-users is required for --type selected-users", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) parsed_users = [u.strip() for u in target_users.split(",")] @@ -180,7 +180,7 @@ def sharing_share( target_users=parsed_users, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -221,7 +221,7 @@ def sharing_unshare( try: result = service.unshare(alias=project, bucket_id=bucket_id) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -286,7 +286,7 @@ def sharing_link( name=name, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -326,7 +326,7 @@ def sharing_unlink( try: result = service.unlink(alias=project, bucket_id=bucket_id) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -363,7 +363,7 @@ def sharing_edges( try: result = service.get_lineage(aliases=project) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: diff --git a/src/keboola_agent_cli/commands/storage.py b/src/keboola_agent_cli/commands/storage.py index e2993a01..cd15e114 100644 --- a/src/keboola_agent_cli/commands/storage.py +++ b/src/keboola_agent_cli/commands/storage.py @@ -10,7 +10,7 @@ import typer from ..config_store import ConfigStore -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import ( check_cli_permission, emit_hint, @@ -72,7 +72,7 @@ def storage_buckets( if branch is not None and (not project or len(project) != 1): formatter.error( message="--branch requires exactly one --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -90,7 +90,7 @@ def storage_buckets( try: result = service.list_buckets(aliases=project, branch_id=effective_branch) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -176,7 +176,7 @@ def storage_bucket_detail( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -269,7 +269,7 @@ def storage_tables( if branch is not None and (not project or len(project) != 1): formatter.error( message="--branch requires exactly one --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) @@ -291,7 +291,7 @@ def storage_tables( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -377,7 +377,7 @@ def storage_table_detail( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -478,10 +478,10 @@ def storage_create_bucket( branch_id=effective_branch, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -562,10 +562,10 @@ def storage_create_table( branch_id=effective_branch, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -649,7 +649,7 @@ def storage_upload_table( p = Path(file) if not p.is_file(): - formatter.error(message=f"File not found: {file}", error_code="FILE_NOT_FOUND") + formatter.error(message=f"File not found: {file}", error_code=ErrorCode.FILE_NOT_FOUND) raise typer.Exit(code=2) from None if not formatter.json_mode: @@ -670,10 +670,10 @@ def storage_upload_table( branch_id=effective_branch, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -795,10 +795,10 @@ def storage_download_table( keep_slices=keep_slices, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -888,7 +888,7 @@ def storage_delete_table( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -916,7 +916,7 @@ def storage_delete_table( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1005,7 +1005,7 @@ def storage_delete_column( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1036,7 +1036,7 @@ def storage_delete_column( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1119,7 +1119,7 @@ def storage_delete_bucket( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -1202,7 +1202,7 @@ def storage_describe_bucket( try: description = resolve_text_input(text=text, file=file, stdin=stdin) except ConfigError as exc: - formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + formatter.error(message=exc.message, error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None try: @@ -1213,7 +1213,7 @@ def storage_describe_bucket( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1278,7 +1278,7 @@ def storage_describe_table( try: description = resolve_text_input(text=text, file=file, stdin=stdin) except ConfigError as exc: - formatter.error(message=exc.message, error_code="INVALID_ARGUMENT") + formatter.error(message=exc.message, error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None try: @@ -1289,7 +1289,7 @@ def storage_describe_table( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1354,7 +1354,7 @@ def storage_describe_column( if "=" not in entry: formatter.error( message=f"--column must be NAME=DESCRIPTION, got: {entry!r}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None name, _, desc = entry.partition("=") @@ -1362,7 +1362,7 @@ def storage_describe_column( if not name: formatter.error( message=f"Column name cannot be empty in: {entry!r}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None parsed[name] = desc @@ -1375,10 +1375,10 @@ def storage_describe_column( branch_id=effective_branch, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1501,10 +1501,10 @@ def _on_item(obj_type: str, obj_id: str, current: int, total: int) -> None: completed=result["applied_count"] + result["error_count"], ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1624,7 +1624,7 @@ def storage_file_list( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1688,7 +1688,7 @@ def storage_file_info( try: result = service.get_file_info(alias=project, file_id=file_id) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1772,7 +1772,7 @@ def storage_file_upload( p = Path(file) if not p.is_file(): - formatter.error(message=f"File not found: {file}", error_code="FILE_NOT_FOUND") + formatter.error(message=f"File not found: {file}", error_code=ErrorCode.FILE_NOT_FOUND) raise typer.Exit(code=2) from None if not formatter.json_mode: @@ -1789,7 +1789,7 @@ def storage_file_upload( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1856,7 +1856,7 @@ def storage_file_download( if not file_id and not tag: formatter.error( message="Either --file-id or --tag must be provided", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -1874,10 +1874,10 @@ def storage_file_download( output_path=output, ) except ValueError as exc: - formatter.error(message=str(exc), error_code="INVALID_ARGUMENT") + formatter.error(message=str(exc), error_code=ErrorCode.INVALID_ARGUMENT) raise typer.Exit(code=2) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -1930,7 +1930,7 @@ def storage_file_tag( if not add and not remove: formatter.error( message="At least one of --add or --remove must be provided", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -1942,7 +1942,7 @@ def storage_file_tag( remove_tags=remove, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -2003,7 +2003,7 @@ def storage_file_delete( dry_run=dry_run, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -2105,7 +2105,7 @@ def storage_load_file( branch_id=effective_branch, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -2202,7 +2202,7 @@ def storage_unload_table( formatter = get_formatter(ctx) formatter.error( message=f"--file-type must be 'csv' or 'parquet', got {file_type!r}", - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, ) raise typer.Exit(code=2) from None if should_hint(ctx): @@ -2247,7 +2247,7 @@ def storage_unload_table( keep_slices=keep_slices, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) diff --git a/src/keboola_agent_cli/commands/sync.py b/src/keboola_agent_cli/commands/sync.py index dc560ec5..3433e5a3 100644 --- a/src/keboola_agent_cli/commands/sync.py +++ b/src/keboola_agent_cli/commands/sync.py @@ -9,7 +9,7 @@ import typer -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ._helpers import check_cli_permission, get_formatter, get_service, map_error_to_exit_code sync_app = typer.Typer(help="Sync project configurations with local filesystem") @@ -81,12 +81,22 @@ def sync_init( "--git-branching", help="Enable git-branching mode (maps git branches to Keboola branches)", ), + adopt_existing: bool = typer.Option( + False, + "--adopt-existing", + help="Adopt an existing .keboola/manifest.json (e.g. written by kbc) " + "instead of failing. Validates the manifest's project_id against the alias " + "and normalises the file. Idempotent.", + ), ) -> None: """Initialize a sync working directory for a Keboola project. Creates the .keboola/ directory with manifest.json containing project metadata and naming conventions. Optionally enables git-branching mode for branch-to-branch mapping. + + Use --adopt-existing to register a directory that was already initialised + by the official kbc CLI without overwriting the manifest. """ formatter = get_formatter(ctx) service = get_service(ctx, "sync_service") @@ -97,12 +107,13 @@ def sync_init( alias=project, project_root=project_root, git_branching=git_branching, + adopt_existing=adopt_existing, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except FileExistsError as exc: - formatter.error(message=str(exc), error_code="ALREADY_EXISTS") + formatter.error(message=str(exc), error_code=ErrorCode.ALREADY_EXISTS) raise typer.Exit(code=1) from None except KeboolaApiError as exc: formatter.error( @@ -115,9 +126,17 @@ def sync_init( if formatter.json_mode: formatter.output(result) else: - formatter.success( - f"Initialized sync for project '{result['project_alias']}' (ID: {result['project_id']})" - ) + status = result.get("status", "initialized") + if status == "adopted": + formatter.success( + f"Adopted manifest for project '{result['project_alias']}' " + f"(ID: {result['project_id']})" + ) + else: + formatter.success( + f"Initialized sync for project '{result['project_alias']}' " + f"(ID: {result['project_id']})" + ) formatter.console.print(f" API host: {result['api_host']}") if result["git_branching"]: formatter.console.print( @@ -439,13 +458,13 @@ def sync_pull( if all_projects and project: formatter.error( message="Cannot use --project with --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) if not all_projects and not project: formatter.error( message="Specify --project ALIAS or --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) @@ -464,7 +483,7 @@ def sync_pull( max_samples=max_samples, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -481,7 +500,7 @@ def sync_pull( try: service.init_sync(project, project_root) except Exception as exc: - formatter.error(message=str(exc), error_code="INIT_ERROR") + formatter.error(message=str(exc), error_code=ErrorCode.INIT_ERROR) raise typer.Exit(code=1) from None try: @@ -498,10 +517,10 @@ def sync_pull( max_samples=max_samples, ) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error( @@ -539,7 +558,7 @@ def sync_status( try: result = service.status(project_root=project_root) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None if formatter.json_mode: @@ -608,13 +627,13 @@ def sync_diff( if all_projects and project: formatter.error( message="Cannot use --project with --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) if not all_projects and not project: formatter.error( message="Specify --project ALIAS or --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) @@ -623,7 +642,7 @@ def sync_diff( try: data = service.diff_all(base_dir) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -637,10 +656,10 @@ def sync_diff( try: result = service.diff(alias=project, project_root=project_root) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -784,13 +803,13 @@ def sync_push( if all_projects and project: formatter.error( message="Cannot use --project with --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) if not all_projects and not project: formatter.error( message="Specify --project ALIAS or --all-projects", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) @@ -804,7 +823,7 @@ def sync_push( allow_plaintext_fallback=allow_plaintext, ) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -824,10 +843,10 @@ def sync_push( allow_plaintext_fallback=allow_plaintext, ) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -902,10 +921,10 @@ def sync_branch_link( branch_name=branch_name, ) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None except KeboolaApiError as exc: formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) @@ -943,10 +962,10 @@ def sync_branch_unlink( try: result = service.branch_unlink(project_root=project_root) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -973,7 +992,7 @@ def sync_branch_status( try: result = service.branch_status(project_root=project_root) except FileNotFoundError as exc: - formatter.error(message=str(exc), error_code="NOT_INITIALIZED") + formatter.error(message=str(exc), error_code=ErrorCode.NOT_INITIALIZED) raise typer.Exit(code=1) from None if formatter.json_mode: diff --git a/src/keboola_agent_cli/commands/tool.py b/src/keboola_agent_cli/commands/tool.py index 0e8b0b56..73aa6d56 100644 --- a/src/keboola_agent_cli/commands/tool.py +++ b/src/keboola_agent_cli/commands/tool.py @@ -11,7 +11,7 @@ import typer from ..config_store import ConfigStore -from ..errors import ConfigError +from ..errors import ConfigError, ErrorCode from ..output import OutputFormatter, format_tool_result, format_tools_table from ._helpers import ( check_cli_permission, @@ -48,7 +48,7 @@ def _read_input(value: str, formatter: OutputFormatter) -> str: if not file_path.is_file(): formatter.error( message=f"Input file not found: {file_path}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None return file_path.read_text(encoding="utf-8") @@ -97,7 +97,7 @@ def tool_list( if branch_str and not project: formatter.error( message="--branch requires --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -106,7 +106,7 @@ def tool_list( try: result = service.list_tools(aliases=aliases, branch_id=branch_str) except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -169,7 +169,7 @@ def tool_call( if branch_str and not project: formatter.error( message="--branch requires --project (branch ID is per-project)", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -182,14 +182,14 @@ def tool_call( except json.JSONDecodeError as exc: formatter.error( message=f"Invalid JSON in --input: {exc}", - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None if not isinstance(parsed_input, dict): formatter.error( message='--input must be a JSON object (e.g. \'{"key": "value"}\')', - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) raise typer.Exit(code=2) from None @@ -203,13 +203,13 @@ def tool_call( ) except ConfigError as exc: # ConfigError covers: unknown tool, missing params, config issues - error_code = "CONFIG_ERROR" + error_code = ErrorCode.CONFIG_ERROR exit_code = 5 if "Missing required parameter" in exc.message: - error_code = "MISSING_PARAMETER" + error_code = ErrorCode.MISSING_PARAMETER exit_code = 2 elif "Unknown MCP tool" in exc.message: - error_code = "CONFIG_ERROR" + error_code = ErrorCode.CONFIG_ERROR exit_code = 5 formatter.error(message=exc.message, error_code=error_code) raise typer.Exit(code=exit_code) from None diff --git a/src/keboola_agent_cli/commands/workspace.py b/src/keboola_agent_cli/commands/workspace.py index 048acdfd..a45bb199 100644 --- a/src/keboola_agent_cli/commands/workspace.py +++ b/src/keboola_agent_cli/commands/workspace.py @@ -9,7 +9,7 @@ import typer from rich.markup import escape -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..output import format_query_results, format_workspaces_table from ._helpers import ( check_cli_permission, @@ -104,7 +104,7 @@ def workspace_create( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -136,7 +136,7 @@ def workspace_list( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -192,7 +192,7 @@ def workspace_detail( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -232,7 +232,7 @@ def workspace_delete( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -275,7 +275,7 @@ def workspace_password( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -337,7 +337,7 @@ def workspace_load( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -393,13 +393,13 @@ def workspace_query( if sql and file: formatter.error( message="Specify either --sql or --file, not both.", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) if not sql and not file: formatter.error( message="Specify either --sql or --file.", - error_code="USAGE_ERROR", + error_code=ErrorCode.USAGE_ERROR, ) raise typer.Exit(code=2) @@ -426,7 +426,7 @@ def workspace_query( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None @@ -478,7 +478,7 @@ def workspace_gc( formatter.error(message=exc.message, error_code=exc.error_code, retryable=exc.retryable) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None if formatter.json_mode: @@ -583,5 +583,5 @@ def workspace_from_transformation( ) raise typer.Exit(code=exit_code) from None except ConfigError as exc: - formatter.error(message=exc.message, error_code="CONFIG_ERROR") + formatter.error(message=exc.message, error_code=ErrorCode.CONFIG_ERROR) raise typer.Exit(code=5) from None diff --git a/src/keboola_agent_cli/errors.py b/src/keboola_agent_cli/errors.py index 7d3a5746..6a7f8dda 100644 --- a/src/keboola_agent_cli/errors.py +++ b/src/keboola_agent_cli/errors.py @@ -1,5 +1,98 @@ """Error types and helpers for Keboola Agent CLI.""" +from enum import StrEnum + + +class ErrorCode(StrEnum): + """Stable machine-readable error codes emitted by kbagent. + + ``str`` mixin means values compare equal to their plain-string equivalents + and serialise as plain strings in JSON output -- no wire-format change. + + Versioning: adding a new code = minor bump; renaming / removing = major bump. + """ + + # Auth / access + INVALID_TOKEN = "INVALID_TOKEN" + ACCESS_DENIED = "ACCESS_DENIED" + PERMISSION_DENIED = "PERMISSION_DENIED" + + # Network / transport + TIMEOUT = "TIMEOUT" + CONNECTION_ERROR = "CONNECTION_ERROR" + RETRY_EXHAUSTED = "RETRY_EXHAUSTED" + + # API / generic + API_ERROR = "API_ERROR" + NOT_FOUND = "NOT_FOUND" + ALREADY_EXISTS = "ALREADY_EXISTS" + VALIDATION_ERROR = "VALIDATION_ERROR" + INVALID_ARGUMENT = "INVALID_ARGUMENT" + INVALID_FORMAT = "INVALID_FORMAT" + USAGE_ERROR = "USAGE_ERROR" + MISSING_PARAMETER = "MISSING_PARAMETER" + UNKNOWN_ERROR = "UNKNOWN_ERROR" + + # Configuration + CONFIG_ERROR = "CONFIG_ERROR" + NOT_INITIALIZED = "NOT_INITIALIZED" + INIT_ERROR = "INIT_ERROR" + + # Jobs + QUEUE_JOB_FAILED = "QUEUE_JOB_FAILED" + QUEUE_JOB_TIMEOUT = "QUEUE_JOB_TIMEOUT" + STORAGE_JOB_FAILED = "STORAGE_JOB_FAILED" + STORAGE_JOB_TIMEOUT = "STORAGE_JOB_TIMEOUT" + QUERY_JOB_FAILED = "QUERY_JOB_FAILED" + QUERY_JOB_TIMEOUT = "QUERY_JOB_TIMEOUT" + + # Variables + NO_VARIABLE_ROWS = "NO_VARIABLE_ROWS" + MALFORMED_VARIABLES_ROW = "MALFORMED_VARIABLES_ROW" + + # Storage + UPLOAD_FAILED = "UPLOAD_FAILED" + EXPORT_EMPTY_MANIFEST = "EXPORT_EMPTY_MANIFEST" + EXPORT_NO_FILE = "EXPORT_NO_FILE" + EXPORT_NO_URL = "EXPORT_NO_URL" + NOT_SLICED = "NOT_SLICED" + FILE_NO_URL = "FILE_NO_URL" + + # I/O + FILE_NOT_FOUND = "FILE_NOT_FOUND" + DIR_NOT_FOUND = "DIR_NOT_FOUND" + READ_ERROR = "READ_ERROR" + WRITE_ERROR = "WRITE_ERROR" + INPUT_ERROR = "INPUT_ERROR" + + # Lineage + NODE_NOT_FOUND = "NODE_NOT_FOUND" + + # Sharing + INVALID_SHARING_TYPE = "INVALID_SHARING_TYPE" + NOT_LINKED_BUCKET = "NOT_LINKED_BUCKET" + + # KAI (AI Service) + KAI_ERROR = "KAI_ERROR" + KAI_NOT_ENABLED = "KAI_NOT_ENABLED" + + # Workspace / Query + MISSING_QUERY = "MISSING_QUERY" + WORKSPACE_NOT_FOUND = "WORKSPACE_NOT_FOUND" + + # Sync + PARENT_CONFIG_NOT_TRACKED = "PARENT_CONFIG_NOT_TRACKED" + + # Encryption + ENCRYPTION_FAILED = "ENCRYPTION_FAILED" + + # Job / queue (extensions from 0.22.0) + JOB_TIMEOUT_TERMINATED = "JOB_TIMEOUT_TERMINATED" + + # Flow (new in 0.22.0) + INVALID_FLOW_DAG = "INVALID_FLOW_DAG" + SCHEDULE_DELETE_FAILED = "SCHEDULE_DELETE_FAILED" + def mask_token(token: str) -> str: """Mask a Keboola Storage API token for safe display. @@ -40,7 +133,7 @@ def __init__( self, message: str, status_code: int = 0, - error_code: str = "UNKNOWN_ERROR", + error_code: str | ErrorCode = ErrorCode.UNKNOWN_ERROR, retryable: bool = False, details: dict | None = None, ) -> None: @@ -72,14 +165,14 @@ def __init__(self, operation: str, message: str = "") -> None: _ERROR_CODE_TO_TYPE: dict[str, str] = { - "INVALID_TOKEN": "authentication", - "TIMEOUT": "network", - "CONNECTION_ERROR": "network", - "RETRY_EXHAUSTED": "network", - "NOT_FOUND": "not_found", - "CONFIG_ERROR": "configuration", - "VALIDATION_ERROR": "validation", - "PERMISSION_DENIED": "authorization", + ErrorCode.INVALID_TOKEN: "authentication", + ErrorCode.TIMEOUT: "network", + ErrorCode.CONNECTION_ERROR: "network", + ErrorCode.RETRY_EXHAUSTED: "network", + ErrorCode.NOT_FOUND: "not_found", + ErrorCode.CONFIG_ERROR: "configuration", + ErrorCode.VALIDATION_ERROR: "validation", + ErrorCode.PERMISSION_DENIED: "authorization", } diff --git a/src/keboola_agent_cli/http_base.py b/src/keboola_agent_cli/http_base.py index af9625f9..d48d57c0 100644 --- a/src/keboola_agent_cli/http_base.py +++ b/src/keboola_agent_cli/http_base.py @@ -21,7 +21,7 @@ MAX_RETRY_AFTER_SECONDS, RETRYABLE_STATUS_CODES, ) -from .errors import KeboolaApiError, mask_token +from .errors import ErrorCode, KeboolaApiError, mask_token logger = logging.getLogger(__name__) @@ -169,7 +169,7 @@ def _do_request( raise KeboolaApiError( message=f"Request timed out connecting to {url_label} (token: {self._masked_token})", status_code=0, - error_code="TIMEOUT", + error_code=ErrorCode.TIMEOUT, retryable=True, ) from exc @@ -189,7 +189,7 @@ def _do_request( raise KeboolaApiError( message=f"Cannot connect to {url_label} (token: {self._masked_token})", status_code=0, - error_code="CONNECTION_ERROR", + error_code=ErrorCode.CONNECTION_ERROR, retryable=True, ) from exc @@ -199,7 +199,7 @@ def _do_request( raise KeboolaApiError( message=f"Request failed after {MAX_RETRIES} retries to {url_label} (token: {self._masked_token})", status_code=0, - error_code="RETRY_EXHAUSTED", + error_code=ErrorCode.RETRY_EXHAUSTED, retryable=True, ) @@ -234,7 +234,7 @@ def _raise_api_error(self, response: httpx.Response, base_url: str | None = None raise KeboolaApiError( message=f"Invalid or expired token (token: {self._masked_token}): {api_message}", status_code=status, - error_code="INVALID_TOKEN", + error_code=ErrorCode.INVALID_TOKEN, retryable=False, ) @@ -242,7 +242,7 @@ def _raise_api_error(self, response: httpx.Response, base_url: str | None = None raise KeboolaApiError( message=f"Access denied (token: {self._masked_token}): {api_message}", status_code=status, - error_code="ACCESS_DENIED", + error_code=ErrorCode.ACCESS_DENIED, retryable=False, ) @@ -250,7 +250,7 @@ def _raise_api_error(self, response: httpx.Response, base_url: str | None = None raise KeboolaApiError( message=f"Resource not found: {api_message}", status_code=status, - error_code="NOT_FOUND", + error_code=ErrorCode.NOT_FOUND, retryable=False, ) @@ -258,6 +258,6 @@ def _raise_api_error(self, response: httpx.Response, base_url: str | None = None raise KeboolaApiError( message=f"API error {status} from {url_label} (token: {self._masked_token}): {api_message}", status_code=status, - error_code="API_ERROR", + error_code=ErrorCode.API_ERROR, retryable=retryable, ) diff --git a/src/keboola_agent_cli/services/_encryption.py b/src/keboola_agent_cli/services/_encryption.py index c8d2c107..44ef1beb 100644 --- a/src/keboola_agent_cli/services/_encryption.py +++ b/src/keboola_agent_cli/services/_encryption.py @@ -12,7 +12,7 @@ import logging from typing import Any -from ..errors import KeboolaApiError +from ..errors import ErrorCode, KeboolaApiError logger = logging.getLogger(__name__) @@ -191,7 +191,7 @@ def encrypt_secrets_in_config( f"Use --allow-plaintext-on-encrypt-failure to override." ), status_code=0, - error_code="ENCRYPTION_FAILED", + error_code=ErrorCode.ENCRYPTION_FAILED, ) from exc return configuration diff --git a/src/keboola_agent_cli/services/branch_service.py b/src/keboola_agent_cli/services/branch_service.py index 0142d6e3..51aac872 100644 --- a/src/keboola_agent_cli/services/branch_service.py +++ b/src/keboola_agent_cli/services/branch_service.py @@ -10,7 +10,7 @@ from typing import Any from ..constants import METADATA_NOT_FOUND -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..models import ProjectConfig from .base import BaseService @@ -408,7 +408,7 @@ def get_branch_metadata( f"Metadata key '{key}' not found on branch '{branch_id}' of project '{alias}'." ), status_code=404, - error_code="NOT_FOUND", + error_code=ErrorCode.NOT_FOUND, retryable=False, ) return { diff --git a/src/keboola_agent_cli/services/config_service.py b/src/keboola_agent_cli/services/config_service.py index bfd5010d..fdc9414b 100644 --- a/src/keboola_agent_cli/services/config_service.py +++ b/src/keboola_agent_cli/services/config_service.py @@ -12,7 +12,7 @@ from pathlib import Path from typing import Any -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..json_utils import compute_diff, deep_merge, set_nested_value from ..models import ProjectConfig from ..sync.manifest import Manifest, load_manifest, save_manifest @@ -291,7 +291,7 @@ def update_config( if not has_content and not has_metadata: raise KeboolaApiError( status_code=400, - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, message=( "At least one of --name, --description, --configuration, " "--configuration-file, or --set must be provided." @@ -734,7 +734,7 @@ def get_config_metadata_value( raise KeboolaApiError( message=f"Metadata key '{key}' not found on config '{component_id}/{config_id}'.", status_code=404, - error_code="NOT_FOUND", + error_code=ErrorCode.NOT_FOUND, retryable=False, ) diff --git a/src/keboola_agent_cli/services/flow_service.py b/src/keboola_agent_cli/services/flow_service.py index e221808e..51aa9c8a 100644 --- a/src/keboola_agent_cli/services/flow_service.py +++ b/src/keboola_agent_cli/services/flow_service.py @@ -14,7 +14,7 @@ import logging from typing import Any -from ..errors import KeboolaApiError +from ..errors import ErrorCode, KeboolaApiError from ..models import ProjectConfig from .base import BaseService @@ -254,7 +254,7 @@ def create_flow( raise KeboolaApiError( message=f"Flow DAG validation failed: {'; '.join(dag_errors)}", status_code=400, - error_code="INVALID_FLOW_DAG", + error_code=ErrorCode.INVALID_FLOW_DAG, retryable=False, ) @@ -324,7 +324,7 @@ def update_flow( raise KeboolaApiError( message=f"Flow DAG validation failed: {'; '.join(dag_errors)}", status_code=400, - error_code="INVALID_FLOW_DAG", + error_code=ErrorCode.INVALID_FLOW_DAG, retryable=False, ) @@ -603,7 +603,7 @@ def remove_flow_schedule( raise KeboolaApiError( message=f"Failed to delete schedules: {'; '.join(errors)}", status_code=0, - error_code="SCHEDULE_DELETE_FAILED", + error_code=ErrorCode.SCHEDULE_DELETE_FAILED, retryable=False, ) diff --git a/src/keboola_agent_cli/services/job_service.py b/src/keboola_agent_cli/services/job_service.py index e6ebe661..0bf3415c 100644 --- a/src/keboola_agent_cli/services/job_service.py +++ b/src/keboola_agent_cli/services/job_service.py @@ -18,7 +18,7 @@ KILLABLE_JOB_STATUSES, VALID_POLL_STRATEGIES, ) -from ..errors import KeboolaApiError +from ..errors import ErrorCode, KeboolaApiError from ..models import ProjectConfig from .base import BaseService @@ -341,7 +341,7 @@ def run_job( - If the local deadline elapses before the remote job finishes, issues ``kill_job`` to cancel the remote work, waits briefly for termination, and raises ``KeboolaApiError`` with - ``error_code="JOB_TIMEOUT_TERMINATED"``. If the kill call itself + ``error_code=ErrorCode.JOB_TIMEOUT_TERMINATED``. If the kill call itself fails we fall back to the original ``QUEUE_JOB_TIMEOUT`` error so the caller can tell "local gave up" from "remote was cancelled". @@ -376,13 +376,13 @@ def run_job( f"Expected one of: {sorted(VALID_POLL_STRATEGIES)}." ), status_code=0, - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) if log_tail_lines < 0: raise KeboolaApiError( message=f"log_tail_lines must be >= 0, got {log_tail_lines}.", status_code=0, - error_code="INVALID_ARGUMENT", + error_code=ErrorCode.INVALID_ARGUMENT, ) projects = self.resolve_projects([alias]) @@ -492,7 +492,7 @@ def _handle_wait_error( f"status={cancelled.get('status')!r}." ), status_code=504, - error_code="JOB_TIMEOUT_TERMINATED", + error_code=ErrorCode.JOB_TIMEOUT_TERMINATED, retryable=False, details={"job": cancelled, "logTail": tail}, ) from exc @@ -571,7 +571,7 @@ def resolve_variable_values_id( f"`--no-variables` to skip resolution." ), status_code=0, - error_code="NO_VARIABLE_ROWS", + error_code=ErrorCode.NO_VARIABLE_ROWS, ) # A row without a usable `id` would otherwise coerce to `""`, which @@ -587,7 +587,7 @@ def resolve_variable_values_id( f"with empty variable bindings." ), status_code=0, - error_code="MALFORMED_VARIABLES_ROW", + error_code=ErrorCode.MALFORMED_VARIABLES_ROW, ) return str(first_row_id) diff --git a/src/keboola_agent_cli/services/kai_service.py b/src/keboola_agent_cli/services/kai_service.py index 15f4b6c7..3cb48a1c 100644 --- a/src/keboola_agent_cli/services/kai_service.py +++ b/src/keboola_agent_cli/services/kai_service.py @@ -11,7 +11,7 @@ from kai_client import KaiClient, KaiError from ..constants import KAI_FEATURE_FLAG, KAI_REQUEST_TIMEOUT, KAI_STREAM_TIMEOUT -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from .base import BaseService logger = logging.getLogger(__name__) @@ -74,7 +74,7 @@ def _check_kai_enabled(self, alias: str) -> None: "Enable the 'AI Agent Chat' feature in project settings." ), status_code=0, - error_code="KAI_NOT_ENABLED", + error_code=ErrorCode.KAI_NOT_ENABLED, ) # ------------------------------------------------------------------ @@ -129,7 +129,7 @@ async def _ping() -> dict[str, Any]: raise KeboolaApiError( message=f"Kai ping failed: {exc.message}", status_code=0, - error_code="KAI_ERROR", + error_code=ErrorCode.KAI_ERROR, ) from exc def ask(self, alias: str, message: str) -> dict[str, Any]: @@ -161,7 +161,7 @@ async def _ask() -> dict[str, Any]: raise KeboolaApiError( message=f"Kai ask failed: {exc.message}", status_code=0, - error_code="KAI_ERROR", + error_code=ErrorCode.KAI_ERROR, ) from exc def chat_message(self, alias: str, message: str, chat_id: str | None = None) -> dict[str, Any]: @@ -198,7 +198,7 @@ async def _chat() -> dict[str, Any]: raise KeboolaApiError( message=f"Kai chat failed: {exc.message}", status_code=0, - error_code="KAI_ERROR", + error_code=ErrorCode.KAI_ERROR, ) from exc def get_history(self, alias: str, limit: int = 10) -> dict[str, Any]: @@ -238,5 +238,5 @@ async def _history() -> dict[str, Any]: raise KeboolaApiError( message=f"Kai history failed: {exc.message}", status_code=0, - error_code="KAI_ERROR", + error_code=ErrorCode.KAI_ERROR, ) from exc diff --git a/src/keboola_agent_cli/services/sharing_service.py b/src/keboola_agent_cli/services/sharing_service.py index 78655395..4f3f5639 100644 --- a/src/keboola_agent_cli/services/sharing_service.py +++ b/src/keboola_agent_cli/services/sharing_service.py @@ -14,7 +14,7 @@ from typing import Any from ..constants import ENV_KBC_MASTER_TOKEN -from ..errors import KeboolaApiError +from ..errors import ErrorCode, KeboolaApiError from ..models import ProjectConfig from .base import BaseService @@ -241,7 +241,7 @@ def unlink( message=f"Bucket '{bucket_id}' is not a linked bucket. " "Use this command only for linked (shared) buckets.", status_code=400, - error_code="NOT_LINKED_BUCKET", + error_code=ErrorCode.NOT_LINKED_BUCKET, retryable=False, ) client.delete_bucket(bucket_id=bucket_id, force=True) diff --git a/src/keboola_agent_cli/services/storage_service.py b/src/keboola_agent_cli/services/storage_service.py index 4eafb34f..221ac937 100644 --- a/src/keboola_agent_cli/services/storage_service.py +++ b/src/keboola_agent_cli/services/storage_service.py @@ -11,6 +11,7 @@ from typing import Any from ..constants import VALID_COLUMN_TYPES +from ..errors import ErrorCode from ..models import ProjectConfig from .base import BaseService @@ -672,7 +673,7 @@ def download_table( raise KeboolaApiError( message="Export job completed but no file ID in results", status_code=500, - error_code="EXPORT_NO_FILE", + error_code=ErrorCode.EXPORT_NO_FILE, retryable=False, ) @@ -683,7 +684,7 @@ def download_table( raise KeboolaApiError( message=f"No download URL for file {file_id}", status_code=500, - error_code="EXPORT_NO_URL", + error_code=ErrorCode.EXPORT_NO_URL, retryable=False, ) @@ -699,7 +700,7 @@ def download_table( "produced a single file. Re-run without --keep-slices." ), status_code=400, - error_code="NOT_SLICED", + error_code=ErrorCode.NOT_SLICED, retryable=False, ) slice_info = client.download_sliced_file_to_dir(file_detail, output_path) @@ -1121,7 +1122,7 @@ def download_file( raise KeboolaApiError( message=f"No files found matching tags: {tag_str}", status_code=404, - error_code="FILE_NOT_FOUND", + error_code=ErrorCode.FILE_NOT_FOUND, retryable=False, ) file_id = files[0]["id"] @@ -1160,7 +1161,7 @@ def download_file( raise KeboolaApiError( message=f"No download URL for file {file_id}", status_code=500, - error_code="FILE_NO_URL", + error_code=ErrorCode.FILE_NO_URL, retryable=False, ) bytes_written = client.download_file(download_url, effective_output) @@ -1375,7 +1376,7 @@ def unload_table_to_file( raise KeboolaApiError( message=f"file_type must be 'csv' or 'parquet', got {file_type!r}", status_code=400, - error_code="VALIDATION_ERROR", + error_code=ErrorCode.VALIDATION_ERROR, retryable=False, ) @@ -1400,7 +1401,7 @@ def unload_table_to_file( raise KeboolaApiError( message="Export job completed but no file ID in results", status_code=500, - error_code="EXPORT_NO_FILE", + error_code=ErrorCode.EXPORT_NO_FILE, retryable=False, ) @@ -1458,7 +1459,7 @@ def unload_table_to_file( "file is a single non-sliced CSV. Drop the flag." ), status_code=400, - error_code="NOT_SLICED", + error_code=ErrorCode.NOT_SLICED, retryable=False, ) effective_output = output_path or f"{table_short}.csv" @@ -1471,7 +1472,7 @@ def unload_table_to_file( raise KeboolaApiError( message=f"No download URL for file {file_id}", status_code=500, - error_code="FILE_NO_URL", + error_code=ErrorCode.FILE_NO_URL, retryable=False, ) bytes_written = client.download_file(download_url, effective_output) diff --git a/src/keboola_agent_cli/services/sync_service.py b/src/keboola_agent_cli/services/sync_service.py index 84ea29d3..cb03693e 100644 --- a/src/keboola_agent_cli/services/sync_service.py +++ b/src/keboola_agent_cli/services/sync_service.py @@ -33,7 +33,7 @@ STORAGE_DIR_NAME, STORAGE_SAMPLES_DIR_NAME, ) -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..sync.code_extraction import extract_code_files, merge_code_files from ..sync.config_format import ( api_config_to_local, @@ -78,6 +78,7 @@ def init_sync( alias: str, project_root: Path, git_branching: bool = False, + adopt_existing: bool = False, ) -> dict[str, Any]: """Initialize a sync working directory for a project. @@ -88,13 +89,19 @@ def init_sync( alias: Project alias from config store. project_root: Root directory for the sync working tree. git_branching: Enable git-branching mode. + adopt_existing: If True and a manifest already exists, validate it + against the alias's project_id and normalise it (idempotent + upgrade of a ``kbc``-written manifest) instead of refusing. Returns: Dict with initialization stats and created file paths. Raises: - ConfigError: If the project alias is not found. - FileExistsError: If manifest already exists (use pull instead). + ConfigError: If the project alias is not found, or if + ``adopt_existing`` is True but the manifest's project_id does + not match the alias's project. + FileExistsError: If manifest already exists and adopt_existing is + False (use ``sync pull`` to update). """ projects = self.resolve_projects([alias]) project = projects[alias] @@ -102,9 +109,12 @@ def init_sync( keboola_dir = project_root / KEBOOLA_DIR_NAME manifest_path = keboola_dir / "manifest.json" if manifest_path.exists(): + if adopt_existing: + return self._adopt_existing_manifest(alias, project_root, project) raise FileExistsError( f"Manifest already exists at {manifest_path}. " - "Use 'sync pull' to update, or delete .keboola/ to reinitialize." + "Use 'sync pull' to update, 'sync init --adopt-existing' to adopt a " + "kbc-written manifest, or delete .keboola/ to reinitialize." ) # Fetch project info from API @@ -184,6 +194,48 @@ def init_sync( "files_created": created_files, } + def _adopt_existing_manifest( + self, + alias: str, + project_root: Path, + project: Any, + ) -> dict[str, Any]: + """Validate and normalise an existing manifest written by kbc or kbagent. + + Idempotent: loads the manifest, confirms project_id matches the alias, + then saves it back through kbagent's serialiser (fills missing optional + fields with defaults, normalises camelCase keys, preserves all + existing content). + + Raises: + ConfigError: If the manifest's project_id doesn't match the alias. + """ + existing = load_manifest(project_root) + + client = self._client_factory(project.stack_url, project.token) + with client: + token_info = client.verify_token() + + if existing.project.id != token_info.project_id: + raise ConfigError( + f"Manifest project_id={existing.project.id} does not match alias " + f"'{alias}' project_id={token_info.project_id}; refusing to overwrite. " + "Check that --project points to the correct alias." + ) + + api_host = project.stack_url.replace("https://", "").rstrip("/") + save_manifest(project_root, existing) + + return { + "status": "adopted", + "project_id": token_info.project_id, + "project_alias": alias, + "api_host": api_host, + "git_branching": existing.git_branching.enabled, + "default_branch": existing.git_branching.default_branch, + "files_created": [], + } + # ------------------------------------------------------------------ # pull # ------------------------------------------------------------------ @@ -1278,7 +1330,7 @@ def _push_row_change( f"{parent_config_id} is not tracked in the manifest." ), status_code=0, - error_code="PARENT_CONFIG_NOT_TRACKED", + error_code=ErrorCode.PARENT_CONFIG_NOT_TRACKED, ) project_id = manifest.project.id if manifest.project else None diff --git a/src/keboola_agent_cli/services/workspace_service.py b/src/keboola_agent_cli/services/workspace_service.py index 8edc0ee4..36d9e84f 100644 --- a/src/keboola_agent_cli/services/workspace_service.py +++ b/src/keboola_agent_cli/services/workspace_service.py @@ -8,7 +8,7 @@ import logging from typing import Any -from ..errors import ConfigError, KeboolaApiError +from ..errors import ConfigError, ErrorCode, KeboolaApiError from ..models import ProjectConfig from .base import BaseService @@ -233,7 +233,7 @@ def _create_workspace_via_job( raise KeboolaApiError( message=f"Sandbox job completed but no workspace found for config {config_id}", status_code=500, - error_code="WORKSPACE_NOT_FOUND", + error_code=ErrorCode.WORKSPACE_NOT_FOUND, retryable=False, ) diff --git a/src/keboola_agent_cli/sync/manifest.py b/src/keboola_agent_cli/sync/manifest.py index 81749bde..6ff05938 100644 --- a/src/keboola_agent_cli/sync/manifest.py +++ b/src/keboola_agent_cli/sync/manifest.py @@ -112,7 +112,9 @@ class Manifest(BaseModel): version: int = MANIFEST_VERSION project: ManifestProject allow_target_env: bool = Field(default=True, alias="allowTargetEnv") - git_branching: ManifestGitBranching = Field(alias="gitBranching") + git_branching: ManifestGitBranching = Field( + default_factory=ManifestGitBranching, alias="gitBranching" + ) sort_by: str = Field(default="id", alias="sortBy") naming: ManifestNaming allowed_branches: list[str] = Field(default_factory=list, alias="allowedBranches") diff --git a/tests/test_e2e.py b/tests/test_e2e.py index 5ded11b4..8a4e5266 100644 --- a/tests/test_e2e.py +++ b/tests/test_e2e.py @@ -4749,3 +4749,150 @@ def test_fetch_job_events_returns_list_on_real_job(self) -> None: # A completed python-transformation job always emits at least one event # (startup + completion). Guard against an empty-but-silent regression. assert len(events) > 0 + + +# =========================================================================== + + +@pytest.mark.e2e +@skip_without_credentials +class TestE2ESyncAdoptExisting: + """E2E test for 'sync init --adopt-existing' against a real Keboola project. + + Simulates a directory that was set up by the kbc CLI (or a previous kbagent + version) by writing a minimal valid manifest, then verifies that: + 1. kbagent sync init --adopt-existing succeeds (status=adopted) + 2. kbagent sync status works on the adopted directory + """ + + @pytest.fixture(autouse=True) + def setup(self, tmp_path: Path) -> None: + self.token = os.environ[ENV_TOKEN] + raw_url = os.environ.get(ENV_URL, "connection.keboola.com") + self.url = raw_url if raw_url.startswith("https://") else f"https://{raw_url}" + self.alias = f"{RUN_ID}-adopt" + + self.config_dir = tmp_path / "config" + self.config_dir.mkdir() + self.project_dir = tmp_path / "project" + self.project_dir.mkdir() + + # Register the project + result = _invoke( + self.config_dir, + [ + "--json", + "project", + "add", + "--project", + self.alias, + "--url", + self.url, + "--token", + self.token, + ], + ) + assert result.exit_code == 0, f"project add failed: {result.output}" + + def _run(self, *args: str): + return _invoke(self.config_dir, ["--json", *args]) + + def _run_ok(self, *args: str) -> dict: + return _json_ok(self._run(*args)) + + def test_adopt_existing_manifest(self) -> None: + """init --adopt-existing adopts a kbc-style manifest; sync status works after.""" + import json as _json + + # 1. Write a kbc-style manifest for the real project. + # We first call sync init normally to learn the real project_id, then + # delete and rewrite as a "legacy" manifest. + _step(1, "sync init (normal) to learn project_id") + resp = self._run_ok( + "sync", + "init", + "--project", + self.alias, + "--directory", + str(self.project_dir), + ) + project_id = resp["data"]["project_id"] + branch_id = None + keboola_dir = self.project_dir / ".keboola" + manifest_path = keboola_dir / "manifest.json" + raw = _json.loads(manifest_path.read_text()) + if raw.get("branches"): + branch_id = raw["branches"][0]["id"] + + # 2. Rewrite as a minimal kbc-style manifest (drop gitBranching field). + _step(2, "rewrite as kbc-style manifest (drop gitBranching)") + kbc_manifest = { + "version": 2, + "project": {"id": project_id, "apiHost": self.url.replace("https://", "")}, + "allowTargetEnv": True, + "sortBy": "id", + "naming": {"branch": "{branch_name}"}, + "branches": [{"id": branch_id, "path": "main"}] if branch_id else [], + "configurations": [], + } + manifest_path.write_text(_json.dumps(kbc_manifest, indent=4), encoding="utf-8") + + # 3. sync init --adopt-existing should succeed without error. + _step(3, "sync init --adopt-existing") + resp = self._run_ok( + "sync", + "init", + "--project", + self.alias, + "--directory", + str(self.project_dir), + "--adopt-existing", + ) + inner = resp["data"] + assert inner["status"] == "adopted", f"Expected 'adopted', got {inner['status']}" + assert inner["project_id"] == project_id + assert inner["files_created"] == [] + + # 4. sync status should work on the adopted directory. + _step(4, "sync status on adopted directory") + resp = self._run_ok( + "sync", + "status", + "--directory", + str(self.project_dir), + ) + # Status should be parseable (may show no changes on an empty dir) + inner = resp["data"] + assert "modified" in inner or "unchanged" in inner or "added" in inner + + def test_adopt_existing_rejects_wrong_project(self) -> None: + """init --adopt-existing rejects a manifest with a different project_id.""" + import json as _json + + keboola_dir = self.project_dir / ".keboola" + keboola_dir.mkdir() + # Write a manifest with a clearly wrong project_id + wrong_manifest = { + "version": 2, + "project": {"id": 999999999, "apiHost": "connection.keboola.com"}, + "allowTargetEnv": True, + "sortBy": "id", + "naming": {"branch": "{branch_name}"}, + "branches": [], + "configurations": [], + } + (keboola_dir / "manifest.json").write_text(_json.dumps(wrong_manifest), encoding="utf-8") + + result = self._run( + "sync", + "init", + "--project", + self.alias, + "--directory", + str(self.project_dir), + "--adopt-existing", + ) + assert result.exit_code == 5, f"Expected exit 5, got {result.exit_code}: {result.output}" + output = _json.loads(result.output) + assert output["status"] == "error" + assert "999999999" in output["error"]["message"] diff --git a/tests/test_errors.py b/tests/test_errors.py index 50dc257e..7cf30336 100644 --- a/tests/test_errors.py +++ b/tests/test_errors.py @@ -1,6 +1,6 @@ """Tests for error types and helpers.""" -from keboola_agent_cli.errors import ConfigError, KeboolaApiError, mask_token +from keboola_agent_cli.errors import ConfigError, ErrorCode, KeboolaApiError, mask_token class TestMaskToken: @@ -91,3 +91,69 @@ def test_is_exception(self) -> None: """ConfigError is a proper Exception subclass.""" err = ConfigError(message="test") assert isinstance(err, Exception) + + +class TestErrorCode: + """Tests for the ErrorCode enum.""" + + def test_str_equality(self) -> None: + """ErrorCode members compare equal to their plain string values.""" + assert ErrorCode.QUEUE_JOB_FAILED == "QUEUE_JOB_FAILED" + assert ErrorCode.PERMISSION_DENIED == "PERMISSION_DENIED" + assert ErrorCode.CONFIG_ERROR == "CONFIG_ERROR" + + def test_is_str(self) -> None: + """ErrorCode is a subtype of str -- usable wherever a str is expected.""" + code = ErrorCode.NOT_FOUND + assert isinstance(code, str) + + def test_json_serialisation(self) -> None: + """ErrorCode serialises to its plain string value in json.dumps.""" + import json + + assert json.dumps(ErrorCode.UPLOAD_FAILED) == '"UPLOAD_FAILED"' + + def test_no_duplicate_values(self) -> None: + """Every ErrorCode member has a unique string value.""" + values = [c.value for c in ErrorCode] + assert len(values) == len(set(values)), "Duplicate ErrorCode values detected" + + def test_known_codes_present(self) -> None: + """Spot-check that key codes defined in the spec are present.""" + required = { + "INVALID_TOKEN", + "PERMISSION_DENIED", + "TIMEOUT", + "CONNECTION_ERROR", + "RETRY_EXHAUSTED", + "API_ERROR", + "NOT_FOUND", + "CONFIG_ERROR", + "QUEUE_JOB_FAILED", + "QUEUE_JOB_TIMEOUT", + "STORAGE_JOB_FAILED", + "NO_VARIABLE_ROWS", + "MALFORMED_VARIABLES_ROW", + "UPLOAD_FAILED", + "UNKNOWN_ERROR", + "ENCRYPTION_FAILED", + "WORKSPACE_NOT_FOUND", + } + member_names = {c.name for c in ErrorCode} + missing = required - member_names + assert not missing, f"Required ErrorCode members missing: {missing}" + + def test_keboolaapierror_default_uses_enum(self) -> None: + """KeboolaApiError default error_code is an ErrorCode member.""" + err = KeboolaApiError(message="oops") + assert err.error_code == ErrorCode.UNKNOWN_ERROR + assert err.error_code == "UNKNOWN_ERROR" + + def test_keboolaapierror_accepts_enum(self) -> None: + """KeboolaApiError can be constructed with an ErrorCode member.""" + err = KeboolaApiError( + message="job died", + error_code=ErrorCode.QUEUE_JOB_FAILED, + ) + assert err.error_code == "QUEUE_JOB_FAILED" + assert err.error_code == ErrorCode.QUEUE_JOB_FAILED diff --git a/tests/test_integration.py b/tests/test_integration.py index cac5d519..27d9a7fb 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -12,6 +12,8 @@ import json import os +import subprocess +import sys from pathlib import Path import pytest @@ -240,3 +242,70 @@ def test_context_command_works(self, integration_config_dir: Path) -> None: assert result.exit_code == 0 assert "kbagent" in result.output assert "--json" in result.output + + +# =========================================================================== +# CI guard: check_error_codes.py catches planted raw strings +# =========================================================================== + + +@pytest.mark.integration +class TestCheckErrorCodesGuard: + """Verify the CI guard script rejects raw error_code string literals.""" + + def test_guard_passes_on_clean_source(self) -> None: + """scripts/check_error_codes.py exits 0 on the current (clean) source.""" + result = subprocess.run( + [sys.executable, "scripts/check_error_codes.py"], + capture_output=True, + text=True, + ) + assert result.returncode == 0, ( + f"Guard failed on clean source:\n{result.stdout}\n{result.stderr}" + ) + + def test_guard_catches_planted_literal(self, tmp_path: Path) -> None: + """Guard exits 1 when a raw string literal is planted in a temp source file.""" + # Write a minimal Python file that uses a raw error_code string + planted = tmp_path / "planted.py" + planted.write_text( + "from keboola_agent_cli.errors import KeboolaApiError\n" + 'raise KeboolaApiError("oops", error_code="QUEUE_JOB_FAILED")\n', + encoding="utf-8", + ) + # Run the guard against only this file by patching SRC_ROOT via env isn't + # practical; instead verify the guard script's logic directly via import. + import ast + + source = planted.read_text(encoding="utf-8") + tree = ast.parse(source) + violations = [] + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + for kw in node.keywords: + if kw.arg == "error_code" and isinstance(kw.value, ast.Constant): + violations.append(kw.value.value) + + assert violations == ["QUEUE_JOB_FAILED"], ( + "Guard logic should detect the planted raw string literal" + ) + + def test_guard_ignores_enum_usage(self, tmp_path: Path) -> None: + """Guard logic does NOT flag error_code=ErrorCode.X (non-Constant node).""" + import ast + + source = ( + "from keboola_agent_cli.errors import ErrorCode, KeboolaApiError\n" + 'raise KeboolaApiError("oops", error_code=ErrorCode.QUEUE_JOB_FAILED)\n' + ) + tree = ast.parse(source) + violations = [] + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + for kw in node.keywords: + if kw.arg == "error_code" and isinstance(kw.value, ast.Constant): + violations.append(kw.value.value) + + assert violations == [], "Enum usage should not be flagged as a violation" diff --git a/tests/test_sync_cli.py b/tests/test_sync_cli.py index 3b0b7a8c..1190ca2f 100644 --- a/tests/test_sync_cli.py +++ b/tests/test_sync_cli.py @@ -1562,3 +1562,159 @@ def test_sync_branch_status_disabled_human(self, tmp_path: Path) -> None: assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" assert "not enabled" in result.output + + +class TestSyncInitAdoptExistingCli: + """Tests for `kbagent sync init --adopt-existing`.""" + + def test_adopt_existing_flag_present_in_help(self) -> None: + """sync init --help shows --adopt-existing flag.""" + result = runner.invoke(app, ["sync", "init", "--help"]) + assert result.exit_code == 0 + assert "--adopt-existing" in _strip_ansi(result.output) + + def test_adopt_existing_json_output(self, tmp_path: Path) -> None: + """sync init --adopt-existing --json returns adopted status.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + + store = _setup_config( + config_dir, + {"prod": {"token": TEST_TOKEN, "project_id": 258}}, + ) + + mock_sync = _make_sync_service_mock() + mock_sync.init_sync.return_value = { + "status": "adopted", + "project_id": 258, + "project_alias": "prod", + "api_host": "connection.keboola.com", + "git_branching": False, + "default_branch": "main", + "files_created": [], + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.SyncService") as MockSyncService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockSyncService.return_value = mock_sync + + result = runner.invoke( + app, + [ + "--json", + "sync", + "init", + "--project", + "prod", + "--directory", + str(tmp_path), + "--adopt-existing", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = json.loads(result.output) + assert output["status"] == "ok" + assert output["data"]["status"] == "adopted" + assert output["data"]["files_created"] == [] + + # Verify adopt_existing=True was passed through to the service + call_kwargs = mock_sync.init_sync.call_args + assert call_kwargs.kwargs.get("adopt_existing") is True + + def test_adopt_existing_human_output_shows_adopted(self, tmp_path: Path) -> None: + """sync init --adopt-existing shows 'Adopted' instead of 'Initialized'.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + + store = _setup_config( + config_dir, + {"prod": {"token": TEST_TOKEN, "project_id": 258}}, + ) + + mock_sync = _make_sync_service_mock() + mock_sync.init_sync.return_value = { + "status": "adopted", + "project_id": 258, + "project_alias": "prod", + "api_host": "connection.europe-west3.gcp.keboola.com", + "git_branching": False, + "default_branch": "main", + "files_created": [], + } + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.SyncService") as MockSyncService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockSyncService.return_value = mock_sync + + result = runner.invoke( + app, + [ + "sync", + "init", + "--project", + "prod", + "--directory", + str(tmp_path), + "--adopt-existing", + ], + ) + + assert result.exit_code == 0, f"Exit code {result.exit_code}: {result.output}" + output = _strip_ansi(result.output) + assert "Adopted" in output or "adopted" in output.lower() + + def test_adopt_existing_config_error_exits_5(self, tmp_path: Path) -> None: + """sync init --adopt-existing returns exit code 5 on project_id mismatch.""" + config_dir = tmp_path / "config" + config_dir.mkdir() + + store = _setup_config( + config_dir, + {"prod": {"token": TEST_TOKEN, "project_id": 258}}, + ) + + from keboola_agent_cli.errors import ConfigError + + mock_sync = _make_sync_service_mock() + mock_sync.init_sync.side_effect = ConfigError( + "Manifest project_id=999 does not match alias 'prod' project_id=258" + ) + + with ( + patch("keboola_agent_cli.cli.ConfigStore") as MockStore, + patch("keboola_agent_cli.cli.ProjectService") as MockProjService, + patch("keboola_agent_cli.cli.SyncService") as MockSyncService, + ): + MockStore.return_value = store + MockProjService.return_value = ProjectService(config_store=store) + MockSyncService.return_value = mock_sync + + result = runner.invoke( + app, + [ + "--json", + "sync", + "init", + "--project", + "prod", + "--adopt-existing", + "--directory", + str(tmp_path), + ], + ) + + assert result.exit_code == 5 + output = json.loads(result.output) + assert output["status"] == "error" + assert "project_id" in output["error"]["message"] diff --git a/tests/test_sync_service.py b/tests/test_sync_service.py index 75374d50..98007886 100644 --- a/tests/test_sync_service.py +++ b/tests/test_sync_service.py @@ -2197,3 +2197,181 @@ def test_handles_branch_not_found_in_api(self) -> None: assert result == "branch-44444" assert len(manifest.branches) == 2 assert manifest.branches[1].path == "branch-44444" + + +# =========================================================================== +# adopt-existing tests +# =========================================================================== + + +class TestAdoptExistingManifest: + """Tests for SyncService.init_sync(adopt_existing=True).""" + + def _kbc_style_manifest(self, project_id: int = 258) -> dict: + """Return a manifest as written by the kbc Go CLI (camelCase, gitBranching present).""" + return { + "version": 2, + "project": {"id": project_id, "apiHost": "connection.keboola.com"}, + "allowTargetEnv": True, + "sortBy": "id", + "gitBranching": {"enabled": False, "defaultBranch": "main"}, + "naming": {"branch": "{branch_name}"}, + "branches": [{"id": 12345, "path": "main"}], + "configurations": [ + { + "branchId": 12345, + "componentId": "keboola.ex-http", + "id": "cfg-001", + "path": "extractor/keboola.ex-http/my-extractor", + "rows": [], + } + ], + } + + def test_adopt_existing_normalises_kbc_manifest( + self, tmp_config_dir: Path, tmp_path: Path + ) -> None: + """adopt_existing=True loads, validates, and saves a kbc manifest without data loss.""" + mock_client = _make_sync_mock_client( + verify_token_response=SAMPLE_VERIFY_TOKEN, # project_id=258 + branches_response=SAMPLE_BRANCHES, + ) + store = setup_single_project(tmp_config_dir) + svc = SyncService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + project_root = tmp_path / "project" + project_root.mkdir() + keboola_dir = project_root / ".keboola" + keboola_dir.mkdir() + manifest_path = keboola_dir / "manifest.json" + import json as _json + + manifest_path.write_text( + _json.dumps(self._kbc_style_manifest(project_id=258)), encoding="utf-8" + ) + + result = svc.init_sync( + alias="prod", + project_root=project_root, + adopt_existing=True, + ) + + assert result["status"] == "adopted" + assert result["project_id"] == 258 + assert result["project_alias"] == "prod" + assert result["files_created"] == [] + + # Original configuration entry preserved in manifest + saved = load_manifest(project_root) + assert len(saved.configurations) == 1 + assert saved.configurations[0].component_id == "keboola.ex-http" + + def test_adopt_existing_rejects_project_id_mismatch( + self, tmp_config_dir: Path, tmp_path: Path + ) -> None: + """adopt_existing raises ConfigError when manifest project_id != alias project_id.""" + mock_client = _make_sync_mock_client( + verify_token_response=SAMPLE_VERIFY_TOKEN, # project_id=258 + branches_response=SAMPLE_BRANCHES, + ) + store = setup_single_project(tmp_config_dir) + svc = SyncService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + project_root = tmp_path / "project" + project_root.mkdir() + keboola_dir = project_root / ".keboola" + keboola_dir.mkdir() + import json as _json + + # Manifest claims a different project (id=999, not 258) + (keboola_dir / "manifest.json").write_text( + _json.dumps(self._kbc_style_manifest(project_id=999)), encoding="utf-8" + ) + + with pytest.raises(ConfigError, match="project_id=999"): + svc.init_sync(alias="prod", project_root=project_root, adopt_existing=True) + + def test_adopt_existing_falls_through_to_normal_init_when_no_manifest( + self, tmp_config_dir: Path, tmp_path: Path + ) -> None: + """adopt_existing=True with no manifest runs normal init (creates manifest).""" + mock_client = _make_sync_mock_client( + verify_token_response=SAMPLE_VERIFY_TOKEN, + branches_response=SAMPLE_BRANCHES, + ) + store = setup_single_project(tmp_config_dir) + svc = SyncService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + project_root = tmp_path / "project" + project_root.mkdir() + + result = svc.init_sync( + alias="prod", + project_root=project_root, + adopt_existing=True, + ) + + # Should behave like a normal init when no manifest exists + assert result["status"] == "initialized" + assert (project_root / ".keboola" / "manifest.json").exists() + + def test_adopt_existing_false_still_raises_on_existing_manifest( + self, tmp_config_dir: Path, tmp_path: Path + ) -> None: + """Default (adopt_existing=False) still raises FileExistsError when manifest exists.""" + mock_client = _make_sync_mock_client( + verify_token_response=SAMPLE_VERIFY_TOKEN, + branches_response=SAMPLE_BRANCHES, + ) + store = setup_single_project(tmp_config_dir) + svc = SyncService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + project_root = tmp_path / "project" + project_root.mkdir() + svc.init_sync(alias="prod", project_root=project_root) + + # Second call without adopt_existing should still fail + with pytest.raises(FileExistsError, match="--adopt-existing"): + svc.init_sync(alias="prod", project_root=project_root) + + def test_adopt_existing_idempotent(self, tmp_config_dir: Path, tmp_path: Path) -> None: + """adopt_existing is idempotent: calling it twice leaves the manifest unchanged.""" + mock_client = _make_sync_mock_client( + verify_token_response=SAMPLE_VERIFY_TOKEN, + branches_response=SAMPLE_BRANCHES, + ) + store = setup_single_project(tmp_config_dir) + svc = SyncService( + config_store=store, + client_factory=lambda url, token: mock_client, + ) + + project_root = tmp_path / "project" + project_root.mkdir() + keboola_dir = project_root / ".keboola" + keboola_dir.mkdir() + import json as _json + + (keboola_dir / "manifest.json").write_text( + _json.dumps(self._kbc_style_manifest(project_id=258)), encoding="utf-8" + ) + + result1 = svc.init_sync(alias="prod", project_root=project_root, adopt_existing=True) + result2 = svc.init_sync(alias="prod", project_root=project_root, adopt_existing=True) + + assert result1["status"] == "adopted" + assert result2["status"] == "adopted" + # Both calls return the same project data + assert result1["project_id"] == result2["project_id"] From bedeef6246c7afbd847dc544f0dd2e6f6eb436a1 Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 23 Apr 2026 14:23:30 +0200 Subject: [PATCH 7/7] docs(skill): complete 0.22.0 reference docs for #201/#202/#204 + README - sync-workflow.md: new "Adopting an existing kbc Go CLI checkout" section for sync init --adopt-existing (#201). - gotchas.md: new entries for exit code 7 (JOB_TIMEOUT_TERMINATED), --poll-strategy fixed, --log-tail-lines N, logTail response semantics (#202); new section for --deny-writes / --deny-destructive (#203); new section for sync init --adopt-existing (#201). - workspace-workflow.md: new "Orphan detection + garbage collection" section for workspace list --orphaned + workspace gc (#204). - commands-reference.md: sync init --adopt-existing flag documented. - README.md: "What it does" table + "All commands" block updated with the 15 new commands shipped in 0.22.0 (flow CRUD, config metadata, workspace GC, storage describe, project pin + firewall, queue polling parity). --- README.md | 31 ++++++++++------ .../kbagent/references/commands-reference.md | 2 +- .../skills/kbagent/references/gotchas.md | 28 ++++++++++++++- .../kbagent/references/sync-workflow.md | 28 +++++++++++++++ .../kbagent/references/workspace-workflow.md | 35 +++++++++++++++++++ 5 files changed, 111 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index b97cf962..fc8815a2 100644 --- a/README.md +++ b/README.md @@ -69,18 +69,19 @@ kbagent workspace query --project prod --workspace-id WS_ID \ | Area | What you get | |------|-------------| | **Multi-project** | All read commands query every connected project in parallel. One command, all projects. | -| **Configurations** | List, search, inspect, scaffold, update, delete configs. Full-text search across all config bodies. | -| **Jobs** | List, inspect, run with `--wait` polling and `--timeout`. Row-level execution for multi-row configs. | -| **Storage** | Buckets, tables, files -- full CRUD. Upload CSV (auto-creates bucket+table). Download by file ID or by tag. | -| **Dev branches** | Create a branch, activate it, and every command auto-targets it. Storage, MCP, sync -- everything follows. | -| **Sync & GitOps** | Pull configs as YAML, edit in IDE, push back. SQL/Python extracted as real files. Diff and status tracking. | +| **Configurations** | List, search, inspect, scaffold, update, delete configs. Full-text search across all config bodies (incl. rows). Metadata CRUD + folder grouping. | +| **Jobs** | List, inspect, run with `--wait` polling (exponential curve), `--timeout` auto-kill, log tail on failure. Row-level execution for multi-row configs. | +| **Flows** | Create, update, delete orchestrator/flow configs with phase/task DAG validation. Attach cron schedules (timezone + enabled/disabled state). | +| **Storage** | Buckets, tables, files -- full CRUD. Upload CSV (auto-creates bucket+table). Download by file ID or by tag. Descriptions on buckets/tables/columns (batch-applicable from YAML). | +| **Dev branches** | Create a branch, activate it, and every command auto-targets it. Storage writes, MCP, sync -- everything follows. Storage reads default to production (safer). | +| **Sync & GitOps** | Pull configs as YAML, edit in IDE, push back. SQL/Python extracted as real files. Diff and status tracking. Adopt existing kbc Go CLI checkouts (`sync init --adopt-existing`). | | **MCP tools** | Call `keboola-mcp-server` tools with auto-expand, multi-project fan-out, branch propagation, schema validation. | -| **Workspaces** | Create Snowflake/BQ workspace, load tables, run SQL. Create from transformation config for instant debugging. | +| **Workspaces** | Create Snowflake/BQ workspace, load tables, run SQL. Create from transformation config for instant debugging. Orphan detection + garbage collection. | | **Sharing** | Cross-project bucket sharing with org/project/user access control. Share, link, unlink. | | **Lineage** | Column-level dependency analysis across projects. SQL/Python parsing, AI-enhanced detection, interactive web browser, Mermaid/HTML/ER export. | | **Kai (AI Assistant)** | Ask Keboola's built-in AI questions about your project. One-shot or chat sessions with full MCP context. | | **Encryption** | Encrypt secrets (`#password`, `#api_token`) via Keboola Encryption API. Works with sync push and MCP. | -| **Permissions** | Firewall for AI agents: read-only, deny-writes, deny-destructive. Code-level enforcement, not prompt tricks. | +| **Permissions** | Firewall for AI agents: read-only, deny-writes, deny-destructive (session-only flags or persisted policy). Project pin + `KBAGENT_PROJECT` env override. Code-level enforcement, stable `ErrorCode` enum, not prompt tricks. | | **Auto-update** | Self-updates on startup. "What's new" after each update. Full changelog via `kbagent changelog`. | ## Setup options @@ -109,25 +110,33 @@ Run `kbagent doctor` to verify your setup. Full command reference with flags: [SKILL.md](plugins/kbagent/skills/kbagent/SKILL.md) ``` -kbagent project add | list | remove | edit | status | refresh +kbagent project add | list | remove | edit | status | refresh | use | current kbagent org setup kbagent component list | detail kbagent config list | detail | search | update | rename | delete | new -kbagent job list | detail | run + metadata-list | get-metadata | set-metadata | delete-metadata | set-folder + variables-set | variables-get | variables-clear +kbagent job list | detail | run | terminate +kbagent flow list | detail | schema | new | update | delete | schedule | schedule-remove kbagent storage buckets | bucket-detail | create-bucket | delete-bucket tables | table-detail | create-table | upload-table | download-table | delete-table | delete-column + describe-bucket | describe-table | describe-column | describe-batch files | file-detail | file-upload | file-download | file-tag | file-delete load-file | unload-table -kbagent sharing list | share | unshare | link | unlink +kbagent sharing list | share | unshare | link | unlink | edges kbagent lineage build | show | info | server kbagent branch list | create | use | reset | delete | merge -kbagent workspace create | list | detail | delete | password | load | query | from-transformation + metadata-list | metadata-get | metadata-set | metadata-delete +kbagent workspace create | list | detail | delete | password | load | query | from-transformation | gc kbagent tool list | call kbagent sync init | pull | status | diff | push | branch-link | branch-unlink | branch-status kbagent kai ping | ask | chat | history kbagent encrypt values kbagent permissions list | show | set | reset | check kbagent init | context | doctor | version | update | changelog + +# Global flags: --json, --verbose, --no-color, --config-dir, --hint client|service +# --deny-writes, --deny-destructive (session-only firewall) ``` ## Documentation diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index a4314603..ff7ba0f4 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -136,7 +136,7 @@ All commands support `--json` for structured output. Multi-project flags (`--pro - `flow schedule-remove --project NAME --flow-id ID [--component-id ID] [--branch ID] [--yes]` -- remove all cron schedules attached to a flow; idempotent ## Sync (GitOps) -- `sync init --project ALIAS [--directory DIR] [--git-branching]` -- initialize sync working directory +- `sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing]` -- initialize sync working directory; `--adopt-existing` (since v0.22.0) adopts a `.keboola/manifest.json` already written by the kbc Go CLI without overwriting (idempotent; validates `project_id` against the alias token) - `sync pull --project ALIAS [--all-projects] [--force] [--dry-run] [--with-samples] [--no-storage] [--no-jobs] [--job-limit N]` -- download configs to local files. For large projects (>100 configs), automatically fetches jobs per-config when the grouped API limit is insufficient - `sync push --project ALIAS [--all-projects] [--dry-run] [--force] [--allow-plaintext-on-encrypt-failure]` -- push local changes (auto-encrypts secrets, fails if encryption fails) - `sync diff --project ALIAS [--all-projects]` -- 3-way diff (local vs base vs remote), detects conflicts diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index ab241f4d..00a30ccb 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -123,8 +123,34 @@ One project failing does not block others. Check the `errors` array: | 1 | General error | | 2 | Usage error (invalid arguments) | | 3 | Authentication error (invalid or expired token) | -| 4 | Network error (timeout, unreachable) | +| 4 | Network error (timeout, unreachable) -- includes `QUEUE_JOB_TIMEOUT` (local gave up AND the remote-kill attempt failed; the remote job may still be running) | | 5 | Configuration error (corrupt config, missing alias) | +| 6 | Permission denied (blocked by firewall / `--deny-writes` / `--deny-destructive`) | +| 7 | `JOB_TIMEOUT_TERMINATED` -- `job run --timeout` elapsed AND the remote job was successfully cancelled (since 0.22.0). Scripts can distinguish "we killed it" from "it failed on its own" (exit 1) from "it's still running" (exit 4). | + +## `job run --wait` polling + log tail (since 0.22.0) + +- Polling follows an exponential curve by default: **2s x 30 -> 5s x 48 -> 15s forever**. For a short test job or a test that needs fast turnaround, pass `--poll-strategy fixed` to force the legacy 1s fixed interval. +- On terminal non-success (`error` / `warning` / `terminated`), kbagent fetches the last N Storage Events and attaches them as `logTail` on the response. Controlled by `--log-tail-lines N` (default 200, max 5000, `0` disables). + - **Errors:** `error.details.logTail` carries the tail when the job surfaces as an exception (exit 1 / `QUEUE_JOB_FAILED`, exit 4 / `QUEUE_JOB_TIMEOUT`). + - **Non-error terminals** (`warning` / `terminated`): `logTail` is attached to the top-level result dict (exit 0). +- `--timeout N` is a **local** deadline. When it elapses, kbagent issues `POST /jobs/{id}/kill` against the Queue API. Two outcomes: + - Kill succeeded -> exit **7** with `details.job` + `details.logTail`. The remote is definitely cancelled. + - Kill failed -> exit **4** with `details.logTail`, `retryable=True`. The remote **may still be running**; investigate before retrying. +- Inspecting events outside of `job run`: `kbagent job detail --project X --job-id N` does not fetch the log tail. To get the raw event stream, call the Storage Events API directly (`GET /v2/storage/events?runId=`) with the project token. + +## `--deny-writes` / `--deny-destructive` firewall (since 0.22.0) + +- Session-only. Flags synthesize a `PermissionPolicy` for the current invocation and merge it with any persisted policy in `config.json`. **Never** written to disk. +- Classes: `--deny-writes` blocks `cli:write` + `tool:write` (covers write+destructive+admin). `--deny-destructive` is narrower -- blocks only `cli:destructive` + `tool:destructive`; pure write ops like `storage create-bucket` stay allowed. +- Blocked operation exits **6** with `error.code = PERMISSION_DENIED`. Read commands stay unaffected. +- Safe to run under either flag without mutating the saved policy -- useful when your agent needs a one-shot read-only run on a machine with a write-enabled config. + +## `sync init --adopt-existing` (since 0.22.0) + +- Adopts a `.keboola/manifest.json` written by the kbc Go CLI **in place** instead of overwriting. Idempotent; re-running is a no-op. +- Validates `project_id` from the manifest against the token via `verify_token`. Mismatch exits 5 (`CONFIG_ERROR`) with guidance -- never silently adopts someone else's checkout. +- If no manifest exists, `--adopt-existing` falls through to the normal init path (no error). ## Token handling diff --git a/plugins/kbagent/skills/kbagent/references/sync-workflow.md b/plugins/kbagent/skills/kbagent/references/sync-workflow.md index 6616ee90..0b44247a 100644 --- a/plugins/kbagent/skills/kbagent/references/sync-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/sync-workflow.md @@ -67,6 +67,34 @@ kbagent sync push --all-projects # apply Each project gets its own subdirectory (named by alias). Projects are processed in parallel. +## Adopting an existing kbc Go CLI checkout (since v0.22.0) + +If you already have a `.keboola/manifest.json` produced by the official +`kbc` Go CLI (keboola-as-code), `kbagent` can adopt it in place instead of +overwriting: + +```bash +cd /path/to/existing-kbc-checkout + +# Adopt the manifest as-is; validates project_id against the alias token +kbagent sync init --project prod --adopt-existing +``` + +Behavior: + +- **Idempotent.** The existing `manifest.json` is re-used, not rewritten. + Re-running `--adopt-existing` is a no-op. +- **Validated.** `project_id` from the manifest is checked against the + token's project via `verify_token`. A mismatch exits 5 (`CONFIG_ERROR`) + with a clear message -- no silent adoption of someone else's checkout. +- **Fall-through.** If no manifest exists, `--adopt-existing` falls + through to the normal init path. +- **Without the flag**, `sync init` still refuses to overwrite an + existing manifest (prior behavior unchanged). + +Use this when migrating a team from the Go CLI to kbagent without +re-pulling all configs. + ## Single-project workflow ```bash diff --git a/plugins/kbagent/skills/kbagent/references/workspace-workflow.md b/plugins/kbagent/skills/kbagent/references/workspace-workflow.md index 1b1283fd..1984d1f6 100644 --- a/plugins/kbagent/skills/kbagent/references/workspace-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/workspace-workflow.md @@ -130,3 +130,38 @@ fully-qualified Snowflake paths (e.g. `"sapi_1507"."in.c-shared-data"."my-table" - **Quoting**: Snowflake converts unquoted identifiers to UPPERCASE. Always double-quote database, schema, and table names -- Keboola names are typically lowercase (e.g. `"sapi_901"."in.c-main"."users"`) - **Query Service**: uses Storage API token for auth -- no Snowflake credentials needed in the query command - **Transactional mode**: add `--transactional` to wrap SQL in a transaction + +## Orphan detection + garbage collection (since v0.22.0) + +Workspaces are backed by `keboola.sandboxes` configs. When a config is deleted +out-of-band (UI cleanup, another CLI, force-delete script), the workspace +record itself can linger. These are **orphaned** workspaces -- they have a +workspace row but no sandbox config. + +```bash +# Show only orphaned workspaces +kbagent workspace list --project prod --orphaned + +# Preview what would be deleted (no side effects) +kbagent workspace gc --project prod --dry-run + +# Delete all orphaned workspaces with confirmation +kbagent workspace gc --project prod + +# Skip interactive confirmation (useful in CI or agent workflows) +kbagent workspace gc --project prod --yes +``` + +Behavior: + +- **`workspace list --orphaned`** filters the normal list to workspaces whose + sandbox config cannot be resolved. Output shape matches `workspace list`. +- **`workspace gc`** deletes each orphaned workspace one by one. Per-workspace + failures accumulate into `errors[]` without stopping the batch -- one + locked sandbox does not prevent the rest from being cleaned up. +- **`--dry-run`** surfaces the would-be-deleted list via `data.would_delete[]` + in JSON mode and a Rich table in human mode. +- Multi-project: `workspace list --orphaned` / `workspace gc` accept + repeatable `--project` or run against all connected projects when omitted. +- Registered as `destructive` in the permission engine -- blocked by + `--deny-destructive` / `--deny-writes`.