Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def
- `storage buckets [--project NAME] [--branch ID]` -- list buckets with sharing/linked info (branch-aware)
- `storage bucket-detail --project NAME --bucket-id ID [--branch ID]` -- bucket detail with backend-native direct-access paths (branch-aware). Output adapts to backend: Snowflake -> `snowflake_database` / `snowflake_schema` / per-table `snowflake_path` quoted with `"..."`. BigQuery -> `bigquery_dataset` (and `bigquery_project` when surfaced via API `databaseName`) / per-table `bigquery_path` quoted with backticks. Always-present backend-agnostic keys: `sql_dialect` (`"snowflake"` / `"bigquery"`) and per-table `sql_path` -- prefer these in agent code instead of branching on backend yourself
- `storage tables [--project NAME ...] [--bucket-id ID] [--branch ID] [--include-usage]` -- list tables across all connected projects in parallel (multi-project by default, same as `storage buckets`); repeat `--project` to target a subset; `--bucket-id` is applied independently per project (missing buckets become per-project errors); `--branch` requires exactly one `--project`. `--include-usage` (0.88.0+) adds `used_by` per table: the configurations naming it in their **storage input/output mapping** only -- a table id inside a transformation's SQL is NOT a reference. Costs one extra component listing per project (not per table), which is the slow call in a big project; unreadable components degrade to an empty `used_by`
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped"
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped". Since 0.88.0 (#624) it also RESOLVES column descriptions written by anyone -- the UI, a component, or kbagent -- into `column_details[].description`, with precedence native definition -> `columnMetadata` `KBC.description` -> legacy flat `KBC.column.*` (an alias table falls back to the source table's `columnMetadata`, matching the MCP server). The response always carries `legacy_column_descriptions`, naming the columns still backed by the pre-0.88.0 convention -- human mode warns and points at `storage describe-migrate`. Reading never writes, so it is safe under a read-only token or `--deny-writes`
- `storage create-bucket --project NAME --stage STAGE --name NAME [--description D] [--backend B] [--branch ID]` -- create bucket (branch-aware). With `--branch ID` on a project lacking the `storage-branches` feature (legacy fake-branch), response carries `legacy_branch_storage: true` and human mode prints a warning -- the runner will create a parallel `out.c-<branch_id>-*` bucket at job time. See `storage-types-workflow.md`
- `storage create-table --project NAME --bucket-id ID --name NAME [--column col:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists]` -- create typed table. Base types `STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP` plus native backend types with length (`VARCHAR(40)`, `NUMBER(18,2)`, `TIMESTAMP_TZ`, `VARIANT`, etc.) -- type/length validation delegated to the Storage API. `--not-null` marks a column `nullable=false`; `--default NAME=VALUE` sets a DEFAULT expression (booleans must be lowercase `true`/`false`). In a dev branch, the target bucket is auto-materialized if it has not yet been written to there -- response surfaces this via `auto_created_bucket: bool`. On legacy fake-branch projects (no `storage-branches` feature), `legacy_branch_storage: true` flags that the runner will use a separate `out.c-<branch_id>-*` bucket at job time. `--if-not-exists` (0.47.0+) turns a duplicate-display-name failure into `action: skipped` when the table really exists at the expected id (safe for parallel workers). Since 0.47.1 the skipped envelope reports the EXISTING table's actual `columns`/`primary_key`/`name`, mirrors the request under `requested_columns`/`requested_primary_key`, and sets `schema_drift: true` when they diverge. **`--source-table-id` (0.66.0+, BigQuery only)** copies an existing table's data into the requested partition/clustering layout instead of building from `--column` (schema derived from source -> `--column`/`--not-null`/`--default` forbidden; the two are mutually exclusive). This is the supported way to repartition a populated BigQuery table -- then promote it with `storage swap-tables`. Partition/clustering flags (`--time-partitioning-*`, `--range-partitioning-*`, `--clustering-field`) also work on a plain `--column` create (BigQuery only); time vs range partitioning are mutually exclusive and range bounds are strings. When any source/partition/clustering flag is used, a one-call backend pre-flight rejects non-BigQuery projects (exit 2) before the create. See `storage-types-workflow.md`
- `storage upload-table --project NAME --table-id ID --file PATH [--incremental] [--branch ID]` -- upload CSV (branch-aware)
Expand Down Expand Up @@ -262,7 +262,7 @@ Lifecycle for `keboola.data-apps`. Combines Storage API (config body, git block,
- `data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]` -- list secret keys + derived runtime env-var names. Never echoes encrypted ciphertext in full. `--show-fingerprint` opt-in for a short ciphertext fingerprint.
- `data-app secrets-get --project ALIAS --app-id ID --key 'KEY' [--branch ID]` -- show ONE key from `parameters.dataApp.secrets`. The leading `#` is OPTIONAL (since v0.43.9); the block holds both encrypted secrets (`#`) and plain unencrypted env-var values, and `secrets-list` enumerates both. For an ENCRYPTED secret it stays metadata-only (`encrypted: true`, `value: null`, fingerprint/prefix) -- the decrypted plaintext is NEVER echoed (Encryption API is one-way). For a PLAIN value it returns the literal value (`encrypted: false`), which is already visible via `config detail`. NOT_FOUND on absent key (exact match, no `#KEY`<->`KEY` fuzzing); never enumerates siblings.
- `data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run]` -- destructive (can break a running app at next deploy). Leading `#` OPTIONAL (since v0.43.9): removes both encrypted secrets and plain env-var keys. Idempotent: missing keys exit 0 with `removed: 0`.
- `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures.
- `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures. Since 0.88.0 (#636) the two `setup.sh` rules (`golden-rule.setup-sh-no-pip`, `golden-rule.setup-sh-uv-sync`) match against **comment-stripped code**: a comment reading `# never pip install` no longer BLOCKS, and a comment merely mentioning `uv sync` no longer satisfies the uv-sync rule
- `data-app git-repo --project NAME --app-id ID` (since 0.63.3) -- show the clone URLs (`ssh_url` / `https_url`) of the app's configured git repo + `is_managed_git_repo` (sandboxes-service `GET /apps/{id}/git-repo`). Read-only, project storage token only. **GOTCHA**: returns 409 `no Git repository configured` until the app has been DEPLOYED at least once -- the git block is synced from the Storage config into the Data Science app record at deploy time; a `--no-deploy` app has no git repo from the service's point of view.
- `data-app git-credentials --project NAME --app-id ID` (since 0.63.3) -- list the credentials of the app's MANAGED git repo (`id`, `type`, `permissions`, `name`, `owner_admin_id`, `created_at`). The secret is NEVER returned here. Needs an admin storage token; external repos have none.
- `data-app git-credentials-create --project NAME --app-id ID --type ssh_key|http_token --permissions readOnly|readWrite [--public-key KEY | --public-key-file PATH] [--name LABEL] [--yes]` (since 0.63.3) -- mint a git credential for the app's MANAGED git repo. `ssh_key` requires a public key; `http_token` returns a ONE-TIME secret (shown once, never retrievable again -- mirrors `data-app password`). Needs an admin storage token. Apps from `data-app create --git-repo` are EXTERNAL => 409 `no managed Git repository`. Confirmation unless `--yes`/`--json`. For a managed-repo app this credential authenticates YOUR `git push` of the code; the deploy itself uses the platform's injected clone credentials -- no further wiring needed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,10 @@ kbagent data-app validate-repo \
Walks the repo via the GitHub Contents + Trees API and emits
BLOCKING / WARN / OK per check (Golden-Rule structure, no `pip install`
in `setup.sh`, `requires-python` consistency, nginx/app port match,
etc.). Each check carries a citation back to the help-doc anchor
etc.). The two `setup.sh` rules match **comment-stripped code** *(since
v0.88.0)*, so documenting the rule in a comment (`# never pip install`)
no longer trips it -- and, in the other direction, a commented-out
`uv sync` no longer satisfies the dependency-install rule. Each check carries a citation back to the help-doc anchor
(<https://help.keboola.com/data-apps/python-js/>). Run before
`data-app create` so you don't burn a deploy cycle on a misconfigured
repo. Public repos: drop `--git-pat-env` and use `--git-public`. Total
Expand Down
14 changes: 14 additions & 0 deletions plugins/kbagent/skills/kbagent/references/gotchas.md
Original file line number Diff line number Diff line change
Expand Up @@ -1637,6 +1637,20 @@ events and emits a final `done` SSE frame mirroring the same record.
per-type canon citations. Tracked as a follow-up.
- **GitHub-only.** GitLab / Bitbucket support is a follow-up. Calling
with a non-GitHub URL exits 2 / `INVALID_ARGUMENT`.
- **The `setup.sh` rules read CODE, not comments** *(since v0.88.0)*. Both
`golden-rule.setup-sh-no-pip` and `golden-rule.setup-sh-uv-sync` strip shell
comments before matching. Before 0.88.0 they grepped the raw file, which broke
in both directions: a script whose comment read
`# always uv sync here, never pip install` was rejected as **BLOCKING** --
penalising exactly the author who documented the rule -- while a script whose
only mention of `uv sync` was in a comment **satisfied** the uv-sync rule
despite installing nothing. If you are advising on a repo validated by an
older kbagent, a BLOCKING `setup-sh-no-pip` may be about a comment; check the
file before telling anyone to change what the script runs.
- The stripper is deliberately not a shell parser: it tracks single/double
quotes, so a `#` inside a string survives, but heredocs and `${...#...}`
expansions are out of scope. A `pip install` hidden inside a heredoc is not
detected -- the rule is a pre-flight heuristic, not a guarantee.
- Exit 0 on all checks <= WARN; exit 1 on any BLOCKING. `--strict`
treats WARNs as failures (exit 1) for CI gating.
- **Reading the build / runtime log** is now available via
Expand Down