From ea28f2e98e4d920e462bed0909f544627d166362 Mon Sep 17 00:00:00 2001 From: Petr Date: Sat, 22 Aug 2026 20:44:16 +0200 Subject: [PATCH] docs(plugin): close the reference gaps a 0.88.0 audit found Audited every 0.88.0 changelog bullet against the agent-facing reference surfaces (commands-reference.md, gotchas.md, the *-workflow.md set, keboola-expert.md). Everything was covered except one change, plus two entries that were accurate but incomplete. - gotchas: `data-app validate-repo`'s two setup.sh rules now read comment-stripped code (#636). This had NO coverage anywhere. It matters to an agent because it broke in both directions before 0.88.0: a script whose comment read `# never pip install` was rejected as BLOCKING, and a script whose only `uv sync` was in a comment satisfied the dependency-install rule while installing nothing. An agent advising on a repo validated by an older kbagent needs to know a BLOCKING may be about a comment. The stripper's limits (not a shell parser; heredocs out of scope) are stated too, so nobody reads it as a guarantee. - commands-reference: the same note on the `validate-repo` entry. - commands-reference: the `table-detail` entry documented #621's `definition` thoroughly but said nothing about #624's READ side -- the resolution precedence into `column_details[].description` and the `legacy_column_descriptions` field. The `describe-column` entry pointed at table-detail for read-back; table-detail did not advertise it. - data-app-workflow: the same comment-stripping note in the pre-flight validation section. No behaviour change, docs only. `check_version_gates.py` now resolves 419 markers (up from 418). --- .../kbagent/references/commands-reference.md | 4 ++-- .../skills/kbagent/references/data-app-workflow.md | 5 ++++- .../kbagent/skills/kbagent/references/gotchas.md | 14 ++++++++++++++ 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/plugins/kbagent/skills/kbagent/references/commands-reference.md b/plugins/kbagent/skills/kbagent/references/commands-reference.md index 2b04ad91..d420c3ec 100644 --- a/plugins/kbagent/skills/kbagent/references/commands-reference.md +++ b/plugins/kbagent/skills/kbagent/references/commands-reference.md @@ -159,7 +159,7 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def - `storage buckets [--project NAME] [--branch ID]` -- list buckets with sharing/linked info (branch-aware) - `storage bucket-detail --project NAME --bucket-id ID [--branch ID]` -- bucket detail with backend-native direct-access paths (branch-aware). Output adapts to backend: Snowflake -> `snowflake_database` / `snowflake_schema` / per-table `snowflake_path` quoted with `"..."`. BigQuery -> `bigquery_dataset` (and `bigquery_project` when surfaced via API `databaseName`) / per-table `bigquery_path` quoted with backticks. Always-present backend-agnostic keys: `sql_dialect` (`"snowflake"` / `"bigquery"`) and per-table `sql_path` -- prefer these in agent code instead of branching on backend yourself - `storage tables [--project NAME ...] [--bucket-id ID] [--branch ID] [--include-usage]` -- list tables across all connected projects in parallel (multi-project by default, same as `storage buckets`); repeat `--project` to target a subset; `--bucket-id` is applied independently per project (missing buckets become per-project errors); `--branch` requires exactly one `--project`. `--include-usage` (0.88.0+) adds `used_by` per table: the configurations naming it in their **storage input/output mapping** only -- a table id inside a transformation's SQL is NOT a reference. Costs one extra component listing per project (not per table), which is the slow call in a big project; unreadable components degrade to an empty `used_by` -- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped" +- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped". Since 0.88.0 (#624) it also RESOLVES column descriptions written by anyone -- the UI, a component, or kbagent -- into `column_details[].description`, with precedence native definition -> `columnMetadata` `KBC.description` -> legacy flat `KBC.column.*` (an alias table falls back to the source table's `columnMetadata`, matching the MCP server). The response always carries `legacy_column_descriptions`, naming the columns still backed by the pre-0.88.0 convention -- human mode warns and points at `storage describe-migrate`. Reading never writes, so it is safe under a read-only token or `--deny-writes` - `storage create-bucket --project NAME --stage STAGE --name NAME [--description D] [--backend B] [--branch ID]` -- create bucket (branch-aware). With `--branch ID` on a project lacking the `storage-branches` feature (legacy fake-branch), response carries `legacy_branch_storage: true` and human mode prints a warning -- the runner will create a parallel `out.c--*` bucket at job time. See `storage-types-workflow.md` - `storage create-table --project NAME --bucket-id ID --name NAME [--column col:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists]` -- create typed table. Base types `STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP` plus native backend types with length (`VARCHAR(40)`, `NUMBER(18,2)`, `TIMESTAMP_TZ`, `VARIANT`, etc.) -- type/length validation delegated to the Storage API. `--not-null` marks a column `nullable=false`; `--default NAME=VALUE` sets a DEFAULT expression (booleans must be lowercase `true`/`false`). In a dev branch, the target bucket is auto-materialized if it has not yet been written to there -- response surfaces this via `auto_created_bucket: bool`. On legacy fake-branch projects (no `storage-branches` feature), `legacy_branch_storage: true` flags that the runner will use a separate `out.c--*` bucket at job time. `--if-not-exists` (0.47.0+) turns a duplicate-display-name failure into `action: skipped` when the table really exists at the expected id (safe for parallel workers). Since 0.47.1 the skipped envelope reports the EXISTING table's actual `columns`/`primary_key`/`name`, mirrors the request under `requested_columns`/`requested_primary_key`, and sets `schema_drift: true` when they diverge. **`--source-table-id` (0.66.0+, BigQuery only)** copies an existing table's data into the requested partition/clustering layout instead of building from `--column` (schema derived from source -> `--column`/`--not-null`/`--default` forbidden; the two are mutually exclusive). This is the supported way to repartition a populated BigQuery table -- then promote it with `storage swap-tables`. Partition/clustering flags (`--time-partitioning-*`, `--range-partitioning-*`, `--clustering-field`) also work on a plain `--column` create (BigQuery only); time vs range partitioning are mutually exclusive and range bounds are strings. When any source/partition/clustering flag is used, a one-call backend pre-flight rejects non-BigQuery projects (exit 2) before the create. See `storage-types-workflow.md` - `storage upload-table --project NAME --table-id ID --file PATH [--incremental] [--branch ID]` -- upload CSV (branch-aware) @@ -262,7 +262,7 @@ Lifecycle for `keboola.data-apps`. Combines Storage API (config body, git block, - `data-app secrets-list --project ALIAS --app-id ID [--branch ID] [--show-fingerprint]` -- list secret keys + derived runtime env-var names. Never echoes encrypted ciphertext in full. `--show-fingerprint` opt-in for a short ciphertext fingerprint. - `data-app secrets-get --project ALIAS --app-id ID --key 'KEY' [--branch ID]` -- show ONE key from `parameters.dataApp.secrets`. The leading `#` is OPTIONAL (since v0.43.9); the block holds both encrypted secrets (`#`) and plain unencrypted env-var values, and `secrets-list` enumerates both. For an ENCRYPTED secret it stays metadata-only (`encrypted: true`, `value: null`, fingerprint/prefix) -- the decrypted plaintext is NEVER echoed (Encryption API is one-way). For a PLAIN value it returns the literal value (`encrypted: false`), which is already visible via `config detail`. NOT_FOUND on absent key (exact match, no `#KEY`<->`KEY` fuzzing); never enumerates siblings. - `data-app secrets-remove --project ALIAS --app-id ID --key 'KEY' [--key ...] [--branch ID] [--yes] [--dry-run]` -- destructive (can break a running app at next deploy). Leading `#` OPTIONAL (since v0.43.9): removes both encrypted secrets and plain env-var keys. Idempotent: missing keys exit 0 with `removed: 0`. -- `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures. +- `data-app validate-repo --git-repo URL [--git-branch BRANCH] [--git-public/--no-git-public] [--git-pat-env VAR | --git-pat-file PATH] [--type python-js] [--strict]` -- pre-flight Golden-Rule check for a data-app git repo (https://help.keboola.com/data-apps/python-js/). GitHub-only; ≤5 API calls (1 tree + ≤4 contents) regardless of repo size. `--type` restricted to `python-js` in 0.28.0; streamlit / pure-Python / R / Node-only follow-up. `--strict` treats WARNs as failures. Since 0.88.0 (#636) the two `setup.sh` rules (`golden-rule.setup-sh-no-pip`, `golden-rule.setup-sh-uv-sync`) match against **comment-stripped code**: a comment reading `# never pip install` no longer BLOCKS, and a comment merely mentioning `uv sync` no longer satisfies the uv-sync rule - `data-app git-repo --project NAME --app-id ID` (since 0.63.3) -- show the clone URLs (`ssh_url` / `https_url`) of the app's configured git repo + `is_managed_git_repo` (sandboxes-service `GET /apps/{id}/git-repo`). Read-only, project storage token only. **GOTCHA**: returns 409 `no Git repository configured` until the app has been DEPLOYED at least once -- the git block is synced from the Storage config into the Data Science app record at deploy time; a `--no-deploy` app has no git repo from the service's point of view. - `data-app git-credentials --project NAME --app-id ID` (since 0.63.3) -- list the credentials of the app's MANAGED git repo (`id`, `type`, `permissions`, `name`, `owner_admin_id`, `created_at`). The secret is NEVER returned here. Needs an admin storage token; external repos have none. - `data-app git-credentials-create --project NAME --app-id ID --type ssh_key|http_token --permissions readOnly|readWrite [--public-key KEY | --public-key-file PATH] [--name LABEL] [--yes]` (since 0.63.3) -- mint a git credential for the app's MANAGED git repo. `ssh_key` requires a public key; `http_token` returns a ONE-TIME secret (shown once, never retrievable again -- mirrors `data-app password`). Needs an admin storage token. Apps from `data-app create --git-repo` are EXTERNAL => 409 `no managed Git repository`. Confirmation unless `--yes`/`--json`. For a managed-repo app this credential authenticates YOUR `git push` of the code; the deploy itself uses the platform's injected clone credentials -- no further wiring needed. diff --git a/plugins/kbagent/skills/kbagent/references/data-app-workflow.md b/plugins/kbagent/skills/kbagent/references/data-app-workflow.md index d7cf6bef..b8bcf958 100644 --- a/plugins/kbagent/skills/kbagent/references/data-app-workflow.md +++ b/plugins/kbagent/skills/kbagent/references/data-app-workflow.md @@ -228,7 +228,10 @@ kbagent data-app validate-repo \ Walks the repo via the GitHub Contents + Trees API and emits BLOCKING / WARN / OK per check (Golden-Rule structure, no `pip install` in `setup.sh`, `requires-python` consistency, nginx/app port match, -etc.). Each check carries a citation back to the help-doc anchor +etc.). The two `setup.sh` rules match **comment-stripped code** *(since +v0.88.0)*, so documenting the rule in a comment (`# never pip install`) +no longer trips it -- and, in the other direction, a commented-out +`uv sync` no longer satisfies the dependency-install rule. Each check carries a citation back to the help-doc anchor (). Run before `data-app create` so you don't burn a deploy cycle on a misconfigured repo. Public repos: drop `--git-pat-env` and use `--git-public`. Total diff --git a/plugins/kbagent/skills/kbagent/references/gotchas.md b/plugins/kbagent/skills/kbagent/references/gotchas.md index 62a57481..22a88932 100644 --- a/plugins/kbagent/skills/kbagent/references/gotchas.md +++ b/plugins/kbagent/skills/kbagent/references/gotchas.md @@ -1637,6 +1637,20 @@ events and emits a final `done` SSE frame mirroring the same record. per-type canon citations. Tracked as a follow-up. - **GitHub-only.** GitLab / Bitbucket support is a follow-up. Calling with a non-GitHub URL exits 2 / `INVALID_ARGUMENT`. +- **The `setup.sh` rules read CODE, not comments** *(since v0.88.0)*. Both + `golden-rule.setup-sh-no-pip` and `golden-rule.setup-sh-uv-sync` strip shell + comments before matching. Before 0.88.0 they grepped the raw file, which broke + in both directions: a script whose comment read + `# always uv sync here, never pip install` was rejected as **BLOCKING** -- + penalising exactly the author who documented the rule -- while a script whose + only mention of `uv sync` was in a comment **satisfied** the uv-sync rule + despite installing nothing. If you are advising on a repo validated by an + older kbagent, a BLOCKING `setup-sh-no-pip` may be about a comment; check the + file before telling anyone to change what the script runs. +- The stripper is deliberately not a shell parser: it tracks single/double + quotes, so a `#` inside a string survives, but heredocs and `${...#...}` + expansions are out of scope. A `pip install` hidden inside a heredoc is not + detected -- the rule is a pre-flight heuristic, not a guarantee. - Exit 0 on all checks <= WARN; exit 1 on any BLOCKING. `--strict` treats WARNs as failures (exit 1) for CI gating. - **Reading the build / runtime log** is now available via