fix(storage): write column descriptions where UI and MCP read them (#624)

CLAUDE.md

@@ -469,6 +469,23 @@ kbagent storage describe-bucket --project NAME --bucket-id ID [--text STR | --fi
 kbagent storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID]
 kbagent storage describe-column --project NAME --table-id ID --column NAME=DESC [--column ...] [--branch ID]
 kbagent storage describe-batch --project NAME --from-file YAML [--branch ID]
+kbagent storage describe-migrate --project ALIAS [--table-id ID ...] [--bucket-id ID] [--prune-orphans] [--dry-run] [--yes] [--branch ID]
+# Column descriptions (0.88.0+, #624): describe-column/describe-batch write through the native
+#   `PUT .../tables/{id}/definition` endpoint (async job) with `isDescriptionSystemManaged: false`
+#   (stops the next Output Mapping run from overwriting the text). The backend mirrors the value into
+#   `columnMetadata` `KBC.description`, so the UI, the MCP server and the Snowflake COMMENT /
+#   BigQuery description all see it. Before 0.88.0 kbagent wrote a flat `KBC.column.{name}.description`
+#   key on the TABLE's metadata -- read by nothing but kbagent, so documented columns looked blank
+#   everywhere else. Unknown column names now FAIL FAST before any write (behavior change; the flat
+#   write accepted typos silently). `table-detail` reads with precedence native definition ->
+#   columnMetadata KBC.description -> legacy flat key, always returns `legacy_column_descriptions`
+#   and warns in human mode when legacy keys remain; it never writes. `describe-migrate` converts
+#   legacy keys in bulk (scope: --table-id / --bucket-id / whole project; scan-then-confirm,
+#   --dry-run reports only; per-table errors accumulate). describe-column/describe-batch also migrate
+#   leftovers on the table they touch. Rules: a column whose visible description already differs is
+#   skipped as `conflict` (newer value wins), an entry for a dropped column is skipped as `orphan`
+#   unless --prune-orphans. Migrated flat entries are DELETED so a later clear cannot be resurrected
+#   by the read fallback.
 kbagent storage files --project NAME [--tag TAG ...] [--limit N] [--offset N] [--query Q] [--branch ID]
 kbagent storage file-upload --project NAME --file PATH [--name NAME] [--tag TAG ...] [--permanent] [--branch ID]
 kbagent storage file-download --project NAME [--file-id ID | --tag TAG ...] [--output FILE]

plugins/kbagent/agents/keboola-expert.md

@@ -250,6 +250,8 @@ its absence is NOT a promise the entry is version-independent (see §1 Rule 6).
   `--name` (the API rejects empty) and fails on an existing table name --
   restore under a new name, verify, then `swap-tables`. `snapshot-delete` only
   forecloses restores; the source table is untouched.
+- **Column descriptions** (0.88.0+, #624): native endpoint; legacy
+  `KBC.column.*` invisible to UI/MCP; `describe-migrate`. gotchas.md.
 - **`bucket-detail` is dialect-aware**: read `sql_dialect` + per-table
   `sql_path` (already correctly quoted) -- don't branch on the backend yourself.
 

plugins/kbagent/skills/kbagent/SKILL.md

@@ -167,10 +167,6 @@ When working inside a git repository or project directory, run `kbagent init` (o
 | Swap two storage tables (any branch, including the default/production branch) | `kbagent storage swap-tables --project PROJECT --table-id TABLE-ID --target-table-id TARGET-TABLE-ID` |
 | Clone (pull) a production table into a development branch | `kbagent storage clone-table --project PROJECT --table-id TABLE-ID` |
 | Delete one or more storage buckets | `kbagent storage delete-bucket --project PROJECT --bucket-id BUCKET-ID` |
-| Set the description on a storage bucket | `kbagent storage describe-bucket --project PROJECT --bucket-id BUCKET-ID` |
-| Set the description on a storage table | `kbagent storage describe-table --project PROJECT --table-id TABLE-ID` |
-| Set descriptions on one or more columns of a storage table | `kbagent storage describe-column --project PROJECT --table-id TABLE-ID --column COLUMN` |
-| Apply descriptions to buckets, tables, and columns from a YAML file | `kbagent storage describe-batch --project PROJECT --from-file FROM-FILE` |
 | List Storage Files with optional tag filtering | `kbagent storage files --project PROJECT` |
 | Show Storage File metadata (without downloading) | `kbagent storage file-detail --project PROJECT --file-id FILE-ID` |
 | Upload a local file to Storage Files | `kbagent storage file-upload --project PROJECT --file FILE` |
@@ -184,6 +180,11 @@ When working inside a git repository or project directory, run `kbagent init` (o
 | Show one snapshot's detail (source table, creation time, description) | `kbagent storage snapshot-detail --project PROJECT --snapshot-id SNAPSHOT-ID` |
 | Delete one or more table snapshots (the source tables are untouched) | `kbagent storage snapshot-delete --project PROJECT --snapshot-id SNAPSHOT-ID` |
 | Create a NEW table from an existing snapshot (snapshot restore) | `kbagent storage table-from-snapshot --project PROJECT --snapshot-id SNAPSHOT-ID --bucket-id BUCKET-ID --name NAME` |
+| Set the description on a storage bucket | `kbagent storage describe-bucket --project PROJECT --bucket-id BUCKET-ID` |
+| Set the description on a storage table | `kbagent storage describe-table --project PROJECT --table-id TABLE-ID` |
+| Set descriptions on one or more columns of a storage table | `kbagent storage describe-column --project PROJECT --table-id TABLE-ID --column COLUMN` |
+| Apply descriptions to buckets, tables, and columns from a YAML file | `kbagent storage describe-batch --project PROJECT --from-file FROM-FILE` |
+| Convert legacy KBC.column.* descriptions to the native definition endpoint | `kbagent storage describe-migrate --project PROJECT` |
 | List Data Streams sources in a project | `kbagent stream list --project PROJECT` |
 | Create an OTLP (or HTTP) source and return its endpoint | `kbagent stream create-source --project PROJECT --name NAME` |
 | Show a source's endpoints, protocol, and destination tables | `kbagent stream detail [SOURCE-ID] --project PROJECT` |

plugins/kbagent/skills/kbagent/references/commands-reference.md

@@ -178,8 +178,9 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def
 - `storage snapshot-delete --project NAME --snapshot-id ID [--snapshot-id ...] [--dry-run] [--yes]` (since v0.75.0) -- delete snapshots (destructive: forecloses restores; source tables untouched). Batch-tolerant, exit 1 when any ID failed
 - `storage describe-bucket --project NAME --bucket-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a bucket description (stored as `KBC.description` in bucket metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage bucket-detail`
 - `storage describe-table --project NAME --table-id ID [--text STR | --file PATH | --stdin] [--branch ID]` -- set a table description (stored as `KBC.description` in table metadata, upsert). Provide exactly one of `--text`, `--file`, `--stdin`. Read back via `storage table-detail`
-- `storage describe-column --project NAME --table-id ID --column NAME=DESCRIPTION [--column ...] [--branch ID]` -- set one or more column descriptions. Stored as `KBC.column.{name}.description` keys in the table's metadata (Keboola has no user-writable column-metadata endpoint). Read back in `storage table-detail` under `column_details[].description`
-- `storage describe-batch --project NAME --from-file PATH [--branch ID]` -- apply bucket/table/column descriptions from a YAML file (top-level `buckets`, `tables`, `columns` sections, all optional). Partial-failure tolerant: per-item errors are collected and reported, the batch does not abort. Non-zero exit only when at least one item failed
+- `storage describe-column --project NAME --table-id ID --column NAME=DESCRIPTION [--column ...] [--branch ID]` -- set one or more column descriptions. *(since v0.88.0)* Writes through the native `PUT /v2/storage/branch/{branch}/tables/{id}/definition` endpoint (the one the web UI uses; async `tableDefinitionUpdate` storage job) with `isDescriptionSystemManaged: false`, so the next component run's Output Mapping cannot overwrite the text. The backend mirrors the value into `columnMetadata` `KBC.description`, so the Keboola UI, the MCP server (`get_tables`) and the Snowflake `COMMENT` / BigQuery column description all see it. Unknown column names are rejected BEFORE any write (behavior change -- the pre-0.88.0 flat-metadata write accepted typos silently). Legacy flat `KBC.column.{name}.description` entries on the same table are migrated in the same write and then deleted. Read back in `storage table-detail` under `column_details[].description`
+- `storage describe-batch --project NAME --from-file PATH [--branch ID]` -- apply bucket/table/column descriptions from a YAML file (top-level `buckets`, `tables`, `columns` sections, all optional). Column items go through the same native write (and same fail-fast + auto-migration) as `describe-column`. Partial-failure tolerant: per-item errors are collected and reported, the batch does not abort. Non-zero exit only when at least one item failed
+- `storage describe-migrate --project ALIAS [--table-id ID ...] [--bucket-id ID] [--prune-orphans] [--dry-run] [--yes] [--branch ID]` *(since v0.88.0)* -- bulk-convert legacy pre-0.88.0 flat `KBC.column.*.description` metadata to the native definition endpoint. Scope is explicit `--table-id` (repeatable), a single `--bucket-id`, or every table in the project; the two scope flags are mutually exclusive (exit 2). Scans first and prints the summary, then asks for confirmation -- `--dry-run` reports without writing, `--yes` skips the prompt. A column whose currently visible description already differs is skipped as `conflict` (the newer value wins); an entry for a column that no longer exists is skipped as `orphan` unless `--prune-orphans` deletes it. Migrated flat entries are deleted after a successful write, so a later `describe-column` clearing the text cannot be resurrected by the read fallback. Per-table failures are accumulated into `errors[]` and never abort the run. Permission class `write`
 
 ## Storage Files
 - `storage files --project NAME [--tag TAG ...] [--limit N] [--offset N] [--query Q] [--branch ID]` -- list Storage Files, optionally filtered by tag/query

plugins/kbagent/skills/kbagent/references/gotchas.md

@@ -2648,15 +2648,34 @@ so setting a branch's description will **not** update the dashboard.
 `kbagent storage describe-bucket / describe-table / describe-column / describe-batch`
 write descriptive metadata onto storage objects. Three behaviors are easy to miss:
 
-- **Column descriptions use a metadata-key convention, not a column endpoint.**
-  The Keboola Storage API has no user-writable column-level metadata endpoint,
-  so `describe-column` stores each description as a `KBC.column.{name}.description`
-  entry on the **table's** metadata (upsert). `storage table-detail` reads them
-  back via the same key and surfaces them under `column_details[].description`.
-  Renaming or deleting a column does NOT automatically clean these entries up
-  (they remain on the table's metadata under the old name). Same convention for
-  table and bucket descriptions: stored as `KBC.description` (provider=user) on
-  the object's metadata.
+- **Column descriptions go through the native definition endpoint** *(since
+  v0.88.0)*. `describe-column` / `describe-batch` write via
+  `PUT /v2/storage/branch/{branch}/tables/{id}/definition` -- the endpoint the
+  web UI uses (async `tableDefinitionUpdate` storage job) -- with
+  `isDescriptionSystemManaged: false`, which is what stops the next component
+  run's Output Mapping from overwriting a hand-authored description. The backend
+  mirrors the written value into `columnMetadata` `KBC.description` for typed AND
+  untyped tables, so one write is visible to the Keboola UI, to the MCP server
+  (`get_tables`), and in the Snowflake `COMMENT` / BigQuery column description.
+  *Pre-0.88.0 behaviour:* kbagent stored each description as a flat
+  `KBC.column.{name}.description` entry on the **table's** metadata. Nothing but
+  kbagent itself ever read that key -- columns documented that way look blank in
+  the UI, are invisible to the MCP server, and never reach the warehouse. That
+  mirroring is one-way: a `POST .../metadata` write never reaches the native
+  field. Convert leftovers with `kbagent storage describe-migrate` (bulk,
+  scan-then-confirm, `--dry-run` first); `describe-column` / `describe-batch`
+  also migrate remaining legacy entries on whatever table they touch. A column
+  whose visible description already differs is skipped as `conflict` (newer value
+  wins), an entry for a dropped column is skipped as `orphan` unless
+  `--prune-orphans`. **Migrated flat entries are DELETED** -- that is deliberate:
+  leaving them would let the read fallback resurrect an old description after
+  someone clears the column's text. `table-detail` reads with the precedence
+  native definition -> `columnMetadata` `KBC.description` -> legacy flat key,
+  always returns `legacy_column_descriptions`, and warns in human mode when
+  legacy keys remain (it never writes -- safe under a read-only token or
+  `--deny-writes`). Unknown column names now fail fast BEFORE any write; the old
+  flat write accepted typos silently. Table and bucket descriptions are
+  unaffected: still `KBC.description` (provider=user) on the object's metadata.
 - **`describe-batch` is partial-failure-tolerant.** Item-level errors are
   collected into `result.errors[]` but the batch keeps processing the remaining
   items. The CLI exits non-zero only if `error_count > 0`, so in scripts always