`storage describe-column` writes column descriptions where the MCP server never reads them

[frantisekrehor]

storage describe-column writes column descriptions where the MCP server never reads them

kbagent: v0.86.0
keboola-mcp-server: 1.74.6
Impact: column descriptions written by kbagent are invisible to every AI client using the Keboola MCP server, while kbagent storage table-detail reports them as present.

Summary

There are two incompatible conventions for storing column descriptions:

	Key	Location
kbagent writes	KBC.column.{name}.description	flat entry in table metadata
MCP server reads	KBC.description	inside columnMetadata[{name}]

They never meet. Descriptions written via kbagent storage describe-column (or the columns: section of storage describe-batch) are silently invisible to get_table_detail / search in the MCP server — which is precisely the audience those descriptions are written for.

The failure is silent in both directions: kbagent storage table-detail reads back its own convention, so the descriptions look correctly applied.

Root cause

client/storage_tables.py:126-129 documents the assumption behind the current behaviour:

Column-level descriptions use the namespaced key convention KBC.column.{colname}.description stored at table-metadata level (Keboola Storage API does not expose a user-writable column-metadata endpoint; columnMetadata is populated exclusively by components).

That assumption is incorrect. The same endpoint kbagent already uses, POST /v2/storage/tables/{id}/metadata, accepts a columnsMetadata payload with provider: user. The MCP server does exactly this in clients/storage.py:944-953.

The mirrored read path has the same problem — services/storage_service.py:664-668 populates col_descriptions only from flat KBC.column.* keys, ignoring KBC.description inside columnMetadata, even though columnMetadata is already loaded on line 654.

Reproduction

Verified live on project 9432 (out.c-out-reporting.customer).

Writing through the columnsMetadata payload:

payload = {
    "provider": "user",
    "columnsMetadata": {
        "customer_id": [{"key": "KBC.description", "value": "Customer ID from the ERP system."}]
    },
}
client._request("POST", "/v2/storage/tables/out.c-out-reporting.customer/metadata", json=payload)

GET /v2/storage/tables/{id} afterwards:

customer_id  [... , ('KBC.description', 'Customer ID from the ERP system.', 'user')]

The write succeeds with provider: user, contradicting the docstring.

Reading the same table back through kbagent after 70 columns were described this way:

$ kbagent --json storage table-detail --project <alias> --table-id out.c-out-reporting.price_history_today
description:      "Today's price snapshot: ..."   # table description: read back fine
column_metadata:  0                               # <- the 70 column descriptions are not seen
column_details with a description: 0 of 70

Where the MCP server reads from

• tools/storage/tools.py:793-799 — column description resolved from columnMetadata[col] via MetadataField.DESCRIPTION, with a fallback to sourceTable.columnMetadata
• tools/storage/search.py:107-110 — same lookup for search matching

Neither path inspects KBC.column.* keys.

Suggested fix

1. set_table_metadata (or a sibling method): send column descriptions as a columnsMetadata JSON payload with provider: user instead of flat metadata[i][key] entries.
2. storage_service.py:664-668: read KBC.description from columnMetadata[col] as the primary source, keeping the KBC.column.* lookup as a fallback for already-written data.
3. Correct the docstrings in client/storage_tables.py:126-129 and services/storage_service.py:2534-2538, plus skills/kbagent/references/storage-describe-workflow.md, which currently states that descriptions written this way will be visible in MCP get_tables.

A migration for existing KBC.column.* entries would be useful — any project already documented this way is silently invisible to its AI clients.

Workaround

Bypass the CLI and post columnsMetadata through keboola_agent_cli.lib.Client(...).raw:

payload = {"provider": "user", "columnsMetadata": {col: [{"key": "KBC.description", "value": desc}] for col, desc in cols.items()}}
client._request("POST", f"/v2/storage/tables/{quote(table_id, safe='')}/metadata", json=payload)

Applied to 529 columns across 20 tables in project 9432; all became visible to the MCP server.

[padak]

Great find — confirmed against the sources of keboola/ui and keboola/connection, and the picture turns out to be broader than two conventions. It changes the recommended fix, so writing it up before implementation.

There are four conventions, not two

	Write	Read
Keboola UI	PUT /v2/storage/branch/{branchId}/tables/{tableId}/definition with columns: [{name, description}] + isDescriptionSystemManaged: false (async, tableDefinitionUpdate job)	native definition.columns[].definition.description, fallback to columnMetadata → KBC.description (storage/helpers.ts:660)
MCP server	POST .../metadata with columnsMetadata	only columnMetadata[col] → KBC.description (+ sourceTable.columnMetadata)
kbagent	POST .../metadata, flat KBC.column.{name}.description	flat KBC.column.* only
exists, unused for descriptions	POST /v2/storage/branch/{branchId}/columns/{tableId}.{col}/metadata (dedicated user-writable column-metadata endpoint; the UI uses it for datatypes)	—

So the docstring's claim ("no user-writable column-metadata endpoint") is disproved twice over — but more importantly, kbagent is the only outlier, not one of two equal conventions.

Key fact: the backend mirrors the native write into columnMetadata

In keboola/connection, TableDefinitionUpdateService routes both the untyped-table description-only patch (syncColumnDescriptions) and the typed-table path (withStorageColumnDescription → syncTypedColumnInfo) into replaceStorageColumnMetadata(...) with key KBC.description. One native write is therefore visible to the UI (native field), the MCP server (columnMetadata), and the warehouse (Snowflake COMMENT / BigQuery description).

The mirroring is one-way. useUpdateDescription.ts in the UI documents that the legacy POST /metadata write "never reaches the native field" (routing dropped with DMD-1273). And connection ships BackfillNativeColumnDefinitionsCommand, actively migrating existing KBC.description metadata into native fields — the platform is moving away from the metadata path, so building the fix on columnsMetadata would swim against that current.

Why columnsMetadata alone is not enough

The suggested fix (send columnsMetadata with provider: user) does make descriptions visible to the MCP server — verified by your live repro. But it carries two silent losses:

1. No propagation to the backend. The description never reaches Snowflake COMMENT / BigQuery — invisible to any tool reading the warehouse directly.
2. isDescriptionSystemManaged stays true. Per DMD-1662 ("Output Mapping may overwrite a system-managed description but must never overwrite a user-managed one; defaults to system-managed"), a description written via metadata remains exposed to silent overwrite by the next component run. The flag is a PUT .../definition parameter — the metadata endpoint cannot set it. The UI pins it to false on every manual edit; that is the only thing protecting hand-written descriptions.

Revised fix (implementation in progress)

1. Write path: describe-column / describe-batch switch to PUT .../definition with columns: [{name, description}] + isDescriptionSystemManaged: false, exactly like the UI. Async (storage job, polled). MCP visibility comes for free via the backend mirror. Unknown column names now fail fast before the write (the flat convention accepted anything).

2. Read path: table-detail resolves descriptions with precedence definition.columns[].definition.description → columnMetadata[col].KBC.description (incl. sourceTable for aliases, MCP parity) → legacy flat KBC.column.*, and reports a legacy_column_descriptions list + human-mode warning when flat keys are found. Read commands never write (--deny-writes firewall, read-only tokens).

3. Migration, three-tier:

   • table-detail warns when legacy keys exist (detection is free — metadata is already in the response);
   • describe-column/describe-batch auto-convert remaining legacy keys on the same table during their write (already a write op, upsert semantics);
   • new storage describe-migrate --project ALIAS [--table-id ...|--bucket-id ID] [--prune-orphans] [--dry-run] [--yes] for bulk conversion of already-documented projects.

   Conversion rules: conflict (target already has a differing description in native/columnMetadata) → skip + report, the newer convention wins; orphan keys (column renamed/deleted) → skip, delete only with --prune-orphans; successfully migrated flat entries are deleted — otherwise a description later cleared through the new path would be resurrected by the fallback.

4. Docs: correct the docstrings and every surface that repeats the wrong assumption (storage-describe-workflow.md, gotchas.md, commands-reference.md, agent context).

One caveat: this is source-reading of ui/connection plus your live repro, not a full live verification of PUT .../definition from kbagent — that's the first thing the implementation validates.