Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,11 @@ kbagent storage table-detail --project NAME --table-id ID [--branch ID]
# ID is unchanged either way, and `create-table` only echoes the layout you REQUESTED (its
# --if-not-exists skip path nulls the layout keys outright). Human mode adds Time
# partitioning / Range partitioning / Clustering / Partition filter required / Partitions
# (a COUNT) and prints nothing new when there is no layout; --json passes `definition`
# (a COUNT) and prints nothing new when there is no layout. The human
# Columns table also carries a Description column (NOT on 0.88.0), populated from
# `column_details[].description` and shown only when some column has one -- the one
# surface #624 left blank, so `describe-column` then `table-detail` now verifies
# itself (long text wraps, never truncates). --json passes `definition`
# through verbatim, including the unbounded `partitions[]` (one entry per physical
# partition from INFORMATION_SCHEMA.PARTITIONS). `definition` is present on EVERY response
# -- untyped tables get one too -- so null means the stack omitted the key, NOT "untyped".
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def
- `storage buckets [--project NAME] [--branch ID]` -- list buckets with sharing/linked info (branch-aware)
- `storage bucket-detail --project NAME --bucket-id ID [--branch ID]` -- bucket detail with backend-native direct-access paths (branch-aware). Output adapts to backend: Snowflake -> `snowflake_database` / `snowflake_schema` / per-table `snowflake_path` quoted with `"..."`. BigQuery -> `bigquery_dataset` (and `bigquery_project` when surfaced via API `databaseName`) / per-table `bigquery_path` quoted with backticks. Always-present backend-agnostic keys: `sql_dialect` (`"snowflake"` / `"bigquery"`) and per-table `sql_path` -- prefer these in agent code instead of branching on backend yourself
- `storage tables [--project NAME ...] [--bucket-id ID] [--branch ID] [--include-usage]` -- list tables across all connected projects in parallel (multi-project by default, same as `storage buckets`); repeat `--project` to target a subset; `--bucket-id` is applied independently per project (missing buckets become per-project errors); `--branch` requires exactly one `--project`. `--include-usage` (0.88.0+) adds `used_by` per table: the configurations naming it in their **storage input/output mapping** only -- a table id inside a transformation's SQL is NOT a reference. Costs one extra component listing per project (not per table), which is the slow call in a big project; unreadable components degrade to an empty `used_by`
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped". Since 0.88.0 (#624) it also RESOLVES column descriptions written by anyone -- the UI, a component, or kbagent -- into `column_details[].description`, with precedence native definition -> `columnMetadata` `KBC.description` -> legacy flat `KBC.column.*` (an alias table falls back to the source table's `columnMetadata`, matching the MCP server). The response always carries `legacy_column_descriptions`, naming the columns still backed by the pre-0.88.0 convention -- human mode warns and points at `storage describe-migrate`. Reading never writes, so it is safe under a read-only token or `--deny-writes`
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Since 0.88.0 (#621) also returns the raw Storage API `definition`: on BigQuery that carries `timePartitioning` / `rangePartitioning` / `clustering` / `requirePartitionFilter` / `partitions[]`, and it is the only way to verify a repartition landed. Human mode prints the layout and a partition COUNT; `--json` passes `definition` through verbatim. Present on every response (untyped tables too), so `null` means the stack omitted the key, not "untyped". Since 0.88.0 (#624) it also RESOLVES column descriptions written by anyone -- the UI, a component, or kbagent -- into `column_details[].description`, with precedence native definition -> `columnMetadata` `KBC.description` -> legacy flat `KBC.column.*` (an alias table falls back to the source table's `columnMetadata`, matching the MCP server). The response always carries `legacy_column_descriptions`, naming the columns still backed by the pre-0.88.0 convention -- human mode warns and points at `storage describe-migrate`. Human mode also renders a `Description` column in the Columns table (only when at least one column has one; long text wraps rather than truncating); on 0.88.0 descriptions were visible in `--json` only. Reading never writes, so it is safe under a read-only token or `--deny-writes`
- `storage create-bucket --project NAME --stage STAGE --name NAME [--description D] [--backend B] [--branch ID]` -- create bucket (branch-aware). With `--branch ID` on a project lacking the `storage-branches` feature (legacy fake-branch), response carries `legacy_branch_storage: true` and human mode prints a warning -- the runner will create a parallel `out.c-<branch_id>-*` bucket at job time. See `storage-types-workflow.md`
- `storage create-table --project NAME --bucket-id ID --name NAME [--column col:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists]` -- create typed table. Base types `STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP` plus native backend types with length (`VARCHAR(40)`, `NUMBER(18,2)`, `TIMESTAMP_TZ`, `VARIANT`, etc.) -- type/length validation delegated to the Storage API. `--not-null` marks a column `nullable=false`; `--default NAME=VALUE` sets a DEFAULT expression (booleans must be lowercase `true`/`false`). In a dev branch, the target bucket is auto-materialized if it has not yet been written to there -- response surfaces this via `auto_created_bucket: bool`. On legacy fake-branch projects (no `storage-branches` feature), `legacy_branch_storage: true` flags that the runner will use a separate `out.c-<branch_id>-*` bucket at job time. `--if-not-exists` (0.47.0+) turns a duplicate-display-name failure into `action: skipped` when the table really exists at the expected id (safe for parallel workers). Since 0.47.1 the skipped envelope reports the EXISTING table's actual `columns`/`primary_key`/`name`, mirrors the request under `requested_columns`/`requested_primary_key`, and sets `schema_drift: true` when they diverge. **`--source-table-id` (0.66.0+, BigQuery only)** copies an existing table's data into the requested partition/clustering layout instead of building from `--column` (schema derived from source -> `--column`/`--not-null`/`--default` forbidden; the two are mutually exclusive). This is the supported way to repartition a populated BigQuery table -- then promote it with `storage swap-tables`. Partition/clustering flags (`--time-partitioning-*`, `--range-partitioning-*`, `--clustering-field`) also work on a plain `--column` create (BigQuery only); time vs range partitioning are mutually exclusive and range bounds are strings. When any source/partition/clustering flag is used, a one-call backend pre-flight rejects non-BigQuery projects (exit 2) before the create. See `storage-types-workflow.md`
- `storage upload-table --project NAME --table-id ID --file PATH [--incremental] [--branch ID]` -- upload CSV (branch-aware)
Expand Down
6 changes: 5 additions & 1 deletion plugins/kbagent/skills/kbagent/references/gotchas.md
Original file line number Diff line number Diff line change
Expand Up @@ -2687,7 +2687,11 @@ write descriptive metadata onto storage objects. Three behaviors are easy to mis
native definition -> `columnMetadata` `KBC.description` -> legacy flat key,
always returns `legacy_column_descriptions`, and warns in human mode when
legacy keys remain (it never writes -- safe under a read-only token or
`--deny-writes`). Unknown column names now fail fast BEFORE any write; the old
`--deny-writes`). Its human-mode Columns table shows a `Description`
column. On 0.88.0 it did NOT -- there the descriptions were readable only
through `--json` `column_details[].description`, so a blank-looking terminal
table on that version does not mean the write failed. (Release step: once this
ships, tag this sentence `(since vX.Y.Z)` with the version that carried it.) Unknown column names now fail fast BEFORE any write; the old
flat write accepted typos silently. Table and bucket descriptions are
unaffected: still `KBC.description` (provider=user) on the object's metadata.
- **`describe-batch` is partial-failure-tolerant.** Item-level errors are
Expand Down
29 changes: 27 additions & 2 deletions src/keboola_agent_cli/commands/_storage_table_detail.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@
tests/test_storage_empty_definition.py -- it broke the Go CLI's decoder), so
the value is type-checked before any `.get()` reaches it.

The columns table shows a `Description` column only when some column has one.
Since 0.88.0 (#624) a description written by `storage describe-column` is visible
in the Keboola UI, in the MCP server and in the warehouse's own COMMENT, and
`--json` has always carried it -- this view was the last surface still blank, so
`describe-column` followed by `table-detail` never confirmed its own write.

`format_time_partitioning` / `format_range_partitioning` are shared with
`storage create-table`'s result output on purpose: verifying a repartition means
diffing what create-table said it applied against what table-detail reads back,
Expand Down Expand Up @@ -98,16 +104,35 @@ def render_table_detail(formatter: OutputFormatter, result: dict[str, Any]) -> N

if result["column_details"]:
formatter.console.print()
# Only grow the column when something fills it -- same rule the layout
# block above follows. `description` is set on a `column_details` entry
# only when one of the three tiers resolved (see
# services/_table_detail._column_details), so this is a real "nothing to
# show" test, not a test for empty strings.
show_description = any(col.get("description") for col in result["column_details"])

table = Table(title="Columns")
table.add_column("Name", style="bold cyan")
table.add_column("Type", style="dim")
table.add_column("Nullable", style="dim")
if show_description:
# Capped and wrapped, never truncated. A description is what the
# user came here to verify after `storage describe-column`, so an
# ellipsis would defeat the point -- `overflow="fold"` breaks even a
# long unbroken token instead of hiding its tail. `max_width` stops
# a wide terminal from stretching the cell across the screen; on a
# narrow one Rich shrinks it further and the table still fits.
table.add_column("Description", max_width=60, overflow="fold")

for col in result["column_details"]:
table.add_row(
cells = [
col["name"],
col.get("type", ""),
"yes" if col.get("nullable") else "",
)
]
if show_description:
# User-authored free text: `[note]` is a note, not Rich markup.
cells.append(escape(col.get("description") or ""))
table.add_row(*cells)

formatter.console.print(table)
98 changes: 98 additions & 0 deletions tests/test_storage_table_definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
from typing import Any
from unittest.mock import MagicMock, patch

import pytest
from typer.testing import CliRunner

from keboola_agent_cli.cli import app
Expand Down Expand Up @@ -251,3 +252,100 @@ def test_definition_passes_through_json_mode(self, tmp_path: Path) -> None:
payload = json.loads(result.output)
data = payload.get("data", payload)
assert data["definition"] == BIGQUERY_LAYOUT


# A native `definition` carrying per-column descriptions -- the tier the Keboola
# UI writes and shows, and the one `storage describe-column` has written since
# 0.88.0 (#624). It rides in on the same `definition` object as the layout, so
# the fixture above already covers it.
DESCRIBED_LAYOUT: dict[str, Any] = {
"primaryKeysNames": ["id"],
"columns": [
{"name": "id", "definition": {"type": "INTEGER"}},
{
"name": "created_at",
"definition": {"type": "TIMESTAMP", "description": "Row creation timestamp, UTC."},
},
{
"name": "tenant_id",
"definition": {"type": "STRING", "description": "Owning tenant."},
},
],
}


class TestColumnDescriptions:
"""The Columns table must show what `storage describe-column` just wrote.

0.88.0 (#624) moved the write to the native table-definition endpoint so the
description surfaces in the Keboola UI, the MCP server's `get_tables` and the
warehouse's own COMMENT. `column_details[].description` has carried it in
`--json` all along. Human mode was the one place still showing nothing, so
the obvious way to check your own work -- describe, then `table-detail` --
was the one way that did not work.
"""

def test_description_is_rendered_in_the_columns_table(self, tmp_path: Path) -> None:
result = _invoke_detail(tmp_path, DESCRIBED_LAYOUT)

assert result.exit_code == 0, result.output
assert "Description" in result.output
assert "Row creation timestamp, UTC." in result.output

def test_undescribed_table_grows_no_empty_column(self, tmp_path: Path) -> None:
"""Same guard style as the layout block: no data, no chrome."""
result = _invoke_detail(tmp_path, UNTYPED_DEFINITION)

assert result.exit_code == 0, result.output
assert "Description" not in result.output

def test_markup_in_a_description_is_shown_literally(self, tmp_path: Path) -> None:
"""Descriptions are user text; Rich would eat (or choke on) square brackets."""
definition = {
"columns": [{"name": "id", "definition": {"description": "see [note] and [/x]"}}]
}
result = _invoke_detail(tmp_path, definition)

assert result.exit_code == 0, result.output
assert "[note]" in result.output

def test_long_description_wraps_inside_an_80_column_terminal(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A wrapped cell keeps the table readable; an unwrapped one destroys it.

The `_deterministic_console_width` fixture in conftest pins every test to
a 200-column console, so the narrow case only gets exercised if a test
asks for it by name.
"""
monkeypatch.setenv("COLUMNS", "80")
long_text = (
"Timestamp the row was created, in UTC, as written by the ingestion "
"job; backfilled rows carry the original source timestamp instead."
)
definition = {"columns": [{"name": "created_at", "definition": {"description": long_text}}]}
result = _invoke_detail(tmp_path, definition)

assert result.exit_code == 0, result.output
widest = max((len(line) for line in result.output.splitlines()), default=0)
assert widest <= 80, f"widest line was {widest} chars"
# Wrapped, not truncated -- the last word must survive.
assert "instead." in result.output

def test_wide_terminal_does_not_stretch_the_description_cell(self, tmp_path: Path) -> None:
"""`max_width` is load-bearing: without it the cell fills the whole console.

Runs at the conftest default of 200 columns. A single-line description
would otherwise pull the table out to the full terminal width and put
yards of whitespace between the column name and its text.
"""
long_text = (
"Timestamp the row was created, in UTC, as written by the ingestion "
"job; backfilled rows carry the original source timestamp instead."
)
definition = {"columns": [{"name": "created_at", "definition": {"description": long_text}}]}
result = _invoke_detail(tmp_path, definition)

assert result.exit_code == 0, result.output
widest = max((len(line) for line in result.output.splitlines()), default=0)
assert widest < 120, f"table sprawled to {widest} chars on a 200-column console"