Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"plugins": [
{
"name": "kbagent",
"version": "0.87.0",
"version": "0.87.1",
"source": "./plugins/kbagent",
"description": "AI-friendly interface to Keboola Connection projects — explore configs, jobs, lineage, sync configs as files, manage dev branches, and debug SQL in workspaces",
"category": "development"
Expand Down
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,11 @@ kbagent storage buckets [--project NAME] [--branch ID]
kbagent storage bucket-detail --project NAME --bucket-id ID [--branch ID]
kbagent storage tables [--project NAME ...] [--bucket-id ID] [--branch ID]
kbagent storage table-detail --project NAME --table-id ID [--branch ID]
# table-detail (0.87.1+, #621): also returns `definition` -- the typed-table layout (primaryKeysNames
# plus, on BigQuery, timePartitioning/rangePartitioning/clustering; null when untyped). The read half
# of the repartition flow: after `create-table --source-table-id ...` + `swap-tables` the table id is
# the same either way, so only the layout says whether the swap took. `storage tables` (LIST) does not
# return it. Human mode prints Partitioning/Clustering rows only when a layout is present.
kbagent storage create-bucket --project NAME --stage STAGE --name NAME [--description D] [--backend B] [--branch ID]
kbagent storage create-table --project NAME --bucket-id ID --name NAME [--column COL:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists]
# --column XOR --source-table-id (0.66.0+, BigQuery only): --source-table-id copies an existing table's data into the requested partition/clustering layout (schema derived from source) -> swap into place with swap-tables. Partition/clustering flags work in both modes (BigQuery only); time vs range partitioning are mutually exclusive. A non-BigQuery project fails fast (pre-flight backend check).
Expand Down
2 changes: 1 addition & 1 deletion plugins/kbagent/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "kbagent",
"version": "0.87.0",
"version": "0.87.1",
"description": "AI-friendly interface to Keboola Connection projects — explore configs, jobs, lineage, sync configs as files, manage dev branches, and debug SQL in workspaces",
"author": {
"name": "Keboola",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Requires a **super-admin** Manage API token (same kind as `org setup`). Same def
- `storage buckets [--project NAME] [--branch ID]` -- list buckets with sharing/linked info (branch-aware)
- `storage bucket-detail --project NAME --bucket-id ID [--branch ID]` -- bucket detail with backend-native direct-access paths (branch-aware). Output adapts to backend: Snowflake -> `snowflake_database` / `snowflake_schema` / per-table `snowflake_path` quoted with `"..."`. BigQuery -> `bigquery_dataset` (and `bigquery_project` when surfaced via API `databaseName`) / per-table `bigquery_path` quoted with backticks. Always-present backend-agnostic keys: `sql_dialect` (`"snowflake"` / `"bigquery"`) and per-table `sql_path` -- prefer these in agent code instead of branching on backend yourself
- `storage tables [--project NAME ...] [--bucket-id ID] [--branch ID]` -- list tables across all connected projects in parallel (multi-project by default, same as `storage buckets`); repeat `--project` to target a subset; `--bucket-id` is applied independently per project (missing buckets become per-project errors); `--branch` requires exactly one `--project`
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware)
- `storage table-detail --project NAME --table-id ID [--branch ID]` -- table detail with columns, types, primary key, row count (branch-aware). Also carries `definition` (since v0.87.1) -- the typed-table layout: `primaryKeysNames` plus, on BigQuery, `timePartitioning` / `rangePartitioning` / `clustering`; `null` on an untyped table. This is the read half of the repartition flow (`create-table --source-table-id --time-partitioning-field ...` -> `swap-tables`): the table id is identical whether the swap happened or not, so the layout is the only field that distinguishes them. Human mode adds `Partitioning:` / `Clustering:` rows only when a layout exists. `storage tables` (the LIST endpoint) genuinely does not return it
- `storage create-bucket --project NAME --stage STAGE --name NAME [--description D] [--backend B] [--branch ID]` -- create bucket (branch-aware). With `--branch ID` on a project lacking the `storage-branches` feature (legacy fake-branch), response carries `legacy_branch_storage: true` and human mode prints a warning -- the runner will create a parallel `out.c-<branch_id>-*` bucket at job time. See `storage-types-workflow.md`
- `storage create-table --project NAME --bucket-id ID --name NAME [--column col:TYPE[(length)] ...] [--primary-key COL] [--not-null COL ...] [--default NAME=VALUE ...] [--source-table-id ID] [--source-branch-id N] [--time-partitioning-type DAY|HOUR|MONTH|YEAR] [--time-partitioning-field COL] [--time-partitioning-expiration-ms MS] [--range-partitioning-field COL --range-partitioning-start S --range-partitioning-end E --range-partitioning-interval I] [--clustering-field COL ...] [--branch ID] [--if-not-exists]` -- create typed table. Base types `STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP` plus native backend types with length (`VARCHAR(40)`, `NUMBER(18,2)`, `TIMESTAMP_TZ`, `VARIANT`, etc.) -- type/length validation delegated to the Storage API. `--not-null` marks a column `nullable=false`; `--default NAME=VALUE` sets a DEFAULT expression (booleans must be lowercase `true`/`false`). In a dev branch, the target bucket is auto-materialized if it has not yet been written to there -- response surfaces this via `auto_created_bucket: bool`. On legacy fake-branch projects (no `storage-branches` feature), `legacy_branch_storage: true` flags that the runner will use a separate `out.c-<branch_id>-*` bucket at job time. `--if-not-exists` (0.47.0+) turns a duplicate-display-name failure into `action: skipped` when the table really exists at the expected id (safe for parallel workers). Since 0.47.1 the skipped envelope reports the EXISTING table's actual `columns`/`primary_key`/`name`, mirrors the request under `requested_columns`/`requested_primary_key`, and sets `schema_drift: true` when they diverge. **`--source-table-id` (0.66.0+, BigQuery only)** copies an existing table's data into the requested partition/clustering layout instead of building from `--column` (schema derived from source -> `--column`/`--not-null`/`--default` forbidden; the two are mutually exclusive). This is the supported way to repartition a populated BigQuery table -- then promote it with `storage swap-tables`. Partition/clustering flags (`--time-partitioning-*`, `--range-partitioning-*`, `--clustering-field`) also work on a plain `--column` create (BigQuery only); time vs range partitioning are mutually exclusive and range bounds are strings. When any source/partition/clustering flag is used, a one-call backend pre-flight rejects non-BigQuery projects (exit 2) before the create. See `storage-types-workflow.md`
- `storage upload-table --project NAME --table-id ID --file PATH [--incremental] [--branch ID]` -- upload CSV (branch-aware)
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "keboola-cli"
version = "0.87.0"
version = "0.87.1"
description = "AI-friendly CLI for managing Keboola projects"
readme = "README.md"
requires-python = ">=3.12"
Expand Down
20 changes: 20 additions & 0 deletions src/keboola_agent_cli/changelog.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,26 @@

# Ordered newest-first. Each value is a list of brief one-line descriptions.
CHANGELOG: dict[str, list[str]] = {
"0.87.1": [
"New (#621): `storage table-detail` now returns the table's `definition`, so a "
"BigQuery repartition can finally be verified from kbagent. The field carries "
"`primaryKeysNames` plus, on BigQuery, `timePartitioning`, `rangePartitioning` and "
"`clustering`; `null` for an untyped table. kbagent could "
"already WRITE that layout (`storage create-table --source-table-id "
"--time-partitioning-field --clustering-field`, promoted with `storage swap-tables`) "
"but had no way to read it back: the service built its response from a field allowlist "
"that dropped `definition`, so the write half of the repartition flow was supported and "
"the verify half was not. It matters because the table ID is unchanged whether the swap "
"happened or not -- the layout is the only field that tells a completed repartition "
"from a failed one, and on a Keboola-managed BigQuery project where you hold no "
"`bigquery.jobs.create`, the Storage `definition` is the only view of the registered "
"layout you can reach at all. Human mode adds `Partitioning:` / `Clustering:` rows "
"between the primary key and last-import lines, and only when a layout exists, so "
"Snowflake and untyped-table output is byte-identical. Additive in `--json` too. The "
"`GET /storage/table-detail/...` route picks it up for free. `storage tables` is "
"unchanged: the LIST endpoint genuinely does not return `definition`, and fetching it "
"per table would be a different change with a different cost profile.",
],
"0.87.0": [
"New (#626): `data-app create` gains `--workspace / --no-workspace` and grants "
"Storage access BY DEFAULT. The flag writes `runtime.workspace.enabled: true`, which "
Expand Down
85 changes: 85 additions & 0 deletions src/keboola_agent_cli/commands/_storage_format.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
"""Human-mode rendering helpers for the ``kbagent storage`` group.

Lives in a private module because ``commands/storage.py`` is already past the
1,200-LOC commands-file ceiling (CONTRIBUTING.md > "File-size budgets"), so
rendering that grows -- such as the typed-table layout added for #621 -- moves
here rather than extending it further.
"""

from __future__ import annotations

from typing import Any

from rich.markup import escape
from rich.table import Table

from ..output import OutputFormatter


def _format_table_layout(definition: Any) -> list[tuple[str, str]]:
"""Render a typed table's partition/cluster layout as label/value rows.

``definition`` is the Storage API's typed-table descriptor: ``None`` for an
untyped table, and on BigQuery it carries ``timePartitioning``,
``rangePartitioning`` and ``clustering``. Returns an empty list whenever
there is no layout to show, so Snowflake and untyped-table output stays
byte-identical.
"""
if not isinstance(definition, dict):
return []
rows: list[tuple[str, str]] = []
time_part = definition.get("timePartitioning") or {}
if isinstance(time_part, dict) and time_part.get("type"):
# `field` is absent for ingestion-time partitioning, where BigQuery
# partitions on the load timestamp rather than on a column.
field = time_part.get("field")
rows.append(
("Partitioning", f"{time_part['type']} on {field}" if field else time_part["type"])
)
range_part = definition.get("rangePartitioning") or {}
if isinstance(range_part, dict) and range_part.get("field"):
rng = range_part.get("range") or {}
bounds = (
f" [{rng.get('start')}, {rng.get('end')}) step {rng.get('interval')}"
if isinstance(rng, dict) and rng
else ""
)
rows.append(("Range partitioning", f"{range_part['field']}{bounds}"))
clustering = definition.get("clustering") or {}
if isinstance(clustering, dict) and clustering.get("fields"):
rows.append(("Clustering", ", ".join(str(f) for f in clustering["fields"])))
return rows


def render_table_detail(formatter: OutputFormatter, result: dict[str, Any]) -> None:
"""Print the human-mode ``storage table-detail`` report.

Layout rows appear only for a typed table that actually declares one, so
Snowflake and untyped-table output is unchanged.
"""
formatter.console.print(f"[bold]Table:[/bold] {result['table_id']}")
formatter.console.print(f" Name: {escape(result['display_name'] or result['name'])}")
formatter.console.print(f" Bucket: {result['bucket_id']}")
formatter.console.print(f" Rows: {result['rows_count']:,}")
size_mb = result["data_size_bytes"] / (1024 * 1024)
formatter.console.print(f" Size: {size_mb:.2f} MB")
if result["primary_key"]:
formatter.console.print(f" Primary key: {', '.join(result['primary_key'])}")
for label, value in _format_table_layout(result.get("definition")):
formatter.console.print(f" {label}: {value}")
if result["last_import_date"]:
formatter.console.print(f" Last import: {result['last_import_date']}")

if result["column_details"]:
formatter.console.print()
table = Table(title="Columns")
table.add_column("Name", style="bold cyan")
table.add_column("Type", style="dim")
table.add_column("Nullable", style="dim")
for col in result["column_details"]:
table.add_row(
col["name"],
col.get("type", ""),
"yes" if col.get("nullable") else "",
)
formatter.console.print(table)
6 changes: 6 additions & 0 deletions src/keboola_agent_cli/commands/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,12 @@

kbagent storage table-detail --project NAME --table-id TABLE_ID [--branch ID]
Show detailed table info: columns (with types if available), primary key, row count, size, last import date.
Also returns `definition` -- the typed-table layout (primaryKeysNames plus, on BigQuery,
timePartitioning / rangePartitioning / clustering), null for an untyped table. This is how you
VERIFY a `create-table` + `swap-tables` repartition actually took effect: the table id is
unchanged either way, so only the layout tells the two apart. Human mode prints Partitioning /
Clustering rows when present. The LIST endpoint does not return it, so `storage tables` has no
equivalent field.
Uses production by default; pass --branch to query a dev branch explicitly.

kbagent storage create-bucket --project NAME --stage STAGE --name BUCKET_NAME [--description D] [--backend B] [--branch ID]
Expand Down
30 changes: 2 additions & 28 deletions src/keboola_agent_cli/commands/storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
map_error_to_exit_code,
resolve_branch,
)
from ._storage_format import render_table_detail

storage_app = typer.Typer(help="Browse and manage storage buckets, tables, and files")

Expand Down Expand Up @@ -408,34 +409,7 @@ def storage_table_detail(
if formatter.json_mode:
formatter.output(result)
else:
formatter.console.print(f"[bold]Table:[/bold] {result['table_id']}")
formatter.console.print(f" Name: {escape(result['display_name'] or result['name'])}")
formatter.console.print(f" Bucket: {result['bucket_id']}")
formatter.console.print(f" Rows: {result['rows_count']:,}")
size_mb = result["data_size_bytes"] / (1024 * 1024)
formatter.console.print(f" Size: {size_mb:.2f} MB")
if result["primary_key"]:
formatter.console.print(f" Primary key: {', '.join(result['primary_key'])}")
if result["last_import_date"]:
formatter.console.print(f" Last import: {result['last_import_date']}")

if result["column_details"]:
formatter.console.print()
from rich.table import Table

table = Table(title="Columns")
table.add_column("Name", style="bold cyan")
table.add_column("Type", style="dim")
table.add_column("Nullable", style="dim")

for col in result["column_details"]:
table.add_row(
col["name"],
col.get("type", ""),
"yes" if col.get("nullable") else "",
)

formatter.console.print(table)
render_table_detail(formatter, result)


@storage_app.command("create-bucket", rich_help_panel=_BUCKETS)
Expand Down
11 changes: 9 additions & 2 deletions src/keboola_agent_cli/services/storage_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -668,8 +668,7 @@ def get_table_detail(
column_details = []
for col in columns:
col_info: dict[str, Any] = {"name": col}
meta = column_metadata.get(col, [])
for m in meta:
for m in column_metadata.get(col, []):
key = m.get("key", "")
value = m.get("value", "")
if key == "KBC.datatype.basetype":
Expand Down Expand Up @@ -708,6 +707,14 @@ def get_table_detail(
"rows_count": table.get("rowsCount") or 0,
"data_size_bytes": table.get("dataSizeBytes") or 0,
"is_alias": table.get("isAlias", False),
# Typed-table layout: primaryKeysNames plus, on BigQuery,
# timePartitioning / rangePartitioning / clustering. None for an
# untyped table. This is the only view of the registered layout
# reachable without BigQuery metadata access, and the only field
# that tells a completed `swap-tables` repartition from a failed
# one -- the table id is unchanged either way (#621). The LIST
# endpoint does not return it, so this stays detail-only.
"definition": table.get("definition"),
"last_import_date": table.get("lastImportDate", ""),
"last_change_date": table.get("lastChangeDate", ""),
"created": table.get("created", ""),
Expand Down
Loading
Loading