Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/kbagent/agents/keboola-expert.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ been retired, so its absence is NOT a promise (see §1 Rule 6).
| Re-seed a table without losing schema / PK / dependents | `kbagent storage truncate-table --project P --table-id in.c-foo.data [--dry-run] [--yes]` -- rows only, uniformly async-via-job on every branch; batch via repeated `--table-id` | -- | drop + recreate (loses descriptions, PK, sharing edges, and breaks every downstream reference); deleting rows via raw SQL in a workspace (bypasses the Storage audit trail) |
| Back up / restore a table around a risky change | `kbagent storage snapshot-create --table-id ...` then, to restore, `kbagent storage table-from-snapshot --snapshot-id ID --bucket-id B --name NEW` -- restore is always a NEW table (`--name` REQUIRED, no overwrite): verify it, then `swap-tables`. See [snapshot-workflow.md](../skills/kbagent/references/snapshot-workflow.md) | `storage snapshots` / `snapshot-detail` to find one | exporting to CSV as a "backup" (loses column types + PK); `create-table --snapshot-id` (not a thing) |
| Debug a failed job | `kbagent job detail --project P --job-id J --json` + `kbagent job run ... --log-tail-lines 200` | `kbagent workspace from-transformation` for SQL repro | "I think the issue is..." without reading logs |
| Ad-hoc SQL / row-count / type audit | `kbagent workspace create` + `workspace load` (since vNEXT auto-CLONEs eligible tables, else COPY; `--load-type` forces one and fails loudly if ineligible; COPY > 1 GiB needs `--force` outside a TTY) + `kbagent workspace query --sql "..."` -- results are inline and fast but **capped at `--limit`, default 500**: check `statements[].truncated` / `total_rows`, use `COUNT(*)` for counts, `--full` for the complete set | `kbagent workspace from-transformation` for existing-transform debugging; `workspace list --qs-compatible` for data-app reuse; read-only input-mapping (`KBC_<STACK>_<PROJECT>`) to query prod with no load at all | trusting a default `SELECT *` as the full result; querying Storage via raw Snowflake credentials outside the workspace abstraction |
| Ad-hoc SQL / row-count / type audit | `kbagent workspace create` + `workspace load` (since vNEXT auto-CLONEs eligible tables, else COPY; `--load-type` forces one and fails loudly if ineligible; COPY > 1 GiB needs `--force` outside a TTY; on a `--timeout` [default 300s] the job keeps running server-side and now exits 4/retryable, not 1 -- retry or poll `GET /v2/storage/jobs/{id}`, don't treat it as a hard failure) + `kbagent workspace query --sql "..."` -- results are inline and fast but **capped at `--limit`, default 500**: check `statements[].truncated` / `total_rows`, use `COUNT(*)` for counts, `--full` for the complete set | `kbagent workspace from-transformation` for existing-transform debugging; `workspace list --qs-compatible` for data-app reuse; read-only input-mapping (`KBC_<STACK>_<PROJECT>`) to query prod with no load at all | trusting a default `SELECT *` as the full result; querying Storage via raw Snowflake credentials outside the workspace abstraction |
| Export a FILTERED or INCREMENTAL slice of a table (no workspace) | `kbagent storage download-table --table-id ... --where-column status --where-value active [--where-operator eq\|neq] [--changed-since "-2 days"]` -- server-side filter on the credential-only export path | `kbagent workspace query` with a `WHERE` clause when you need real SQL | downloading the whole table then filtering locally |
| Run Keboola SQL / read-write Storage Files from INSIDE a Python process you control | `from keboola_agent_cli import Client` -- stateless `Client(url, token)`; `.query(workspace_id, sql)`, `.files.upload/.read_bytes/.list`; no subprocess, no `serve`, no config-dir. See [library-workflow.md](../skills/kbagent/references/library-workflow.md) | the CLI or `kbagent serve` REST when you are NOT already inside Python | shelling out to the `kbagent` binary from Python you control; using it for open-ended exploration (fixed set of typed ops) |
| Inspect dev branch | `kbagent branch list --project P`, `kbagent branch use --project P --branch ID` | -- | acting on `main` when a dev branch exists |
Expand Down
4 changes: 2 additions & 2 deletions plugins/kbagent/skills/kbagent/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: >
Covers: exploring and searching configurations, job history, data
lineage, dev branches, workspace SQL debugging, GitOps config sync
(pull/push/diff/clone), bucket sharing and linking, encrypting secrets,
Storage tables, files, and snapshots (backup/restore), data apps
Storage tables, files, and snapshots, data apps
(deploy/logs/secrets), flows and schedules, members and invitations,
feature flags, OTLP data streams, scoped Storage tokens, the semantic
layer (models, metrics), the Developer Portal, browser login.
Expand All @@ -16,7 +16,7 @@ description: >
feature flag, flow schedule, invite member, SQL transformation edit,
sync action, keboola docs, table snapshot, auth, login, sign in,
PAYG credits, flow notifications, alert recipients, config trash,
restore config, undelete config.
restore config, undelete config, zero-copy clone, workspace load type.
---

# kbagent -- Keboola Agent CLI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,9 @@ kbagent branch use --project prod --branch <ID>

# 2. In a workspace, build a typed CTAS into a sibling table
kbagent workspace create --project prod
# Default auto-picks clone (fast, any size) when eligible, else copy --
# on a large table that falls back to copy, --json refuses without --force.
# See workspace-workflow.md "Load types" for the full clone/copy/view rules.
kbagent workspace load --workspace-id W --tables in.c-foo.data
kbagent workspace query --workspace-id W --sql "
CREATE TABLE \"in.c-foo.data_change_log\" AS
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,12 @@ kbagent --json workspace create --project ALIAS --name "typify-<table-slug>"
# -> workspace_id, host, database, schema, user, password
# SAVE THE PASSWORD -- it cannot be retrieved later.

# 2b. Load the typeless table
# 2b. Load the typeless table. Default (no --load-type) auto-picks clone
# when eligible -- zero-copy, seconds, any size. If it falls back to copy
# (see tables[].clone_ineligible_reason in the response) and the table is
# over 1 GiB, --json mode refuses without --force (no prompt to answer) --
# add --force once you've confirmed the copy cost is expected. See
# workspace-workflow.md "Load types" for the full clone/copy/view rules.
kbagent --json workspace load \
--project ALIAS \
--workspace-id W_ID \
Expand Down
44 changes: 34 additions & 10 deletions plugins/kbagent/skills/kbagent/references/workspace-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,23 @@ kbagent --json workspace load \
--tables in.c-bucket.table3 \
--preserve

# Force a zero-copy CLONE and skip the size guard on a very large table
# Force clone explicitly -- fails loudly (HTTP 400) instead of silently
# falling back to copy if the table turns out not to be clone-eligible.
# --force is irrelevant here: the size guard only ever applies to copy.
kbagent --json workspace load \
--project ALIAS \
--workspace-id WS_ID \
--tables in.c-bucket.table1 \
--load-type clone

# Force a real COPY of a table you know is huge and not clone-eligible --
# --force is required (there is no prompt under --json), and a COPY really
# moves the data so it can outrun the 300s default --timeout.
kbagent --json workspace load \
--project ALIAS \
--workspace-id WS_ID \
--tables in.c-bucket.huge_table \
--load-type clone \
--load-type copy \
--force \
--timeout 600

Expand All @@ -93,24 +104,37 @@ kbagent --json workspace query \

## Load types: clone / copy / view

*(since vNEXT, #687)* `workspace load` supports three load types. **Default (no `--load-type`)**:
kbagent decides per table, mirroring the server's own eligibility rules --
zero-copy `clone` when the table is on the same backend as the workspace
with a full (unfiltered) load and no external-schema/Analytics-Hub source
bucket, otherwise a plain `copy`. Check `tables[].clone_ineligible_reason`
in the JSON to see why a given table fell back to `copy`.
*(since vNEXT, #687)* **Cheapest first**: skip the load entirely if you can
(see "Don't load at all" below) > `clone` (zero-copy, seconds, any table
size) > `view` (zero-storage, narrower eligibility) > `copy` (real data
movement, size-guarded). `workspace load` supports all three as
`--load-type`. **Default (no `--load-type`)**: kbagent decides per table,
mirroring the server's own eligibility rules -- zero-copy `clone` when the
table is on the same backend as the workspace with a full (unfiltered) load
and no external-schema/Analytics-Hub source bucket, otherwise a plain
`copy`. Check `tables[].clone_ineligible_reason` in the JSON to see why a
given table fell back to `copy`.

- **`--load-type clone`** -- zero-copy, metadata-only, finishes in seconds
regardless of table size. Fails loudly (server HTTP 400) if any requested
table is not clone-eligible -- it does not silently fall back to `copy`.
- **`--load-type copy`** -- always a real data copy. A table over 1 GiB
needs interactive confirmation (TTY) or `--force` (JSON / scripted runs)
before the load starts.
needs interactive confirmation or `--force`. The confirmation prompt only
exists in human mode (no `--json`); `--json` (the default per Rule 1
above) never prompts, so a scripted `--json` load of a table that might
cross 1 GiB needs `--force` up front or it is refused outright. Check
`dataSizeBytes` via `storage table-detail` beforehand if you want to know
which one you'll need without a failed dry run.
- **`--load-type view`** -- zero-storage, read-only view, no data movement
at all. BigQuery: any same-backend bucket. Snowflake: only an
external-schema bucket with the project feature
`input-mapping-read-only-storage`.

**Verify what actually happened**: read `tables[].load_type` and
`tables[].data_size_bytes` in the JSON result rather than assume -- with
the auto-default, two requests that look identical can each take a
different path per table.

Loading a large table can take a while even as a copy job; `--timeout`
defaults to 300s for `workspace load` (other storage-job commands keep
60s). If it times out, the storage job **keeps running server-side** --
Expand Down