Skip to content

feat(storage): native column types + dev-branch auto-materialize (0.25.0) - #223

Merged
padak merged 4 commits into
mainfrom
feat/storage-native-types
Apr 24, 2026
Merged

feat(storage): native column types + dev-branch auto-materialize (0.25.0)#223
padak merged 4 commits into
mainfrom
feat/storage-native-types

Conversation

@padak

@padak padak commented Apr 24, 2026

Copy link
Copy Markdown
Member

Closes #192. Closes #222.

Summary

kbagent storage create-table now accepts native backend column types with length, plus per-column --not-null / --default flags. In a dev branch the target bucket auto-materializes on the first write (mirrors the official Keboola Go CLI's EnsureBucketExists). Bumps the CLI to v0.25.0.

kbagent storage create-table --project p --bucket-id in.c-sales --name orders \
  --column 'pk:VARCHAR(40)' \
  --column 'amount:NUMERIC(18,2)' \
  --column 'ts:TIMESTAMP_TZ' \
  --column 'meta:VARIANT' \
  --column 'is_paid:BOOLEAN' \
  --primary-key pk \
  --not-null pk --not-null amount \
  --default amount=0 --default is_paid=false

The Storage API derives basetype automatically and validates type/length per backend; the CLI does only syntactic validation (regex). Removing the hard-coded VALID_COLUMN_TYPES whitelist means BigQuery/Redshift/Synapse native types pass through without per-backend code changes.

What's in the box

Layer Change
Service (storage_service.py) New helpers _parse_column_spec, _parse_default_assignments, _ensure_bucket_exists_in_branch. create_table gains not_null_columns + defaults kwargs and surfaces auto_created_bucket: bool in the response. column_details (in get_table_detail) now also returns native_type, length, default from KBC.datatype.* metadata -- backwards-compatible.
CLI (commands/storage.py) New --not-null COL (repeatable) and --default NAME=VALUE (repeatable). Updated --help with two-tier examples (basic + native types). Human-mode output prints a yellow note when auto_created_bucket=true.
Hints (hints/definitions/storage.py) Service mode passes the new kwargs; client mode notes the column-dict shape conversion.
Context (commands/context.py) Updated signature + native-types examples + branch materialize behaviour.
Constants (constants.py) Removed VALID_COLUMN_TYPES (whitelist no longer maintained -- API validates).
Tests 13 new service-level tests in tests/test_storage_write.py. New E2E class TestE2EStorageNativeTypesAndBranchMaterialize round-trips VARCHAR(40)/NUMERIC(18,2)/TIMESTAMP_TZ/VARIANT/BOOLEAN to Snowflake (27s wall clock).
Skill / agent New references/storage-types-workflow.md, updated gotchas.md, updated keboola-expert.md (tool matrix + inline gotchas), regenerated SKILL.md decision table.
Docs New TUTORIAL §8 with Snowflake type cheat sheet + retype-after-profiling example. New VHS demo docs/demos/demo-storage-types.tape -> 408 KB rendered GIF. README + CLAUDE.md command surface updated.
Versioning pyproject.toml, plugin.json, marketplace.json synced via make version-sync. New changelog entry.

Test plan

  • make check passes (ruff lint + format-check + skill-check + changelog-check + 2311 unit tests + error-codes scan)
  • make skill-gen shows no diff after commit
  • E2E test TestE2EStorageNativeTypesAndBranchMaterialize passes against the live padak-2-0 Snowflake project (27.88s)
  • Manual round-trip on padak-2-0: create-bucket -> create-table with VARCHAR(40)/NUMERIC(18,2)/TIMESTAMP_TZ/VARIANT/BOOLEAN -> table-detail (column_details shows native_type + length + nullable + default) -> delete-bucket cascade
  • Dev-branch auto-materialize verified end-to-end on padak-2-0 (response carries auto_created_bucket: true on first write, false on subsequent writes to the same bucket)
  • kbagent storage create-table --help displays the new flags and examples
  • kbagent --hint client and kbagent --hint service both produce valid Python for the new flags
  • kbagent context documents the native-type syntax + branch materialize behaviour

Demo

storage create-table with native types

Backwards compatibility

  • All existing --column name:TYPE forms keep working (STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP).
  • Service signature gains two new kwargs with None defaults -- existing callers unaffected.
  • column_details adds three new optional keys; old keys (type, nullable, description) unchanged.
  • One CLI test (test_create_table_invalid_column_type) was rewritten as test_create_table_malformed_column_spec because the whitelist is gone -- unknown type strings now flow to the API instead of being CLI-rejected.

padak added 4 commits April 25, 2026 00:06
…5.0)

Closes #192 and #222.

storage create-table now accepts any native backend type with length
(VARCHAR(40), NUMBER(18,2), TIMESTAMP_TZ, VARIANT, ...), plus per-column
--not-null and --default flags. The hard-coded whitelist in constants.py
is gone -- type/length validation is delegated to the Storage API, which
has precise per-backend rules and returns actionable errors.

In a dev branch, create-table auto-materializes the target bucket on 404
(mirrors the Go CLI's EnsureBucketExists). Response surfaces this via
auto_created_bucket: bool; production writes (no --branch) never
materialize anything.

storage table-detail column_details now also surfaces native_type,
length, and default from KBC.datatype.* metadata (backwards-compatible).

Docs:
- New reference: plugins/kbagent/skills/kbagent/references/storage-types-workflow.md
- New TUTORIAL section 8 with a retype-after-profiling example + Snowflake cheat sheet
- New VHS demo: docs/demos/demo-storage-types.tape -> docs/assets/demo-storage-types.gif
- Updated keboola-expert.md (tool matrix + inline gotchas)
- Updated gotchas.md with the 0.25.0 create-table behaviour

Tests: 13 new service-level tests in tests/test_storage_write.py, plus
a new E2E class TestE2EStorageNativeTypesAndBranchMaterialize that
round-trips the native types through to Snowflake against a live project.
The previous tape chained branch create -> bash proc-substitution into
$BRANCH_ID -> create-table --branch -> branch delete. Live recording
produced errors because:
- bash proc-sub renders poorly in VHS (visible $ interpolation)
- if a previous run left active_branch_id stale in the sandbox config,
  any command without explicit --branch hit 'Branch id ... does not
  exists'

Rewritten as a flat four-step scenario against production:
  1. storage create-bucket --stage in --name demo-types
  2. storage create-table with VARCHAR(40) / NUMERIC(18,2) / TIMESTAMP_TZ
     / VARIANT / BOOLEAN + --not-null + --default
  3. storage table-detail | jq .data.column_details  -- shows native_type,
     length, nullable, default survived the Snowflake round-trip
  4. storage delete-bucket --force --yes  -- cascades to the table

No shell variables, no dev branch, no state that bleeds between runs.
The previous tape used "Type "kbagent ... \\" + Enter" to split a long
command across multiple lines. VHS escapes \\" inside the Type string
literal as a real backslash, so the shell received:
  kbagent ... --project padak-2-0 \
  --bucket-id in.c-demo-types ...
which bash interpreted as a complete (broken) command followed by
several rogue --bucket-id / --column / --primary-key invocations
('command not found' on each).

Fix: use multiple Type "..." calls WITHOUT Enter between them. VHS
appends the chunks into one continuous shell line; the terminal soft-
wraps it visually but bash sees a single command. Verified
end-to-end against the kbagent-e2e sandbox -- bucket gets created,
table created, column_details inspected, bucket cleanly deleted.

Also added a comment block at the top of the tape documenting this
gotcha for the next person regenerating the demo.
Sleep after the create-bucket Enter: 2500ms -> 4500ms.
Sleep after the create-table Enter: 4500ms -> 6500ms.

Both screens were clearing too fast to read the success output.
@padak
padak merged commit b093b62 into main Apr 24, 2026
1 check passed
@padak
padak deleted the feat/storage-native-types branch April 24, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant