feat(storage): native column types + dev-branch auto-materialize (0.25.0) - #223
Merged
Conversation
…5.0) Closes #192 and #222. storage create-table now accepts any native backend type with length (VARCHAR(40), NUMBER(18,2), TIMESTAMP_TZ, VARIANT, ...), plus per-column --not-null and --default flags. The hard-coded whitelist in constants.py is gone -- type/length validation is delegated to the Storage API, which has precise per-backend rules and returns actionable errors. In a dev branch, create-table auto-materializes the target bucket on 404 (mirrors the Go CLI's EnsureBucketExists). Response surfaces this via auto_created_bucket: bool; production writes (no --branch) never materialize anything. storage table-detail column_details now also surfaces native_type, length, and default from KBC.datatype.* metadata (backwards-compatible). Docs: - New reference: plugins/kbagent/skills/kbagent/references/storage-types-workflow.md - New TUTORIAL section 8 with a retype-after-profiling example + Snowflake cheat sheet - New VHS demo: docs/demos/demo-storage-types.tape -> docs/assets/demo-storage-types.gif - Updated keboola-expert.md (tool matrix + inline gotchas) - Updated gotchas.md with the 0.25.0 create-table behaviour Tests: 13 new service-level tests in tests/test_storage_write.py, plus a new E2E class TestE2EStorageNativeTypesAndBranchMaterialize that round-trips the native types through to Snowflake against a live project.
The previous tape chained branch create -> bash proc-substitution into
$BRANCH_ID -> create-table --branch -> branch delete. Live recording
produced errors because:
- bash proc-sub renders poorly in VHS (visible $ interpolation)
- if a previous run left active_branch_id stale in the sandbox config,
any command without explicit --branch hit 'Branch id ... does not
exists'
Rewritten as a flat four-step scenario against production:
1. storage create-bucket --stage in --name demo-types
2. storage create-table with VARCHAR(40) / NUMERIC(18,2) / TIMESTAMP_TZ
/ VARIANT / BOOLEAN + --not-null + --default
3. storage table-detail | jq .data.column_details -- shows native_type,
length, nullable, default survived the Snowflake round-trip
4. storage delete-bucket --force --yes -- cascades to the table
No shell variables, no dev branch, no state that bleeds between runs.
The previous tape used "Type "kbagent ... \\" + Enter" to split a long
command across multiple lines. VHS escapes \\" inside the Type string
literal as a real backslash, so the shell received:
kbagent ... --project padak-2-0 \
--bucket-id in.c-demo-types ...
which bash interpreted as a complete (broken) command followed by
several rogue --bucket-id / --column / --primary-key invocations
('command not found' on each).
Fix: use multiple Type "..." calls WITHOUT Enter between them. VHS
appends the chunks into one continuous shell line; the terminal soft-
wraps it visually but bash sees a single command. Verified
end-to-end against the kbagent-e2e sandbox -- bucket gets created,
table created, column_details inspected, bucket cleanly deleted.
Also added a comment block at the top of the tape documenting this
gotcha for the next person regenerating the demo.
Sleep after the create-bucket Enter: 2500ms -> 4500ms. Sleep after the create-table Enter: 4500ms -> 6500ms. Both screens were clearing too fast to read the success output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #192. Closes #222.
Summary
kbagent storage create-tablenow accepts native backend column types with length, plus per-column--not-null/--defaultflags. In a dev branch the target bucket auto-materializes on the first write (mirrors the official Keboola Go CLI'sEnsureBucketExists). Bumps the CLI to v0.25.0.The Storage API derives
basetypeautomatically and validates type/length per backend; the CLI does only syntactic validation (regex). Removing the hard-codedVALID_COLUMN_TYPESwhitelist means BigQuery/Redshift/Synapse native types pass through without per-backend code changes.What's in the box
storage_service.py)_parse_column_spec,_parse_default_assignments,_ensure_bucket_exists_in_branch.create_tablegainsnot_null_columns+defaultskwargs and surfacesauto_created_bucket: boolin the response.column_details(inget_table_detail) now also returnsnative_type,length,defaultfromKBC.datatype.*metadata -- backwards-compatible.commands/storage.py)--not-null COL(repeatable) and--default NAME=VALUE(repeatable). Updated--helpwith two-tier examples (basic + native types). Human-mode output prints a yellow note whenauto_created_bucket=true.hints/definitions/storage.py)commands/context.py)constants.py)VALID_COLUMN_TYPES(whitelist no longer maintained -- API validates).tests/test_storage_write.py. New E2E classTestE2EStorageNativeTypesAndBranchMaterializeround-trips VARCHAR(40)/NUMERIC(18,2)/TIMESTAMP_TZ/VARIANT/BOOLEAN to Snowflake (27s wall clock).references/storage-types-workflow.md, updatedgotchas.md, updatedkeboola-expert.md(tool matrix + inline gotchas), regeneratedSKILL.mddecision table.docs/demos/demo-storage-types.tape-> 408 KB rendered GIF. README + CLAUDE.md command surface updated.pyproject.toml,plugin.json,marketplace.jsonsynced viamake version-sync. New changelog entry.Test plan
make checkpasses (ruff lint + format-check + skill-check + changelog-check + 2311 unit tests + error-codes scan)make skill-genshows no diff after commitTestE2EStorageNativeTypesAndBranchMaterializepasses against the livepadak-2-0Snowflake project (27.88s)padak-2-0: create-bucket -> create-table with VARCHAR(40)/NUMERIC(18,2)/TIMESTAMP_TZ/VARIANT/BOOLEAN -> table-detail (column_detailsshowsnative_type+length+nullable+default) -> delete-bucket cascadepadak-2-0(response carriesauto_created_bucket: trueon first write,falseon subsequent writes to the same bucket)kbagent storage create-table --helpdisplays the new flags and exampleskbagent --hint clientandkbagent --hint serviceboth produce valid Python for the new flagskbagent contextdocuments the native-type syntax + branch materialize behaviourDemo
Backwards compatibility
--column name:TYPEforms keep working (STRING/INTEGER/NUMERIC/FLOAT/BOOLEAN/DATE/TIMESTAMP).Nonedefaults -- existing callers unaffected.column_detailsadds three new optional keys; old keys (type,nullable,description) unchanged.test_create_table_invalid_column_type) was rewritten astest_create_table_malformed_column_specbecause the whitelist is gone -- unknown type strings now flow to the API instead of being CLI-rejected.