fix(storage): stamp KBC.createdBy.branch.id on auto-materialized buckets (#224) - #225
Merged
Conversation
Closes #224. When `storage create-table --branch <ID>` triggers the auto-materialize path (POST /v2/storage/branch/<id>/buckets), Storage API does not populate KBC.createdBy.branch.id system metadata. On projects with the "branched storage" feature enabled, every subsequent transformation output mapping into that bucket fails with: Trying to create a table in the development bucket "X" on branch "Y" (ID "Z"), but the bucket is not assigned to any development branch. The error is raised by keboola/output-mapping (Storage/BucketCreator::checkDevBucketMetadata), which scans bucket metadata for KBC.createdBy.branch.id or KBC.lastUpdatedBy.branch.id and throws when neither is present. Fix: after create_bucket in the dev-branch materialize path, set KBC.createdBy.branch.id = <branch_id> with provider=system. Failures are logged but do not abort the create-table call, so users without bucket-metadata permission do not regress -- the runner will surface a clearer message later if the bucket really cannot be assigned. Same bug exists in keboola-as-code Go CLI's EnsureBucketExists, but that fix belongs in a separate repo. Client changes: - KeboolaClient.set_bucket_metadata gains optional provider= keyword (default "user" preserved). Auto-materialize uses provider="system". - New KeboolaClient.list_bucket_metadata helper for E2E verification. Tests: - test_set_bucket_metadata_system_provider (provider parameter) - TestListBucketMetadata (new helper) - test_branch_auto_materialize_bucket_on_404 (asserts metadata stamp) - test_branch_auto_materialize_metadata_failure_does_not_abort (graceful) - test_branch_no_materialize_when_bucket_exists (no spurious metadata) - TestE2EStorageNativeTypesAndBranchMaterialize (asserts metadata round-trip) Verified end-to-end against kbagent E2E project (10546) with branched storage enabled: dev-branch transformation output mapping now succeeds without manual metadata patches.
…lize Refs #224, follow-up to #225. Three places now describe the metadata stamping behaviour added in 0.25.1: - `kbagent context` (commands/context.py): one-line note that auto- materialized buckets get KBC.createdBy.branch.id stamped, so transformation runners on branched-storage projects accept them as output destinations. - Plugin reference `storage-types-workflow.md`: new "Branched-storage metadata stamp" subsection explains the why (output-mapping's checkDevBucketMetadata throws "not assigned to any development branch" without it), the what (KBC.createdBy.branch.id with provider=system), and the failure mode (best-effort; logged 403/5xx does not abort create-table). Cross-links to keboola/connection issue for the upstream Storage API fix. - Plugin reference `gotchas.md`: new bullet under the storage create-table section warning that a bucket created bypassing kbagent (raw POST to the Storage API) will be missing the stamp, and providing the manual re-stamp endpoint as an escape hatch. No code changes; pure documentation that closes the contributor checklist gap from CONTRIBUTING.md (kbagent context + plugin references are mandatory updates for behavioural changes).
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #224.
When
kbagent storage create-table --branch <ID>auto-materializes a bucket in a dev branch, kbagent now stamps it with theKBC.createdBy.branch.idsystem metadata thatkeboola/output-mappingrequires on branched-storage projects. Without it, every subsequent transformation output mapping into that bucket aborts with:The error originates in
output-mapping'sStorage/BucketCreator::checkDevBucketMetadata: it scans bucket metadata forKBC.createdBy.branch.id(orKBC.lastUpdatedBy.branch.id) and throws when neither is present. Storage API does not auto-populate either key onPOST /v2/storage/branch/<id>/buckets, so any client driving the table-create path -- kbagent CLI, the official Go CLI'sEnsureBucketExists, third-party SDKs -- hits this on projects with the branched-storage feature enabled.This patch fixes the kbagent path. The Go CLI carries the same bug; that fix belongs upstream.
Why kbagent users hit this and Web-UI users do not
The Web UI never goes through the bare
POST /branch/<id>/bucketsendpoint -- it always materializes through the table-create flow thatoutput-mappingitself drives, so its own metadata write happens inline. CLI users (kbagent, Go CLI) callPOST /branch/<id>/bucketsdirectly via the auto-materialize helper, then hand the bucket off to a transformation runner that runscheckDevBucketMetadataon a bucket nobody stamped.Implementation
src/keboola_agent_cli/services/storage_service.pyclient.create_bucket(branch_id=...)in_ensure_bucket_exists_in_branch, callclient.set_bucket_metadata(...)withKBC.createdBy.branch.id = str(branch_id)andprovider="system". Wraps the metadata write intry/except KeboolaApiError: a 403/5xx is logged and the create-table call still proceeds, so users without bucket-metadata permission do not regress.src/keboola_agent_cli/client.pyset_bucket_metadatagains optionalprovider: str = "user"keyword (default unchanged for existing CLI describe paths). Auto-materialize passesprovider="system"-- the API rejects user-provider writes on the reservedKBC.*namespace. New helperlist_bucket_metadata(bucket_id, branch_id=None)for E2E verification.src/keboola_agent_cli/changelog.py+pyproject.toml0.25.1; one-line changelog entry under that version (make version-synckeepsplugin.json/marketplace.jsonaligned).Failure mode handling for the metadata write is intentional: if it fails, the bucket still exists in the branch -- the user gets the original
output-mappingerror if they try to write through a transformation runner, which is no worse than today, and they get logger.warning that explains why.Tests
Unit (run with
make test):TestSetBucketMetadata::test_set_bucket_metadata_system_provider-- verifiesprovider="system"lands in the form body, required forKBC.*keys.TestListBucketMetadata(2 tests) -- new client helper covering production and--branchpaths.TestCreateTableService::test_branch_auto_materialize_bucket_on_404-- assertsset_bucket_metadatais called with the exact payload aftercreate_bucket.TestCreateTableService::test_branch_auto_materialize_metadata_failure_does_not_abort-- new test:set_bucket_metadataraising 403 must not abortcreate_table.TestCreateTableService::test_branch_no_materialize_when_bucket_exists-- assertsset_bucket_metadatais not called when the bucket already exists (we only stamp buckets we just created).E2E (
make test-e2e, requiresE2E_API_TOKEN+E2E_URL):TestE2EStorageNativeTypesAndBranchMaterialize::test_native_types_and_branch_materialize-- extended to assert exactly oneKBC.createdBy.branch.identry exists on the auto-materialized bucket, withvalue == str(branch_id)andprovider == "system".End-to-end manual verification on the kbagent E2E project (
10546, branched storage enabled): clean dev branch +storage create-table --branch <id>+ Snowflake transformation withdestination: in.c-issue224.MY_TABLE_typednow succeeds (job 1305349305, 1 row imported, manifest updated,download-tablereturns CSV body). Pre-fix runs of the same scenario reliably produced the issue #224 error (job 1305347861).Test plan
make check(lint + format + skill + version + changelog + error-codes + tests).make test-e2eagainst a project with branched storage enabled (the bug only fires on those; legacy projects silently lose the storage manifest sync, which is its own latent issue but out of scope here).kbagent storage create-table --project <branched-project> --branch <ID> --bucket-id in.c-fresh --name t --column id:INTEGERand confirmkbagent storage bucket-detailshows the new metadata key undermetadata: [...].