Skip to content

fix(storage): stamp KBC.createdBy.branch.id on auto-materialized buckets (#224) - #225

Merged
padak merged 2 commits into
mainfrom
fix/issue-224-bucket-metadata
Apr 27, 2026
Merged

fix(storage): stamp KBC.createdBy.branch.id on auto-materialized buckets (#224)#225
padak merged 2 commits into
mainfrom
fix/issue-224-bucket-metadata

Conversation

@padak

@padak padak commented Apr 27, 2026

Copy link
Copy Markdown
Member

Summary

Closes #224.

When kbagent storage create-table --branch <ID> auto-materializes a bucket in a dev branch, kbagent now stamps it with the KBC.createdBy.branch.id system metadata that keboola/output-mapping requires on branched-storage projects. Without it, every subsequent transformation output mapping into that bucket aborts with:

Trying to create a table in the development bucket "X" on branch "Y" (ID "Z"),
but the bucket is not assigned to any development branch.

The error originates in output-mapping's Storage/BucketCreator::checkDevBucketMetadata: it scans bucket metadata for KBC.createdBy.branch.id (or KBC.lastUpdatedBy.branch.id) and throws when neither is present. Storage API does not auto-populate either key on POST /v2/storage/branch/<id>/buckets, so any client driving the table-create path -- kbagent CLI, the official Go CLI's EnsureBucketExists, third-party SDKs -- hits this on projects with the branched-storage feature enabled.

This patch fixes the kbagent path. The Go CLI carries the same bug; that fix belongs upstream.

Why kbagent users hit this and Web-UI users do not

The Web UI never goes through the bare POST /branch/<id>/buckets endpoint -- it always materializes through the table-create flow that output-mapping itself drives, so its own metadata write happens inline. CLI users (kbagent, Go CLI) call POST /branch/<id>/buckets directly via the auto-materialize helper, then hand the bucket off to a transformation runner that runs checkDevBucketMetadata on a bucket nobody stamped.

Implementation

File Change
src/keboola_agent_cli/services/storage_service.py After client.create_bucket(branch_id=...) in _ensure_bucket_exists_in_branch, call client.set_bucket_metadata(...) with KBC.createdBy.branch.id = str(branch_id) and provider="system". Wraps the metadata write in try/except KeboolaApiError: a 403/5xx is logged and the create-table call still proceeds, so users without bucket-metadata permission do not regress.
src/keboola_agent_cli/client.py set_bucket_metadata gains optional provider: str = "user" keyword (default unchanged for existing CLI describe paths). Auto-materialize passes provider="system" -- the API rejects user-provider writes on the reserved KBC.* namespace. New helper list_bucket_metadata(bucket_id, branch_id=None) for E2E verification.
src/keboola_agent_cli/changelog.py + pyproject.toml Bump to 0.25.1; one-line changelog entry under that version (make version-sync keeps plugin.json / marketplace.json aligned).

Failure mode handling for the metadata write is intentional: if it fails, the bucket still exists in the branch -- the user gets the original output-mapping error if they try to write through a transformation runner, which is no worse than today, and they get logger.warning that explains why.

Tests

Unit (run with make test):

  • TestSetBucketMetadata::test_set_bucket_metadata_system_provider -- verifies provider="system" lands in the form body, required for KBC.* keys.
  • TestListBucketMetadata (2 tests) -- new client helper covering production and --branch paths.
  • TestCreateTableService::test_branch_auto_materialize_bucket_on_404 -- asserts set_bucket_metadata is called with the exact payload after create_bucket.
  • TestCreateTableService::test_branch_auto_materialize_metadata_failure_does_not_abort -- new test: set_bucket_metadata raising 403 must not abort create_table.
  • TestCreateTableService::test_branch_no_materialize_when_bucket_exists -- asserts set_bucket_metadata is not called when the bucket already exists (we only stamp buckets we just created).

E2E (make test-e2e, requires E2E_API_TOKEN + E2E_URL):

  • TestE2EStorageNativeTypesAndBranchMaterialize::test_native_types_and_branch_materialize -- extended to assert exactly one KBC.createdBy.branch.id entry exists on the auto-materialized bucket, with value == str(branch_id) and provider == "system".

End-to-end manual verification on the kbagent E2E project (10546, branched storage enabled): clean dev branch + storage create-table --branch <id> + Snowflake transformation with destination: in.c-issue224.MY_TABLE_typed now succeeds (job 1305349305, 1 row imported, manifest updated, download-table returns CSV body). Pre-fix runs of the same scenario reliably produced the issue #224 error (job 1305347861).

Test plan

  • CI green: make check (lint + format + skill + version + changelog + error-codes + tests).
  • make test-e2e against a project with branched storage enabled (the bug only fires on those; legacy projects silently lose the storage manifest sync, which is its own latent issue but out of scope here).
  • Smoke: kbagent storage create-table --project <branched-project> --branch <ID> --bucket-id in.c-fresh --name t --column id:INTEGER and confirm kbagent storage bucket-detail shows the new metadata key under metadata: [...].

padak added 2 commits April 27, 2026 16:24
Closes #224.

When `storage create-table --branch <ID>` triggers the auto-materialize
path (POST /v2/storage/branch/<id>/buckets), Storage API does not populate
KBC.createdBy.branch.id system metadata. On projects with the "branched
storage" feature enabled, every subsequent transformation output mapping
into that bucket fails with:

  Trying to create a table in the development bucket "X" on branch "Y"
  (ID "Z"), but the bucket is not assigned to any development branch.

The error is raised by keboola/output-mapping
(Storage/BucketCreator::checkDevBucketMetadata), which scans bucket
metadata for KBC.createdBy.branch.id or KBC.lastUpdatedBy.branch.id and
throws when neither is present.

Fix: after create_bucket in the dev-branch materialize path, set
KBC.createdBy.branch.id = <branch_id> with provider=system. Failures are
logged but do not abort the create-table call, so users without
bucket-metadata permission do not regress -- the runner will surface a
clearer message later if the bucket really cannot be assigned.

Same bug exists in keboola-as-code Go CLI's EnsureBucketExists, but that
fix belongs in a separate repo.

Client changes:
- KeboolaClient.set_bucket_metadata gains optional provider= keyword
  (default "user" preserved). Auto-materialize uses provider="system".
- New KeboolaClient.list_bucket_metadata helper for E2E verification.

Tests:
- test_set_bucket_metadata_system_provider (provider parameter)
- TestListBucketMetadata (new helper)
- test_branch_auto_materialize_bucket_on_404 (asserts metadata stamp)
- test_branch_auto_materialize_metadata_failure_does_not_abort (graceful)
- test_branch_no_materialize_when_bucket_exists (no spurious metadata)
- TestE2EStorageNativeTypesAndBranchMaterialize (asserts metadata round-trip)

Verified end-to-end against kbagent E2E project (10546) with branched
storage enabled: dev-branch transformation output mapping now succeeds
without manual metadata patches.
…lize

Refs #224, follow-up to #225.

Three places now describe the metadata stamping behaviour added in 0.25.1:

- `kbagent context` (commands/context.py): one-line note that auto-
  materialized buckets get KBC.createdBy.branch.id stamped, so
  transformation runners on branched-storage projects accept them as
  output destinations.
- Plugin reference `storage-types-workflow.md`: new "Branched-storage
  metadata stamp" subsection explains the why (output-mapping's
  checkDevBucketMetadata throws "not assigned to any development branch"
  without it), the what (KBC.createdBy.branch.id with provider=system),
  and the failure mode (best-effort; logged 403/5xx does not abort
  create-table). Cross-links to keboola/connection issue for the
  upstream Storage API fix.
- Plugin reference `gotchas.md`: new bullet under the storage create-table
  section warning that a bucket created bypassing kbagent (raw POST to
  the Storage API) will be missing the stamp, and providing the manual
  re-stamp endpoint as an escape hatch.

No code changes; pure documentation that closes the contributor
checklist gap from CONTRIBUTING.md (kbagent context + plugin references
are mandatory updates for behavioural changes).
@padak
padak merged commit faeff12 into main Apr 27, 2026
1 check passed
@padak
padak deleted the fix/issue-224-bucket-metadata branch April 27, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transformation output fails when writing to dev-branch-only table: bucket not assigned to development branch

1 participant