Skip to content

Support shared/linked buckets from other projects in direct Storage access #35

Description

@beckavac

Problem

When planning to remove input mapping from a Snowflake transformation and replace table aliases with direct Keboola Storage paths (format: sapi_<project_id>."<bucket-id>"."<table-name>"), shared buckets sourced from other projects are not visible in the current project's Snowflake database.

Concrete example

Project Slevomat (ID: 226) has bucket in.c-db which is a shared bucket from a different project (not owned by project 226). When running SHOW SCHEMAS IN DATABASE sapi_226, the schema in.c-db does not appear — even though it is accessible via input mapping.

This means the direct-access path sapi_226."in.c-db"."orders" may not work, and the correct path likely points to a different Snowflake database (the source project's database).

Expected behavior

kbagent should be able to:

  1. Detect shared/linked buckets — when listing input tables or analyzing a transformation's input mapping, flag which buckets are shared from other projects vs. owned locally.
  2. Resolve the correct Snowflake path for shared buckets — return the full qualified path including the source project's database (e.g., sapi_<source_project_id>."<bucket-id>"."<table-name>").
  3. Ideally expose this via a command like:
kbagent --json bucket info --project slevomat --bucket in.c-db
# Should return: owner project, Snowflake database, direct access path

Use case

This is needed for the optimization pattern of removing input mapping from large transformations to eliminate the data-copy overhead (in this case: 17 minutes for 209 tables). Without knowing the correct Snowflake path for shared buckets, the migration cannot be completed safely.

Context

  • Project: Slevomat (ID: 226), stack: connection.keboola.com
  • Transformation: Slevomat Main Task (config ID: 746772863)
  • Affected bucket: in.c-db — 161 tables (77% of all input tables in the transformation)
  • The bucket appears in input mapping as in.c-db.* but its schema is absent from SHOW SCHEMAS IN DATABASE sapi_226

Proposed solution

  • Add a bucket info subcommand (or extend config detail) to resolve the Snowflake-level path for each input table, including cross-project shared buckets
  • In config detail output, add field snowflake_path alongside each input table entry

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions