Skip to content

Single-config sync: pull/push individual configuration by component+config ID #59

Description

@padak

Problem

Current sync pull and sync push operate on the entire project -- all configurations at once. For many real-world use cases, users need to work with a single specific configuration:

  • CI/CD deployment of data apps -- push just one Streamlit/data-app config after build
  • Quick iteration on a single transformation -- pull one config, edit, push back
  • Selective config management -- update just a specific extractor config without touching anything else
  • Lightweight usage -- no need for full sync init + project workspace when you just need one config

Proposed Solution

New CLI options for existing commands

# Pull a single configuration
kbagent sync pull --project my-proj --config keboola.python-transformation-v2/12345

# Push a single configuration
kbagent sync push --project my-proj --config keboola.python-transformation-v2/12345

# Shorthand: component-id and config-id as separate flags
kbagent sync pull --project my-proj --component-id keboola.python-transformation-v2 --config-id 12345

Standalone single-config mode (no sync init required)

For CI/CD and quick one-off operations, a lightweight mode that doesn't need a full .keboola/manifest.json:

# Pull single config to a directory (creates _config.yml + code files)
kbagent config pull --project my-proj --component-id keboola.python-transformation-v2 --config-id 12345 --output-dir ./my-config/

# Push single config from a directory
kbagent config push --project my-proj --component-id keboola.python-transformation-v2 --config-id 12345 --input-dir ./my-config/

# Push with auto-create (if config doesn't exist yet, create it)
kbagent config push --project my-proj --component-id keboola.python-transformation-v2 --input-dir ./my-config/ --create

CI/CD example (data app deployment)

# GitHub Actions example
- name: Deploy data app config to Keboola
  run: |
    kbagent config push \
      --project production \
      --component-id keboola.data-apps \
      --config-id ${{ vars.DATA_APP_CONFIG_ID }} \
      --input-dir ./data-app-config/

Technical Analysis

Complexity: Medium

The building blocks already exist:

  1. API client methods -- get_config_detail(), create_config(), update_config() + row equivalents are all implemented in client.py
  2. Config format conversion -- api_config_to_local() / local_config_to_api() in sync/config_format.py handle bidirectional YAML <-> API JSON
  3. Code extraction -- extract_code_files() / merge_code_files() in sync/code_extraction.py handle SQL/Python file splitting
  4. Row support -- list_config_rows(), create_config_row(), update_config_row() are implemented

What needs to be built

Option A: Filter mode (within existing sync)

  • Add --config filter to pull() and push() in SyncService
  • For pull: fetch only get_config_detail() + list_config_rows() instead of list_components_with_configs()
  • For push: filter diff changeset to only the specified config
  • Manifest tracking still applies (subset update)
  • Effort: ~2-3 days

Option B: Standalone mode (no sync init)

  • New config pull / config push commands in commands/config.py
  • Lightweight: reads/writes a single directory with _config.yml + code files
  • No manifest needed -- uses component_id + config_id directly
  • Reuses existing format conversion and code extraction
  • Needs: detect create vs update, handle rows, encrypted values
  • Effort: ~3-5 days

Recommended: Both options

  • Option A is quick and useful for developers already using sync
  • Option B unlocks the CI/CD use case which is the primary driver

Edge cases to handle

  • Encrypted values (#encrypted# markers) -- on push, skip fields that haven't changed to avoid overwriting secrets
  • Config rows -- need to handle row-level CRUD (add/update/delete individual rows)
  • Dev branch support -- both modes should respect --branch flag
  • Conflict detection in filter mode -- what if remote changed since last pull?
  • Config creation -- config push --create needs to handle first-time deployment

Open Questions

  1. Should standalone mode support rows, or just top-level config for v1?
  2. Should config push auto-encrypt values marked with # prefix? (Current sync push has this logic)
  3. Naming: config pull/push vs sync pull --config vs new deploy command group?

Metadata

Metadata

Assignees

No one assigned

    Labels

    epic/write-protectionSub-issue of #63 (Write Protection + Single-Config Sync)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions