Skip to content

feat(config): add config clone to duplicate a configuration whole - #596

Merged
padak merged 4 commits into
mainfrom
feat/config-clone-587
Aug 18, 2026
Merged

feat(config): add config clone to duplicate a configuration whole#596
padak merged 4 commits into
mainfrom
feat/config-clone-587

Conversation

@padak

@padak padak commented Aug 17, 2026

Copy link
Copy Markdown
Member

Closes #587 (remainder — the schema-validation bug was #589, already merged).

The problem

There was no way to copy a configuration, so people rebuilt the body from config detail output — typically copying configuration["parameters"] and nothing else. A configuration's root also carries storage, runtime and authorization, and dropping one is silent: the reporter lost runtime.parallelism, Keboola fell back to parallelism: 1, and a 65-row Snowflake writer ran strictly sequentially — 140 minutes instead of the expected 60-90, caught only by hand-comparing per-row job timestamps afterwards.

The fix

kbagent config clone --project P --component-id C --config-id ID --name N [--target-project P2] [--set PATH=VALUE ...] [--secret PATH=VALUE ...] [--dry-run]

Two paths, split by encryption — not an arbitrary design choice:

Same project (default): server-side copy via POST .../configs/{id}/versions/{v}/create. Nothing is rebuilt. Verified live against a real project: a 2-row source with a runtime sibling produced a 2-row clone carrying runtime.parallelism intact. --set PATH=VALUE is applied afterwards as a normal update on the copy, so an override can never be why a key went missing.

Cross project (--target-project): reassembled client-side, rows recreated one by one — because a Keboola ciphertext is scoped to the project it was encrypted in; copying it verbatim would produce a configuration that looks complete and fails at runtime, in a project nobody is watching. Any KBC:: value — in the parent body or in a row (rows[N].path) — makes the clone fail fast (exit 5), listing every affected path, until re-supplied via --secret PATH=VALUE (encrypted in the target project on write). --dry-run reports the same paths in missing_secrets instead of refusing, which is how you discover what to gather.

Storage bucket/table IDs are copied verbatim, never remappedsync clone is the command that does that; duplicating it here would be scope creep.

Structure

New services/_config_clone.py (not config_service.py, which make loc-check already flags as over its per-layer budget). New commands/_config_clone_cmd.py for the same reason on the commands side — mirrors the existing commands/_storage_snapshots.py pattern. KeboolaClient.create_config_copy is the one new client method.

Testing

  • 15 service tests (both paths, encrypted-value refusal at parent AND row level, row copying, client selection). Mutation-tested: disabling encrypted-value detection or row copying each fail exactly the tests that claim to cover them.
  • 9 CLI tests, including the human-output renderer. A first live run crashed on a human_formatter arity bug (lambda taking 1 arg instead of 2) that no service test could catch; re-introducing that bug fails 6 of the 9 CLI tests.
  • E2E step 19d: clones a config carrying a runtime sibling and two rows, asserts both survive.
  • make check: 5591 passed, 12 skipped.

Docs

CLAUDE.md, context.py (AGENT_CONTEXT), commands-reference.md, gotchas.md, keboola-expert.md (had to trim ~90 bytes elsewhere to stay under its 62000-byte cap). Version bumped to 0.85.0 with a full changelog entry.

Review

@zajca — could you take a look at this one specifically? The cross-project path (encrypted-value detection + refusal + row-by-row recreation) is the part I'd most want a second pair of eyes on.


Open in Devin Review

@padak
padak requested a review from zajca August 17, 2026 20:06

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 9 potential issues.

View 1 additional finding in Devin Review.

Open in Devin Review

Comment thread src/keboola_agent_cli/services/_config_clone.py Outdated
Comment thread src/keboola_agent_cli/services/_encryption.py Outdated
Comment thread src/keboola_agent_cli/services/_config_clone.py
Comment thread src/keboola_agent_cli/commands/_config_clone_cmd.py
Comment thread scripts/file_size_baseline.json Outdated
Comment thread src/keboola_agent_cli/services/_config_clone.py
Comment thread src/keboola_agent_cli/services/_config_clone.py
Comment thread src/keboola_agent_cli/services/_config_clone.py
Comment thread src/keboola_agent_cli/services/_encryption.py Outdated
zajca
zajca previously requested changes Aug 17, 2026

@zajca zajca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested based on the completed review findings.

padak added a commit that referenced this pull request Aug 18, 2026
Devin review on #596; zajca requested changes on the strength of it.

BLOCKING -- a re-supplied row secret landed in the parent body. Row
ciphertext is reported under a `rows[N].` prefix and the refusal check
accepted a `--secret` at that exact path, but the substitution applied
every override to the parent. So a cross-project clone of a config with
per-row credentials reported success while the copied rows still held
the source project's undecryptable ciphertext -- precisely the outcome
this command exists to prevent, discoverable only at runtime, in the
other project. Proven before fixing: set_nested_value put the value
under a literal "rows[0]" key in the parent.

List-element secret paths were unusable. `parameters.values.[0].#token`
raises ValueError in set_nested_value, so the CLI refused the clone with
an instruction that then crashed when followed. Paths now use a plain
index segment and point at the `value` field that actually gets
replaced: `parameters.values.0.value`.

Ciphertext under a plain (non-`#`) key is now refused. collect_secrets
only picks up `#` keys, so a `--secret` supplied for such a path would
have been written to the target project in plaintext. Refusing and
pointing at `kbagent encrypt values` + `--set` is the only option that
neither breaks the clone nor leaks the credential.

`--set` is encrypted on both paths. The same-project path wrote
overrides through a direct update_config, bypassing the Encryption API
-- and `--set 'parameters.db.#password=...'` is the documented way to
repoint a copy at another database, so this was a live plaintext write
that version history would keep. `--allow-plaintext-on-encrypt-failure`
was a no-op on that path; now it means what it says.

Also: cross-project inherits the source description instead of blanking
it; `--target-branch` is rejected on a same-project clone when it
differs from `--branch`, because the server-side copy writes into the
source's branch and silently using the wrong one is not acceptable; a
row failing mid-copy reports the created config id and how many rows
landed, so the partial clone can be found and removed.

Added the matching POST .../{config_id}/clone route -- CONTRIBUTING.md
mandates 1:1 CLI/HTTP coverage and I had skipped it without saying so.

The loc baseline is back to main's values. Raising a grandfathered
ceiling to fit two new lines is exactly what CONTRIBUTING.md forbids;
`config oauth-url` moved into its own private module instead, so
commands/config.py SHRANK to 1953 code lines from 2007.

7 new tests covering each fix.
@padak
padak force-pushed the feat/config-clone-587 branch from f8683f0 to edcb35e Compare August 18, 2026 01:37
@padak

padak commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

@zajca — review findings addressed in edcb35e, all nine threads replied to and resolved. Summary of what actually changed, since three of them were real defects rather than style:

The blocking one was real and I had missed it. A re-supplied row secret (--secret 'rows[0].parameters.#token=…') was applied to the parent body, not the row. The refusal check accepted the path, set_nested_value created a literal "rows[0]" key in the parent, and the copied row kept the source project's undecryptable ciphertext — while the command reported success. That is exactly the failure this command exists to prevent, and it would only have surfaced at runtime in the other project. My own test for row secrets only covered detection via --dry-run, never application, which is why it passed.

Two more that would have leaked or crashed:

  • --set 'parameters.db.#password=…' on the same-project path bypassed the Encryption API entirely and wrote plaintext to Storage (kept in version history). Repointing a copy at another database is the documented use case, so this was live traffic. Both paths now encrypt.
  • Ciphertext under a plain non-# key: I verified that a supplied replacement was not picked up by collect_secrets, so it reached the target project in the clear. Those paths are now refused outright rather than accepted, pointing at kbagent encrypt values + --set instead.

Also fixed: list-element secret paths were emitted as parameters.values.[0].#token, which raises ValueError when fed back — so the CLI was refusing the clone with an instruction that then crashed. Cross-project now inherits the source description; --target-branch is rejected on a same-project clone instead of silently writing to the source branch; a row failing mid-copy reports the created config id and how many rows landed.

Two process points you were right to flag via the review: I added the missing POST .../{config_id}/clone route (the 1:1 rule — I hadn't consciously skipped it, which is what the rule catches), and I reverted file_size_baseline.json to main's values. Raising a grandfathered ceiling by 2 lines to fit my own addition is precisely what CONTRIBUTING.md forbids; config oauth-url moved to its own module instead, so commands/config.py shrank to 1953 code lines from 2007.

7 new tests, make check green (5644 passed). Ready for another look when you have time.

padak added 4 commits August 17, 2026 23:48
Closes #587.

There was no way to copy a configuration, so people rebuilt the body from
`config detail` output -- typically taking `configuration["parameters"]`
and nothing else. A configuration's root also carries `storage`,
`runtime` and `authorization`, and dropping one is silent. The reporter
lost `runtime.parallelism`; Keboola falls back to `parallelism: 1`, so a
65-row Snowflake writer ran strictly sequentially -- 140 minutes instead
of the expected 60-90. Nothing errored. It surfaced only by
hand-comparing per-row job timestamps afterwards.

Two paths, and the split is not arbitrary -- it is encryption.

Within a project the Storage API copies server-side
(POST .../configs/{id}/versions/{v}/create), so nothing is rebuilt at
all. Verified live against a real project: the clone came back with
`runtime.parallelism` AND both rows of a 2-row source, so no client-side
row copying is needed. `--set PATH=VALUE` edits are applied afterwards as
a normal update on the new config, which means an override can never be
the reason a key went missing.

Across projects that endpoint is unavailable, and would be wrong anyway:
a Keboola ciphertext is scoped to the project it was encrypted in.
Copying it verbatim produces a configuration that looks complete and
fails at runtime, in a project nobody is watching. So the cross-project
path assembles the body itself, recreates rows one by one, and refuses
(exit 5) while any `KBC::` value is unaccounted for -- listing every
path, from the parent body and from rows (`rows[N].path`). Each is
re-supplied with `--secret PATH=VALUE` and encrypted in the TARGET
project on write. `--dry-run` reports those paths in `missing_secrets`
rather than refusing, which is how a caller discovers what to gather.

Storage mappings are copied verbatim: bucket and table IDs are NOT
remapped, and cross-project output says so out loud. `sync clone` is the
command that remaps; duplicating that here would be scope creep.

The flow lives in a new services/_config_clone.py rather than in
config_service.py, which `make loc-check` already reports as over its
per-layer budget with an explicit "split it first" instruction.

Tests: 15 service tests (both paths, encrypted-value refusal, row
copying, client selection) and 9 CLI tests. The cross-project tests were
written after the module, so I mutation-tested them -- disabling
encrypted detection and disabling row copying each fail exactly the
tests that claim to cover them. The CLI suite exists because a first
live run crashed on a human_formatter arity bug that no service test
could catch; re-introducing that bug fails 6 of the 9.

E2E step 19d duplicates a config carrying a runtime sibling and two rows,
and asserts both survive.
commands/config.py was at its grandfathered loc-check ceiling (2007 code
lines); the clone command pushed it 2 lines over. Moved the command body
into a new commands/_config_clone_cmd.py, mounted via register(), mirroring
the existing commands/_storage_snapshots.py pattern. Net effect: config.py
drops the ~230-line clone command down to a 2-line import+register hook.

Ran `make loc-baseline` to record the new (lower) code-line counts for
every grandfathered file, per its own documented purpose ("after a
split"). This is that split.

WIP checkpoint -- pausing for the weekend. Remaining before this is
ready for review:
- Verify Devin/kbagent-pr-reviewer haven't already run on stale state
- Open the PR against main (branch: feat/config-clone-587)
- Request review from zajca per the user's explicit instruction
Devin review on #596; zajca requested changes on the strength of it.

BLOCKING -- a re-supplied row secret landed in the parent body. Row
ciphertext is reported under a `rows[N].` prefix and the refusal check
accepted a `--secret` at that exact path, but the substitution applied
every override to the parent. So a cross-project clone of a config with
per-row credentials reported success while the copied rows still held
the source project's undecryptable ciphertext -- precisely the outcome
this command exists to prevent, discoverable only at runtime, in the
other project. Proven before fixing: set_nested_value put the value
under a literal "rows[0]" key in the parent.

List-element secret paths were unusable. `parameters.values.[0].#token`
raises ValueError in set_nested_value, so the CLI refused the clone with
an instruction that then crashed when followed. Paths now use a plain
index segment and point at the `value` field that actually gets
replaced: `parameters.values.0.value`.

Ciphertext under a plain (non-`#`) key is now refused. collect_secrets
only picks up `#` keys, so a `--secret` supplied for such a path would
have been written to the target project in plaintext. Refusing and
pointing at `kbagent encrypt values` + `--set` is the only option that
neither breaks the clone nor leaks the credential.

`--set` is encrypted on both paths. The same-project path wrote
overrides through a direct update_config, bypassing the Encryption API
-- and `--set 'parameters.db.#password=...'` is the documented way to
repoint a copy at another database, so this was a live plaintext write
that version history would keep. `--allow-plaintext-on-encrypt-failure`
was a no-op on that path; now it means what it says.

Also: cross-project inherits the source description instead of blanking
it; `--target-branch` is rejected on a same-project clone when it
differs from `--branch`, because the server-side copy writes into the
source's branch and silently using the wrong one is not acceptable; a
row failing mid-copy reports the created config id and how many rows
landed, so the partial clone can be found and removed.

Added the matching POST .../{config_id}/clone route -- CONTRIBUTING.md
mandates 1:1 CLI/HTTP coverage and I had skipped it without saying so.

The loc baseline is back to main's values. Raising a grandfathered
ceiling to fit two new lines is exactly what CONTRIBUTING.md forbids;
`config oauth-url` moved into its own private module instead, so
commands/config.py SHRANK to 1953 code lines from 2007.

7 new tests covering each fix.
Main already carried an unreleased 0.84.2 (billing credits #597,
config state-get/state-set #598, kbc->kbagent CI/CD skill #402), and
v0.84.1 is the newest published release. Folding clone into that same
unreleased version ships one release instead of two, and leaves 0.85.0
free for the `tool` group removal it is already promised to (epic #390
phase 3) -- the doc references to that removal deliberately still say
0.85.0.

Version files, the changelog key (clone notes merged above the existing
0.84.2 entries) and the since-tags in CLAUDE.md, context.py, gotchas.md,
commands-reference.md, keboola-expert.md and the E2E docstring all move
to 0.84.2.
@padak
padak force-pushed the feat/config-clone-587 branch from edcb35e to a5c8bbc Compare August 18, 2026 03:54
@padak
padak dismissed zajca’s stale review August 18, 2026 04:08

Dismissing as stale, with the PR author's authorisation. This review was submitted against f8683f0; HEAD is now a5c8bbc. All nine findings it referenced (via the Devin pass) have been fixed, each replied to on its thread and resolved — including the blocking one where a re-supplied row secret was written to the parent body instead of the row. Full summary: #596 (comment). Re-review welcome; not blocking the 0.84.2 release on it.

@padak
padak merged commit b47a3c4 into main Aug 18, 2026
4 checks passed
@padak
padak deleted the feat/config-clone-587 branch August 18, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

config new: cloning a config by hand silently drops sibling keys like runtime (parallelism) — no clone command, examples show only parameters

2 participants