feat(sync): run the runtime-safety script guard on the push path (follow-up to #686) - #696
Merged
Merged
Conversation
padak
marked this pull request as ready for review
August 25, 2026 13:01
`normalize_blocks_codes_script` -- the #245/#274 guard that turns a `parameters.blocks[].codes[].script` string into an array and re-splits an element packing several `;`-separated statements -- ran on `config update` (0.28.0) and `transformation edit/create` (0.30.8) but never on `sync push`, the one remaining deploy route to the Storage API. After #686 parts 2+3 the guard is a no-op on the GitOps path by construction (`merge_code_files` rebuilds the blocks through the single canonical producer), so this is a regression backstop rather than a correctness fix. It does still cover the shape that bypasses code extraction entirely: a hand-authored `_config.yml` carrying `parameters.blocks` inline with no companion `transform.sql`, which `merge_code_files` passes through verbatim -- a shape the Storage API accepts and the job runtime rejects. - new `guard_script_shape()` in `_sync_push_ops.py` wraps the helper unchanged and shapes its records into push-envelope warnings; - called in `push_create` / `push_update` after `merge_code_files` + `local_config_to_api`, before encryption and send; - called in the Phase C variables backfill too: it re-PUTs the WHOLE body, so it is the last write a freshly-created transformation receives; - rows are deliberately NOT guarded: code extraction is config-level only (`merge_code_files` is never called for a row) and the sibling `config row-create` / `row-update` path is likewise unguarded; - records surface as `warnings[]` entries with `change_type: "script_normalization"` (`path` / `action` / `after_length` kept), so human mode prints them through the existing push-warning loop and `--json` carries them structurally. `config update` keeps its dedicated `normalizations` key; a push envelope spans many configs, so each record carries its own identity. `warnings[]` element type widened to `dict[str, Any]` -- the records carry a non-string `after_length`. Docs: `(since vNEXT)` notes in gotchas.md and sync-workflow.md.
padak
force-pushed
the
feat/issue-686-followup-push-runtime-guard
branch
from
August 25, 2026 13:21
0f1bd55 to
4b49eb5
Compare
padak
added a commit
that referenced
this pull request
Aug 25, 2026
CONTRIBUTING.md release step 4 keeps version tags off headings: resolving the placeholder changes the generated anchor slug and breaks inbound links. PR #697 did this pass, but #694 and #696 landed their headings around the same time and fell outside its grep. - gotchas.md: two headings, tag moved to the section's first body line - sync-workflow.md: heading reworded to 'Migrating a legacy sync tree', with the version stated on a body line No inbound links reference the affected anchors (verified by grep).
This was referenced Aug 25, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #694 (
claude/issue-686-pr-draft-867ec7) — the diff below is only this PR's commit. Follow-up to #694 / issue #686, as its "Follow-up" note announced. Retarget the base tomainonce #694 merges.Why
normalize_blocks_codes_scriptis the runtime-safety guard forparameters.blocks[].codes[].script: it turns a string into an array (#245) andre-splits an element that packs several
;-separated statements (#274). TheStorage API accepts both broken shapes silently; the Keboola runtime then
fails the job with
Expected "array", but got "string"orActual statement count N did not match the desired statement count 1.It has run on
config updatesince 0.28.0 and ontransformation edit/createsince 0.30.8.
sync push— the GitOps deploy route — was the one path to theStorage API without it.
Belt and braces, not a fix
After #694's parts 2+3 the guard is a no-op on this path by construction:
merge_code_filesrebuildsparameters.blocksfromtransform.sqlthrough thesingle canonical producer (
canonical_sql_script), and the boundary markersmake that round-trip lossless. Wiring it in is a regression backstop — if a
future change to code extraction or the canonical producer regresses the shape,
the push is corrected instead of deploying a job that crashes at runtime.
Its semantics are untouched; it is called, not modified.
It is also not purely theoretical today. One shape bypasses code extraction
entirely: a hand-authored
_config.ymlcarryingparameters.blocksinlinewith no companion
transform.sql._merge_sql_transformationreturns earlywhen the file is absent, so those parameters reach the API verbatim. Two of the
new tests fail on
origin/claude/issue-686-pr-draft-867ec7for exactly thatreason (the string arrives at the API as a string).
What changed
guard_script_shape()inservices/_sync_push_ops.py— wraps the helperunchanged and shapes its records into push-envelope warnings.
push_createandpush_update, aftermerge_code_files()local_config_to_api()produce the APIconfigurationbody, beforeencryption and send.
_sync_bindings. _apply_variable_binding): it re-PUTs the whole configuration, so it is thelast write a freshly-created transformation receives — an unguarded body there
would undo what
push_createhad just fixed. Phase D (flow task remap) is notguarded: flows carry no
blocks/codes, so the call would be dead code.merge_code_filesis never called for a row, and SQL/Python transformationsare row-less components whose blocks live in the root config's
parameters.The sibling CLI path (
config row-create/config row-update) is likewiseunguarded, so guarding here would diverge from the convention this PR is
mirroring. Say the word if you want it symmetric anyway.
What is surfaced, where
config update's convention is a dedicatednormalizationskey on itsenvelope. A push envelope spans many configs, so each record instead carries its
own identity and joins the existing
warnings[]list added by #694:{ "change_type": "script_normalization", "component_id": "keboola.snowflake-transformation", "config_id": "cfg-sql", "config_path": "transformation/keboola.snowflake-transformation/raw-data-processing", "path": "parameters.blocks[0].codes[0].script", "action": "sql_split", "before_type": "str", "after_type": "list", "after_length": 2, "message": "Normalized …/cfg-sql … before the write (…): the local files held a script shape the Keboola runtime rejects. Run 'kbagent sync pull' …" }--json: structural, underwarningson the push envelope.commands/sync.py— no change needed there, which is the point of reusing thechannel. Both are pinned by tests.
warnings[]element type widened fromdict[str, str]todict[str, Any](
after_lengthis an int).Tests —
tests/test_sync_push_script_guard.py(11)TDD: the three integration tests below were written first and failed on the
parent branch (the crashing shape reached the API verbatim).
scripton UPDATE → sent as an array, record surfaced with the rightaction/after_length/ identity;sql_resplit);scripton CREATE (hand-authored, never-pulled config dir);transform.sql→ push flow produceszero records, sends the canonical array, and still diffs in sync;
["SELECT 1", "SELECT 2"]);guard_script_shape(record shaping, silence on acanonical body,
warnings=Nonestill fixes the body);--jsonsurfacing.make checkgreen: 6200 passed, 12 skipped; ruff / ty / version-gates /command-sync / changelog / error-codes / sentinel-guards / loc-check all clean.
Release notes
No version bump, no
changelog.pyentry (per the release process). Docs tagged(since vNEXT)ingotchas.mdandsync-workflow.md.