fix(component): emit _keboola block in flow scaffold _config.yml - #652
Conversation
_build_flow_config_yml() was the only scaffold builder that never appended
the _keboola: {component_id: ...} footer every other category gets from
_build_config_yml(). Without it, sync push resolves the component of an
untracked local config from _keboola.component_id (_find_untracked_configs
in sync_service.py), falls back to "unknown", and a scaffolded flow can
never be pushed from disk.
Append the same footer (plus the "config_id will be assigned by Keboola on
first push" NOTE comment _build_config_yml uses) to the flow builder.
Add a regression test parametrized across every scaffold category
(extractor, sql/python transformation, custom Python app, flow) asserting
_config.yml always carries a _keboola block with the right component_id.
Fixes #650
| that behaviour is unchanged, and its `--log-tail-lines` is capped at the same | ||
| maximum as `job detail`'s. | ||
|
|
||
| ## A scaffolded `keboola.flow` config can now be pushed from disk (since vNEXT) |
There was a problem hiding this comment.
🟡 Gotcha tagged with placeholder version instead of a release
The new gotcha entry is tagged (since vNEXT), but the repo rules require every gotcha to name a released (since vX.Y.Z) version. vNEXT names no release. The GATE_RE in scripts/check_version_gates.py:51 matches only numeric versions, so the placeholder passes CI and can ship unnoticed.
Prompt for agents
CONTRIBUTING.md ('references/gotchas.md ... always tag with (since vX.Y.Z)') and CLAUDE.md convention #17 require every new gotcha to be tagged with a real released version, not a placeholder. This entry uses '(since vNEXT)'. Replace vNEXT with the actual version this fix ships in (the version being bumped in pyproject.toml / added to changelog.py for this release). If the exact number is not yet known because the version bump happens in a separate release PR, coordinate so the marker is set to the shipped version before or at release time; the scripts/check_version_gates.py gate will not catch vNEXT because its regex only matches numeric versions.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This is deliberate, not an oversight — it's the convention this repo is introducing in PR #648: feature/fix PRs land without a version bump or a changelog.py entry, and any doc gate that would otherwise need a real version number (here, the (since vX.Y.Z) tag on a new gotcha) is tagged with the literal placeholder vNEXT instead. A later, separate release PR bumps pyproject.toml, adds the changelog.py entry, and rewrites every vNEXT placeholder introduced since the last release to the actual shipped version.
GATE_RE in scripts/check_version_gates.py:51 only matches a numeric X.Y.Z, so it intentionally does not flag vNEXT — that's what lets this land without either a false-positive CI failure now or a stale/guessed version number that would need correcting later. No code change here; flagging for visibility since the convention isn't yet written down anywhere Devin's rule-check can see it.
| lines = [ | ||
| "#", | ||
| "# NOTE: config_id will be assigned by Keboola on first push", |
There was a problem hiding this comment.
🔍 Flow scaffold still omits the version field
_build_flow_config_yml emits no top-level version: line, unlike _build_config_yml which writes version: 2 (services/component_service.py:90). This PR adds only the _keboola block. Worth confirming sync push of a scaffolded flow succeeds end-to-end without a version field.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Good catch — this was a real lead, and chasing it down surfaced a bigger bug than the missing version: line alone.
_build_flow_config_yml also put phases/tasks at the top level of _config.yml. local_config_to_api (sync/config_format.py:173-207) only promotes parameters, input/output (under storage), and processors to the API configuration body on sync push — anything else at the top level is dropped unless it was preserved under _configuration_extra. So a scaffolded flow's phases/tasks were silently discarded on push, producing a flow with an empty configuration. It's also the mirror image of api_config_to_local (~line 110-166), which nests a pulled flow's phases/tasks under _configuration_extra — so the scaffold shape didn't match the shape that actually round-trips.
Fixed in 9474c45: the scaffold now emits top-level version: 2 (closing the gap you flagged) and nests phases/tasks under _configuration_extra, matching a pulled flow's local shape. Added test_scaffold_flow_round_trips_through_local_config_to_api in tests/test_component_service.py, which runs the generated scaffold through local_config_to_api and asserts phases/tasks survive into the returned configuration dict — it failed against the pre-fix top-level shape and passes now.
…extra Devin's review flagged that the flow scaffold's _config.yml has no top-level `version:` line; verifying that lead surfaced a deeper bug the version gap was a symptom of. local_config_to_api() (sync/config_format.py) only promotes `parameters`, `input`/`output` (under storage), and `processors` from a local _config.yml to the API configuration body -- everything else at the top level is dropped unless it was preserved under `_configuration_extra`. The flow scaffold from _build_flow_config_yml() put `phases`/`tasks` directly at the top level, so `sync push` of a scaffolded flow silently lost its entire definition and created a flow with an empty configuration. This is also the reverse of api_config_to_local(), which nests a pulled flow's phases/tasks under `_configuration_extra` -- the scaffold shape did not match the shape that round-trips. Fix: emit `version: 2` at the top level (closing the version gap Devin found) and nest phases/tasks under `_configuration_extra`, matching what a pulled flow's _config.yml looks like. The _keboola footer from the previous commit is unchanged. Added a regression test that runs the generated scaffold through local_config_to_api() and asserts `phases`/`tasks` survive into the API configuration dict, plus updated the existing flow scaffold test for the new nesting.
Root cause
_build_flow_config_yml()(src/keboola_agent_cli/services/component_service.py) generated thekeboola.flowscaffold's_config.ymlwithout the_keboola:footer block. Every other scaffold category (extractor/writer, SQL transformation, Python transformation, custom Python app) gets one from_build_config_yml():sync pushresolves the component of an untracked local config from_keboola.component_id(_find_untracked_configsinservices/sync_service.py, which readskeboola_meta.get("component_id", "unknown")). Without the block, a scaffolded flow always resolved to"unknown", so the documentedconfig new-> edit ->sync pushworkflow (scaffold-workflow.md) was broken for exactly this category -- a scaffolded flow could never be pushed from disk.Fix
Append the same
_keboola: {component_id: ...}footer to_build_flow_config_yml()(it already receivescomponent_id), plus the same "config_id will be assigned by Keboola on first push" NOTE comment convention_build_config_yml()uses. No other builder was touched, andgenerate_scaffold's public shape is unchanged.Tests (TDD)
TestGenerateScaffold::test_scaffold_every_category_has_keboola_block, parametrized across all five scaffold categories (extractor, SQL transformation, Python transformation, custom Python app, flow). Before the fix, 4/5 passed andflowfailed with_config.yml for keboola.flow must contain a _keboola block-- confirming the regression. After the fix, all 5 pass.uv run pytest tests/ -x -q(excluding e2e, which needs live env vars) -> 5936 passed, 61 skipped, 0 failed.ruff check src/ tests/andruff format --check .-> clean.scripts/check_version_gates.pyandscripts/check_command_sync.py-> no drift.Docs
scaffold-workflow.mdalready documented "The_keboola.component_idfield in_config.ymlis required for push to work" as a general rule -- it wasn't contradicted, just unenforced for flows, so no changes needed there.gotchas.mdentry tagged(since vNEXT)per this repo's release convention (version bump happens in a separate joint release PR).Fixes #650