feat: SDK hardening -- typed contracts, idempotent run_job, sync clone (#426/#427/#428)

[padak]

Collector PR for the three SDK-hardening issues. Each landed as its own reviewed sub-PR into this branch:

• feat(lib): typed SDK return models + py.typed marker (#428) #429 → feat(sdk): typed return models + py.typed for the service layer #428: typed SDK return models + py.typed
• feat(job): client-side idempotency key for run_job (#427) #430 → feat(sdk): idempotency keys for run_job (and push) — safe replay after interruption #427: client-side idempotency key for run_job
• feat(sync): clone composite + flow-task configId remap (#426) #431 → feat(sync): clone_project composite — clone + parameterize + push from a reference snapshot #426: sync clone composite + flow-task configId remap

All three harden the importable in-process SDK (keboola_agent_cli.Client + service layer) seeded by the 0.61.0 library facade, so a programmatic consumer (the FIIA Scaffold Kit) gets typed contracts, replay-safe job runs, and a tested clone primitive instead of hand-rolling them.

#428 — typed contracts + py.typed

• PEP 561 py.typed marker (forced into the wheel) → downstream mypy/ty/IDEs treat the SDK as typed.
• Typed return models (JobResult, QueryResult, UploadTableResult, SyncPushResult, ConfigDetailResult) exported from the package root; all extra="allow" so backend drift never raises, populate_by_name + AliasChoices so model_validate(service_dict) works on raw API keys. Strategy: typed at the facade — the dict-returning service layer + --json CLI output are unchanged.
• Typed facade wrappers: Client.run_job/query_result/config_detail/upload_table.

#427 — idempotent run_job (client-side)

• The Queue API POST /jobs has no server idempotency token (verified against the live spec v1.3.8 + the keboola/job-queue server source — an internal deduplicationId exists but is daemon-only). So dedup is client-side: JobIdempotencyStore (atomic, fcntl-locked, 0600) + a probe-before-create policy.
• job run --idempotency-key KEY [--force-rerun]; Client.run_job(idempotency_key=..., idempotency_store=...). Prior non-failed → returned (idempotent_replay); prior failed → re-run; key reused for a different config → INVALID_ARGUMENT.

#426 — sync clone + flow remap

• kbagent sync clone / SyncService.clone_project → typed CloneResult: copy a reference tree, apply declarative bucket_map/variable_values/instance_rename overrides, push fresh.
• New push Phase D remaps keboola.flow task configIds reference→ULID via created_id_map (generic; benefits any fresh-create push). Fresh-target guard + idempotent re-run.

Versioning

No version bump in this branch — it stays at 0.62.0. CI's version-check only enforces plugin.json↔pyproject consistency (both 0.62.0). The version bump to 0.63.0 + the changelog entry are the agreed joint final step before merge; a ready-to-apply changelog entry is provided in a PR comment.

Verification

make check green on the merged branch: 4089 passed, 8 skipped. Per-phase doc-sync (CLAUDE.md, AGENT_CONTEXT, commands-reference, gotchas (since v0.63.0), workflow refs) done; command-sync-check + skill-check pass.

---

[devin-ai-integration]

Devin Review found 2 potential issues.

[padak]

Review of #432 — feat: SDK hardening -- typed contracts, idempotent run_job, sync clone (#426/#427/#428)

Generated by kbagent-pr-reviewer subagent. Verdict and findings below
are advisory; the human author retains every veto. CI-coverable issues
(lint, format, tests) are confirmed via make check, not duplicated here.

Summary

This collector PR delivers three hardening items for the importable in-process SDK: PEP 561 typed contracts (py.typed + pydantic result models at the facade), client-side idempotency for job run, and a sync clone composite that copies a reference synced tree, applies declarative overrides, and pushes fresh. The implementation is well-structured: Phase D (flow-task configId remap) is correctly scoped to created_configs only, so it cannot corrupt a normal update push; the idempotency store uses established atomic fcntl write patterns; the typed-at-facade strategy correctly leaves the service-layer dicts and --json CLI output unchanged. make check passes with 4089 tests. Verdict: APPROVE — no blocking findings. Four non-blocking items below are worth addressing before or shortly after the version bump to 0.63.0.

Verdict

• Verdict: APPROVE
• Blocking findings: 0
• Non-blocking findings: 4
• Nits: 2

Blocking findings

(none)

Non-blocking findings

[NB-1] plugins/kbagent/agents/keboola-expert.md:97 — "Cross-project migration" matrix row not updated for sync clone

The §2 Tool Selection Matrix row for "Cross-project migration" still points to kbagent sync pull + edit + kbagent sync push --dry-run as the first choice. After this PR, kbagent sync clone (0.63.0+) is the correct first-choice when the user's intent is "provision a new instance from a golden reference" — the existing row covers the manual edit-then-push migration pattern. An agent that reads only the matrix will recommend the manual multi-step flow for the clone use case, missing the safer composite. The file has 51 KB headroom under the 62 KB budget. Fix: add one row | Clone a reference project into a fresh target project | \kbagent sync clone --source ./golden --target alias --target-dir ./clone` (0.63.0+) | -- | manual copy + id-surgery + push | and add a note to the §3 gotchas: "sync clone --dry-runcreates--target-dir` on disk (applies overrides, saves manifest) but does NOT push — the directory persists after a dry run."

[NB-2] plugins/kbagent/skills/kbagent/references/library-workflow.md:143 — CloneResult documented as returned by service but service returns dict

The line reads: "(CloneResult is returned by the service-layer SyncService.clone_project -- see GitOps sync)". The actual method signature (src/keboola_agent_cli/services/sync_service.py:1820) docstring says "Returns: A dict matching CloneResult" — the service returns a plain dict, not a CloneResult instance. The result models for JobResult, QueryResult, etc. are validated at the facade layer via model_validate; CloneResult has no such facade wrapper (there is no Client.clone_project). A programmatic consumer who imports CloneResult and calls SyncService.clone_project directly will get a dict, not a typed object. Fix: change the sentence to "(CloneResult documents the dict shape returned by SyncService.clone_project; callers can wrap it via CloneResult.model_validate(result))" OR add CloneResult.model_validate(push_result) to the return in clone_project. Also: CloneResult is absent from tests/test_result_models.py despite being exported from the package root — add a contract test mirroring TestJobResult.test_maps_real_queue_keys.

[NB-3] tests/test_e2e.py:4117 — sync clone E2E only exercises --dry-run; real push not covered

The E2E step for sync clone uses --dry-run and re-points the clone at the SAME project, so it exercises copy + manifest repoint + diff but not the actual push (Phase D flow remap, fresh-target guard, idempotent re-run). Per CONTRIBUTING.md §Tests: "Every CLI command MUST have a corresponding E2E test in tests/test_e2e.py" — the intent is push coverage. A full live clone push requires a second dedicated test project (fresh, empty), which may not be available in the nightly suite. If that second project is not available, document the skip explicitly in the test class docstring with the reason, and consider adding a nightly-only TestE2EFullClone class gated on a E2E_CLONE_TARGET_URL / E2E_CLONE_TARGET_TOKEN pair to catch Phase D regressions against the real API.

[NB-4] src/keboola_agent_cli/services/sync_service.py — adds 321 LOC to a file already 2.7× the hard ceiling

sync_service.py grows from 3 712 lines (main) to 4 033 lines (+321). The hard ceiling for services/*.py is 1 500 LOC (CONTRIBUTING.md §Code Quality Patterns). The file has been over budget for several releases and is called out by name in the guidelines as the known extreme case. This PR adds material and does not split first, as the guideline requires ("the next PR that adds material to it should split first"). The new Phase D code (_resolve_flow_task_bindings, _remap_flow_tasks_in_place, _apply_flow_task_binding) and clone_project are natural candidates for extraction into _flow_bindings.py and sync/clone.py (the clone helpers are already in sync/clone.py; the service orchestration could follow). Recommend opening a split issue tagged tech-debt and linking it from the PR description so the debt is tracked.

Nits

• [NIT-1] src/keboola_agent_cli/commands/sync.py:517 — _format_clone_result(formatter: Any, result: dict[str, Any]) uses Any for formatter. All sibling _format_* helpers use the concrete OutputFormatter type. Narrowing it improves IDE navigation and catches a formatter.success() call that only exists on OutputFormatter (not on arbitrary Any).

• [NIT-2] src/keboola_agent_cli/services/job_idempotency_store.py:232 — the force_rerun=True path silently bypasses the component/config collision guard. The --help text says "Ignore any stored entry and always create a fresh job" and the error message for the collision case says "Use a distinct key, or --force-rerun to override." — so bypassing is intentional. But a one-line comment inside the if existing is not None and not force_rerun: guard would make it explicit that force_rerun is the documented escape hatch for the collision scenario, preventing a future reader from "fixing" it by hoisting the collision check outside the not force_rerun branch.

Verification log

• gh auth status → authenticated as padak ✓
• gh pr view 432 --json state,files,additions,deletions → OPEN, 27 files, +3206/-46, title matches feat: prefix ✓
• git rev-parse --abbrev-ref HEAD → feat/sdk-hardening-426-427-428 matches <branch> ✓
• CONTRIBUTING.md Plugin synchronization map walked row by row:
  • src/keboola_agent_cli/commands/context.py (AGENT_CONTEXT): sync clone + job run --idempotency-key documented ✓
  • CLAUDE.md ## All CLI Commands: both signatures updated ✓
  • permissions.py OPERATION_REGISTRY: "sync.clone": "write" added at line 937 ✓
  • plugins/kbagent/skills/kbagent/SKILL.md: sync clone row added ✓
  • plugins/kbagent/skills/kbagent/references/commands-reference.md: updated ✓
  • plugins/kbagent/skills/kbagent/references/gotchas.md: three new sections, all tagged (since v0.63.0) ✓
  • plugins/kbagent/skills/kbagent/references/library-workflow.md: updated ✓
  • plugins/kbagent/skills/kbagent/references/sync-workflow.md: sync clone section added ✓
  • plugins/kbagent/agents/keboola-expert.md: NOT updated → NON-BLOCKING [NB-1]
  • src/keboola_agent_cli/hints/definitions/*.py: not checked (no new command group)
• Layer compliance: grep typer/httpx in services/lib → empty ✓
• Phase D safety check: _resolve_flow_task_bindings only iterates created_configs (newly created this push); a normal update-only push has created_configs=[] and flow_binding.tasks_remapped=0 → no mutation of existing flows on non-clone pushes ✓
• Phase D scope: only remaps tasks whose (componentId, configId) appears in created_id_map (reference→ULID map keyed by pre-writeback id) → an existing flow that references a pre-existing (not created-this-push) config is left untouched ✓
• flow_binding scope: always assigned inside the with client: block before line 1774; early returns at lines 1505 (no_changes) and 1517 (dry_run) exit before that block ✓
• Idempotency store atomicity: record() opens lock file with O_RDONLY|O_CREAT, acquires LOCK_EX via fcntl.flock, does read-modify-write under lock, writes to .tmp via os.fdopen context manager (fd closed on exception), then os.replace (atomic rename). Pattern mirrors ConfigStore ✓
• force_rerun=True + different component/config: collision guard is inside not force_rerun branch → intentional bypass per help text ("ignore stored entry") → NIT-2 (clarity comment, not a bug) ✓
• CloneResult usage: exported from package root, but SyncService.clone_project returns dict not a typed CloneResult instance → [NB-2]
• make check (worktree, uv sync --extra server) → 4089 passed, 8 skipped, 15 warnings ✓
• sync_service.py LOC: 3712 (main) → 4033 (+321) — pre-existing budget violation, worsened slightly → [NB-4]
• Convention checks: no magic numbers, no raw error_code strings, no bare except:, no print() in production code, no tokens in logs ✓
• Backward compat: Client.query() still returns list[dict] (delegates to _run_query().rows) ✓; service-layer dicts unchanged; --json output unchanged ✓
• Behavior verification: could not run E2E against a live project (no fresh empty target project available); claimed behavior verified through code analysis and unit test coverage instead. Per-run note: test_e2e.py sync clone step exercises only --dry-run path against self → [NB-3]

Open questions for the author

• Is the sync clone --dry-run behavior (creating --target-dir on disk while not pushing) intentional and stable public contract? The help text ("Apply overrides and show the would-be diff without pushing") describes it correctly, but a user who runs --dry-run expecting zero disk side effects will find a new directory at --target-dir. If this is intentional, a note in the sync-workflow.md clone section would prevent confusion.

[padak]

Review findings addressed (commit 118e687)

Thanks to Devin Review and the kbagent-pr-reviewer. Resolution:

Devin

• 🟡 AGENT_CONTEXT: sync clone inserted mid-paragraph of sync push — real bug, fixed. The clone block split the push one-liner from its continuation, mis-attributing push semantics (encryption fail-closed, fresh-CREATE, --branch) to clone. Moved the clone block to after the full push description.
• 🚩 Version not bumped to 0.63.0 — intentional. The version bump + changelog entry are the agreed joint final step before merge (CI's version-check only enforces plugin.json↔pyproject consistency; both stay 0.62.0). A ready-to-apply 0.63.0 changelog entry is prepared for that step.

kbagent-pr-reviewer

• NB-1 keboola-expert.md tool matrix — fixed: added a sync clone row for "provision a new project from a reference".
• NB-2 library-workflow.md CloneResult wording + missing test — fixed: clarified CloneResult documents the dict shape clone_project returns (service returns a plain dict; wrap via model_validate); added TestCloneResult contract tests.
• NB-3 E2E clone only --dry-run — documented: added a docstring note explaining the single-project E2E harness has no fresh second project for a live clone push (push path is unit-covered); flagged a nightly full-clone E2E as follow-up.
• NB-4 sync_service.py LOC budget — acknowledged, deferred: tracked as a tech-debt split task (extract Phase-C/D bindings + clone orchestration). Not done here to keep the feature PR focused.
• NIT-1 _format_clone_result(formatter: Any) — not applied: verified that all sibling _format_* helpers in sync.py use Any; keeping it consistent with the established pattern.
• NIT-2 force_rerun collision-guard bypass — fixed: added a comment that the bypass is the documented escape hatch.

make check green (4092 passed). Re-running CI + Devin on the fix commit.