feat(job): client-side idempotency key for run_job (#427) - #430
Merged
Conversation
Add an optional `--idempotency-key` to `job run` (and `Client.run_job`)
so an agentic orchestrator can replay an interrupted, side-effecting
build step without creating a duplicate job.
The Keboola Queue API `POST /jobs` accepts NO client-supplied idempotency
token -- verified against the live OpenAPI spec (v1.3.8) and the server
source (an internal `deduplicationId` exists but is daemon-only, never
read from the public create-job request). So dedup is implemented
client-side:
- JobIdempotencyStore: a persistent `key -> {job_id, component, config}`
map (atomic write under an fcntl lock, 0600), mirroring ConfigStore.
- run_idempotent_job: probe-before-create. A prior still-running or
non-failed job is returned (no new side effect); a prior FAILED run is
re-run; `--force-rerun` always creates fresh. Reusing a key for a
different component/config raises INVALID_ARGUMENT rather than returning
the wrong job. A purged (404) prior re-runs; any other fetch error
propagates (never silently duplicates).
- Service path defaults the store to `<config-dir>/job_idempotency.json`;
the config-dir-free SDK facade requires the consumer to supply a store
(constructor or per-call) and raises if a key is given without one.
- JobResult gains `idempotent_replay`; CLI prints a note, --json carries it.
Exports JobIdempotencyStore from the package root. Docs synced (CLAUDE.md
command list, AGENT_CONTEXT, commands-reference, gotchas since v0.63.0,
library-workflow). Unit tests cover the store, the dispatch policy, the
service wiring, the facade, and the CLI flag. make check green (4068).
padak
added a commit
that referenced
this pull request
Jun 17, 2026
#426/#427/#428) (#432) * feat(lib): typed SDK return models + py.typed marker (#428) (#429) Typed pydantic return models (JobResult/QueryResult/UploadTableResult/SyncPushResult/ConfigDetailResult) + py.typed marker + typed facade wrappers. Part 1/3 of the SDK-hardening collector. * feat(job): client-side idempotency key for run_job (#427) (#430) Optional --idempotency-key on job run + Client.run_job: client-side dedup store (Queue API has no server token, verified vs live spec). Part 2/3 of the SDK-hardening collector. * feat(sync): clone composite + flow-task configId remap (#426) (#431) kbagent sync clone + SyncService.clone_project: copy a reference tree, apply bucket/variable/instance overrides, push fresh with flow-task + variable-link remap (new push Phase D). Part 3/3 of the SDK-hardening collector. * fix(review): address #432 review findings (Devin + kbagent-pr-reviewer) - context.py AGENT_CONTEXT (Devin BUG): the `sync clone` block was inserted between the `sync push` one-liner and its multi-paragraph continuation, so push semantics (encryption fail-closed, fresh-CREATE, --branch) mis-read as clone properties. Move the clone block AFTER the full push description. - keboola-expert.md (NB-1): add a `sync clone` row to the §2 tool matrix so the agent recommends the composite for "provision a new project from a reference", not the manual pull+edit+push flow. - library-workflow.md (NB-2): clarify CloneResult DOCUMENTS the dict shape that SyncService.clone_project returns (the service returns a plain dict; wrap via model_validate) -- it is not returned as a typed instance. - test_result_models.py (NB-2): add CloneResult contract tests (embedded SyncPushResult, ok property, dry_run-without-push) + cover it in the base loop. - test_e2e.py (NB-3): document why the clone E2E step is --dry-run only (no fresh second project in the single-project E2E harness; push path is unit-covered). - job_idempotency_store.py (NIT-2): comment that force_rerun intentionally bypasses the collision guard (don't hoist the check out of the branch). - sync-workflow.md: note that `sync clone --dry-run` still writes --target-dir. NIT-1 (formatter: Any) intentionally NOT applied: all sibling _format_* helpers in sync.py use `Any` -- matching the established pattern. NB-4 (sync_service.py LOC split) tracked as a follow-up tech-debt task. make check green (4092).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 3 of the SDK-hardening collector branch (#426/#427/#428). Builds on the typed
JobResultfrom #428.What
Closes #427. Optional
--idempotency-keyonjob run(+Client.run_job(idempotency_key=...)) so a replayed, interrupted build step does not create a duplicate side-effecting job.Why client-side
The Keboola Queue API
POST /jobsaccepts no client-supplied idempotency token. Verified against the live OpenAPI spec (queue.keboola.com/docs/swagger.yaml, v1.3.8) and the server source (keboola/job-queueapps/public-api/src/JobDefinition.php): an internaldeduplicationIdexists (DB-unique, ~Nov 2023) but is daemon-only for fan-out child jobs and is never read from the public create-job request. So dedup must be client-side.How
JobIdempotencyStore— persistentkey -> {job_id, component_id, config_id, branch_id}map; atomic write under anfcntllock at0600, mirroringConfigStore. Corrupt-file tolerant (never wedges a run).run_idempotent_job— probe-before-create policy:idempotent_replay=true, no new side effect)error/terminated/cancelled) → re-run--force-rerun→ always freshINVALID_ARGUMENT(never returns the wrong job)<config-dir>/job_idempotency.json. The config-dir-free SDK facade requires the consumer to supply a store (constructor or per-call) and raisesValueErrorif a key is passed with no store.JobResult.idempotent_replayadded; CLI prints a note in human mode,--jsoncarries the flag.Scope / limits
Dedup is per store file (per machine). A replay from a different machine that does not share the file is not deduplicated. A follow-up upstream request to expose the Queue's internal
deduplicationIdwould enable DB-enforced cross-machine dedup; until then this is the safe interim.Docs
JobIdempotencyStoreexported from the package root. Synced: CLAUDE.md command list,AGENT_CONTEXT,commands-reference.md,gotchas.md(since v0.63.0),library-workflow.md.Tests
tests/test_job_idempotency_store.py(store CRUD, atomic/0600, corrupt-file, full dispatch policy incl. 404/500/collision/force-rerun), service wiring intests/test_services.py, facade intests/test_lib.py, CLI flag forwarding + replay note intests/test_cli.py.make checkgreen (4068 passed). No new CLI command → no E2E command required (the dedup logic is client-side, covered by unit tests).