feat(flow): add flow triggers -- table triggers, not just cron (#714) - #719
Conversation
kbagent could see one of the (at least) three ways a flow gets started
automatically. `schedule list` / `schedule find` / `search` read
`keboola.scheduler` configs only, so a flow driven by a TABLE TRIGGER --
a separate Storage API resource, not a component config at all -- came back
"nothing found". Twice in one investigation that was read as "this flow has
no trigger", for a flow that was already live; the wrong conclusion was that
a working notification pipeline still needed manual scheduling.
Adds `kbagent flow triggers --project P --flow-id ID`, returning cron
schedules AND table triggers in one call, plus `GET /flows/{p}/{id}/triggers`
over `serve`.
The central point is the honesty of the negative answer, so cross-project
triggers -- a trigger-queue app config in ANOTHER project -- are reported as
`cross_project_triggers_checked: false` rather than as an empty list. An empty
list reads as "checked, found none", which is the exact false negative this
command exists to prevent. Detecting them means scanning every connected
project and resolving each candidate's parameters back to this project and
flow; that is not implemented, so it is declared rather than guessed.
`schedule list` / `schedule find` now say "No cron schedules found. Table
triggers and cross-project triggers were NOT checked -- see `kbagent flow
triggers`" instead of "No schedules found".
Wire contract taken from the Storage API's own controller rather than guessed:
the route is declared `isAvailableInBranch: false`, so table triggers are
production-only and are never advertised as branch-scoped. `?configurationId=`
is sent but the result is narrowed again client-side -- the Notification
Service accepts `?event=` and ignores it (#600), and an unverified server-side
filter would silently attribute another flow's trigger to this one.
The command lives in `commands/_flow_triggers.py` because `commands/flow.py`
is exactly at the 800-code-line commands soft ceiling; it is mounted flat onto
`flow_app`, so the permission key stays `flow.triggers` and `flow --help` is
unchanged.
Independent verification against keboola/connection (TriggerRepository:: findAllByFilter, exact match, AND-ed) confirms both ?component= and ?configurationId= are applied server-side, and the server's own testTriggersRestrictionsForReadOnlyUser E2E test confirms listing needs no elevated privilege -- any project Storage token sees every trigger. The docstrings and gotchas entry claimed this could not be confirmed; keep the client-side re-narrowing (defense in depth after #600) but state the verified facts instead of the open question.
padak
left a comment
There was a problem hiding this comment.
Independent verification of #714 / this PR
I verified the issue's claims and this PR's wire-contract assertions independently — against the keboola/connection source (commit de02f5d, src/Storage/Triggers/**, src/Controller/Storage/Triggers/**, plus its E2E suites tests/E2E/Storage/Common/TriggersTest.php and Backend/SOX/TriggersTest.php), the public keboola/storage-api-php-client (createTrigger/listTriggers/... on Client.php), and the actual keboola/component-orchestration-trigger-queue-v2 source.
Issue claim: confirmed
A full-repo grep (pre-PR) for storage/triggers, coolDown, tableTrigger, trigger-queue, kds-team.app-orchestration-trigger-queue-v2 returns zero hits — kbagent's only trigger concept really was keboola.scheduler configs. The false-negative failure mode described in #714 is real.
Wire contract: every assertion in this PR checks out
GET /v2/storage/triggerswith?component=/?configurationId=— confirmed, and the server does apply both filters (TriggerRepository::findAllByFilter, exact match, AND-ed). The PR had honestly flagged this as unverifiable; I pushed 2500380 updating the docstrings/gotchas to state the confirmed fact while keeping the client-side re-narrowing as defense in depth (the #600 precedent stands).- Production-only — confirmed: every trigger route declares
isAvailableInBranch: false, and a SOX E2E test demonstrates a dev-branch table is unreachable through this resource. - Response shape — confirmed 1:1 (
idis a string,tablesis[{tableId}]objects,lastRunnullable = never fired,componentis free-form — the legacyorchestrationappears in real data). - Auth — better than assumed: list/detail are
AsReadOnlyAction, scoped only by the token's project; the server's owntestTriggersRestrictionsForReadOnlyUserasserts even a read-only token sees every trigger. The stored project token is always sufficient (also now recorded in gotchas).
cross_project_triggers_checked: false is the only correct design, not a scope cut
The trigger-queue component derives the destination project at runtime by parsing the encrypted #kbcToken (parameters['#kbcToken'].split('-')[0] in its component.py), and resolves flow-vs-legacy-orchestrator by live-probing the destination project's API. The only readable hint is the optional, non-authoritative trigger_metadata.project_id UI cache. A reliable scanner cannot be built from config parameters alone, so declaring "not checked" instead of returning a false empty list is exactly right. Worth capturing in a follow-up issue if a best-effort (trigger_metadata-based, explicitly labelled heuristic) scan is ever wanted.
Live smoke test (branch build, real project)
flow triggers against a real flow on europe-west3.gcp: HTTP 200, JSON shape as documented, human mode renders the not-checked warning; schedule list prints the new "No cron schedules found. Table triggers and cross-project triggers were NOT checked" wording. (No live table trigger existed to exercise the populated path — that path is covered by the unit tests using the controller's verbatim example payload, which matches the shape asserted by connection's own E2E tests.)
Not in this PR (fine, but should not get lost)
Part 3 of #714 — the job detail / job list hint ("N runs had no matching cron schedule — check flow triggers") — is not implemented. Suggest a follow-up issue rather than growing this PR.
Verdict: correct, honestly scoped, well tested. Good to merge.
A flow run with no matching cron schedule is not necessarily manual -- table triggers and cross-project triggers also start flows and neither appears in `schedule list`. That dead-end is exactly how the false 'this flow has no trigger' conclusion in #714 happened. `job detail` on a keboola.flow / keboola.orchestrator job now attaches trigger_hint (JSON and serve included -- the misled consumer in #714 was an agent reading tool output) pointing at `kbagent flow triggers`, and human mode renders it plus a new 'Created by token' line: the token that created the job (a human's email vs a scheduler's or trigger's token) is the one factual provenance signal the Queue payload carries. Doc surfaces updated: CLAUDE.md, context.py, commands-reference.md, gotchas.md, keboola-expert.md.
|
Pushed 801be8a implementing part 3 of #714 on top of the verification review above: |
Batches the seven PRs merged since v0.91.0 into one version bump, one changelog entry and one set of resolved version gates: - #719 (#714) `flow triggers` -- table triggers, not just cron - #717 (#711) a 401 is no longer automatically blamed on the token - #722 (#704) setup completes in chat; skill covers setup + logout - #718 (#716) `--conversation-id` global flag - #706 223 stale version gates retired at the 0.80.0 floor - #702 release process enforced rather than remembered - #721 `get_flow_detail` docstring fix Includes a curated What's-new entry (#717's error rework is UI-visible on the Semantic Layer page) and the step 8-11 silent-drift review.
What
kbagent could see one of the (at least) three ways a Keboola flow gets started automatically.
schedule list/schedule find/searchreadkeboola.schedulerconfigs only, so a flow drivenby a table trigger — a separate Storage API resource, not a component config at all — came back
"nothing found". Twice in one investigation that was read as "this flow has no trigger", for a flow
that was already live; the wrong conclusion was that a working notification pipeline still needed
manual scheduling.
keboola.schedulerconfig/v2/storage/triggersImplements parts 1, 2 and 3 of the issue.
The design point: an honest negative
Cross-project triggers are reported as
cross_project_triggers_checked: false, deliberately not asan empty list. An empty list reads as "checked, found none" — which is the exact false negative the
command exists to prevent. Detecting them means scanning every connected project and resolving each
candidate's
parametersback to this project + flow; that shape isn't something I could verify, soit is declared rather than guessed.
Part 2 landed too:
schedule list/schedule findnow sayNo cron schedules found. Table triggers and cross-project triggers were NOT checked -- see kbagent flow triggersinstead ofNo schedules found.Wire contract — read from the source, not guessed
Verified against Keboola's own
Controller/Storage/Triggers/TriggerListActionandTriggerResponserather than inferred:
isAvailableInBranch: false→ table triggers are production-only.There is no branch-scoped variant to call, so
--branchnarrows the cron half only andtable_triggers_branch_scopedis alwaysfalse. The command never advertises a branch-scopedanswer it cannot produce.
?component=and?configurationId=are declared filters — but whether the server applies themis not readable from the published source (the filter object is threaded into
getTriggersByRequestFilter, whose body I could not locate). This codebase has been burned byexactly that before: the Notification Service accepts
?event=and ignores it (kbagent notification: fleet-wide audit of Flow Notification subscriptions (Notification Service API) #600). So thefilter is sent and the result narrowed again client-side — correct whichever way the server
behaves, and an unnarrowed response would otherwise attribute another flow's trigger to this one.
tablesis a list of{"tableId": ...}objects, not strings;lastRunis nullable andnullmeans never fired, not disabled (human mode renders it as
never, not a blank cell);componentis not necessarilykeboola.flow— the API's own example is the legacyorchestration, so matching is onconfigurationId.NOT_FOUNDfrom the endpoint degrades to an empty list, but any other error propagates —silently swallowing a 500 as "no triggers" would recreate the very bug this fixes.
Structure
commands/flow.pyis at exactly 800 code lines — the commands soft ceiling — so per CONTRIBUTING'sfile-size budgets the command lives in
commands/_flow_triggers.py, following the existing_storage_snapshots.pyprecedent. It is mounted flat ontoflow_app, so the permission key staysflow.triggers(read) andkbagent flow --helpis unchanged.Layers:
client/triggers.py(new_TriggersMixin) →FlowService.get_flow_triggers→commands/_flow_triggers.py, plusGET /flows/{project}/{config_id}/triggersoverserve.Part 3: unexplained runs get a guided next step
job detailon akeboola.flow/keboola.orchestratorjob now attachestrigger_hint-- "If nocron schedule explains this run, check
kbagent flow triggers ..." -- in JSON and overservetoo,because the consumer that was misled in #714 was an agent reading tool output, not a human at a
terminal. Human mode renders the hint and a new Created by token line: the token that created the
job (a human's email vs a scheduler's or trigger's token) is the one factual provenance signal the
Queue payload carries. Non-flow jobs are untouched.
How it was tested
tests/test_triggers_client.py(5) — L3 wire contract, using the controller's verbatim examplepayload: path, both query filters, numeric-id coercion, and no empty query string when unfiltered.
TestGetFlowTriggersintests/test_flow_service.py(9) — table trigger surfaced, empty resultcarries the not-checked flag, another config's trigger filtered out locally, numeric
configurationIdstill matches, filter sent to the API, cron half included alongside, neveradvertised as branch-scoped, 404 degrades, non-404 propagates.
TestFlowTriggers+TestScheduleNegativeResultWordingin the CLI tests (6) — JSON shape, thenot-checked warning in human output,
neverrendering, exit-code mapping, and the newschedule listwording.test_human_mode_emptyasserted the oldNo schedules foundstring; updated,since that wording is the bug.
make checkexits 0 — 6385 passed,check_command_sync.pygreen at 268 commands.Doc surfaces (convention #17)
This adds a CLI command, so all of them:
CLAUDE.mdcommand list,context.pyAGENT_CONTEXT,commands-reference.md,gotchas.md(tagged(since vNEXT, #714)),keboola-expert.md(new matrixrow + gotcha; prompt is 54 407 B of the 70 000 B budget), regenerated
SKILL.mdanddocs/web-server-endpoints.md.No version bump and no
changelog.pyentry — that belongs to the release PR.Fixes #714