No bulk path to fetch config details (rows + state) for many configs

[ondrej-lorenc]

Context

Discovered while auditing 14 Keboola projects for a single question: "Which Snowflake writers are active, part of a scheduled flow, and load a source table larger than 1 GB in full-load mode?" The audit should be answerable end-to-end from the CLI / MCP tools, but several gaps forced manual workarounds. This issue covers one of them; the audit uncovered three more, filed separately.

Problem

config detail returns one config at a time. When an audit needs rows + state for many configs (e.g. all writers of a given component across the fleet), the only option is N parallel CLI invocations. For 102 Snowflake writers this meant 102 subprocesses; at fleet-scale (thousands of configs) this will not be viable.

Workaround used

Shell script fanning out 102 background kbagent config detail calls and collecting JSON files in /tmp.

Use case

Audits across an entire organization's configs (compliance checks, "which configs use deprecated parameter X", onboarding checklists). Without a bulk path, users hit rate limits and complicated parallel orchestration for what is conceptually one question.

[ondrej-lorenc]

Root cause: the model didn't use the locally synced data

After filing these issues, I discovered that the originating audit could probably have been probably answered entirely from the kbagent sync pull output on disk, without any of the API workarounds that motivated the issues.

So the four issues are not CLI bugs — they are symptoms of the model never reaching for the right tool.

What would actually help: skill guidance

The underlying problem is that the kbagent skill does not steer the model toward the synced filesystem for read-heavy, cross-project audits. The main skill page lists reading-synced-data.md as one of many references, but there is no strong signal that says:

For cross-project read/audit tasks ("find all X where Y", "inventory Z", "which configs have property P"), prefer the synced directory over API commands.

[padak]

Fully closed in v0.23.0 via #218.

Bulk config detail is now a direct CLI/JSON path -- no more 102 parallel subprocess calls:

# Pull rows + state for ALL Snowflake writers across ALL 14 projects in one command
kbagent --json config detail --component-id keboola.wr-db-snowflake --with-state

# Same but for one project + full bodies
kbagent --json config detail --project <alias> --component-id keboola.wr-db-snowflake --with-state

# Only the list summary with rows inlined
kbagent --json config list --component-id keboola.wr-db-snowflake --include-rows

What shipped:

• config detail --component-id ID (omit --config-id) -- bulk mode returning an array of configs for the component. JSON envelope follows the multi-project shape: {"configs": [...], "errors": [...]}. Single-config shape preserved when --config-id is passed (backward compatible -- dedicated regression test).
• config detail --with-state -- attaches runtime state. Bulk mode fetches state inline via include=state on the listing endpoint (NO N+1 -- one API call per project). Single mode reads state directly from the detail response (Storage API embeds state inline; there is no standalone /state endpoint -- documented in gotchas.md).
• config list --include-rows -- opt-in flag for when you want full bodies on the list output.
• Bonus: bulk mode auto-derives componentType from the keboola.ex-* / keboola.wr-* / keboola.app-* / keboola.*-transformation prefix and forwards it to the API -- shrinks response payload when you're scanning a specific component type across a fleet.

Benchmark (live, real project): 13 configs via bulk = 1 API call in ~1.4s. For 102 Snowflake writers that's a single round-trip per project instead of 102 subprocesses.

Security hardening bundled with this fix: UNEXPECTED_ERROR envelopes now truncate exception messages to 256 chars before emission, preventing OAuth refresh tokens or credential-bearing response-buffer fragments from leaking into JSON error output under --with-state (CWE-209). Full detail still reaches debug logs. The --with-state --help text also carries an explicit WARNING: that output may contain credential-bearing data -- do not pipe into shared logs without redaction.

Thanks for the precise reproduction ("102 Snowflake writers this meant 102 subprocesses; at fleet-scale this will not be viable") -- that framing drove the bulk-via-include=state optimization directly. Combined with #195 (schedule find) and the 0.21.2 fixes (#196 config search in rows, #198 storage tables multi-project), the original 14-project audit is fully automatable.

Release: https://github.com/padak/keboola_agent_cli/releases/tag/v0.23.0