feat(workspace): fast inline results for workspace query, --full for complete CSV (0.59.0)

[padak]

Why

kbagent workspace query materialized a CSV file via the warehouse UNLOAD path (GET .../export?fileType=csv) for every query, paying a slow warehouse -> object-storage -> download round-trip even for tiny result sets. For an interactive SQL-debugging tool this is the wrong default.

What changed

The default now reads results inline via the Query Service GET /api/v1/queries/{job}/{stmt}/results endpoint (JSON columns+rows, no file materialization). Measured ~2.5x faster live on a 25-row table (inline ~3.1s vs --full ~7.9s); the gap is the fixed file-export overhead and grows with volume.

• Each statement now carries structured columns + rows (plus row_count, total_rows, truncated) and a synthesized csv_data, so the CLI preview, web UI table, and any --json consumer keep working unchanged (no breaking shape change).
• --limit N (default 500) caps the fast path; it pages by offset and flags the statement truncated when the warehouse has more rows. The CLI prints Showing first N of TOTAL rows. Use --full….
• --full opts back into the complete CSV export (slower, uncapped) for bulk extraction.
• The kbagent serve /workspaces/{p}/{w}/query REST endpoint defaults to full=True to preserve the web UI's complete-CSV "Download" until the frontend learns to paginate (REST clients can pass full=false).

API floor caught during live testing

The /results endpoint enforces 100 <= pageSize <= 100000 (a smaller pageSize 400s with Invalid pageSize parameter, must be between 100 and 100000). Deriving pageSize from --limit broke --limit 5. Fixed: pageSize is a fixed valid value and --limit only trims the accumulated rows locally. Locked in by a regression test.

Layers touched (3-layer design)

Layer	Change
client.py	new get_query_results() on /results; export_query_results kept for --full
services/workspace_service.py	execute_query(full, limit); _collect_inline_results (paginate+trim), _rows_to_csv
output.py	Rich table from structured data + truncation hint, CSV fallback for --full
commands/workspace.py	--full, --limit flags
server/routers/workspaces.py	WorkspaceQuery.full/limit (default full=True)

Tests & docs

• Unit tests: client (/results pagination params), service (inline path, pagination, truncation, small-limit regression, --full, graceful failures), CLI (--full/--limit forwarding), output (table vs CSV fallback). make check green (3909 passed).
• E2E _test_workspace_query now exercises both the fast and --full paths.
• Docs: CLAUDE.md, context.py (AGENT_CONTEXT), commands-reference.md, gotchas.md (since v0.59.0, incl. the pageSize floor), and a new "Fast inline results vs --full — mind the result-set volume" section in workspace-workflow.md.
• Version bumped to 0.59.0 (changelog + make version-sync).

Live verification (demo "shop", Snowflake workspace, 25-row table)

Structured output + NULL handling, --limit 5 truncation (row_count:5, total_rows:25, truncated:true), human-readable Rich table + hint, --full complete 26-row CSV, and the ~2.5x speedup.

---

[devin-ai-integration]

Devin Review found 2 potential issues.

View 3 additional findings in Devin Review.

[padak]

Review of #406 — feat(workspace): fast inline results for workspace query, --full for complete CSV (0.59.0)

Generated by kbagent-pr-reviewer subagent. Verdict and findings below
are advisory; the human author retains every veto. CI-coverable issues
(lint, format, tests) are confirmed via make check, not duplicated here.

Summary

This PR replaces the workspace query result-fetch strategy: instead of always materialising a CSV file through the warehouse UNLOAD path, the default now reads results inline via the Query Service /api/v1/queries/{job}/{stmt}/results endpoint (paginated, capped at --limit rows, default 500). A --full flag reverts to the old CSV-export path for complete result sets. The JSON output shape is additive (new columns, rows, row_count, total_rows, truncated fields; csv_data is synthesised and preserved for backward compat). All three layers are touched correctly, documentation surfaces are updated, make check is green at 3909 tests. One NON-BLOCKING logic gap was found in the truncation signal, and two NON-BLOCKING documentation gaps remain in keboola-expert.md. APPROVE.

Verdict

• Verdict: APPROVE
• Blocking findings: 0
• Non-blocking findings: 3
• Nits: 2

Blocking findings

(none)

Non-blocking findings

[NB-1] src/keboola_agent_cli/services/workspace_service.py:219 — truncated is incorrectly False when numberOfRows is absent from API response and rows hit the limit

The truncation signal is truncated = total_rows is not None and total_rows > len(rows). If the Query Service does not return numberOfRows in its payload (a defensive case), total_rows stays None and truncated is False — even though we stopped collecting because we hit --limit, not because the warehouse ran out of rows. A caller reading truncated: false would believe they received the complete result set when they may not have.

Fix: change the condition to truncated = (total_rows is not None and total_rows > len(rows)) or (len(collected) >= limit and len(rows) == limit) — flag truncation whenever we filled the limit, regardless of whether numberOfRows was returned. Add a test case where the API returns no numberOfRows and len(collected) == limit.

[NB-2] plugins/kbagent/agents/keboola-expert.md:103 and :344-359 — keboola-expert.md §2 matrix and §4.4 workflow do not mention the new default cap

The §2 "Ad-hoc SQL / row-count / type audit" matrix row and the §4.4 "Workspace-based SQL debugging" workflow example both show workspace query --sql "..." without any mention of --full or the 500-row default cap introduced in this PR. The workflow note at line 358 explicitly says to use this for "ROW COUNT COMPARISONS between branches" — a query returning a 501-row table will silently return 500 rows as truncated: true, but an agent that ignores the truncated flag (or just looks at row_count) will draw a wrong comparison.

Per CONTRIBUTING.md §2 the matrix carries the (X.Y.Z+) version floor and is updated when behaviour changes. Fix: add a note to the "Ad-hoc SQL" matrix row and the §4.4 example, e.g. workspace query --sql "SELECT COUNT(*) ..." (aggregate queries are fine); add a one-liner to §3 "Workspace" group: **workspace query default cap 500 rows (0.59.0+)**: results capped at \--limit` (default 500) on the fast path; use `--full` for complete result sets, check `truncated` before trusting row counts.`

[NB-3] tests/test_workspace_service.py — four new execute_query test cases do not assert mock_client.close.call_count == 2

The existing test_execute_query_success (line 1008) and all other happy-path service tests assert mock_client.close.call_count == 2 (one call in _resolve_branch_id, one in the finally block of execute_query). The four new tests test_execute_query_full_uses_csv_export, test_execute_query_inline_pagination, test_execute_query_small_limit_keeps_valid_page_size, and test_execute_query_inline_fetch_fails_gracefully omit this assertion. Per CONTRIBUTING.md > Testing Guidelines: "Verify client.close() is called." A missing close() would leak an httpx connection but no test would catch it.

Fix: add assert mock_client.close.call_count == 2 at the end of each of the four new happy-path / pagination tests.

Nits

• [NIT-1] src/keboola_agent_cli/services/workspace_service.py:216 — when total_rows is exactly equal to QUERY_RESULTS_PAGE_SIZE (e.g. exactly 500 rows in the warehouse, limit=600, page_size=500), the pagination loop will make a second API call to offset=500 before discovering the empty page and breaking. This is not a bug — it's a single wasted round-trip — but could be avoided by checking total_rows is not None and offset + len(page_rows) >= total_rows as an early break condition.

• [NIT-2] src/keboola_agent_cli/constants.py:339 — QUERY_RESULTS_PAGE_SIZE and QUERY_RESULTS_DEFAULT_LIMIT are currently both 500, which may create the impression they are the same concept. A brief inline comment (or at least a divergent example value in the docstring) would help the next reader understand that page_size is an API wire constraint (100..100000) and limit is a user-facing row cap that can be independently tuned.

Verification log

• gh pr view 406 --json title,body,files,additions,deletions,baseRefName,headRefName,labels,state → 21 files, +794/-60, state OPEN, conventional feat(workspace): prefix ✓
• git rev-parse --abbrev-ref HEAD in worktree → claude/upbeat-payne-1ed2e5 matches PR branch ✓
• Layer violation checks (typer/click in services, httpx in commands, formatter in clients) → empty ✓
• grep -E '^\+' diff | grep -E 'bare except' → empty ✓
• grep -E '^\+.*error_code\s*=\s*"[A-Z_]+"' diff → one hit in test fixture only (not production code) ✓
• Token discipline: token="901-55555-fakeTestTokenDoNotUseXXXXXXXX" is the canonical test token ✓
• Plugin synchronization map: no new commands added; only flags added to existing workspace.query; no OPERATION_REGISTRY entry needed (command already registered at src/keboola_agent_cli/permissions.py:107) ✓
• CLAUDE.md ## All CLI Commands → updated with [--full] [--limit N] at correct line ✓
• src/keboola_agent_cli/commands/context.py AGENT_CONTEXT → updated ✓
• plugins/kbagent/skills/kbagent/references/commands-reference.md → updated ✓
• plugins/kbagent/skills/kbagent/references/gotchas.md → new section (since v0.59.0) with version tag ✓
• plugins/kbagent/skills/kbagent/references/workspace-workflow.md → new section added ✓
• plugins/kbagent/agents/keboola-expert.md → gotchas.md cross-reference exists; §2 matrix row for workspace unchanged; §3 has no new workspace gotcha → gap flagged as NB-2
• src/keboola_agent_cli/server/routers/workspaces.py → WorkspaceQuery.full / limit added; full=True default preserves web UI shape ✓
• make check → 3909 passed, 8 skipped, 124 deselected, 15 warnings (coroutine warnings from pre-existing MCP tests) in 83.48s, exit 0 ✓
• Backward compatibility: csv_data synthesised on fast path; full=True default on serve preserves old web UI shape; rows_affected source now stmt.get("numberOfRows", stmt.get("resultRows", 0)) — graceful fallback for older API responses ✓
• Pagination logic review: no infinite loop possible (while exits at len(collected) >= limit); truncation logic has one edge case when numberOfRows absent (flagged as NB-1) ✓
• E2E tests: _test_workspace_query extended to cover both fast path (columns/rows/csv_data asserted) and --full path ✓
• Behavior reproduction: could not run live against real project (no credentials in this session); PR description includes a verified live measurement (~2.5x faster) and the --limit 5 regression story matches the regression test at tests/test_workspace_service.py:1135 ✓

Open questions for the author

(none)