feat: Semantic Layer UI — full parity with kbagent CLI (4 phases)

[padak]

Summary

Native Semantic Layer surface inside kbagent serve --ui, matching the full
kbagent semantic-layer CLI feature set. Built in four iterative passes
against the keboola-ai project's langsmith_semantic_model (15 metrics,
16 datasets, 142 relationships, 6 constraints) so every code path is
exercised against real production-shape data.

What lands

Phase 1+2 — CRUD scaffolding (commit bcc80b7)

• New /semantic-layer page in the Insights sidebar group.
• Schema-driven entity forms (one declaration covers Add + Edit for all
  five entity kinds — metric, dataset, relationship, constraint, glossary).
• Per-model header with Validate / Export / Delete buttons.

Phase 3 — Workflows + backend hardening (commit 34514ce)

• New SemanticLayerDialogs.tsx with five drawers: Diff (project↔project,
  project↔file, file↔file), Promote (cross-project with dry-run preview),
  Import (snapshot replay with item-by-item log), Build (heuristic
  greenfield builder with bucket-filtered table picker), Encrypt token
  (project storage token → KBC::ProjectSecure ciphertext for transformation
  user_properties).
• Filled-keboola primary buttons so the actionable choice no longer looks
  identical to Cancel/Close.
• Backend: heuristic_generate_model now maps warehouse-native column
  types onto the metastore's lowercase closed set (string/integer/...).
  Previously every legacy untyped table 422'd the build push.
• Backend: http_base._raise_api_error now surfaces the real error
  message from the metastore's {"error": 422, "errors": [...]} shape
  instead of bare "API error 422: 422".

Phase 4 — Polish for real-data scale (commit 4d7dcb4)

• tableId/primaryKey/constraintType camelCase → snake normalisation
  at the API boundary so the TABLE column actually populates.
• Relationships ERD toggle (Mermaid erDiagram; mermaid is already in
  deps). Soft 80-edge cap with banner pointing at the dataset chip filter
  for larger models.
• Quick search box across every entity table, plus a "filter by dataset"
  chip for relationships.
• Dataset detail drawer renders fields with role badges + AI keywords
  instead of raw JSON (Raw still one click away).
• Constraints grouped into collapsible <details> per constraint_type
  (inequality → equality → range → ...) with severity icons
  (🔴 critical · ⚠ warning · ⓘ info).

Architecture / business-logic check

• All UI calls route through /api/semantic-layer/* — zero
  metastore HTTP from JS, zero business logic in the frontend.
• CLI ↔ API ↔ UI parity is 1:1 (14 routes mirror 11 CLI subcommands +
  15 add/edit/remove leaves; POST /items/{kind} parameterises the
  five kinds in REST style).
• Backend service layer is the only place fields/types/FQN/push order
  are computed.

Known follow-ups (filed separately, not in this PR)

• semantic-layer model delete leaves orphan children → blocks Build/Import retries #306 — semantic-layer model delete leaves orphan children in
  metastore; per-project name conflicts block subsequent Build / Import.
  Suggested fix is cascade-delete in delete_model (reverse PUSH_ORDER).

Test plan

• Local smoke: kbagent serve --ui against padak-2-0-master and
  keboola-ai
• Phase 1 — Create model (Snowflake / BigQuery)
• Phase 2 — Add dataset, Add metric, Validate, Export → snapshot.json
• Phase 3 — Diff (project↔file), Build (with cleanup of orphan
  datasets), Encrypt token (paste-ready user_properties block)
• Phase 4 — Dataset detail drawer on fct_conversations (64 fields),
  Relationships ERD with dataset chip filter (142 → 31 edges),
  Constraints grouped by type (inequality/range/composition) with
  severity icons

[padak]

Pushed fe49c1a — relationships ERD now uses Mermaid flowchart TB instead of erDiagram.

Why: erDiagram has no rankdir and always lays out wide-and-short. For Keboola's typical hub-and-spoke models (one fact, many dims) that wastes ~70% of the canvas. flowchart TB puts the hub above its dimensions and keeps the diagram readable end-to-end.

Three small changes:

• erDiagram → flowchart TB in the Mermaid source
• Edge labels shortened from <from>_to_<to> (left) to just left / inner (the full names live in the edge list below the diagram)
• Auto-fit switched from Math.max to Math.min(fitX, fitY) so the whole graph fits without scrolling; floor 0.4 / cap 2.5

Verified locally with the langsmith_semantic_model (16 datasets, 142 relationships): hub-filter view auto-fits at ~63%, full-model view at ~40%, both fully readable.

[padak]

Review of #308 — feat: Semantic Layer UI — full parity with kbagent CLI (4 phases)

Generated by kbagent-pr-reviewer subagent. Verdict and findings below
are advisory; the human author retains every veto. CI-coverable issues
(lint, format, tests) are confirmed via make check, not duplicated here.

Summary

This PR adds a full Semantic Layer management page to kbagent serve --ui,
mirroring all kbagent semantic-layer CLI operations (model CRUD, five entity
kinds, validate/export/diff/promote/import/build/encrypt-token). Two backend
bugfixes are bundled: _raise_api_error now surfaces real error text from the
Keboola Metastore's non-standard error shapes ({"error": 422, "errors": [...]},
{"description": "..."}) instead of printing a bare HTTP status code; and
heuristic_generate_model now maps warehouse-native column types to the
metastore's closed vocabulary before pushing, which previously caused HTTP 422
on every legacy untyped table. The UI calls zero Metastore endpoints directly —
all traffic routes through /api/semantic-layer/*. Architecture, layer
boundaries, security, and permissions are clean.

Verdict: COMMENT — no blocking findings; four non-blocking findings, most
of which are a missing test for the two backend bugfixes.

Verdict

• Verdict: COMMENT
• Blocking findings: 0
• Non-blocking findings: 4
• Nits: 1

Blocking findings

(none)

Non-blocking findings

[NB-1] src/keboola_agent_cli/http_base.py:226-231 — duplicate comment block introduced by the diff

The diff inserts a comment block starting with # Real Keboola APIs answer with one of these keys in priority order. at line 226. A second, more precise
version of the same intro sentence starts at line 232 (# Real Keboola APIs answer with one of these keys in priority). Lines 226-231 are a leftover from
an earlier draft; the correct version is the second block (lines 232-240) which
explains the two caveats. Fix: delete the five stale lines (226-231) so the
comment reads as a single cohesive block.

[NB-2] src/keboola_agent_cli/http_base.py:241-255 — no tests for the three new _raise_api_error fallback code paths

The PR introduces three new parser paths in _raise_api_error: (a) error
field that is an int ({"error": 422}) must be rejected in favour of
description/errors; (b) {"description": "..."} (FastAPI default); (c)
{"errors": [...]} / {"detail": [...]} list shapes. test_http_base.py has
no test exercising any of these shapes. These are the exact shapes the Metastore
returns on validation failures (the stated motivation for the fix), so a
regression here would silently bring back the "bare 422" UX. Add three test
cases to tests/test_http_base.py mirroring the real Metastore response shapes.

[NB-3] src/keboola_agent_cli/services/_semantic_layer_internals.py:988-994 — _normalize_field_type has no dedicated unit test; existing heuristic test does not assert the mapping

_normalize_field_type is the fix for the "legacy untyped table 422" bug.
tests/test_semantic_layer_service.py::TestBuildModel::test_heuristic_fallback
exercises heuristic_generate_model with {"name": "AMOUNT", "type": "NUMBER"}
but asserts only the count of datasets/metrics/glossary — not that the
generated field type was normalized to "decimal". A dedicated parametrized
test (empty string -> "string", "NUMBER(18,2)" -> "decimal",
"VARCHAR(255)" -> "string", unknown type -> "string") would pin the
mapping and prevent regression when the FIELD_TYPE_MAP is extended. Also
extend test_heuristic_fallback to assert result["generated"]["datasets"][0]["fields"][0]["type"] == "decimal".

[NB-4] plugins/kbagent/skills/kbagent/references/gotchas.md:185-210 — field-type normalization fix not documented in the build gotcha entry

The existing ## \semantic-layer build` is a HEURISTIC fallback, not full AI
(since v0.41.0)entry describes the heuristic fields[] synthesis but says nothing about field-type normalization. Prior to this PR,build 422'd on every legacy untyped Storage table (Storage returns empty/""or warehouse- native uppercase types likeVARCHAR, NUMBER; the Metastore's closed vocabulary only accepts lowercase string/integer/decimal/...). An AI agent instructed to buildagainst an untyped project on kbagent < this fix will hit 422s and have no documentation to consult. Add a one-liner with a(since v0.41.10)` tag to the existing entry: e.g. "Field types are normalized
from warehouse-native (VARCHAR, NUMBER, ...) to the metastore's closed
lowercase set ({string, integer, decimal, boolean, date, datetime, json}) —
avoids HTTP 422 on legacy untyped Storage tables (since v0.41.10)."

Nits

• [NIT-1] web/frontend/src/pages/SemanticLayer.tsx:1 — at 2099 lines this file is well above the soft ceiling for any layer. It's frontend (no hard ceiling defined in CONTRIBUTING.md for TypeScript), but splitting the entity-table section and the model-CRUD header into sibling files would make the next feature addition (e.g. a new entity kind) faster to navigate.

Verification log

• gh pr view 308 --json title,body,files,additions,deletions,baseRefName,headRefName,labels,state → 7 files, +3588/-2, state: OPEN, branch: feat/semantic-layer-ui ✓
• git rev-parse --abbrev-ref HEAD → feat/semantic-layer-ui (matches PR branch) ✓
• Layer violation greps (typer/click in services; httpx in commands; formatter in clients) → empty output ✓
• grep -E '^\+(from httpx|import httpx...)' diff → no new HTTP calls in wrong layer ✓
• grep '*.keboola.com\|fetch\|axios' SemanticLayer.tsx SemanticLayerDialogs.tsx → no direct Metastore calls from JS; all routes through /api/semantic-layer/* ✓
• grep -E '^\+\s*except\s*:' diff → no bare excepts introduced ✓
• grep -E '^\+.*error_code\s*=\s*"[A-Z_]+"' diff → no raw error-code strings ✓
• grep -E '^\+(from typer|import typer...)' diff → no layer violations ✓
• grep 'OPERATION_REGISTRY' permissions.py → semantic-layer commands already registered (pre-existing from v0.41.0); this PR adds no new CLI commands, so no new entries needed ✓
• Plugin synchronization map: PR adds no new CLI commands; no AGENT_CONTEXT, CLAUDE.md ## All CLI Commands, or commands-reference.md updates required ✓
• grep "Real Keboola APIs answer" http_base.py → duplicate comment block at lines 226-231 and 232-240 detected ✗ (NB-1)
• git diff --stat main...HEAD | grep tests/ → no test files changed in this PR ✗ (NB-2, NB-3)
• make check → 3339 passed, 7 skipped, exit 0 ✓
• Behavior verification: could not run kbagent serve --ui against a real project (no E2E credentials in this session). PR author's test plan (4 phases × multiple scenarios including legacy untyped table build) is detailed and appears complete; marking as unverified by reviewer.
• Token discipline: grep token SemanticLayerDialogs.tsx — token appears only in the encrypt-token dialog (building #metastore_token key) and in UI copy strings; no token is read from JS scope or logged ✓

Open questions for the author

• The _raise_api_error change uses body.get("description") as one of the fallback keys. The Keboola Manage API also uses description for a success body field on some endpoints. If a success response somehow reaches _raise_api_error (should not happen given the < 400 guard, but defensive question), would the description key from a non-error body produce a confusing error message? The guard appears correct; just flagging for author confirmation.

[padak]

Review of #308 — feat: Semantic Layer UI — full parity with kbagent CLI (4 phases)

Generated by kbagent-pr-reviewer subagent (delta re-review after commit b2dbe69).
Verdict and findings below are advisory; the human author retains every veto.
CI-coverable issues (lint, format, tests) are confirmed via make check, not duplicated here.

Summary

This is a delta re-review targeting the fix-up commit b2dbe69 ("test(semantic-layer-ui): address /kbagent:review findings on #308"). All four non-blocking findings from the prior review (NB-1 through NB-4) are now fully addressed. NIT-1 (SemanticLayer.tsx file size) was explicitly deferred by the author and is not re-raised here. No new blocking or non-blocking issues were introduced by the fix-up commit. One new nit is noted regarding a minor type annotation mismatch introduced by NB-3's test. make check passes with 3357 tests (up from 3339).

Verdict: APPROVE — zero blocking findings, zero non-blocking findings.

Verdict

• Verdict: APPROVE
• Blocking findings: 0
• Non-blocking findings: 0
• Nits: 1

Blocking findings

(none)

Non-blocking findings

(none)

Prior findings — resolution status

Finding	Status
NB-1: duplicate comment block in http_base.py:226-231	Resolved — b2dbe69 removes the 6 stale lines; only the precise two-caveat block at lines 226-234 remains.
NB-2: no tests for three new _raise_api_error fallback paths	Resolved — tests/test_http_base.py gains four tests: test_int_error_field_falls_back_to_description, test_description_field_used_when_no_message, test_errors_list_shape_serialised, test_detail_list_shape_serialised. Each reproduces the real Metastore/FastAPI error shape.
NB-3: _normalize_field_type has no dedicated unit test; test_heuristic_fallback did not assert field type	Resolved — TestNormalizeFieldType parametrizes 14 cases (empty, None, parameterised types, case variants, unknown UDT fallthrough). test_heuristic_fallback now asserts ds_fields[0]["type"] == "decimal".
NB-4: field-type normalization not documented with (since vX.Y.Z) tag	Resolved — gotchas.md line 211 adds **Field-type normalization (since v0.41.10)** with the full closed-set description.
NIT-1: SemanticLayer.tsx at 2099 lines above soft ceiling	Explicitly deferred — acknowledged in commit message; not re-raised.

Nits

• [NIT-2] tests/test_semantic_layer_service.py:1924 — TestNormalizeFieldType.test_normalize_field_type passes None as the basetype argument but _normalize_field_type is annotated basetype: str. The runtime if not basetype: guard handles None correctly and ty does not flag the parametrize call site, but the annotation and the test contract are inconsistent. The production call site (line 1028) cannot produce None via col.get("type", "") or col.get("native_type", "") unless both dict values are explicitly None (not absent), which would bypass the "" default. If the intent is to document that None-safe behavior is guaranteed, change the signature to basetype: str | None. If None is not a real input, drop the parametrize case. Either way is fine; this is cosmetic.

Verification log

• gh pr view 308 --json title,state → state=OPEN, title matches ✓
• git rev-parse --abbrev-ref HEAD → feat/semantic-layer-ui matches <branch> ✓
• git log --oneline feat/semantic-layer-ui ^main → 6 commits; top is b2dbe69 ✓
• git show b2dbe69 --stat → 4 files changed: gotchas.md (+8), http_base.py (-6), test_http_base.py (+99), test_semantic_layer_service.py (+49) ✓
• NB-1: grep -n "Real Keboola APIs" http_base.py → 1 hit (line 226); duplicate gone ✓
• NB-2: grep -n "def test_" test_http_base.py → 4 new tests for _raise_api_error fallback shapes ✓
• NB-3: grep -n "class TestNormalizeFieldType" test_semantic_layer_service.py → found; 14 parametrize cases; test_heuristic_fallback extended with ds_fields[0]["type"] == "decimal" assertion ✓
• NB-4: grep -n "since v0.41.10" gotchas.md → line 211, Field-type normalization entry ✓
• make check → 3357 passed, 7 skipped, exit 0 ✓
• make typecheck on modified files (http_base.py, _semantic_layer_internals.py) → 0 error[ diagnostics in those files ✓
• Bare except:, raw error_code strings, print() in production, token-in-log checks → all clean ✓
• b2dbe69 http_base.py diff: pure comment removal (6 lines), no logic change ✓

Open questions for the author

(none)

[padak]

Review of #308 — feat: Semantic Layer UI — full parity with kbagent CLI (4 phases)

Generated by kbagent-pr-reviewer subagent (delta re-review after commit b2dbe69).
Verdict and findings below are advisory; the human author retains every veto.
CI-coverable issues (lint, format, tests) are confirmed via make check, not duplicated here.

Summary

This is a delta re-review targeting the fix-up commit b2dbe69 ("test(semantic-layer-ui): address /kbagent:review findings on #308"). All four non-blocking findings from the prior review (NB-1 through NB-4) are now fully addressed. NIT-1 (SemanticLayer.tsx file size) was explicitly deferred by the author and is not re-raised here. No new blocking or non-blocking issues were introduced by the fix-up commit. One new nit is noted regarding a minor type annotation mismatch introduced by NB-3's test. make check passes with 3357 tests (up from 3339).

Verdict: APPROVE — zero blocking findings, zero non-blocking findings.

Verdict

• Verdict: APPROVE
• Blocking findings: 0
• Non-blocking findings: 0
• Nits: 1

Blocking findings

(none)

Non-blocking findings

(none)

Prior findings — resolution status

Finding	Status
NB-1: duplicate comment block in http_base.py:226-231	Resolved — b2dbe69 removes the 6 stale lines; only the precise two-caveat block at lines 226-234 remains.
NB-2: no tests for three new _raise_api_error fallback paths	Resolved — tests/test_http_base.py gains four tests: test_int_error_field_falls_back_to_description, test_description_field_used_when_no_message, test_errors_list_shape_serialised, test_detail_list_shape_serialised. Each reproduces the real Metastore/FastAPI error shape.
NB-3: _normalize_field_type has no dedicated unit test; test_heuristic_fallback did not assert field type	Resolved — TestNormalizeFieldType parametrizes 14 cases (empty, None, parameterised types, case variants, unknown UDT fallthrough). test_heuristic_fallback now asserts ds_fields[0]["type"] == "decimal".
NB-4: field-type normalization not documented with (since vX.Y.Z) tag	Resolved — gotchas.md line 211 adds **Field-type normalization (since v0.41.10)** with the full closed-set description.
NIT-1: SemanticLayer.tsx at 2099 lines above soft ceiling	Explicitly deferred — acknowledged in commit message; not re-raised.

Nits

• [NIT-2] tests/test_semantic_layer_service.py:1924 — TestNormalizeFieldType.test_normalize_field_type passes None as the basetype argument but _normalize_field_type is annotated basetype: str. The runtime if not basetype: guard handles None correctly and ty does not flag the parametrize call site, but the annotation and the test contract are inconsistent. The production call site (line 1028) cannot produce None via col.get("type", "") or col.get("native_type", "") unless both dict values are explicitly None (not absent), which would bypass the "" default. If the intent is to document that None-safe behavior is guaranteed, change the signature to basetype: str | None. If None is not a real input, drop the parametrize case. Either way is fine; this is cosmetic.

Verification log

• gh pr view 308 --json title,state → state=OPEN, title matches ✓
• git rev-parse --abbrev-ref HEAD → feat/semantic-layer-ui matches <branch> ✓
• git log --oneline feat/semantic-layer-ui ^main → 6 commits; top is b2dbe69 ✓
• git show b2dbe69 --stat → 4 files changed: gotchas.md (+8), http_base.py (-6), test_http_base.py (+99), test_semantic_layer_service.py (+49) ✓
• NB-1: grep -n "Real Keboola APIs" http_base.py → 1 hit (line 226); duplicate gone ✓
• NB-2: grep -n "def test_" test_http_base.py → 4 new tests for _raise_api_error fallback shapes ✓
• NB-3: grep -n "class TestNormalizeFieldType" test_semantic_layer_service.py → found; 14 parametrize cases; test_heuristic_fallback extended with ds_fields[0]["type"] == "decimal" assertion ✓
• NB-4: grep -n "since v0.41.10" gotchas.md → line 211, Field-type normalization entry ✓
• make check → 3357 passed, 7 skipped, exit 0 ✓
• make typecheck on modified files (http_base.py, _semantic_layer_internals.py) → 0 error[ diagnostics in those files ✓
• Bare except:, raw error_code strings, print() in production, token-in-log checks → all clean ✓
• b2dbe69 http_base.py diff: pure comment removal (6 lines), no logic change ✓

Open questions for the author

(none)