perf(agent): reduce LLM workflow overhead by SkyeAv · Pull Request #138 · SkyeAv/Tablassert

SkyeAv · 2026-09-04T19:30:54Z

Reduces agent workflow overhead by moving inspection context and coverage improvement into deterministic supervisor code. The LLM receives a smaller tool surface and bounded context while builds, scores, and persisted configs remain behaviorally equivalent.

LLM Workflow

Tool surface: make_tools(full) exposes exactly read_table, pmc_article_context, derive_config, and build_and_audit; derive-only modes remain unchanged.
Context injection: Adds bounded, fenced column digests with separator statistics and fail-visible fallback behavior.
Prompt budget: Removes supervisor-only helpers from the LLM prompt and transcript while preserving the derive → build → answer workflow.

Audit and Persistence

Audit reports: Adds a compact cached wrapper with a stable 12-key schema and capped unresolved details; the pure build_and_audit report remains complete.
Config compaction: Derives removable defaults from Pydantic models, preserves semantic nulls/provenance/template overrides, re-validates output, and falls back to the exact original config on failure.
State metrics: Records config_chars for the exact persisted configuration.

Measurement

Offline baselines: Adds deterministic FakeModel workflow metrics and pre-redesign budget baselines.
Accuracy invariant: Verifies workflow deltas and golden-fixture KGX/quality equivalence before and after compaction.

Design

The redesign deliberately keeps coverage and proposal helpers in the deterministic supervisor rather than exposing them to the LLM, reducing tool churn without removing existing derive modes. Live-model validation and GEPA re-optimization are deferred; all current efficiency evidence is hermetic and offline.

Docs

Updates docs/agent.md with the revised workflow, context injection, and config persistence behavior.

Testing

uv run pytest -n 0 → 1230 passed, 17 skipped
uv run pytest tests/test_agent_compact.py tests/test_agent_supervisor.py -q → 56 passed
uv run ruff check . → all checks passed
uv run ruff format --check . → 83 files already formatted
uv run pyright → 0 errors, 0 warnings, 0 informations
Independent final code review → PASS, no blockers or should-fix items

Summary by CodeRabbit

New Features
- Added automatic column separator analysis to improve detection of values split across columns.
- Added deterministic configuration cleanup that reduces saved configuration size while preserving results.
- Added configuration size tracking for improved transparency.
Improvements
- Streamlined the agent workflow to reduce unnecessary steps and tool calls.
- Improved audit output with concise, relevant results and clearer unresolved-item reporting.
- Coverage refinement is now handled automatically through a consistent supervisor workflow.
Documentation
- Updated agent documentation to describe the redesigned workflow and analysis process.

coderabbitai · 2026-09-04T19:31:00Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 924d0217-83bd-4272-a6cf-17ff9aecc08b

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 01d718ec-dc35-43ff-9976-823f34261f66

📥 Commits

Reviewing files that changed from the base of the PR and between 631dedf and 5684322.

📒 Files selected for processing (11)

docs/agent.md
src/tablassert/agent.py
tests/test_agent_assembly.py
tests/test_agent_build.py
tests/test_agent_compact.py
tests/test_agent_propose.py
tests/test_agent_speed.py
tests/test_agent_supervisor.py
tests/test_agent_workflow_metrics.py
tests/test_cover_agent_propose.py
tests/test_distill.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The agent pipeline now supplies column digests, uses a four-tool LLM workflow, performs coverage improvement in deterministic supervisor code, compacts valid configurations, and records compact audit and configuration-size metrics.

Changes

Agent workflow redesign

Layer / File(s)	Summary
Column digests and digest-first context `src/tablassert/agent.py`, `tests/test_agent_speed.py`, `docs/agent.md`	Previewed tables and worksheets now include bounded separator statistics, samples, and fallback notes. Prompt guidance and tests use the digest for `explode_by` and `split_by` detection.
Audit and configuration compaction `src/tablassert/agent.py`, `tests/test_agent_build.py`, `tests/test_agent_compact.py`, `tests/test_agent_supervisor.py`, `tests/test_agent_workflow_metrics.py`	Audit output is compacted to high-signal fields. Valid configurations remove provable defaults, preserve semantics, revalidate, and record persisted character counts.
Reduced tool surface and supervisor workflow `src/tablassert/agent.py`, `docs/agent.md`, `tests/test_agent_assembly.py`, `tests/test_agent_workflow_metrics.py`, `tests/test_agent_propose.py`, `tests/test_cover_agent_propose.py`, `tests/test_distill.py`	Full mode exposes four LLM tools. Coverage helpers run in deterministic supervisor code. Tests measure tool surfaces, workflow budgets, callback metrics, and helper isolation.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to 56843

No actionable merge-blocking risk remains in the reviewed changes.

Sequence Diagram(s)

sequenceDiagram
  participant Agent
  participant build_and_audit
  participant Supervisor
  participant map_coverage
  participant propose_config_edit
  Agent->>build_and_audit: validate derived configuration
  build_and_audit-->>Agent: return compact audit observation
  Agent-->>Supervisor: return configuration and audit result
  Supervisor->>map_coverage: inspect coverage gaps
  map_coverage-->>Supervisor: return coverage feedback
  Supervisor->>propose_config_edit: generate ranked candidate edits
  propose_config_edit-->>Supervisor: return candidate configuration
  Supervisor->>build_and_audit: evaluate candidate build

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: reducing LLM workflow overhead in the agent.
Description check	✅ Passed	The description explains the motivation, major changes, design decisions, testing results, and deferred work. It does not use all template headings and omits the Related Issues and Checklist sections,…
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 120 functions across 10 files. (1 skipped: …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

Resolve merge conflict in branch more-efficent-workflows

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch more-efficent-workflows

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # tests/test_agent_supervisor.py

Cut 16.6.1 and bump the package version in pyproject.toml, uv.lock, and CITATION.cff. Patch: the agent's LLM workflow carries a far smaller tool surface, tighter injected context, and smaller persisted configs (#138). Full mode registers exactly four tools — read_table, pmc_article_context, derive_config, build_and_audit — because coverage improvement moved into deterministic supervisor code: map_coverage and propose_config_edit are pure helpers the supervisor calls in its own improve loop, and only the opt-in --reflexion path still spends an LLM call on improvement. Task text now injects bounded per-column column_digest separator statistics (first 500 data rows), so the canonical derive → build → answer path is three steps or fewer with zero inspection tool calls, and the agent rebuilds only on a coded build error, at most twice. The LLM sees a compact twelve-key build_and_audit observation (unresolved capped at 20 with a visible +N more marker) while the supervisor keeps the full report, and the accepted best config is compacted deterministically before persistence (compact_config strips keys equal to their Pydantic defaults; the accuracy-invariance test pins the identical KGX and quality_score, and failure falls back to the exact original config). state.json now records config_chars per article. Builds, scores, and persisted configs remain behaviorally equivalent. Changelog: - Versioned the changelog as 16.6.1 and wrote the Performance entry for the agent workflow redesign (#138). No other entries were pending — nothing else has merged since 16.6.0. Docs: docs/agent.md was updated by #138 itself (revised workflow, context injection, and config persistence sections); the release needs none beyond the changelog. Testing: - uv run pytest -q -> 1248 passed, 3 skipped (96% coverage) - uv run ruff check . && uv run ruff format --check . && uv run pyright -> clean / 0 errors - uv lock --check -> up to date - uv run mkdocs build --strict -> clean

SkyeAv added 7 commits September 4, 2026 10:08

test: [US-001] capture agent workflow efficiency baseline

ccf85ec

feat(agent): [US-002] slim LLM workflow tool surface

18ae82a

perf(agent): [US-003] compact audit tool reports

6b05e39

perf(agent): [US-004] inject column digests into context

d1e0804

perf(agent): [US-005] compact persisted configs

75aa9d1

feat(agent): [US-006] prove workflow efficiency deltas

67adaee

fix(agent): revalidate compacted configs

5684322

Merge remote-tracking branch 'origin/main' into more-efficent-workflows

77f3055

# Conflicts: # tests/test_agent_supervisor.py

SkyeAv merged commit af7a060 into main Sep 4, 2026
5 checks passed

Navigation Menu

perf(agent): reduce LLM workflow overhead - #138

perf(agent): reduce LLM workflow overhead#138
SkyeAv merged 8 commits into
mainfrom
more-efficent-workflows

SkyeAv commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

SkyeAv commented Sep 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LLM Workflow

Audit and Persistence

Measurement

Design

Docs

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

SkyeAv commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading