Skip to content

keboola-expert.md budget: the test is fine, but a stale CI merge commit let an over-budget merge through #585

Description

@padak

Corrected after investigation. The original text of this issue claimed the budget is unenforced and proposed adding a test. That was wrong: tests/test_agent_prompt.py:31 already defines PROMPT_BYTE_BUDGET = 62_000 and asserts against it. It was missed by a grep 62000 — the constant is written 62_000. The real gap is different and is described below.

The gap: green checks that describe a merge which no longer exists

.github/workflows/ci.yml uses actions/checkout@v5 with no ref:, so on a pull_request event it checks out the merge commit. The budget test therefore does measure the merged result — it is the right test, wired the right way. But a merge commit is only recomputed when the PR is updated. A PR that sits while main moves keeps reporting a green result for a merge that no longer reflects reality.

What happened on PR #580

When main keboola-expert.md merged with #580 Outcome
2026-08-12, CI run 73ded7a (61494 B) 61623 B under budget — CI passed correctly
2026-08-13, after #565 + trim 7445d76 61990 B 62119 B over budget, but CI never re-ran

CI ran exactly twice on that branch: once on the original commit, once when the fix was pushed. In between, main grew by 496 B. The over-budget merge was caught by hand with a git merge-tree simulation during review; no automation would have caught it, even though the test exists and works.

This generalises past this one file: any invariant that depends on both sides of a merge (the licence-declaration test from #577 has the same shape) is only as fresh as the last CI run on that branch.

Options

  1. Require branches to be up to date before merging (repo ruleset). Closes the hole exactly. Cost: every PR needs a rebase/update when main moves, which is real friction on a repo with this merge rate.
  2. Merge queue. Same guarantee without the manual rebase, at the cost of setup and slower merges.
  3. Accept and document. Keep it a reviewer responsibility, but say so explicitly in CONTRIBUTING.md so it is a known check rather than folklore — and note that a stale green check is not evidence about the merge.

Option 1 is the honest fix; option 3 is the cheap one. Worth a deliberate decision rather than drifting into 3 by default.

Two smaller things found alongside

  • CONTRIBUTING.md says "60 KB" in three places (lines 381, 464, 594) while the enforced budget is 62 000 B. The test's own comment records the bump from 60 kB to 62 kB in v0.48.0; the prose was never updated. Worth fixing so the number in the checklist matches the number that fails the build.
  • Headroom is now 15 bytes. After fix(search): make --search-type config-based case-insensitive (#569) #580 the merged file sits at 61985 B. The next addition to that file hits the ceiling immediately, whichever option above is chosen.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions