Skip to content

docs: add reproducible savings measurement guide - #1847

Merged
DeusData merged 1 commit into
mainfrom
docs/issue-361-measuring-savings-resume
Aug 28, 2026
Merged

docs: add reproducible savings measurement guide#1847
DeusData merged 1 commit into
mainfrom
docs/issue-361-measuring-savings-resume

Conversation

@DeusData

Copy link
Copy Markdown
Owner

Closes #361

Summary

  • add a reproducible guide for measuring answer quality, indexing/query latency, stability, model-token use, and total client tool calls
  • link the README performance claim to the measurement guide and explicitly avoid treating the published headline ratio as universally reproducible
  • freeze repository state with clean detached worktrees, require fresh graph preflight evidence, and keep quality separate from efficiency metrics
  • classify indexing as full-source, artifact-assisted, or incremental using successful import evidence
  • require directly measured, comparable usage windows without inferring or duplicating session totals

Recommendation and trade-offs

The selected approach is a controlled paired Graph/file-by-file experiment: one isolated one-question session per condition, identical inputs and budgets, source-grounded quality grading, and client-captured usage.

Benefits:

  • produces auditable measurements tied to an exact repository SHA, question set, model, machine, and run count
  • prevents a fast query or low token count from standing in for answer correctness
  • avoids misusing CBM's query counters as a proxy for model tokens or non-CBM tool calls
  • distinguishes full-source indexing from artifact-assisted and incremental paths

Trade-offs:

  • clean worktrees, isolated sessions, retained artifacts, and blind grading require more setup and disk space than a headline-only comparison
  • requiring a usage window to be directly reported by both clients can reduce the number of comparable metrics, but avoids fabricated or duplicated totals
  • the guide is a methodology and reproducibility recipe, not a turnkey benchmark harness

Alternatives considered:

  • publishing only a single savings ratio was rejected because it cannot be generalized without the original inputs and artifacts
  • substituting CBM query_count for total tool calls was rejected because it cannot observe file tools, orchestration, retries, or other client activity
  • forcing two token windows for every client was rejected because unsupported windows would have to be inferred or misattributed

Validation

  • exact base: 909051ccee2d070f33e15fda71bfe61c9076eea6
  • exact head: 31724178b2623b074f1ab87b1588f91a0dd5760d
  • independent exact-commit review: PASS
  • current-source checks: CLI flags and Git-context fields, diagnostics records/counters, artifact import success/failure signals, soak entry point/artifacts/headers, grading rubrics, formulas, and zero-denominator handling
  • documentation checks: relative targets and anchors, one-line CSV schema, shell syntax, failure-injection checks, exact two-file scope, git diff --check, and DCO trailer
  • disposable detached-worktree setup and cleanliness flow: PASS; that block was unchanged by the later prose-only methodology corrections
  • Linux arm64 container lint: no-skips, cppcheck, clang-format, and NOLINT all PASS on the final tree

Platform boundaries

This change touches documentation only; no product code or tests changed. A prior full Linux arm64 run on the pre-rebase documentation tree reported 7,393 passing tests, 0 failures, and 8 platform skips. The amd64 attempt was blocked by unrelated QEMU/ASan infrastructure, macOS by host ENOSPC, and the real-Windows endpoint was unavailable. Those unavailable legs are left to hosted CI and downstream review rather than represented as local greens.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit 489b177 into main Aug 28, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Question] How to measure gain/savings?

1 participant