docs: add reproducible savings measurement guide - #1847
Merged
Conversation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #361
Summary
Recommendation and trade-offs
The selected approach is a controlled paired Graph/file-by-file experiment: one isolated one-question session per condition, identical inputs and budgets, source-grounded quality grading, and client-captured usage.
Benefits:
Trade-offs:
Alternatives considered:
query_countfor total tool calls was rejected because it cannot observe file tools, orchestration, retries, or other client activityValidation
909051ccee2d070f33e15fda71bfe61c9076eea631724178b2623b074f1ab87b1588f91a0dd5760dgit diff --check, and DCO trailerPlatform boundaries
This change touches documentation only; no product code or tests changed. A prior full Linux arm64 run on the pre-rebase documentation tree reported 7,393 passing tests, 0 failures, and 8 platform skips. The amd64 attempt was blocked by unrelated QEMU/ASan infrastructure, macOS by host ENOSPC, and the real-Windows endpoint was unavailable. Those unavailable legs are left to hosted CI and downstream review rather than represented as local greens.