[Question] How to measure gain/savings?

[nisrulz]

• I was wondering if there is a way to see the savings/gains . Something like this

[DeusData]

Good question — there's no written guidance on measuring the token/time savings yet. Will put something together; thanks for raising it.

[koriyoshi2041]

Thanks for confirming there is no written guidance yet. I did a quick read through the existing measurement surfaces and it looks like there may be a small docs-first path here before adding any new command.

What I found:

• README already states two different savings frames: the preprint-level 10x fewer tokens / 2.1x fewer tool calls result, and the smaller five-query example of about 3.4K tokens vs about 412K.
• docs/BENCHMARK.md has the quality/attempt methodology, but it does not yet explain how a user should reproduce a "gain" measurement on their own repo.
• CBM_DIAGNOSTICS=1 already exposes query count/error/time stats through /tmp/cbm-diagnostics-<pid>.json, and scripts/soak-test.sh records latency/diagnostic CSVs, but those are closer to stability/perf metrics than token/tool savings.

Would a first useful contribution be a short docs/MEASURING_SAVINGS.md that separates:

1. quality: pass/partial/fail benchmark answer quality;
2. latency/stability: existing diagnostics and soak-test CSVs;
3. token/tool-call savings: how to compare a fixed question set against a file-by-file baseline, with caveats that CBM itself does not know the agent's final token count unless the client records it?

If that scope matches what you had in mind, I can draft the docs-only first pass and keep it tied to the existing benchmark/diagnostics surfaces rather than inventing a new CLI command prematurely.

[DeusData]

Yes, that docs-only scope matches what I had in mind. Please keep the three measurement classes separate: answer quality, latency/stability, and token/tool-call savings. The important caveat is exactly the one you identified: the server can report its own query timing and counts, but final model tokens and tool calls must be measured by the client or evaluation harness. A focused docs/MEASURING_SAVINGS.md linked to the existing benchmark and diagnostics material would be a useful first contribution without adding a new CLI surface.

[juliengloria]

How to be sure my mcp is working well ?

[DeusData]

Answered on main in 489b1771 (PR #1847, "docs: add reproducible savings measurement guide").

Thank you for asking it in the first place — "how do I measure the gain?" is the right question to put to a tool that claims one, and it deserved a reproducible procedure rather than a number in a README. The guide documents the method so the figure can be re-derived on your own repositories instead of taken on trust.