rule coverage feature for unit tests

[moritz-gross]

Hi everybody,

Neil and I were thinking about adding a way to see which rules are covered by unit tests and which rules are not.

Some open questions include:

• should this be available using the existing Python tools, e.g. uv run audit-translations --rule-coverage (seems the best to me)
• other than which rules are hit, are there other relevant statistics to know?
• are we interested in the rules hit be a specific test, or vice-versa in the tests using a specific rule, or any other more detailed breakdowns?
• I assume displaying the results only in a command line is too messy, so I suggest opening a browser window to show something similar to this
• @daisy/mathcat-translators any other thoughts in general?

My suggestion is that I just start with a basic version, which should not be that hard to do, and then I iterate based on feedback collected here.

[moritz-gross]

update: I've create a first prototype on the branch rule-coverage.
On this branch, run uv run audit-translations --rule-coverage, that's it (might take a few minutes though).

[tayfuryldz]

For the first version I'd collect both directions even if the report initially shows only one: rule to tests and test to rules. Once instrumentation records rule IDs per test, the reverse index is nearly free. JSON output alongside HTML would also make later CI use possible without forcing a coverage threshold now.