feat(testcomp): BenchExec packaging, and always emit a test suite (H1.4) - #61
Merged
Merged
Conversation
H1.4 was the last unstarted item in H1, and H1 is the front that decides
whether Map2Check can be submitted to Test-Comp at all.
Emit the suite for every outcome, not only for a violation. It used to live
inside `if (foundViolation)`, so a run that decided nothing left no
test-suite/ directory behind -- and the competition harness reads an absent
directory as a tool that crashed, not as an honest zero. metadata.xml is now
always written; foundViolation decides whether the suite carries a test case,
not whether the suite exists.
map2check-testcomp-wrapper.py is a separate entry point from the SV-COMP
wrapper on purpose. That one answers "does this program satisfy the property"
and prints a verdict; this one produces an artefact and lets TestCov score it.
The verdict vocabulary, the exit codes and the output are all different, and
folding both into one script would mean behaviour that depends on a property
string in two unrelated ways. It resolves the binary against its own path
rather than "./map2check", because BaseTool2 dropped the working_directory()
hook the old wrapper relied on: the process now starts in the benchmark's
directory, not the tool's.
utils/moduleBenchExec/map2check_testcomp.py is the tool-info module, written
against the introspected benchexec 3.35 API rather than from memory. It
refuses cover-branches instead of accepting it: branch coverage needs one test
case per input vector and the runtime writes one nondet log per run (H1.3), so
accepting it would look like participation while scoring zero and hiding why.
The harness walltime becomes the tool's own budget, so it is Map2Check that
stops and finishes writing metadata.xml rather than the harness killing it
mid-write.
The existing map2check.py is left alone. It is the SV-COMP adaptor and still
uses BaseTool 1.x -- deprecated, but still present in benchexec 3.35, so it is
legacy rather than dead, and migrating it is a separate change with its own
risk.
Verified against a full build and install, not in isolation:
- violation -> metadata.xml + testcase-1.xml, specification verbatim
- no violation -> metadata.xml, 0 test cases
- tool-info -> 15/15 against the real benchexec package
- TestCov corpus -> 6/6, including the two NOT_COVERED cases that now reach
TestCov with an empty suite instead of taking the old
"no suite produced" shortcut
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes H1.4, the last unstarted item in H1 — conformance, which is the front that decides whether Map2Check can be submitted to Test-Comp at all.
1. The suite is now emitted for every outcome
Emission lived inside
if (foundViolation). A run that decided nothing left notest-suite/directory at all — and the competition harness reads an absent directory as a tool that crashed, not as an honest zero.metadata.xmlis now always written.foundViolationdecides whether the suite carries a test case, not whether the suite exists.metadata.xml+testcase-1.xmlmetadata.xml, 0 test cases2.
utils/map2check-testcomp-wrapper.pySeparate from the SV-COMP wrapper deliberately. That one answers "does this program satisfy the property" and prints a verdict; this one produces an artefact and lets TestCov score it. Different verdict vocabulary, different exit codes, different output — folding both into one script would mean behaviour depending on a property string in two unrelated ways.
It resolves the binary against its own path, not
./map2check:BaseTool2dropped theworking_directory()hook the old wrapper relied on, so the process now starts in the benchmark's directory, not the tool's. (The binary must still be namedmap2check— it derivesMAP2CHECK_PATHfrom/proc/self/exe.)3.
utils/moduleBenchExec/map2check_testcomp.pyThe tool-info module, in
BaseTool2, written against the introspected benchexec 3.35 API rather than from memory — the whole risk here is the API not being what we assumed.Two decisions worth calling out:
cover-branchesis refused, not accepted. Branch coverage needs one test case per input vector and the runtime writes one nondet log per run (H1.3). Accepting it would look like participation while scoring zero and hiding the reason. RaisingUnsupportedFeatureExceptionmakes BenchExec skip the task instead.metadata.xml, rather than the harness killing it mid-write and leaving a truncated file — which is worse than an empty suite.determine_resultonly distinguishes ran from broke. Claiming anything stronger would be the tool self-reporting whether its suite covers the error, which is exactly what the competition does not allow.4.
tests/integration/test_benchexec_toolinfo.py— 15 assertionsNothing in this repository exercised the tool-info module, and its failure mode is nasty: it surfaces on competition infrastructure, days after submission, as every task erroring out. Tested against the real benchexec package, not a stub.
Wired into the TestCov job, which already has benchexec (testcov depends on it).
5.
scripts/package-release.shThe wrapper goes into the release zip, because the module's
REQUIRED_PATHSdeclares it and BenchExec copies exactly those paths to the execution node — a file missing from the zip becomes an error on competition infrastructure, not here.Verification
Against a full build and install, not the pieces in isolation:
The TestCov corpus mattered most here: the two
NOT_COVEREDcases used to take the runner's "no test suite produced" shortcut, and now reach TestCov with an empty suite. TestCov handles that and reportsResult: UNKNOWN, so the gate holds — this was the one way the change could have broken it.Not in this PR
fm-tools— lives outside this repository.map2check.py(SV-COMP) toBaseTool2— it is onBaseTool1.x, which is deprecated but still present in benchexec 3.35, so it is legacy rather than dead. Separate change, separate risk.