Skip to content

tests: migrate to data-driven fixtures for feature presence - #2986

Merged
williballenthin merged 14 commits into
masterfrom
data-fixtures-2
May 11, 2026
Merged

tests: migrate to data-driven fixtures for feature presence#2986
williballenthin merged 14 commits into
masterfrom
data-fixtures-2

Conversation

@williballenthin

@williballenthin williballenthin commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator

closes #2743

This PR refactors our test suite with the primary goal of making the feature tests data-driven; that is, we now have a collection of JSON files that describe where backends should find various features. This enables non-Python implementations of capa to demonstrate that they extract the features we'd expect.

Along the way, I also made some other cleanups. In particular, I tried to simplify fixtures.py a bit. I removed a lot of the symbolic extractors (e.g., pma01_03_extractor) since many were used only once or twice in other modules; I inlined this logic with more explicit code. I figure, more obvious code -> easier to mechanically port to another language. Pytest fixtures as magic test function parameter names is not that easy to reason about...

To the reviewers, I request a thorough review of not just the code style, but of the underlying implementation, design, decisions, etc. I think you should set aside an hour for this.

Here's how I'd recommend that you review these changes:

  • read the new test_viv_extractor.py to see how a backend test now looks
  • read tests/fixtures/features/README.md and static.json to see how the features are declared
  • read fixtures.py for familiarity. don't read the diff, read the final.
  • read various diffs when they're reasonable, but i'm afraid there's a bunch of churn, so it may not be easy to parse. focus on the end result, imho, and possibly ask Gemini for any hotspots.

To be clear, there's moderate risk here: I've translated the existing feature test cases to a new format, and I might have missed or corrupted some of them. In some scenarios, I tried to consolidate feature test cases (like when IDA only worked on some samples, and Ghidra on a different set) and it's possible that I mangled the underlying intent. However, I did my best and the test suite probably high quality. I believe we should accept this risk, but you should think about this, too.

Checklist

  • This submission includes AI-generated code and I have provided details in the description.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

Comment thread tests/fixtures/features/static.json
Comment thread tests/fixtures.py Outdated
gemini-code-assist[bot]

This comment was marked as outdated.

@williballenthin
williballenthin force-pushed the data-fixtures-2 branch 6 times, most recently from f8d4c8b to ad6aeb0 Compare April 15, 2026 10:13
@williballenthin

This comment was marked as resolved.

@github-actions
github-actions Bot dismissed their stale review April 21, 2026 12:46

CHANGELOG updated or no update needed, thanks! 😄

@williballenthin
williballenthin marked this pull request as ready for review April 21, 2026 13:50

@mr-tz mr-tz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work, great step into a data-driven test direction - a few comments to discuss inline

Comment thread capa/rules/__init__.py
Comment thread capa/rules/__init__.py
Comment thread tests/fixtures/features/static.json
Comment thread tests/fixtures/features/static.json
Comment thread tests/fixtures/features/static.json
Comment thread tests/fixtures/features/vmray.json
Comment thread tests/data
Comment thread tests/test_dynamic_span_of_calls_scope.py
Comment thread tests/test_idalib_features.py Outdated
Comment thread rules
@williballenthin
williballenthin force-pushed the data-fixtures-2 branch 2 times, most recently from e1fb1ff to f0c8c3f Compare May 11, 2026 07:06

@mr-tz mr-tz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking like it's on a good track!

@williballenthin

Copy link
Copy Markdown
Collaborator Author

awaiting CI outcome. i have no further pending TODOs, so recommend a full review and decision on whether to merge.

@mr-tz

mr-tz commented May 11, 2026

Copy link
Copy Markdown
Collaborator

Ha! Just went through it so assuming CI passes, we're good to go IMHO.

@williballenthin
williballenthin merged commit 61adf15 into master May 11, 2026
29 checks passed
@williballenthin
williballenthin deleted the data-fixtures-2 branch May 11, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

migrate feature tests fixtures to a JSON representation

2 participants