Align indexed and patterned hash meta fields (#74) - #88
Merged
Conversation
Issue YARAHQ#74 documented that align_yara_rule_hashes only matches a fixed list of exact key names, so indexed/patterned hash fields used across real rule sets (hash1..hash31, md5_1..md5_6, SHA256_1/2, hash_1..hash_9, hash1_sha256, thumbprint1-4, sample_md5, sample_sha1, parent_hash, etc.) pass through unnormalized instead of collapsing to 'hash' like the plain-named variants already do. Add a regex match for the indexed/patterned key shapes alongside the existing exact-name list, plus an explicit exclusion for a few hash-adjacent fields that are not raw sample hash values (logic_hash is the pipeline's own dedup key, malpedia_hash/yarahub_reference_md5 are third-party reference fields, exemplar_hashes is a list not a hash string) so they're not swept in by a broader pattern. Verified against the full key list from the issue - the new pattern picks up 84 indexed/typo'd hash-shaped keys while leaving the 5 keys above and other non-hash fields (sharing, tlp, etc.) untouched.
Contributor
|
Thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #74
align_yara_rule_hashesinmain/rule_processors.pyonly matches a fixed list of exact key names (hash,md5,sha1,sha256, etc.). The issue's key dump shows a ton of hash fields that don't match anything on that list:hash1throughhash31,md5_1throughmd5_6,SHA256_1/SHA256_2,hash_1throughhash_9,hash1_sha256,thumbprint1-thumbprint4,sample_md5,sample_sha1,parent_hash, and more. All of those pass straight through to the output instead of collapsing intohashthe waymd5/sha1/sha256already do.This adds a regex check alongside the existing name list to catch the indexed/patterned variants (
hash1,hash_1,md5_1,sha256_1,hash1_sha256,thumbprint2, etc.) and aligns them tohashthe same way. I also added a small exclusion list for a few fields that look hash-ish but shouldn't get swept in:logic_hashis the pipeline's own dedup key (set earlier inprocess_yara_rules), andmalpedia_hash/yarahub_reference_md5/exemplar_hashesare third-party reference fields, not a rule's own sample hash.I didn't try to solve the whole issue here (it lists 200+ meta keys total, covering author/description/TLP casing too) - that's a much bigger and more opinionated change. This just closes the hash-field gap the issue calls out specifically, since that's the largest and most mechanical chunk of it.
Verification:
sharing/tlp.tests/test_rule_processors_hashes.pycovering both cases (indexed keys aligning tohash, and the exclusions staying put).test_rule_collector.py,test_rule_output_guardrails.py) but they fail the same way on a clean checkout of master with no changes, so that's pre-existing drift, not something this touches.flake8 main tests --select=E9,F63,F7,F82(the syntax-error check CI runs) is clean.