fix hasher/matcher bugs that show up with SMDA 4 and mcrit

[r0ny123]

A handful of these have been sitting in the hasher/matcher for a while. They started to actually hurt once mcrit began indexing SMDA 4 reports (including AArch64).

Intel-only PIC hashes

picblocks/blockhasher.py calculateBlockhash always did:

instruction.getEscapedBinary(IntelInstructionEscaper, escape_intraprocedural_jumps=True, ...)

SMDA 4.x has SmdaReport.getInstructionEscaper() / SmdaFunction._escaper (intel / aarch64 / cil / dalvik). mcrit MemoryStorage / MongoDbStorage call getBlockhashesForFunction(..., hash_size=8) on every add, so non-Intel functions were stored under Intel-style escaping.

The hasher now uses the function/report escaper (checking smda_function._escaper, report.getInstructionEscaper(), or SmdaFunction.getInstructionEscaper(architecture)) and only falls back to Intel if nothing else is available. The SHA-256 prefix is unpacked little-endian (<I / <Q) so the integer is the same as SMDA 4.x on a big-endian host.

Matcher percentages vs block_bytes

extractBlockhashes adds block_size to block_bytes for every hashed block. match() credited each family once per (hash, size) (the family_ids set was reset per hash+size, then reused across query fids).

A sample with the same 16-byte matching block in 10 functions plus 16 unmatched bytes scored 16/176 instead of 160/176. The family loop now credits once per query fid.

While I was in there: unmatched_hashes was initialized to 0 and never written, and a hash hit / size miss bumped unmatched_score but not unmatched_blocks. saveDb() also ensures parent directories are created (os.makedirs) if saving to a new directory path, and load() handles missing filenames and empty blockhashes safely.

Dump vs PE routing

processFile used "dump" in filepath, so /data/dumps/malware.exe went through disassembleBuffer. It now matches Malpedia-style names (dump_0x... / dump7_0x...) on the basename.

processBuffer still treats _0x in the filename, or an explicit baseaddress (including 0), as a mapped buffer. baseaddress=0 used to be falsy and dropped the override, so the PE mapper ran instead.

Dump regexes and base/bitness parsers also support 5–6 digit hex base addresses (such as dump_0x400000) and standard architecture identifiers (x86, i386, amd64, win32, win64).

Smaller things

• SMDA error reports (status="error", xcfg is None on 1.12.7) no longer crash hashing
• num_functions_hashed was incremented next to num_blocks inside the per-block loop
• hash_malpedia.py skipped 9/10 files (file_index % 10 != 0), read e_lfanew as a WORD at 0x3C (now DWORD), and Pool(cpu_count() - 2) dies on 1–2 CPUs. _identifyPython now covers Python 3.10+ DLLs (python310.dll, python311.dll, etc.), and path extraction handles both / and \ separators.
• utils/import_picblocksdb_to_mongo.py refactored into import_db() with fast insert_many batch insertion and guarded CLI execution.
• setup.py had unpinned "smda" while requirements.txt pinned 1.12.7 — both now smda>=4.2.13 (mcrit's floor)
• Web UI fixes: result page never passed block_b into report.html; default DB timestamp was %Y-%d-%d; hidden bitness radio value="0" won over the 32-bit control; added HTML escaping for family links; /stats falls back to db/stats.json if Mongo is offline; /blocks and /api/blocks validate payloads (returning 400 on empty API requests and accepting bitness/baseaddress parameters).
• README still said python -m blocks.blockhashmatcher and python -m utils.make_stats.py; Makefile pointed at nose / package blocks

Tooling, Typing & CI

• Added pyproject.toml configuring ruff and Astral ty. Codebase is formatted and passes both ruff check / ruff format and ty check with zero diagnostics.
• pytest covers the hasher, matcher, and the demo form/API (59 tests). The new workflow runs a dedicated lint job (ruff + ty), unit tests, and a second job that starts Mongo 7 + mcrit server/worker, checks out SMDA's xored fixtures, and submits cutwail / aarch64_static / bashlite through the same getBlockhashesForFunction path mcrit uses. requests is guarded in test_mcrit_live.py so unit test discovery doesn't fail when running offline.
• Actions are pinned to full SHAs (actions/checkout@3d3c42e..., actions/setup-python@5fda3b9...), persist-credentials: false, and permissions: contents: read.