fix(ci): the invisible-character gate never matched anything - #62
fix(ci): the invisible-character gate never matched anything#62hyperpolymath wants to merge 1 commit into
Conversation
MEASURED 2026-08-27: this gate's pattern caught 0 OF 6 invisible-character test
cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi
override or word joiner.
ROOT CAUSE: the pattern used UTF-8 BYTE sequences (\xc2\xa0) while grep -P
matches CHARACTERS. Bytes c2 a0 are ONE character U+00A0; \xc2\xa0 asks for TWO
characters, U+00C2 then U+00A0, which is never present.
grep -P '\xc2\xa0' -> miss
grep -P '\x{a0}' -> MATCH
Only \x00 worked, being single-byte in both readings.
FIXED: codepoint escapes; C0 control characters \x01-\x08,\x0B,\x0C,\x0E-\x1F
added (TAB/LF/CR excluded); and grep -a, without which grep skips any NUL-bearing
file as binary.
The C0 range matters: a stray BACKSPACE byte made a workflow unparseable in
developer-ecosystem, so it never ran, and this linter called it clean.
Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
VERIFIED: YAML re-parsed, and the corrected pattern was confirmed to catch a real
NBSP before the change was kept.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (6)
|
| Layer / File(s) | Summary |
|---|---|
Update invisible-character scanning .github/workflows/dogfood-gate.yml |
The scan matches Unicode code points, includes additional control characters, and uses grep -aPrl to process binary files as text. |
Estimated code review effort: 2 (Simple) | ~5 minutes
Merge Risk: ⚪ Minimal · up to 900bf
This change corrects the workflow gate so it can detect the intended invisible characters and control bytes. No actionable merge-blocking risk remains beyond normal checks and review.
Poem
A rabbit checks each hidden mark,
With Unicode light against the dark.
Binary files now join the queue,
The gate sees what it must review.
Soft spaces hop into the park.
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Linked Issues check | The workflow change addresses codepoint escapes, C0 controls, and NUL scanning with grep -a. It does not show the separate leading-BOM check or the required updates to stdlib/ByteDetector.affine and c… | Add the separate byte-wise leading-BOM check and update stdlib/ByteDetector.affine and config.ncl with the matching C0-control handling. Alternatively, narrow the linked issue or split the remaining requirements into a separate pull request… |
✅ Passed checks (4 passed)
| Check name | Status | Explanation |
|---|---|---|
| Title check | ✅ Passed | The title clearly identifies the main change: fixing the CI gate for invisible characters. |
| Description check | ✅ Passed | The description gives a clear root cause, lists the implemented changes, and records verification results. It does not use all template headings or include the checklist, but it contains the main requ… |
| Out of Scope Changes check | ✅ Passed | The two-line change is limited to the invisible-character CI gate and is related to the linked issue. No unrelated changes are shown. |
| Docstring Coverage | ✅ Passed | No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
Full details: Description check
Explanation
The description gives a clear root cause, lists the implemented changes, and records verification results. It does not use all template headings or include the checklist, but it contains the main required information.
Full details: Linked Issues check
Explanation
The workflow change addresses codepoint escapes, C0 controls, and NUL scanning with grep -a. It does not show the separate leading-BOM check or the required updates to stdlib/ByteDetector.affine and config.ncl for compiled-linter alignment.
Resolution
Add the separate byte-wise leading-BOM check and update stdlib/ByteDetector.affine and config.ncl with the matching C0-control handling. Alternatively, narrow the linked issue or split the remaining requirements into a separate pull request and link it explicitly.
Full details: Docstring Coverage
Explanation
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
- Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
- Create stacked PR
- Commit on current branch
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull Request Overview
The PR correctly addresses the non-functional invisible-character gate by transitioning from byte-level hex sequences to Unicode codepoint escapes compatible with 'grep -P'. However, there is a significant risk that the PCRE engine will fail silently for characters above \xFF (such as U+200B) unless UTF-8 mode is explicitly enabled in the pattern.
While the logic for detecting C0 controls and handling null bytes with 'grep -a' is sound, the implementation lacks automated tests to verify these patterns. Furthermore, the CI command execution can be optimized and made more transparent by removing error silencing and improving file batching.
About this PR
- The PR description mentions that the previous gate caught '0 of 6 test cases', implying the existence of test cases, but no automated tests or fixtures (e.g., sample files containing these invisible characters) are included to prevent regression of these regex patterns.
Test suggestions
- Verify detection of Non-Breaking Space (U+00A0) using codepoint escape
- Verify detection of C0 control characters like Backspace (\x08)
- Verify detection of Byte Order Mark (U+FEFF)
- Verify that a file containing a Null byte (\x00) is flagged and not skipped as binary
- Verify detection of Soft Hyphen (U+00AD) and Word Joiner (U+2060)
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify detection of Non-Breaking Space (U+00A0) using codepoint escape
2. Verify detection of C0 control characters like Backspace (\x08)
3. Verify detection of Byte Order Mark (U+FEFF)
4. Verify that a file containing a Null byte (\x00) is flagged and not skipped as binary
5. Verify detection of Soft Hyphen (U+00AD) and Word Joiner (U+2060)
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| # non-breaking spaces, null bytes, and other invisible Unicode in source files. | ||
| set +e | ||
| PATTERNS='\xc2\xa0|\xe2\x80\x8b|\xe2\x80\x8c|\xe2\x80\x8d|\xef\xbb\xbf|\xc2\xad|\xe2\x80\x8e|\xe2\x80\x8f|\xe2\x80\xaa|\xe2\x80\xab|\xe2\x80\xac|\xe2\x80\xad|\xe2\x80\xae|\x00' | ||
| PATTERNS='\x00|[\x01-\x08\x0B\x0C\x0E-\x1F]|\x{a0}|\x{ad}|\x{200b}|\x{200c}|\x{200d}|\x{200e}|\x{200f}|\x{202a}|\x{202b}|\x{202c}|\x{202d}|\x{202e}|\x{2060}|\x{feff}' |
There was a problem hiding this comment.
🔴 HIGH RISK
To ensure the PCRE engine correctly handles Unicode codepoints above \xFF (such as \x{200b}), you should prepend the (*UTF) verb to the pattern string. This prevents silent failures or 'hexadecimal value is greater than \xFF' errors in environments where UTF-8 mode isn't the default for PCRE.
| -o -name '*.idr' -o -name '*.zig' -o -name '*.v' -o -name '*.jl' \ | ||
| -o -name '*.gleam' -o -name '*.hs' -o -name '*.ml' -o -name '*.sh' \) \ | ||
| -exec grep -Prl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | ||
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: Improve efficiency and visibility by batching the grep calls and removing redundant flags and error silencing. The addition of the -a flag is essential for files containing null bytes, but using -exec ... {} + is more efficient than -exec ... {} ;. Additionally, 2>/dev/null should be removed so that regex syntax errors are visible in CI logs.
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | |
| -exec grep -aPl "$PATTERNS" {} + > /tmp/empty-lint-results.txt |
Measured 2026-08-27: this gate caught 0 of 6 invisible-character test cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi override or word joiner.
Root cause
The pattern used UTF-8 byte sequences (
\xc2\xa0) whilegrep -Pmatches characters. Bytesc2 a0are one character U+00A0;\xc2\xa0asks for two, U+00C2 then U+00A0 — never present.Only
\x00worked, being single-byte in both readings. The gate ran, passed, and could not see what it exists to see.Fixed
\x01-\x08,\x0B,\x0C,\x0E-\x1Fadded (TAB/LF/CR excluded)grep -a— without it grep skips any NUL-bearing file as binaryThe C0 range matters: a stray backspace byte made a workflow unparseable in
developer-ecosystem, so it never ran — and this linter called it clean.Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
Verified: YAML re-parsed, and the corrected pattern was confirmed to catch a real NBSP before the change was kept.