fix(ci): the invisible-character gate never matched anything - #196
Conversation
MEASURED 2026-08-27: this gate's pattern caught 0 OF 6 invisible-character test
cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi
override or word joiner.
ROOT CAUSE: the pattern used UTF-8 BYTE sequences (\xc2\xa0) while grep -P
matches CHARACTERS. Bytes c2 a0 are ONE character U+00A0; \xc2\xa0 asks for TWO
characters, U+00C2 then U+00A0, which is never present.
grep -P '\xc2\xa0' -> miss
grep -P '\x{a0}' -> MATCH
Only \x00 worked, being single-byte in both readings.
FIXED: codepoint escapes; C0 control characters \x01-\x08,\x0B,\x0C,\x0E-\x1F
added (TAB/LF/CR excluded); and grep -a, without which grep skips any NUL-bearing
file as binary.
The C0 range matters: a stray BACKSPACE byte made a workflow unparseable in
developer-ecosystem, so it never ran, and this linter called it clean.
Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
VERIFIED: YAML re-parsed, and the corrected pattern was confirmed to catch a real
NBSP before the change was kept.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (14)
🔇 Additional comments (2)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe workflow replaces byte-oriented invisible-character patterns with PCRE Unicode code-point escapes. It also adds binary-safe grep processing with ChangesInvisible-character gate
Estimated code review effort: 2 (Simple) | ~5 minutes Merge Risk: ⚪ Minimal · up to This PR corrects the workflow gate’s matching behavior for invisible characters and NUL-bearing files. The change is localized, and no actionable merge-blocking risk remains beyond normal checks and review. Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Linked Issues checkExplanation The changes address Unicode codepoint escapes and binary-safe scanning, and the description states that C0 controls were added [ Resolution Add a separate byte-wise leading-BOM check. Update stdlib/ByteDetector.affine and config.ncl with the matching C0-control logic, including is_c0_control/1. Add or run tests that verify invisible characters, a leading BOM, NUL and backspace bytes, clean files, and legitimate TAB, LF and CR characters. Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull Request Overview
This PR improves the invisible-character linter gate by transitioning to Unicode codepoint matching using PCRE and ensuring files containing null bytes are not skipped. While these changes address the stated issue of the gate failing to match intended characters, there are two primary areas of concern.
First, there is a total lack of automated regression testing for this change; none of the required test scenarios for character detection or null-byte handling are currently covered. Second, the command implementation is inefficient and potentially unreliable. Using -exec ... {} \; in a large repository will impact performance, and silencing stderr while using the PCRE engine risks having the linter silently skip files if the regex engine encounters encoding errors. Codacy analysis indicates the PR is otherwise up to standards.
About this PR
- The PR lacks automated regression tests or a 'dirty' test file containing the invisible characters (NBSP, BOM, C0 controls, etc.) to ensure the linter functions correctly and prevents future regressions.
Test suggestions
- Verify detection of Non-Breaking Space (U+00A0) using the new codepoint escape.
- Verify detection of C0 control characters (e.g., backspace \x08).
- Verify that files containing null bytes are scanned rather than skipped by grep.
- Verify detection of Unicode Byte Order Mark (U+FEFF).
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify detection of Non-Breaking Space (U+00A0) using the new codepoint escape.
2. Verify detection of C0 control characters (e.g., backspace \x08).
3. Verify that files containing null bytes are scanned rather than skipped by grep.
4. Verify detection of Unicode Byte Order Mark (U+FEFF).
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| -o -name '*.idr' -o -name '*.zig' -o -name '*.v' -o -name '*.jl' \ | ||
| -o -name '*.gleam' -o -name '*.hs' -o -name '*.ml' -o -name '*.sh' \) \ | ||
| -exec grep -Prl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null | ||
| -exec grep -aPrl "$PATTERNS" {} \; > /tmp/empty-lint-results.txt 2>/dev/null |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: The current command has several areas for improvement:
- Performance: Using
-exec ... {} \;spawns a separategrepprocess for every file. Switching to{} +allowsfindto pass multiple files to a singlegrepinstance, significantly improving speed in large repositories. - Redundancy: The
-rflag ingrepis unnecessary becausefindhandles the directory traversal. - Reliability: Using
2>/dev/nullhides errors fromgrep -P. If the PCRE engine fails on a malformed file, the error will be swallowed, potentially allowing characters to bypass the check.
Consider this optimization:
find ... -exec grep -aPl "$PATTERNS" {} +
🔍 Hypatia Security ScanFindings: 56 issues detected
View findings[
{
"reason": "Required file missing (condition: public_repo)",
"type": "missing_requirement",
"file": "SECURITY.md",
"action": "create",
"rule_module": "cicd_rules",
"severity": "high"
},
{
"reason": "Agda postulate assumes without proof -- potential soundness hole (1 occurrences, CWE-704)",
"type": "agda_postulate",
"file": "/home/runner/work/valence-shell/valence-shell/proofs/agda/FilesystemModel.agda",
"action": "flag",
"rule_module": "code_safety",
"severity": "critical"
},
{
"reason": "believe_me undermines formal verification (2 occurrences, CWE-704)",
"type": "believe_me",
"file": "/home/runner/work/valence-shell/valence-shell/proofs/idris2/src/Filesystem/Axioms.idr",
"action": "flag",
"rule_module": "code_safety",
"severity": "critical"
},
{
"reason": "unsafe block -- requires SAFETY comment (16 occurrences, CWE-676)",
"type": "unsafe_block",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/commands.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
},
{
"reason": "as_ptr exposes raw pointer that may dangle or alias unsafely (6 occurrences, CWE-676)",
"type": "as_ptr",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/commands.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
},
{
"reason": "expect() in hot path (4 occurrences, CWE-754)",
"type": "expect_in_hot_path",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/parser.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
},
{
"reason": "unsafe block -- requires SAFETY comment (2 occurrences, CWE-676)",
"type": "unsafe_block",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/process_sub.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
},
{
"reason": "as_ptr exposes raw pointer that may dangle or alias unsafely (2 occurrences, CWE-676)",
"type": "as_ptr",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/process_sub.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
},
{
"reason": "unwrap() without prior check -- DoS via panic (1 occurrences, CWE-754)",
"type": "unwrap_without_check",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/test_command.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "high"
},
{
"reason": "expect() in hot path (1 occurrences, CWE-754)",
"type": "expect_in_hot_path",
"file": "/home/runner/work/valence-shell/valence-shell/impl/rust-cli/src/external.rs",
"action": "flag",
"rule_module": "code_safety",
"severity": "medium"
}
]Powered by Hypatia Neurosymbolic CI/CD Intelligence |
Measured 2026-08-27: this gate caught 0 of 6 invisible-character test cases. It has never detected an NBSP, zero-width space, BOM, soft hyphen, bidi override or word joiner.
Root cause
The pattern used UTF-8 byte sequences (
\xc2\xa0) whilegrep -Pmatches characters. Bytesc2 a0are one character U+00A0;\xc2\xa0asks for two, U+00C2 then U+00A0 — never present.Only
\x00worked, being single-byte in both readings. The gate ran, passed, and could not see what it exists to see.Fixed
\x01-\x08,\x0B,\x0C,\x0E-\x1Fadded (TAB/LF/CR excluded)grep -a— without it grep skips any NUL-bearing file as binaryThe C0 range matters: a stray backspace byte made a workflow unparseable in
developer-ecosystem, so it never ran — and this linter called it clean.Canonical fix: hyperpolymath/empty-linter#70. 1 file(s) here.
Verified: YAML re-parsed, and the corrected pattern was confirmed to catch a real NBSP before the change was kept.