fix(windows-sweep): POSIX path + UTF-8 encoding for Make/jq/SQL/verify/demo scripts (#519, #521) - #545
Merged
Merged
Conversation
…521) Capture/validate_make.sh and fingerprint_{jq,make}.sh had two Windows compat bugs that broke them on Git Bash with native GNU Make / Python: #519 — POSIX path passed to GNU Make / Python (capture_make.sh, validate_make.sh): - ${mk_path} interpolated into heredoc `include` directive. On Windows Git Bash, ${mk_path} is POSIX-style (/c/Users/...); native Windows GNU Make misreads it as relative to current drive (C:\c\Users\...), producing 'file not found' on every cluster. - Same issue affects MK_PATH env var passed to Python in recompute_inputs_hashes(), which then writes the POSIX path to a temp Makefile via f.write(f"include {mk_path}\n"). Fix: add tool_path() helper (cygpath -m when available, no-op on Linux/Mac — same pattern proven in PR #524/#526 for capture_tcl.sh and 19 other stacks). Convert mk_path to mk_path_make before heredoc interpolation and before passing to Python. #521 — Embedded python3 snippets without UTF-8 encoding (fingerprint_jq.sh, fingerprint_make.sh, capture_make.sh, validate_make.sh): - stable_stringify() in fingerprint_jq.sh + fingerprint_make.sh uses json.dumps(obj, ensure_ascii=False) which produces non-ASCII stdout for unicode inputs. On Windows native Python (default cp1252 stdout), this crashes with UnicodeEncodeError. - Multiple `python3 -c` snippets in capture_make.sh + validate_make.sh call json.load(sys.stdin) on input that may contain UTF-8 multi-byte chars. On Windows native Python (default cp1252 stdin), this crashes with UnicodeDecodeError. Fix: prefix every affected `python3 -c` invocation with `PYTHONIOENCODING=utf-8` env var. This forces UTF-8 for stdin, stdout, and stderr without changing the snippet logic. No-op on Linux/Mac (UTF-8 is already the default there). Files changed: - scripts/capture_make.sh — tool_path() helper + mk_path_make + PYTHONIOENCODING on 6 python3 -c calls - scripts/validate_make.sh — tool_path() helper + mk_path_make + PYTHONIOENCODING on recompute_inputs_hashes + 2 call_args python3 -c calls - scripts/fingerprint_jq.sh — PYTHONIOENCODING on stable_stringify - scripts/fingerprint_make.sh — PYTHONIOENCODING on stable_stringify Verified locally on Linux: - bash -n syntax check passes for all 4 files - PYTHONIOENCODING=utf-8 is a no-op on Linux (UTF-8 already default) - tool_path() returns the path unchanged on Linux (cygpath not present) Needs Windows CI verification: cygpath conversion + PYTHONIOENCODING behavior on native Windows Python should be confirmed with a real capture -> validate cycle on a Make cluster with non-ASCII inputs. Out of scope (untouched, already fixed by recent PRs): - capture/validate_{tcl,rust,crystal,csharp,dart,go,haskell,julia, kotlin,nim,scala,swift,zig}.sh — fixed in #524/#526/#528/#532/#534/#530 - capture/validate_{c,cpp,fsharp,java,bash}.sh — confirmed NOT to use bash->python/node path pattern (delegate to compiled binary / dotnet / java / jq), so no #519 issue - fingerprint_{bash,haskell,swift,tcl}.sh — Python snippets only do base36 conversion (ASCII hex input, ASCII digit output), no UTF-8 risk
verify_*.sh and proof/*/demo-refactor-flow.sh have embedded Python snippets that open() source files (Java, Rust, Lua, C, C++) which contain non-ASCII characters (em-dashes — in comments, ❌ emoji in error messages, etc.). On Windows native Python, open() defaults to cp1252, which raises UnicodeDecodeError on the multi-byte UTF-8 sequences. The same snippets also print() non-ASCII strings (✅ emoji) to stdout, which raises UnicodeEncodeError on cp1252 stdout. Fix per the user's #521 specification: - Prefix every `python3 << 'PYEOF'` heredoc invocation with `PYTHONIOENCODING=utf-8` env var (forces UTF-8 for stdin/stdout/ stderr — no-op on Linux/Mac where UTF-8 is already default). - Add `encoding='utf-8'` to every `open(path)` and `open(path, 'w')` call so file I/O uses UTF-8 explicitly (PYTHONIOENCODING doesn't affect open() default encoding). Files changed: - scripts/verify_java_stack.sh — 2 Python snippets (open Java source) - scripts/verify_rust_stack.sh — 2 Python snippets (open Rust source) - scripts/verify_lua_stack.sh — 1 Python snippet (open Lua source) - proof/c/demo-refactor-flow.sh — 3 Python snippets (open C source) - proof/cpp/demo-refactor-flow.sh — 3 Python snippets (open C++ source) - proof/c_bitops/demo-refactor-flow.sh — 2 Python snippets (open C source) - proof/java/demo-refactor-flow.sh — 2 Python snippets (open Java source) Verified locally on Linux: - bash -n syntax check passes for all 7 files - proof/c/demo-refactor-flow.sh: ran end-to-end successfully (capture -> validate PASS -> valid refactor PASS -> breaking refactor FAIL -> restore). All .regret files restored to HEAD state after test. - PYTHONIOENCODING=utf-8 is a no-op on Linux - encoding='utf-8' on open() is a no-op on Linux (already default) Needs Windows CI verification: confirm the fix resolves UnicodeDecodeError/UnicodeEncodeError on native Windows Python with cp1252 locale. Out of scope (POSIX path issues in verify/demo scripts): - verify_*.sh and demo-*.sh pass POSIX-style paths to Python via env vars ($JAVA_FILE, $LIB_RS, $FIXTURE, $DEMO_SRC). On Windows native Python, these paths (/c/Users/...) would not resolve. This is a #519 issue but the user's #1 scope is restricted to scripts/capture_*.sh and scripts/validate_*.sh. Per scope-discipline skill, documenting as 'Found but not fixed' for BOS to decide.
Three Node.js scripts spawn python3 child processes without setting
PYTHONIOENCODING, causing UnicodeDecodeError/UnicodeEncodeError on
Windows native Python (default cp1252 stdin/stdout/stderr):
- scripts/capture_sql.mjs (line 117): spawnSync('python3', ['-c', script])
where the Python script does json.loads(sys.stdin.read()) on a SQL
request JSON that may contain UTF-8 multi-byte chars (unicode strings
in bind_params or setup_sql).
- scripts/validate_sql.mjs (line 50): same pattern, same issue.
- scripts/contest.mjs (line 387): execFileSync('python3', [_chain_step.py,
payload]) where _chain_step.py prints json.dumps(result,
ensure_ascii=False) to stdout (line 133) and ❌ emoji to stderr
(lines 21/64/87/103). Both crash with UnicodeEncodeError on cp1252.
Fix: add `env: { ...process.env, PYTHONIOENCODING: 'utf-8' }` to the
spawn options. This forces UTF-8 for the Python child's stdin/stdout/
stderr without changing the script logic. No-op on Linux/Mac (UTF-8 is
already the default there).
Note: capture_sql.mjs and validate_sql.mjs were touched in #529 for a
CRLF fix in parseRegret(), but that PR did NOT address the python3
encoding issue. contest.mjs has not been touched in any Windows-fix PR.
Verified locally on Linux:
- node --check syntax passes for all 3 files
- tests/sql-stack.test.js: 4/4 PASS (capture writes .regret, validate
PASSes, breaking change FAILs, cross-stack parity holds)
- PYTHONIOENCODING=utf-8 is a no-op on Linux
Needs Windows CI verification: confirm the fix resolves
UnicodeDecodeError (capture_sql/validate_sql) and UnicodeEncodeError
(contest.mjs chain step) on native Windows Python with cp1252 locale.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Closed
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Windows compatibility sweep for issues #519 (POSIX paths passed to Node.js/Python via bash stack scripts) and #521 (embedded Python snippets without UTF-8 encoding). Audited all 38
scripts/capture_*.sh+scripts/validate_*.shfiles plus allverify_*.shandproof/*/demo-*.shscripts. Fixed 14 files; the rest were either already fixed in recent PRs (#524, #526, #528, #532, #534, #530) or don't use the affected patterns.Issues closed
capture_*.sh/validate_*.shwith the POSIX path issue)Files changed
POSIX path fix (#519) + UTF-8 fix (#521)
scripts/capture_make.shtool_path()helper +mk_path_makePYTHONIOENCODING=utf-8on 6python3 -ccallsscripts/validate_make.shtool_path()helper +mk_path_makePYTHONIOENCODING=utf-8onrecompute_inputs_hashes+ 2call_argscallsUTF-8 fix only (#521)
scripts/fingerprint_jq.shPYTHONIOENCODING=utf-8onstable_stringifyscripts/fingerprint_make.shPYTHONIOENCODING=utf-8onstable_stringifyscripts/capture_sql.mjsenv: { ...process.env, PYTHONIOENCODING: 'utf-8' }onspawnSync('python3', ...)scripts/validate_sql.mjsspawnSync('python3', ...)scripts/contest.mjsexecFileSync('python3', ['_chain_step.py', ...])scripts/verify_java_stack.shencoding='utf-8'onopen()+PYTHONIOENCODING=utf-8(2 Python snippets)scripts/verify_rust_stack.shscripts/verify_lua_stack.shproof/c/demo-refactor-flow.shproof/cpp/demo-refactor-flow.shproof/c_bitops/demo-refactor-flow.shproof/java/demo-refactor-flow.shAudit table — all
scripts/capture_*.shandscripts/validate_*.shcapture_bash.sh/validate_bash.shfingerprint_bash.sh(jq-based, no node -e); CRLF guard added in #536. Nobash→node/pythonpath issue.capture_c.sh/validate_c.shbash→node/pythonpath issue.capture_cpp.sh/validate_cpp.shbash→node/pythonpath issue.capture_crystal.sh/validate_crystal.shcapture_csharp.sh/validate_csharp.shcapture_dart.sh/validate_dart.shcapture_fsharp.sh/validate_fsharp.shdotnet run; nobash→node/pythonpath issue. CRLF guard in #536.capture_go.sh/validate_go.shcapture_haskell.sh/validate_haskell.shcapture_java.sh/validate_java.shjava RegretJava.java; nobash→node/pythonpath issue.capture_jq.sh/validate_jq.shfingerprint_jq.shUTF-8 fix is in this PR.)capture_julia.sh/validate_julia.shcapture_kotlin.sh/validate_kotlin.shcapture_make.shtool_path()cygpath helper +mk_path_make+PYTHONIOENCODING=utf-8on 6python3 -ccalls.validate_make.shtool_path()cygpath helper +mk_path_make+PYTHONIOENCODING=utf-8onrecompute_inputs_hashes+ 2call_argscalls.capture_nim.sh/validate_nim.shcapture_rust.sh/validate_rust.shcapture_scala.sh/validate_scala.shcapture_swift.sh/validate_swift.shcapture_tcl.sh/validate_tcl.shcapture_zig.sh/validate_zig.shEmbedded Python snippets audit (non-
capture_*.sh/validate_*.sh)fingerprint_bash.shfingerprint_haskell.shfingerprint_swift.shfingerprint_tcl.shfingerprint_jq.shstable_stringifyusesjson.dumps(obj, ensure_ascii=False)— needsPYTHONIOENCODING=utf-8.fingerprint_make.shfingerprint_jq.sh.capture_sql.mjsspawnSync('python3', ['-c', script])reads JSON viasys.stdin.read()— needsPYTHONIOENCODING=utf-8env.validate_sql.mjscapture_sql.mjs.contest.mjsexecFileSync('python3', ['_chain_step.py', ...])—_chain_step.pyprintsjson.dumps(result, ensure_ascii=False)+ ❌ emoji to stderr.parity_test_bash.shpython3 -ccalls; sourcesfingerprint_bash.shonly.verify_css_stack.shnode -eforrequire.resolve('postcss')— no Python, no path issue (postcss is in node_modules).verify_go_stack.shnode -ewith$SKILL_DIR/scripts/fingerprint.js—SKILL_DIRis set from$(dirname ...)which Git Bash resolves natively. No Python.verify_java_stack.shopen($JAVA_FILE)— Java source has em-dashes + ❌ emoji. Addedencoding='utf-8'+PYTHONIOENCODING=utf-8.verify_rust_stack.shopen($LIB_RS)— Rust source has em-dashes. Same fix.verify_lua_stack.shopen($FIXTURE/strings.lua)— Lua source has em-dashes. Same fix.verify_perl_stack.shproof/c/demo-refactor-flow.shopen($DEMO_SRC)— C source has em-dashes.proof/cpp/demo-refactor-flow.shproof/c_bitops/demo-refactor-flow.shproof/java/demo-refactor-flow.shVerification
npm testresults (relevant subset — full suite takes 5+ min)The 5 failures are all in
issue-272-contest-callee-revalidation.test.jsandissue-283-contest-config-options.test.js— pre-existing onmain(confirmed bygit checkout main && node --test ...→ same 5 failures). They are NOT caused by my changes.Sample capture + validate output for fixed stacks (Linux)
Make stack (
proof/make_slugify/):jq stack (
proof/jq_slugify/) — verifiesfingerprint_jq.shUTF-8 fix:C demo-refactor-flow (
proof/c/) — verifies UTF-8 fix in demo script:Syntax checks
bash -npasses for all 11 modified shell scriptsnode --checkpasses for all 3 modified.mjsfilesCross-platform note
cygpathconversion +PYTHONIOENCODING=utf-8behavior on native Windows Python should be confirmed with a real capture → validate cycle on Make + jq + SQL stacks with non-ASCII inputstool_path()returns path unchanged; UTF-8 already default)The fixes use the same pattern proven in PR #524 (Tcl) and #526 (19 stacks) —
cygpath -mwhen available, no-op otherwise. ThePYTHONIOENCODING=utf-8env var is a standard Python convention that's a no-op on Linux/Mac (UTF-8 is already the default there).Reference
cygpath -mpattern for Tcl (pattern source for this PR).cspath fixFound but not fixed (out of scope)
Per scope-discipline skill, documenting these for BOS to decide:
scripts/capture_awk.mjs+scripts/validate_awk.mjs— pass POSIX-styleawkFiletospawn(bin, ['-f', awkFile, ...]). On Windows Git Bash,awkFileis POSIX-style (/c/Users/...); native Windowsawk.exemay misread it. Out of scope because feat: support non-JSON-serializable types (ArrayBuffer, Map, Set, NaN, etc.) #1 in the task brief only coversscripts/capture_*.shandscripts/validate_*.sh(not.mjs).verify_*.shandproof/*/demo-*.shpass POSIX paths to Python via env vars ($JAVA_FILE,$LIB_RS,$FIXTURE,$DEMO_SRC). On Windows native Python, these paths would not resolve. Out of scope for the same reason — feat: support non-JSON-serializable types (ArrayBuffer, Map, Set, NaN, etc.) #1 only coverscapture_*.sh/validate_*.sh. The UTF-8 fix ([RED TEAM] Embedded python3 snippets in demo/verify scripts crash on Windows due to missing explicit UTF-8 encoding #521) was applied because feat: add Go stack support (Community Preview) — tested against bfgo #2 explicitly includes "embedded python3 snippets inscripts/*.sh,proof/*/verify-*.mjs,proof/*/demo-*.sh".