Decode command output as UTF-8; stop guessing the remote code page (#142) - #143
Merged
Conversation
) The remote shell was created with console code page 437 while its output was decoded with the charset from Win32_OperatingSystem.CodeSet. That property is the remote machine's ANSI code page, not the console (OEM) one, so the two could only agree on a US-English host. On a French Windows Server 2022, `dir` came back with "num‚ro de s‚rie" and "786ÿ432": CP437's 0x82 (é) and 0xFF (NBSP) read as windows-1252. Create the shell with code page 65001 and decode as UTF-8 unconditionally. The output charset is then known without asking, it is lossless for every locale (CJK included, which no single-byte OEM page can carry), and the SELECT CodeSet FROM Win32_OperatingSystem round trip made before every command is gone. Verified against a French Windows Server 2022: "Le numéro de série", the NBSP thousands separator, and remote-generated Japanese, Greek and Cyrillic all come back byte-exact. A few legacy tools (net.exe, chcp.com) ignore the console code page and write OEM bytes regardless; charset(Charset) decodes those, and the README and site docs now say so. WindowsRemoteProcessUtils.getWindowsEncodingCharset() is deprecated: the library no longer calls it, and it never answered the question it was being asked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #142.
The bug
The remote shell was created with console code page 437 while its output was decoded with the
charset from
Win32_OperatingSystem.CodeSet— the remote machine's ANSI code page. Those twocan only agree on a US-English host. On a French Windows Server 2022,
dir /Acame back as:CP437's
0x82(é) and0xFF(NBSP) read as windows-1252‚andÿ. Confirmed as a decodingbug, not a console-rendering artifact: the
Stringthe library returned already held U+201A.The fix
Create the shell with
WINRS_CODEPAGE=65001and decode command output as UTF-8 unconditionally.The output charset is then known without asking, it is lossless for every locale, and the
SELECT CodeSet FROM Win32_OperatingSystemround trip previously made before every command isgone.
Win32_OperatingSystem.CodeSet(ANSI)Envelopes.createShell:WINRS_CODEPAGE437 → 65001.WindowsRemoteExecutor.SHELL_OUTPUT_CHARSETis the single place naming the shell's charset;WinRMClient.detectCharsetand both detection calls inWinRMCommandExecutorare gone.CommandRequest.charset(Charset)is unchanged as an explicit override.WindowsRemoteProcessUtils.getWindowsEncodingCharset()is deprecated (not removed): the libraryno longer calls it, and it never answered the question it was being asked.
Verified on a real French Windows Server 2022
Byte-checked rather than eyeballed — the thousands separator arrives as
C2 A0(U+00A0 NBSP), andremote-generated Japanese, Greek and Cyrillic round-trip exactly:
None of that is representable in CP437 at all.
Known residual limitation (documented, not a regression)
A few legacy tools —
net.exeandchcp.comare the ones I found — ignore the console code pageand write text pre-converted to the machine's OEM code page. Captured from the host via
certutil,net userreally does sendr\x82gion(CP850) through a 65001 shell, so its accentsarrive as
U+FFFD. This was equally broken before (it producedr‚gioninstead), andcharset(Charset.forName("IBM850"))decodes it exactly. Called out in the README,commands.md,and covered by a test.
Tests
commandDecodesOutputAsUtf8WithoutProbingTheRemoteCodeSet— noWin32_OperatingSystemrequestreaches the wire before the first command.
commandOutputKeepsNonAsciiCharactersOfEveryLocale— French and Japanese output survives.explicitCharsetOverridesTheShellDefault— the OEM escape hatch, using the exact bytesnet.exesends.
WsmanProtocolTest,WinRmCliTest,WinRMCommandExecutorTest).mvn clean verify site: 148 tests pass, 0 checkstyle / 0 PMD / 0 SpotBugs, no new Javadoc warnings.Docs
README gains a Character encoding section;
commands.md,cli.mdandtimeouts-and-errors.mdno longer describe the detection step, and both README and
commands.mdcarry a "Changed in2.0.00" note explaining why the old behaviour was wrong.
🤖 Generated with Claude Code