Skip to content

Decode command output as UTF-8; stop guessing the remote code page (#142) - #143

Merged
bertysentry merged 1 commit into
mainfrom
142-utf8-command-output
Jul 28, 2026
Merged

Decode command output as UTF-8; stop guessing the remote code page (#142)#143
bertysentry merged 1 commit into
mainfrom
142-utf8-command-output

Conversation

@bertysentry

Copy link
Copy Markdown
Contributor

Fixes #142.

The bug

The remote shell was created with console code page 437 while its output was decoded with the
charset from Win32_OperatingSystem.CodeSet — the remote machine's ANSI code page. Those two
can only agree on a US-English host. On a French Windows Server 2022, dir /A came back as:

 Le num‚ro de s‚rie du volume est E6B6-D774
23/07/2026  15:46           786ÿ432 NTUSER.DAT

CP437's 0x82 (é) and 0xFF (NBSP) read as windows-1252 and ÿ. Confirmed as a decoding
bug, not a console-rendering artifact: the String the library returned already held U+201A.

The fix

Create the shell with WINRS_CODEPAGE=65001 and decode command output as UTF-8 unconditionally.
The output charset is then known without asking, it is lossless for every locale, and the
SELECT CodeSet FROM Win32_OperatingSystem round trip previously made before every command is
gone.

before after
shell code page 437 65001
decode charset Win32_OperatingSystem.CodeSet (ANSI) UTF-8
extra WQL query per command yes no
  • Envelopes.createShell: WINRS_CODEPAGE 437 → 65001.
  • WindowsRemoteExecutor.SHELL_OUTPUT_CHARSET is the single place naming the shell's charset;
    WinRMClient.detectCharset and both detection calls in WinRMCommandExecutor are gone.
  • CommandRequest.charset(Charset) is unchanged as an explicit override.
  • WindowsRemoteProcessUtils.getWindowsEncodingCharset() is deprecated (not removed): the library
    no longer calls it, and it never answered the question it was being asked.

Verified on a real French Windows Server 2022

 Le volume dans le lecteur C n’a pas de nom.
 Le numéro de série du volume est E6B6-D774
 Répertoire de C:\Users\Administrateur

Byte-checked rather than eyeballed — the thousands separator arrives as C2 A0 (U+00A0 NBSP), and
remote-generated Japanese, Greek and Cyrillic round-trip exactly:

日本語 Δε При é     →  E6 97 A5 E6 9C AC E8 A8 9E  CE 94 CE B5  D0 9F D1 80 D0 B8  C3 A9  C2 A0

None of that is representable in CP437 at all.

Known residual limitation (documented, not a regression)

A few legacy tools — net.exe and chcp.com are the ones I found — ignore the console code page
and write text pre-converted to the machine's OEM code page. Captured from the host via
certutil, net user really does send r\x82gion (CP850) through a 65001 shell, so its accents
arrive as U+FFFD. This was equally broken before (it produced r‚gion instead), and
charset(Charset.forName("IBM850")) decodes it exactly. Called out in the README, commands.md,
and covered by a test.

Tests

  • commandDecodesOutputAsUtf8WithoutProbingTheRemoteCodeSet — no Win32_OperatingSystem request
    reaches the wire before the first command.
  • commandOutputKeepsNonAsciiCharactersOfEveryLocale — French and Japanese output survives.
  • explicitCharsetOverridesTheShellDefault — the OEM escape hatch, using the exact bytes net.exe
    sends.
  • Updated the three tests that asserted the old behaviour (WsmanProtocolTest, WinRmCliTest,
    WinRMCommandExecutorTest).

mvn clean verify site: 148 tests pass, 0 checkstyle / 0 PMD / 0 SpotBugs, no new Javadoc warnings.

Docs

README gains a Character encoding section; commands.md, cli.md and timeouts-and-errors.md
no longer describe the detection step, and both README and commands.md carry a "Changed in
2.0.00" note explaining why the old behaviour was wrong.

🤖 Generated with Claude Code

)

The remote shell was created with console code page 437 while its output was
decoded with the charset from Win32_OperatingSystem.CodeSet. That property is
the remote machine's ANSI code page, not the console (OEM) one, so the two
could only agree on a US-English host. On a French Windows Server 2022, `dir`
came back with "num‚ro de s‚rie" and "786ÿ432": CP437's 0x82 (é) and 0xFF
(NBSP) read as windows-1252.

Create the shell with code page 65001 and decode as UTF-8 unconditionally. The
output charset is then known without asking, it is lossless for every locale
(CJK included, which no single-byte OEM page can carry), and the
SELECT CodeSet FROM Win32_OperatingSystem round trip made before every command
is gone.

Verified against a French Windows Server 2022: "Le numéro de série", the NBSP
thousands separator, and remote-generated Japanese, Greek and Cyrillic all come
back byte-exact.

A few legacy tools (net.exe, chcp.com) ignore the console code page and write
OEM bytes regardless; charset(Charset) decodes those, and the README and site
docs now say so. WindowsRemoteProcessUtils.getWindowsEncodingCharset() is
deprecated: the library no longer calls it, and it never answered the question
it was being asked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bertysentry
bertysentry merged commit 8a09c71 into main Jul 28, 2026
5 checks passed
@bertysentry
bertysentry deleted the 142-utf8-command-output branch July 28, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Command output is mis-decoded on non-English Windows: shell pinned to CP437, output decoded as the ANSI code page

1 participant