Honour OutputHandler.Encoding regardless of a byte order mark - #77
Merged
Merged
Conversation
Process builds its StandardOutput and StandardError readers with byte-order-mark detection switched on, so output beginning with a BOM was decoded with the encoding the BOM named rather than the one the caller asked for. Output starting FF FE came back as UTF-16LE text under a strict UTF-8 handler, and a lone FF FE was consumed as a mark, leaving a run that reported no output and no error for bytes it should have rejected. Read the raw pipes with the caller's encoding instead, and drop a leading U+FEFF so a mark matching that encoding is still stripped as it was before. Separately, the read loop restarted whichever read had completed, and a faulted task is a completed one. A decode failure raised while the process was still running was overwritten by a fresh read and never observed, which is what made such failures look intermittent: the repository's own suite lost one roughly every six runs. Stop the loop when either read has faulted so the exception reaches the caller. Four new tests cover both faults, and all four fail without the change. The existing 35 pass unchanged. Fixes #76 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RUsrYfSAuSQ7VWws7vzsr
The Windows CI leg failed the two byte-order-mark tests. The cause was the harness, not the fix: cmd /c type transcodes a file carrying a UTF-16 byte order mark rather than copying its bytes, so the invalid bytes never reached the decoder and no failure was raised. PowerShell now writes the raw bytes to the standard output stream instead. Rather than trust that emitter too, each test first confirms the bytes really reach the pipe unchanged, and reports the platform as inconclusive with the bytes it saw when they do not. That check drives the emitter through Process directly and copies the raw BaseStream, never touching the code under test: a control that went through RunCommand would measure the very defect these tests exist to catch and blame the emitter for it, turning a regression into an inconclusive result instead of a failure. Verified both ways, with the source reverted and with a deliberately mangling emitter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RUsrYfSAuSQ7VWws7vzsr
|
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #76.
Two independent defects made
OutputHandler.Encodingunreliable. Both are measured atmain(
afa8bc1), .NET SDK 10.0.401, Linux.1. A byte order mark replaced the requested encoding
AsyncProcessStreamReaderreadprocess.StandardOutput, andProcessbuilds thatStreamReaderwith
detectEncodingFromByteOrderMarks: true. Output beginning with a BOM was decoded with theencoding the BOM named, not the one the caller asked for.
FF FE 68 00 69 00(UTF-16LE BOM +hi)hi, decoded as UTF-16LE, no errorDecoderFallbackExceptionFE FF 00 68 00 69(UTF-16BE BOM +hi)hi, decoded as UTF-16BE, no errorDecoderFallbackExceptionFF FEaloneDecoderFallbackExceptionEF BB BF hello(UTF-8 BOM)hellohello— unchangedFFaloneDecoderFallbackExceptionThe third row is what
ktsu-dev/GitBranchStateCache#27reported as a decode failure that fires"only sometimes". It was not intermittent — those two bytes were eaten as a mark, leaving nothing
to decode.
The fix reads
BaseStreamwith the caller's encoding anddetectEncodingFromByteOrderMarks: false.Turning detection off also turns off the BOM stripping that came with it, so
StripLeadingByteOrderMarkdrops a single leading U+FEFF from each stream's first chunk. Everyencoding decodes its own BOM to U+FEFF, so that covers all of them without guessing — and a BOM
belonging to a different encoding no longer decodes to U+FEFF, which is precisely the case that
should now surface as bad bytes. Row four shows the benign case is byte-for-byte unchanged.
2. A faulted read was discarded while the process was still running
The loop restarted whichever read had completed — and a faulted task is a completed one:
A decode failure raised while the process was still alive was overwritten by a fresh read, which
then returned 0 at EOF, and the call reported success. Whether the exception surfaced depended
purely on whether the process exited before the next pass.
This is not theoretical — it flaked this PR's own tests at about one run in six before I found
it. The loop now stops when either read has faulted, so
Task.WhenAllrethrows. This applies toany exception from a read or an
OutputHandlercallback, not only decoding.AsyncProcessStreamReadernow owns its twoStreamReaders, so it implementsIDisposableandRunAsyncholds it in ausing.Task.WhenAllcompletes both reads before rethrowing, so neitherreader is disposed with a read in flight.
The test harness, and why it is shaped this way
The first push failed the Windows leg, and the cause was the harness rather than the fix:
cmd /c typetranscodes a file carrying a UTF-16 byte order mark instead of copying its bytes,so the invalid bytes never reached the decoder and no failure was raised. PowerShell now writes the
raw bytes to the standard output stream instead.
Rather than trust that emitter either, each test first confirms the bytes really reach the pipe
unchanged and reports the platform as inconclusive — quoting the bytes it actually saw — when they
do not. That control drives the emitter through
Processdirectly and copies the rawBaseStream,deliberately never touching the code under test: an earlier version of it went through
RunCommand, which meant that on unfixed source it measured the very defect these tests exist tocatch and blamed the emitter for it, turning a regression into an inconclusive result instead of a
failure. That was caught and fixed before this push.
Verification
the faulted-task guard: 2 fail. Both restored: all pass. So neither fix is carrying the other.
failures and 2 skips naming the exact bytes (
Expected [FFFE], the pipe carried [FE]); with thesource reverted it reports genuine failures and no emitter excuses. It cannot produce a false
green or a false red.
in 6;
ADecodeFailureIsNotDiscardedWhenTheProcessKeepsRunningmakes that race deterministic andfails every time without the fix.
— the two pre-existing UAC-elevation tests plus the race test. The three byte-order-mark tests
run and pass on Windows, so the PowerShell emitter is byte-faithful there and that coverage is
real, not skipped.
Caveats
Assert.Inconclusive, because it needs a shell that canemit bytes and then stay alive. The code it covers is platform independent, so the Linux and
macOS legs cover it.
INFOmaintainability smells in the new tests, none posted as review comments:
MSTEST0037twice(
Assert.ContainsoverAssert.IsTrue) andMSTEST0061([OSCondition]over anAssert.Inconclusiveplatform guard). Left alone rather than spending a CI cycle on them; happyto fold them in if this PR is touched again.
Not in this PR
Decode failures still reach the caller wrapped in
AggregateException(fromReadCallback'sreadTask.Result), socatch (DecoderFallbackException)does not catch one unchanged —ktsu-dev/GitBranchStateCache#27notes this. Unwrapping is a visible change to the exceptioncallers see, so it is left for its own decision.
CommandOptionsalso still has no way to redirector close the child's stdin, which is the other half of what that issue needs.
🤖 Generated with Claude Code
https://claude.ai/code/session_014RUsrYfSAuSQ7VWws7vzsr