Decode %UTF8% parameter values centrally in ParameterCollection.Parse - #54
Open
Andrei-Errapart wants to merge 1 commit into
Open
Decode %UTF8% parameter values centrally in ParameterCollection.Parse#54Andrei-Errapart wants to merge 1 commit into
Andrei-Errapart wants to merge 1 commit into
Conversation
Altium writes non-ASCII parameter values twice: a %UTF8% variant (UTF-8
bytes) first, then a plain-ANSI twin. Parse stripped the prefix without
decoding, so both entries collapsed onto one key and the first-match
indexer returned the mojibake variant ("KON 4.7µF" instead of
"KON 4.7µF" for SchComponent.Description, and any other %UTF8% key).
Decode the value when the prefix is stripped, so the UTF-8 variant wins
with the correct string. Remove the now-redundant per-site decodes of
the Text parameter in SchDocReader/SchLibReader (the %UTF8% key is still
detected to preserve TextIsUtf8 for the writer), promote
ComponentDescription to a %UTF8% parameter on write when Windows-1252
cannot represent it (as with SchParameter.Text), and drop the renderer's
mojibake-repair heuristic for parameter-sourced text (it could corrupt
legitimate strings like "Não"; binary pin names keep it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Altium writes non-ASCII parameter values twice: a
%UTF8%variant (UTF-8 bytes) first, then a plain-ANSI twin.ParameterCollection.Parsestripped the%UTF8%prefix without decoding the value, so both entries collapsed onto one key and the first-match indexer returned the mojibake variant —"KON 4.7µF"instead of"KON 4.7µF"forSchComponent.Description, and the same for any other%UTF8%key.Decoding the value at the point the prefix is stripped makes the UTF-8 variant win with the correct string, which in turn lets several downstream workarounds go away.
Changes
AltiumRecordGenerator— in the generatedParameterCollection.Parse, decode the value viaAltiumEncoding.DecodeUtf8ParameterValuewhen the%UTF8%prefix is stripped.SchDocReader/SchLibReader— remove the now-redundant per-site decode of theTextparameter. The%UTF8%key is still detected, purely to preserveTextIsUtf8for the writer.SchLibWriter— promoteComponentDescriptionto%UTF8%ComponentDescriptionwhen Windows-1252 cannot represent the description, matching howSchParameter.Textis written.SchComponentRenderer— stop applying theFixTextEncodingmojibake-repair heuristic to parameter-sourced text (text frames, net labels, harness type labels and entries, string indirection); it could corrupt legitimate strings such as"Não". Binary pin names, which carry no encoding marker, still use it.Test Plan
dotnet test) — 853 passed, 10 skipped, 0 failedAdded to
Utf8ParameterRoundTripTests:ParameterCollection_Utf8AndAnsiDuplicate_ReturnsDecodedValue— a record carrying both%UTF8%ComponentDescriptionand its ANSI twin resolves to the decoded string.SchDoc_NonAsciiComponentDescription_RoundTrips— round-trips a Windows-1252-representable description (KON 4.7µF ±10%, plain key) and one that is not (Shunt 5mΩ, forcing the%UTF8%key).Checklist
FixTextEncodingdoc comment now states that it covers binary pin names onlyNo signatures change, but two behavioural changes are worth calling out:
ParameterCollection.Parsenow returns decoded values for%UTF8%-prefixed keys. Any caller that repaired the mojibake itself (asSchDocReader/SchLibReaderdid) would double-decode and should drop its own fix-up.SchComponentRenderer.FixTextEncoding, so text that was previously "repaired" at render time now renders as stored.