Fix unicode-range tokenization and serialization - #191
Merged
TylerBrinks merged 1 commit intoJul 23, 2026
Conversation
Three defects kept @font-face unicode-range descriptors from working: A start of fewer than six hex digits returned immediately after the wildcard branch, so it never reached the '-<hex>' range-end handling. "U+41-5A" tokenized as just "U+41" and left "-5A" behind as stray tokens. Only the wildcard form is mutually exclusive with the range form, so a '-' now falls through to the shared range handling. The wildcard padding loop used "6 - StringBuffer.Length" as its bound while appending to that same buffer, so the bound shrank on every iteration and only half the available wildcards were consumed: "U+??????" stopped after three. The budget is now captured up front. ValueBuilder had no case for TokenType.Range, so range tokens hit the default arm and marked the whole value invalid, dropping the descriptor entirely under strict (non-tolerant) parsing. RangeToken.Data holds the range without its "U+" prefix, so a retained descriptor serialized as "41-5A" rather than "U+41-5A". ToValue now restores the prefix. Also makes RangeToken.SelectedRange lazy: a legal range such as U+0-10FFFF expands to over a million strings, which the constructor materialized eagerly whether or not anything read it.
jhaygood86
marked this pull request as ready for review
July 22, 2026 21:17
jhaygood86
marked this pull request as draft
July 22, 2026 21:40
jhaygood86
marked this pull request as ready for review
July 22, 2026 21:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
@font-face { unicode-range: … }does not survive parsing. Four separate defects combine:1. A start of fewer than six hex digits can't have a range end.
Lexer.UnicodeRangereturns immediately out of the wildcard branch, so it never reaches the-<hex>handling below it:U+41-5Atokenized as justU+41, leaving-5Abehind as stray tokens. Only the wildcard form is mutually exclusive with the start-end form (CSS Syntax 3 §4.3.6); a-should fall through.2. The wildcard padding loop consumes only half its wildcards. The bound
6 - StringBuffer.Lengthis re-evaluated each iteration while the body appends to that same buffer, so it shrinks asigrows and they meet in the middle.U+??????stopped after three?, leaving???unconsumed.3.
ValueBuilderhas no case forTokenType.Range, so range tokens fall to the default arm, mark the whole value invalid, and the descriptor is dropped entirely under strict (non-tolerant) parsing — the default.4.
RangeTokenserializes without itsU+prefix.Dataholds the bare range, so once a descriptor is retained it round-trips as41-5A, which is not a valid<urange>.Fix
One change each, plus making
RangeToken.SelectedRangelazy — a legalU+0-10FFFFexpands to over a million strings, which the constructor materialized eagerly whether or not anything ever read the property. (Nothing in the library reads it today.)Tests
13 tests.
Tokenization.cscovers the lexer directly: start/end forms (U+41-5A,U+0-7F,U+400-4FF,U+000041-00005A), the wildcard and single-value forms (U+4??,U+??????,U+41), and on-demandSelectedRangeexpansion.FontFace.cscovers the descriptor end-to-end throughIFontFaceRule.Range, including a comma-separated list.10 of the 13 fail on
master. The full suite (1263 existing tests) stays green, and all seven target frameworks build with no new warnings.