Skip to content

Fix unicode-range tokenization and serialization - #191

Merged
TylerBrinks merged 1 commit into
TylerBrinks:masterfrom
jhaygood86:bugfix/unicode-range-token
Jul 23, 2026
Merged

Fix unicode-range tokenization and serialization#191
TylerBrinks merged 1 commit into
TylerBrinks:masterfrom
jhaygood86:bugfix/unicode-range-token

Conversation

@jhaygood86

Copy link
Copy Markdown
Contributor

Problem

@font-face { unicode-range: … } does not survive parsing. Four separate defects combine:

1. A start of fewer than six hex digits can't have a range end. Lexer.UnicodeRange returns immediately out of the wildcard branch, so it never reaches the -<hex> handling below it:

if (StringBuffer.Length != 6)
{
    // ... consume '?' wildcards ...
    return NewRange(FlushBuffer());   // <-- always returns
}

if (current == Symbols.Minus) { /* range end - unreachable unless exactly 6 digits */ }

U+41-5A tokenized as just U+41, leaving -5A behind as stray tokens. Only the wildcard form is mutually exclusive with the start-end form (CSS Syntax 3 §4.3.6); a - should fall through.

2. The wildcard padding loop consumes only half its wildcards. The bound 6 - StringBuffer.Length is re-evaluated each iteration while the body appends to that same buffer, so it shrinks as i grows and they meet in the middle. U+?????? stopped after three ?, leaving ??? unconsumed.

3. ValueBuilder has no case for TokenType.Range, so range tokens fall to the default arm, mark the whole value invalid, and the descriptor is dropped entirely under strict (non-tolerant) parsing — the default.

4. RangeToken serializes without its U+ prefix. Data holds the bare range, so once a descriptor is retained it round-trips as 41-5A, which is not a valid <urange>.

Fix

One change each, plus making RangeToken.SelectedRange lazy — a legal U+0-10FFFF expands to over a million strings, which the constructor materialized eagerly whether or not anything ever read the property. (Nothing in the library reads it today.)

Tests

13 tests. Tokenization.cs covers the lexer directly: start/end forms (U+41-5A, U+0-7F, U+400-4FF, U+000041-00005A), the wildcard and single-value forms (U+4??, U+??????, U+41), and on-demand SelectedRange expansion. FontFace.cs covers the descriptor end-to-end through IFontFaceRule.Range, including a comma-separated list.

10 of the 13 fail on master. The full suite (1263 existing tests) stays green, and all seven target frameworks build with no new warnings.

Three defects kept @font-face unicode-range descriptors from working:

A start of fewer than six hex digits returned immediately after the
wildcard branch, so it never reached the '-<hex>' range-end handling.
"U+41-5A" tokenized as just "U+41" and left "-5A" behind as stray
tokens. Only the wildcard form is mutually exclusive with the range
form, so a '-' now falls through to the shared range handling.

The wildcard padding loop used "6 - StringBuffer.Length" as its bound
while appending to that same buffer, so the bound shrank on every
iteration and only half the available wildcards were consumed:
"U+??????" stopped after three. The budget is now captured up front.

ValueBuilder had no case for TokenType.Range, so range tokens hit the
default arm and marked the whole value invalid, dropping the descriptor
entirely under strict (non-tolerant) parsing.

RangeToken.Data holds the range without its "U+" prefix, so a retained
descriptor serialized as "41-5A" rather than "U+41-5A". ToValue now
restores the prefix.

Also makes RangeToken.SelectedRange lazy: a legal range such as
U+0-10FFFF expands to over a million strings, which the constructor
materialized eagerly whether or not anything read it.
@jhaygood86
jhaygood86 marked this pull request as ready for review July 22, 2026 21:17
@jhaygood86
jhaygood86 marked this pull request as draft July 22, 2026 21:40
@jhaygood86
jhaygood86 marked this pull request as ready for review July 22, 2026 21:43
@TylerBrinks
TylerBrinks merged commit 88fecf9 into TylerBrinks:master Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants