Conversation
liumaker
force-pushed
the
fix/tokenizer-output-validation
branch
from
September 8, 2026 07:24
afd8aee to
4424349
Compare
Soul-Code
requested changes
Sep 8, 2026
Soul-Code
left a comment
Contributor
There was a problem hiding this comment.
提交组织问题:当前 #2 → #3 → #4 → #5 → #6 → #8 → #7 是一条累计提交链,但全部以 main 为 base,导致后一个 PR 包含前面全部修改,标题与实际审核范围不一致。请调整为独立 PR 或正确的 stacked PR;每个问题都应增加能够证明“main 失败、当前独立修改后通过”的最小回归测试。
已复现 main 的 tokenizer 会错误改写:\ddots → \ddot{s}、\cdots → \cdot s、\int → \in t。当前修改能够修复这些问题,核心正则调整应保留。
但本 PR 同时把 tokenize() 的失败契约从返回 (False, source) 改成抛出 CDMPreprocessingError,还新增了显式 \limits 输出;后者会导致当前着色流程渲染失败,必须依赖 #6 才能恢复。
请拆分“命令前缀修复”“错误契约修改”和“limits 规范化”,说明 API 兼容策略并补充回归测试。
This was referenced Sep 8, 2026
Open
liumaker
force-pushed
the
fix/tokenizer-output-validation
branch
from
September 10, 2026 08:20
4424349 to
f60a218
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fastcdm的隐式乘法规则可能在完整 LaTeX 命令内部命中较短前缀,例如破坏
\ddots、\cdots和\int。修改前 tokenizer 的损坏输出可能被当作正常公式继续处理;修改后完整命令保持稳定,执行或输出异常会立即成为明确的预处理失败。