diff --git a/README.md b/README.md index 3bba438..ab0b8ba 100644 --- a/README.md +++ b/README.md @@ -261,15 +261,18 @@ target_locales: [fr, de, es, ja, yue, zh-CN, zh-TW, ar] # One or more source-to-target mappings (required). # {locale} is replaced with each configured target locale. +message_syntax: auto # default; overridden per bundle below bundles: - id: app source: locales/en.json target: locales/{locale}.json format: json + message_syntax: i18next - id: docs source: README.md target: docs/i18n/{locale}.md format: markdown + message_syntax: plain - id: browser source: browser/locales/en-US/browser.ftl target: browser/locales/{locale}/browser.ftl @@ -348,6 +351,35 @@ and locale-specific provider overrides match canonical-equivalent spelling. In the example above, locales without an override—including Japanese—inherit the global Gemini configuration. +File format and message syntax are separate settings. `message_syntax` accepts +`auto` (the default), `i18next`, `icu`, or `plain`, globally and per bundle: + +- `i18next` protects `{{name}}`, nested paths such as `{{user.name}}`, escaping + modifiers such as `{{- name}}`, formatting modifiers, and repeated placeholders. + It enables i18next v4 locale-specific plural keys. Other braces are literal; + `{.sift,.claude,.codex,.agents}` is not parsed as ICU. Custom interpolation + delimiters and nesting expressions are not part of this profile. +- `icu` always parses the message as ICU, including malformed input that cannot + be recognized by automatic detection. Parsing errors never fall back to text. +- `plain` treats braces as text and does not impose an interpolation grammar. +- `auto` infers each message's grammar from its source. Select an explicit mode + for catalogs mixing prose and code. Fluent resources require `auto` and use + their own grammar; Markdown documents cannot select `icu`. + +Validation, provider and TM output checks, adoption, pseudolocalization, and +review approval share the selected syntax. HTML `` contents and Markdown +code spans are preserved exactly. Explicit syntax profiles enforce protected +content even without `--strict`; `--strict` additionally checks translation +quality and glossary rules. Source errors are reported once per bundle with +`source_path`, `blocked_by_source`, and `blocked_locales` in JSON reports. + +The syntax setting is part of the policy hash. This prompt-contract update +makes previously recorded policies stale, including those using `auto`. +Use `translate --dry-run` to inspect them, `translate --refresh-policy` to +regenerate, or `translate --adopt-existing` to validate and record existing +values under the new policy. Adoption leaves entries needing explicit review. +Dry-run reports “Would translate” with planned keys and blocked job counts. + ICU MessageFormat values are parsed structurally. Simple arguments, `select`, `plural`, `selectordinal`, `number`, `date`, and `time` are supported, including nested messages, plural offsets, exact-number selectors, and `#`. Validation diff --git a/docs/execufunction-message-syntax.md b/docs/execufunction-message-syntax.md new file mode 100644 index 0000000..6a0392d --- /dev/null +++ b/docs/execufunction-message-syntax.md @@ -0,0 +1,43 @@ +# Using Internationalizer with ExecuFunction + +The 0.1.2 regression was a grammar mismatch: the marketing catalog contains +literal shell brace syntax inside HTML code elements. Automatic ICU inference +rejected `{.sift,.claude,.codex,.agents}` before any translation could run. + +Configure the web catalog independently from marketing. From ExecuFunction's +repository root, the web mapping is: + +```yaml +source_locale: en +# Populate this from the web application's supported locale manifest. +target_locales: [fr] +bundles: + - id: web + source: exf-app/web/src/i18n/locales/en.json + target: exf-app/web/src/i18n/locales/{locale}.json + format: json + message_syntax: i18next +``` + +This profile uses i18next's default double-brace interpolation, including nested +paths and escaping modifiers, documented in the +[i18next interpolation reference](https://www.i18next.com/translation-function/interpolation). +Internationalizer also preserves formatter modifiers and checks v4 plural-key +families against the target locale. + +The existing marketing config points at `tmp/english-keys.json` and sibling +locale files. Give that bundle its own explicit syntax, chosen for the runtime +that consumes those files: `plain` for literal strings or `i18next` if double-brace +interpolation is used. Coverage for these temporary catalogs measures those +catalogs only; it does not establish coverage of the web app or of the marketing +pipeline's authoritative inputs. Do not add a guessed production path. Verify +the marketing build's actual input and output mapping before adopting it. + +Run `validate --json` and `translate --dry-run` first. Syntax selection removes +the ICU false positive; it can reveal actual translated code or missing +placeholders, which remain errors. Neither dry-run nor validation writes files. + +The acceptance fixture `test/acceptance/testdata/execufunction-syntax` exercises +the exact shell-brace pattern alongside i18next interpolation and an explicit +ICU bundle. Its lifecycle covers both pseudo strategies, adoption without +approval, explicit approval, and invalidation after a syntax-policy change. diff --git a/docs/localization-core-v2.md b/docs/localization-core-v2.md index 848c7dc..4201c6a 100644 --- a/docs/localization-core-v2.md +++ b/docs/localization-core-v2.md @@ -15,6 +15,12 @@ framework runtime, translation CDN, or source-rewriting setup command. ## Message semantics +- Resource format and runtime grammar are distinct. `message_syntax` defaults + to `auto` and supports bundle overrides for `i18next`, `icu`, and `plain`. + Source units carry the resolved grammar; targets never select their grammar. +- Explicit ICU always parses, without a parse-error fallback. i18next protects + double-brace interpolation and v4 plural-key families; plain text treats + braces literally. Fluent resources own their grammar. Markdown is not ICU. - ICU messages are parsed structurally rather than treated as brace-shaped interpolation strings. - Supported arguments are simple interpolation, `select`, `plural`, @@ -54,6 +60,8 @@ inserted. artifacts without a provider or translation-memory lookup. - ICU and Fluent runtime expressions remain intact while linguistic text is transformed. +- HTML code element contents and Markdown code spans are protected from + translation and pseudolocalization. Approval rechecks protected content. - `data-l10n-name` identifies semantic rich-text slots. Translators may reorder named slots, but element identity, protected attributes, nesting, and contained markup must remain compatible. @@ -61,7 +69,7 @@ inserted. ## Policy identity The manifest records style-guide, glossary, and prompt-contract components -separately. The combined policy hash includes those components and the provider +separately. The combined policy hash includes those components, message syntax, and the provider settings, but not the rendered prompt text. Refactoring prompt construction therefore leaves current translations alone; a semantic prompt change requires an explicit prompt-contract version bump. Style guides are read-only inputs and diff --git a/internal/config/config.go b/internal/config/config.go index 4efac7f..d37cad2 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -8,6 +8,7 @@ import ( "strings" localeid "github.com/Tom-R-Main/Internationalizer/internal/locale" + "github.com/Tom-R-Main/Internationalizer/internal/message" "gopkg.in/yaml.v3" ) @@ -22,19 +23,20 @@ const ( ) type Config struct { - SourceLocale string `yaml:"source_locale"` - TargetLocales []string `yaml:"target_locales"` - SourcePath string `yaml:"source_path"` - Bundles []Bundle `yaml:"bundles"` - LLM LLM `yaml:"llm"` - BatchSize int `yaml:"batch_size"` - Concurrency int `yaml:"concurrency"` - StyleGuidesDir string `yaml:"style_guides_dir"` - GlossaryDir string `yaml:"glossary_dir"` - TMPath string `yaml:"tm_path"` - ManifestPath string `yaml:"manifest_path"` - Formats []string `yaml:"formats"` - Validation Validation `yaml:"validation"` + MessageSyntax message.Syntax `yaml:"message_syntax"` + SourceLocale string `yaml:"source_locale"` + TargetLocales []string `yaml:"target_locales"` + SourcePath string `yaml:"source_path"` + Bundles []Bundle `yaml:"bundles"` + LLM LLM `yaml:"llm"` + BatchSize int `yaml:"batch_size"` + Concurrency int `yaml:"concurrency"` + StyleGuidesDir string `yaml:"style_guides_dir"` + GlossaryDir string `yaml:"glossary_dir"` + TMPath string `yaml:"tm_path"` + ManifestPath string `yaml:"manifest_path"` + Formats []string `yaml:"formats"` + Validation Validation `yaml:"validation"` } // Validation configures optional project-specific validation rules. @@ -45,10 +47,11 @@ type Validation struct { // Bundle maps one source file to a locale-specific target path. // Target must contain the literal {locale} placeholder. type Bundle struct { - ID string `yaml:"id"` // required stable identity for explicit bundles - Source string `yaml:"source"` - Target string `yaml:"target"` - Format string `yaml:"format"` + MessageSyntax message.Syntax `yaml:"message_syntax"` + ID string `yaml:"id"` // required stable identity for explicit bundles + Source string `yaml:"source"` + Target string `yaml:"target"` + Format string `yaml:"format"` } type LLM struct { @@ -71,6 +74,9 @@ type LLMOverride struct { } func (c *Config) ApplyDefaults() { + if c.MessageSyntax == "" { + c.MessageSyntax = message.Auto + } if c.SourceLocale == "" { c.SourceLocale = "en" } @@ -237,12 +243,29 @@ func (c *Config) ValidateProject() error { } } bundles := c.EffectiveBundles() + if err := message.ValidateSyntax(c.MessageSyntax); err != nil { + return err + } if len(bundles) == 0 { return fmt.Errorf("source_path or bundles is required") } seen := make(map[string]struct{}, len(bundles)) targets := make(map[string]string, len(bundles)*len(c.TargetLocales)) for _, bundle := range bundles { + if err := message.ValidateSyntax(bundle.MessageSyntax); err != nil { + return fmt.Errorf("bundle %q: %w", bundle.ID, err) + } + format := strings.ToLower(bundle.Format) + extension := strings.ToLower(filepath.Ext(bundle.Source)) + if format == "" && extension == ".ftl" { + format = "fluent" + } + if format == "fluent" && bundle.MessageSyntax != "" && bundle.MessageSyntax != message.Auto { + return fmt.Errorf("bundle %q: Fluent resources require message_syntax: auto", bundle.ID) + } + if (format == "markdown" || (format == "" && (extension == ".md" || extension == ".mdx"))) && bundle.MessageSyntax == message.ICU { + return fmt.Errorf("bundle %q: Markdown documents cannot use message_syntax: icu", bundle.ID) + } if bundle.ID == "" { return fmt.Errorf("explicit bundle id is required") } @@ -320,15 +343,21 @@ func (c *Config) EffectiveBundles() []Bundle { if len(c.Bundles) > 0 { bundles := make([]Bundle, len(c.Bundles)) copy(bundles, c.Bundles) + for i := range bundles { + if bundles[i].MessageSyntax == "" { + bundles[i].MessageSyntax = c.MessageSyntax + } + } return bundles } if c.SourcePath == "" { return nil } return []Bundle{{ - ID: "default", - Source: c.SourcePath, - Target: filepath.Join(filepath.Dir(c.SourcePath), "{locale}"+filepath.Ext(c.SourcePath)), + MessageSyntax: c.MessageSyntax, + ID: "default", + Source: c.SourcePath, + Target: filepath.Join(filepath.Dir(c.SourcePath), "{locale}"+filepath.Ext(c.SourcePath)), }} } @@ -343,6 +372,19 @@ func (b Bundle) TargetPath(locale string) (string, error) { return filepath.Clean(strings.ReplaceAll(b.Target, "{locale}", locale)), nil } +// PluralStyle uses v4 key families for i18next; explicit non-i18next grammars +// must not reinterpret keys just because they end in a plural suffix. +func (b Bundle) PluralStyle(legacyStyle string) string { + switch b.MessageSyntax { + case message.I18next: + return "i18next-v4" + case "", message.Auto: + return legacyStyle + default: + return "" + } +} + func localeOverride(overrides map[string]LLMOverride, requested string) (LLMOverride, bool) { if override, ok := overrides[requested]; ok { return override, true diff --git a/internal/config/syntax_test.go b/internal/config/syntax_test.go new file mode 100644 index 0000000..99d8ce1 --- /dev/null +++ b/internal/config/syntax_test.go @@ -0,0 +1,57 @@ +package config + +import ( + "testing" + + "github.com/Tom-R-Main/Internationalizer/internal/message" +) + +func TestMessageSyntaxInheritanceAndValidation(t *testing.T) { + cfg := Config{SourceLocale: "en", TargetLocales: []string{"fr"}, MessageSyntax: message.I18next, Bundles: []Bundle{ + {ID: "web", Source: "web/en.json", Target: "web/{locale}.json"}, + {ID: "icu", Source: "icu/en.json", Target: "icu/{locale}.json", MessageSyntax: message.ICU}, + {ID: "fluent", Source: "en.ftl", Target: "{locale}.ftl", MessageSyntax: message.Auto}, + }} + cfg.ApplyDefaults() + if err := cfg.ValidateProject(); err != nil { + t.Fatal(err) + } + got := cfg.EffectiveBundles() + if got[0].MessageSyntax != message.I18next || got[1].MessageSyntax != message.ICU || got[2].MessageSyntax != message.Auto { + t.Fatalf("inheritance: %+v", got) + } + if cfg.Bundles[0].MessageSyntax != "" { + t.Fatal("resolution mutated input config") + } + if got[1].PluralStyle("i18next-v4") != "" { + t.Fatal("ICU inherited i18next plural keys") + } + for _, syntax := range []message.Syntax{"typo", "fluent", message.Legacy} { + cfg.MessageSyntax = syntax + if err := cfg.ValidateProject(); err == nil { + t.Fatalf("accepted invalid default %q", syntax) + } + } + cfg.MessageSyntax = message.Auto + cfg.Bundles[0].MessageSyntax = "typo" + if err := cfg.ValidateProject(); err == nil { + t.Fatal("accepted invalid override") + } + cfg.Bundles[0].MessageSyntax = message.Auto + cfg.Bundles[2].MessageSyntax = message.I18next + if err := cfg.ValidateProject(); err == nil { + t.Fatal("accepted non-Fluent resource grammar") + } + cfg.Bundles[2] = Bundle{ID: "doc", Source: "README.md", Target: "{locale}.md", MessageSyntax: message.ICU} + if err := cfg.ValidateProject(); err == nil { + t.Fatal("accepted ICU document") + } + cfg.Bundles[2].Source = "README.MDX" + if err := cfg.ValidateProject(); err == nil { + t.Fatal("accepted ICU MDX document") + } + cfg.Bundles[2] = Bundle{ID: "fluent", Source: "EN.FTL", Target: "{locale}.ftl", MessageSyntax: message.Plain} + if err := cfg.ValidateProject(); err == nil { + t.Fatal("accepted plain Fluent resource with uppercase extension") + } +} diff --git a/internal/formats/units.go b/internal/formats/units.go index 73a8924..1c2dfc0 100644 --- a/internal/formats/units.go +++ b/internal/formats/units.go @@ -3,6 +3,8 @@ package formats import ( "fmt" "sort" + + "github.com/Tom-R-Main/Internationalizer/internal/message" ) // UnitKind describes the semantic role of one independently translatable unit. @@ -21,11 +23,24 @@ const ( // Context is translator-facing information; Structure is a deterministic // adapter-owned signature used to detect semantic message changes. type Unit struct { - ID string `json:"id"` - Value string `json:"value"` - Kind UnitKind `json:"kind"` - Context string `json:"context,omitempty"` - Structure string `json:"structure,omitempty"` + Syntax message.Syntax `json:"syntax,omitempty"` + ID string `json:"id"` + Value string `json:"value"` + Kind UnitKind `json:"kind"` + Context string `json:"context,omitempty"` + Structure string `json:"structure,omitempty"` +} + +// ParseSourceUnits attaches the resolved source grammar for downstream checks. +func ParseSourceUnits(format Format, data []byte, syntax message.Syntax) ([]Unit, error) { + units, err := ParseUnits(format, data) + if err != nil { + return nil, err + } + for i := range units { + units[i].Syntax = message.ResolveSyntax(format.Name(), syntax, units[i].Value) + } + return units, nil } // UnitFormat is implemented by formats with richer semantics than a flat diff --git a/internal/llm/provider.go b/internal/llm/provider.go index 3586a2b..10ef941 100644 --- a/internal/llm/provider.go +++ b/internal/llm/provider.go @@ -8,6 +8,7 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/config" "github.com/Tom-R-Main/Internationalizer/internal/glossary" + "github.com/Tom-R-Main/Internationalizer/internal/message" ) // Entry is a source key-value pair to be translated. @@ -83,9 +84,23 @@ func EffectiveReasoningEffort(cfg config.LLM) string { return cfg.ReasoningEffort } +func interpolationRule(syntaxes []message.Syntax) string { + if len(syntaxes) > 0 { + switch syntaxes[0] { + case message.I18next: + return "- Message syntax: i18next. Preserve every {{variable}} interpolation, including paths, escaping and formatting modifiers. Other braces are literal text, not ICU.\n" + case message.Plain: + return "- Message syntax: plain. Braces are literal text, not runtime interpolation or ICU.\n" + case message.ICU: + return "- Message syntax: ICU MessageFormat. Preserve valid arguments, types, selectors and plural branches.\n" + } + } + return "- Preserve interpolation variables exactly: {{variable}}, {variable}, %{variable}.\n" +} + // BuildSystemPrompt constructs the system prompt for translation, // combining rules, glossary terms, and style guide content. -func BuildSystemPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term) string { +func BuildSystemPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term, syntax ...message.Syntax) string { var b strings.Builder b.WriteString("You are a professional software localizer. Translate the following ") @@ -97,7 +112,7 @@ func BuildSystemPrompt(sourceLocale, targetLocale, styleGuide string, terms []gl b.WriteString("## Rules\n") b.WriteString("- Return a JSON object mapping each key to its translated value.\n") - b.WriteString("- Preserve interpolation variables exactly: {{variable}}, {variable}, %{variable}.\n") + b.WriteString(interpolationRule(syntax)) b.WriteString("- Preserve all HTML tags exactly (,
, etc.).\n") b.WriteString("- Do not translate the JSON keys, only the values.\n") b.WriteString("- Do not add any explanation or commentary, only output valid JSON.\n") @@ -158,7 +173,7 @@ func marshalEntries(entries []Entry) ([]byte, error) { } // BuildDocumentPrompt constructs a prompt for whole-document translation (e.g. Markdown). -func BuildDocumentPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term) string { +func BuildDocumentPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term, syntax ...message.Syntax) string { var b strings.Builder b.WriteString("You are a professional translator. Translate the following document from ") @@ -169,7 +184,7 @@ func BuildDocumentPrompt(sourceLocale, targetLocale, styleGuide string, terms [] b.WriteString("## Rules\n") b.WriteString("- Preserve all Markdown formatting (headings, links, code blocks, lists).\n") - b.WriteString("- Preserve interpolation variables exactly: {{variable}}, {variable}, %{variable}.\n") + b.WriteString(interpolationRule(syntax)) b.WriteString("- Do not translate code blocks or inline code.\n") b.WriteString("- Return a JSON object with every original input key mapped to its translated document unit.\n") b.WriteString("- Do not add commentary, omit keys, or add keys that were not present in the input.\n") diff --git a/internal/message/message.go b/internal/message/message.go index 2a90d9e..06cc6a2 100644 --- a/internal/message/message.go +++ b/internal/message/message.go @@ -142,6 +142,12 @@ func Compare(source, target, targetLocale string) []Issue { if !LooksLike(source) && !LooksLike(target) { return nil } + return CompareICU(source, target, targetLocale) +} + +// CompareICU always parses both values, including malformed messages that do +// not resemble an ICU argument strongly enough for automatic detection. +func CompareICU(source, target, targetLocale string) []Issue { sourceMessage, err := Parse(source) if err != nil { return []Issue{{Code: CodeSyntax, Message: fmt.Sprintf("source ICU message: %v", err)}} diff --git a/internal/message/syntax.go b/internal/message/syntax.go new file mode 100644 index 0000000..eed2037 --- /dev/null +++ b/internal/message/syntax.go @@ -0,0 +1,74 @@ +package message + +import ( + "fmt" + "regexp" + "sort" + "strings" + + "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern" +) + +// Syntax describes the runtime grammar, independently of the resource format. +type Syntax string + +const ( + Auto Syntax = "auto" + ICU Syntax = "icu" + I18next Syntax = "i18next" + Plain Syntax = "plain" + Fluent Syntax = "fluent" + // Legacy preserves the non-ICU interpolation checks used by auto mode. + Legacy Syntax = "legacy" +) + +func ValidateSyntax(syntax Syntax) error { + switch syntax { + case "", Auto, ICU, I18next, Plain: + return nil + default: + return fmt.Errorf("unsupported message_syntax %q (expected auto, icu, i18next, or plain)", syntax) + } +} + +// ResolveSyntax uses the source only: target text must never select its grammar. +// Fluent resources own their grammar; Markdown documents are not ICU messages. +func ResolveSyntax(format string, policy Syntax, source string) Syntax { + if format == "fluent" { + return Fluent + } + if policy != "" && policy != Auto { + return policy + } + if format == "markdown" { + return Legacy + } + for _, location := range typedICUArgument.FindAllStringIndex(source, -1) { + // A formatter inside {{value, number}} belongs to i18next, not ICU. + if location[0] == 0 || source[location[0]-1] != '{' { + return ICU + } + } + if fluentpattern.LooksLike(source) { + return Fluent + } + if LooksLike(source) { + return ICU + } + return Legacy +} + +var typedICUArgument = regexp.MustCompile(`\{\s*[\pL\pN_.-]+\s*,`) + +// I18nextTokens includes escaping and formatting modifiers as part of the +// contract, as well as nested variable paths. Repeated tokens remain repeated. +var I18nextToken = regexp.MustCompile(`\{\{[^{}\r\n]+\}\}`) + +func I18nextTokens(value string) []string { + tokens := I18nextToken.FindAllString(value, -1) + for i, token := range tokens { + tokens[i] = strings.TrimSpace(token[2 : len(token)-2]) + } + sort.Strings(tokens) + return tokens +} diff --git a/internal/message/syntax_test.go b/internal/message/syntax_test.go new file mode 100644 index 0000000..d03f517 --- /dev/null +++ b/internal/message/syntax_test.go @@ -0,0 +1,23 @@ +package message + +import "testing" + +func TestResolveSyntax(t *testing.T) { + for _, test := range []struct { + source, format string + policy, want Syntax + }{ + {"{{amount, number}}", "json", Auto, Legacy}, + {"{n, plural, one {[Guide](https://example.com)} other {[Guide](https://example.com)}}", "json", Auto, ICU}, + {"{ $count ->\n [one] One { -brand }\n *[other] { $count }\n}", "fluent", Auto, Fluent}, + {"{ $count ->\n [one] One { -brand }\n *[other] { $count }\n}", "json", Auto, Fluent}, + {"{.sift,.claude}", "json", I18next, I18next}, + {"{", "json", ICU, ICU}, + {"# Hi {name}", "markdown", Auto, Legacy}, + {"{name}", "json", Plain, Plain}, + } { + if got := ResolveSyntax(test.format, test.policy, test.source); got != test.want { + t.Errorf("ResolveSyntax(%q,%s,%q)=%s want %s", test.format, test.policy, test.source, got, test.want) + } + } +} diff --git a/internal/policy/policy.go b/internal/policy/policy.go index be7a0ed..d0ca269 100644 --- a/internal/policy/policy.go +++ b/internal/policy/policy.go @@ -6,10 +6,11 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/glossary" "github.com/Tom-R-Main/Internationalizer/internal/llm" "github.com/Tom-R-Main/Internationalizer/internal/locale" + "github.com/Tom-R-Main/Internationalizer/internal/message" "github.com/Tom-R-Main/Internationalizer/internal/state" ) -const PromptContractVersion = 2 +const PromptContractVersion = 3 // Resolved is the resolved prompt, provider settings, and stable hash for // one target locale and source format. @@ -25,7 +26,17 @@ type Resolved struct { // Resolve builds the effective translation policy. Callers provide // already-loaded style-guide and glossary content so policy resolution remains // independent of filesystem layout. -func Resolve(cfg *config.Config, targetLocale, format, styleGuide string, terms []glossary.Term) (Resolved, error) { +func Resolve(cfg *config.Config, targetLocale, format, styleGuide string, terms []glossary.Term, bundleSyntax ...message.Syntax) (Resolved, error) { + syntax := cfg.MessageSyntax + if len(bundleSyntax) > 0 { + syntax = bundleSyntax[0] + } + if syntax == "" { + syntax = message.Auto + } + if err := message.ValidateSyntax(syntax); err != nil { + return Resolved{}, err + } sourceLocale := cfg.SourceLocale if sourceLocale == "" { sourceLocale = "en" @@ -40,13 +51,14 @@ func Resolve(cfg *config.Config, targetLocale, format, styleGuide string, terms } effectiveLLM := cfg.LLMForLocale(canonicalTarget) - prompt := llm.BuildSystemPrompt(canonicalSource, canonicalTarget, styleGuide, terms) + prompt := llm.BuildSystemPrompt(canonicalSource, canonicalTarget, styleGuide, terms, syntax) switch format { case "markdown": - prompt = llm.BuildDocumentPrompt(canonicalSource, canonicalTarget, styleGuide, terms) + prompt = llm.BuildDocumentPrompt(canonicalSource, canonicalTarget, styleGuide, terms, syntax) case "fluent": prompt = llm.BuildFluentPrompt(canonicalSource, canonicalTarget, styleGuide, terms) } + prompt += "\nPreserve the complete content of HTML code elements and Markdown code spans exactly." guideHash, err := state.HashValue(styleGuide) if err != nil { @@ -57,16 +69,17 @@ func Resolve(cfg *config.Config, targetLocale, format, styleGuide string, terms return Resolved{}, err } hash, err := state.HashValue(struct { - Version int `json:"version"` - SourceLocale string `json:"source_locale"` - TargetLocale string `json:"target_locale"` - Format string `json:"format"` - Provider string `json:"provider"` - Model string `json:"model"` - Reasoning string `json:"reasoning_effort"` - GuideHash string `json:"guide_hash"` - GlossaryHash string `json:"glossary_hash"` - }{PromptContractVersion, canonicalSource, canonicalTarget, format, effectiveLLM.Provider, effectiveLLM.Model, llm.EffectiveReasoningEffort(effectiveLLM), guideHash, glossaryHash}) + Syntax message.Syntax `json:"message_syntax"` + Version int `json:"version"` + SourceLocale string `json:"source_locale"` + TargetLocale string `json:"target_locale"` + Format string `json:"format"` + Provider string `json:"provider"` + Model string `json:"model"` + Reasoning string `json:"reasoning_effort"` + GuideHash string `json:"guide_hash"` + GlossaryHash string `json:"glossary_hash"` + }{syntax, PromptContractVersion, canonicalSource, canonicalTarget, format, effectiveLLM.Provider, effectiveLLM.Model, llm.EffectiveReasoningEffort(effectiveLLM), guideHash, glossaryHash}) if err != nil { return Resolved{}, err } diff --git a/internal/protectedtext/code.go b/internal/protectedtext/code.go new file mode 100644 index 0000000..f7c7c4e --- /dev/null +++ b/internal/protectedtext/code.go @@ -0,0 +1,56 @@ +// Package protectedtext locates literal code shared by validation and pseudo. +package protectedtext + +import ( + "regexp" + "strings" +) + +type Span struct{ Start, End int } + +// Quoted attributes may contain >; comments must not introduce code elements. +var htmlToken = regexp.MustCompile(`(?s)|"']|"[^"]*"|'[^']*')*>`) + +// HTMLCodeSpans includes the full code element, with nested markup intact. +// An unclosed code element protects through EOF rather than rewriting code. +func HTMLCodeSpans(input string) []Span { + var spans []Span + start, depth := 0, 0 + for _, loc := range htmlToken.FindAllStringIndex(input, -1) { + token := input[loc[0]:loc[1]] + if strings.HasPrefix(token, " Text keep`, []string{`keep`}}, + {`outerinnerend`, []string{`outerinnerend`}}, + {`before unclosed`, []string{`unclosed`}}, + {`text`, nil}, + } { + if got := HTMLCode(test.source); !reflect.DeepEqual(got, test.want) { + t.Errorf("HTMLCode(%q)=%q want %q", test.source, got, test.want) + } + } +} diff --git a/internal/pseudo/generate.go b/internal/pseudo/generate.go index 49fa1e5..4445cac 100644 --- a/internal/pseudo/generate.go +++ b/internal/pseudo/generate.go @@ -10,6 +10,7 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/config" "github.com/Tom-R-Main/Internationalizer/internal/formats" localeid "github.com/Tom-R-Main/Internationalizer/internal/locale" + "github.com/Tom-R-Main/Internationalizer/internal/message" "github.com/Tom-R-Main/Internationalizer/internal/state" ) @@ -58,17 +59,18 @@ func Generate(cfg *config.Config, opts GenerateOptions) ([]GenerateResult, error if err != nil { return nil, err } - policyHash, err := state.HashValue(struct { - Version int `json:"version"` - Strategy Strategy `json:"strategy"` - }{Version: 1, Strategy: opts.Strategy}) - if err != nil { - return nil, err - } results := make([]GenerateResult, 0, len(cfg.EffectiveBundles())) now := time.Now().UTC() for _, bundle := range cfg.EffectiveBundles() { + policyHash, err := state.HashValue(struct { + Version int `json:"version"` + Strategy Strategy `json:"strategy"` + Syntax message.Syntax `json:"message_syntax"` + }{Version: 2, Strategy: opts.Strategy, Syntax: bundle.MessageSyntax}) + if err != nil { + return nil, err + } format, err := formatForBundle(bundle) if err != nil { return nil, fmt.Errorf("bundle %q format: %w", bundle.ID, err) @@ -77,7 +79,7 @@ func Generate(cfg *config.Config, opts GenerateOptions) ([]GenerateResult, error if err != nil { return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err) } - sourceUnits, err := formats.ParseUnits(format, sourceData) + sourceUnits, err := formats.ParseSourceUnits(format, sourceData, bundle.MessageSyntax) if err != nil { return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err) } @@ -158,8 +160,11 @@ func formatForBundle(bundle config.Bundle) (formats.Format, error) { } func transformUnit(format formats.Format, unit formats.Unit, strategy Strategy) (string, error) { + if unit.Syntax == "" { + unit.Syntax = message.ResolveSyntax(format.Name(), message.Auto, unit.Value) + } if format.Name() != "markdown" { - return Transform(unit.Value, strategy) + return TransformSyntax(unit.Value, strategy, unit.Syntax) } lineEnd := strings.IndexByte(unit.Value, '\n') if lineEnd < 0 { @@ -175,12 +180,12 @@ func transformUnit(format formats.Format, unit formats.Unit, strategy Strategy) prefixEnd++ } if prefixEnd == headingStart || prefixEnd >= len(line) || (line[prefixEnd] != ' ' && line[prefixEnd] != '\t') { - return Transform(unit.Value, strategy) + return TransformSyntax(unit.Value, strategy, unit.Syntax) } for prefixEnd < len(line) && (line[prefixEnd] == ' ' || line[prefixEnd] == '\t') { prefixEnd++ } - heading, err := Transform(line[prefixEnd:], strategy) + heading, err := TransformSyntax(line[prefixEnd:], strategy, unit.Syntax) if err != nil { return "", err } @@ -193,14 +198,14 @@ func transformUnit(format formats.Format, unit formats.Unit, strategy Strategy) } body = unit.Value[lineEnd+1:] } - transformedBody, err := transformPadded(body, strategy) + transformedBody, err := transformPadded(body, strategy, unit.Syntax) if err != nil { return "", err } return line[:prefixEnd] + heading + newline + transformedBody, nil } -func transformPadded(value string, strategy Strategy) (string, error) { +func transformPadded(value string, strategy Strategy, syntax message.Syntax) (string, error) { start := 0 for start < len(value) && strings.ContainsRune(" \t\r\n", rune(value[start])) { start++ @@ -212,7 +217,7 @@ func transformPadded(value string, strategy Strategy) (string, error) { if start == end { return value, nil } - transformed, err := Transform(value[start:end], strategy) + transformed, err := TransformSyntax(value[start:end], strategy, syntax) if err != nil { return "", err } diff --git a/internal/pseudo/transform.go b/internal/pseudo/transform.go index c4279d9..89a6690 100644 --- a/internal/pseudo/transform.go +++ b/internal/pseudo/transform.go @@ -11,6 +11,7 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern" "github.com/Tom-R-Main/Internationalizer/internal/message" + "github.com/Tom-R-Main/Internationalizer/internal/protectedtext" ) // Strategy selects a pseudolocalization transformation. @@ -30,6 +31,11 @@ type span struct { // Transform transforms linguistic text while preserving runtime syntax. func Transform(input string, strategy Strategy) (string, error) { + return TransformSyntax(input, strategy, message.ResolveSyntax("", message.Auto, input)) +} + +// TransformSyntax applies the resolved source grammar without guessing again. +func TransformSyntax(input string, strategy Strategy, syntax message.Syntax) (string, error) { if strategy != Accented && strategy != Bidi { return "", fmt.Errorf("unsupported pseudolocalization strategy %q", strategy) } @@ -38,17 +44,18 @@ func Transform(input string, strategy Strategy) (string, error) { } output := "" var err error - if fluentpattern.LooksLike(input) { + switch syntax { + case message.Fluent: output, err = fluentpattern.TransformText(input, transformLiteral) if err != nil { return "", fmt.Errorf("pseudolocalizing Fluent pattern: %w", err) } - } else if message.LooksLike(input) { + case message.ICU: output, err = message.TransformText(input, transformLiteral) if err != nil { return "", fmt.Errorf("pseudolocalizing ICU message: %w", err) } - } else { + default: output = transformLiteral(input) } if strategy == Accented { @@ -89,6 +96,12 @@ func protectedSpans(input string) []span { for _, location := range protectedTokenRe.FindAllStringIndex(input, -1) { spans = append(spans, span{start: location[0], end: location[1]}) } + for _, location := range message.I18nextToken.FindAllStringIndex(input, -1) { + spans = append(spans, span{start: location[0], end: location[1]}) + } + for _, location := range protectedtext.HTMLCodeSpans(input) { + spans = append(spans, span{start: location.Start, end: location.End}) + } spans = append(spans, markdownDestinationSpans(input)...) sort.Slice(spans, func(i, j int) bool { if spans[i].start != spans[j].start { diff --git a/internal/review/review.go b/internal/review/review.go index 278a03b..f8adbbe 100644 --- a/internal/review/review.go +++ b/internal/review/review.go @@ -155,6 +155,9 @@ func entryValidationError(entry state.Entry, reports []validate.Report) error { if len(report.Errors) > 0 { return fmt.Errorf("cannot approve %s/%s/%s: %s", entry.Bundle, entry.Key, entry.Locale, report.Errors[0]) } + if report.BlockedBySource { + return fmt.Errorf("cannot approve %s/%s/%s: source bundle %s is invalid", entry.Bundle, entry.Key, entry.Locale, report.SourcePath) + } for _, missing := range report.Missing { if missing == entry.Key { return fmt.Errorf("cannot approve %s/%s/%s: target key is missing", entry.Bundle, entry.Key, entry.Locale) diff --git a/internal/translate/protected.go b/internal/translate/protected.go index fd9cb4a..0b05fb2 100644 --- a/internal/translate/protected.go +++ b/internal/translate/protected.go @@ -4,6 +4,7 @@ import ( "fmt" "strings" + "github.com/Tom-R-Main/Internationalizer/internal/message" validation "github.com/Tom-R-Main/Internationalizer/internal/validate" ) @@ -15,6 +16,7 @@ func validateTranslationValue(key, source, target, targetLocale string) error { } type valueValidationContext struct { + syntaxes map[string]message.Syntax document bool sourcePath string targetPath string @@ -24,9 +26,17 @@ func validateTranslationValueWithContext(key, source, target, targetLocale strin if strings.TrimSpace(source) != "" && strings.TrimSpace(target) == "" { return fmt.Errorf("blank translation for %q", key) } - protected := validation.ProtectedFindings(key, source, target, targetLocale) + syntax := validationContext.syntaxes[key] + if syntax == "" { + format := "" + if validationContext.document { + format = "markdown" + } + syntax = message.ResolveSyntax(format, message.Auto, source) + } + protected := validation.ProtectedSyntaxFindings(key, source, target, targetLocale, syntax) if validationContext.document { - protected = validation.ProtectedDocumentFindings(key, source, target, targetLocale, validationContext.sourcePath, validationContext.targetPath) + protected = validation.ProtectedDocumentFindings(key, source, target, targetLocale, validationContext.sourcePath, validationContext.targetPath, syntax) } if len(protected) > 0 { return fmt.Errorf("%s for %q", protected[0].Message, key) @@ -34,7 +44,7 @@ func validateTranslationValueWithContext(key, source, target, targetLocale strin if validationContext.document { return nil } - if findings := validation.ICUFindings(key, source, target, targetLocale); len(findings) > 0 { + if findings := validation.SyntaxFindings(key, source, target, targetLocale, syntax); len(findings) > 0 { return fmt.Errorf("%s for %q", findings[0].Message, key) } return nil diff --git a/internal/translate/syntax_test.go b/internal/translate/syntax_test.go new file mode 100644 index 0000000..06fe86e --- /dev/null +++ b/internal/translate/syntax_test.go @@ -0,0 +1,59 @@ +package translate + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/Tom-R-Main/Internationalizer/internal/llm" + "github.com/Tom-R-Main/Internationalizer/internal/message" +) + +func TestI18nextProviderBoundary(t *testing.T) { + const code = `<root>/{.sift,.claude,.codex,.agents}/skills` + const target = "Lire " + code + " pour {{user.name}}" + for _, candidate := range []string{target, strings.Replace(target, ".claude", ".changed", 1), strings.Replace(target, "{{user.name}}", "{{name}}", 1)} { + t.Run(candidate, func(t *testing.T) { + dir := t.TempDir() + sourcePath := filepath.Join(dir, "en.json") + data, err := json.Marshal(map[string]string{"docs": "Read " + code + " for {{user.name}}"}) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(sourcePath, data, 0o644); err != nil { + t.Fatal(err) + } + cfg := testConfig(dir, sourcePath) + cfg.MessageSyntax = message.I18next + provider := &fakeProvider{response: &llm.TranslateResponse{Translations: map[string]string{"docs": candidate}}} + results, err := Run(context.Background(), cfg, provider, Options{}) + if (err == nil) != (candidate == target) { + t.Fatalf("error=%v results=%+v", err, results) + } + if candidate != target { + if _, err := os.Stat(filepath.Join(dir, "fr.json")); !os.IsNotExist(err) { + t.Fatalf("invalid target written: %v", err) + } + } else { + if !strings.Contains(provider.requests[0].SystemPrompt, "Message syntax: i18next") { + t.Fatal("missing syntax prompt") + } + cfg.MessageSyntax = message.Plain + results, err := Run(context.Background(), cfg, provider, Options{DryRun: true}) + if err != nil || results[0].KeysPolicyStale != 1 { + t.Fatalf("syntax change not stale: %v %+v", err, results) + } + if err := os.Remove(filepath.Join(dir, "fr.json")); err != nil { + t.Fatal(err) + } + results, err = Run(context.Background(), cfg, provider, Options{}) + if err != nil || provider.calls != 2 || results[0].KeysCached != 0 { + t.Fatalf("reused incompatible TM: calls=%d results=%+v error=%v", provider.calls, results, err) + } + } + }) + } +} diff --git a/internal/translate/translate.go b/internal/translate/translate.go index 54aab98..2fa9f5e 100644 --- a/internal/translate/translate.go +++ b/internal/translate/translate.go @@ -13,6 +13,7 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/formats" "github.com/Tom-R-Main/Internationalizer/internal/glossary" "github.com/Tom-R-Main/Internationalizer/internal/llm" + "github.com/Tom-R-Main/Internationalizer/internal/message" "github.com/Tom-R-Main/Internationalizer/internal/policy" "github.com/Tom-R-Main/Internationalizer/internal/state" "github.com/Tom-R-Main/Internationalizer/internal/styleguide" @@ -34,6 +35,10 @@ type Options struct { // Result holds the outcome of one bundle and locale. Lifecycle counters are // pre-run observations; manual, source-stale, and policy-stale may overlap. type Result struct { + BlockedBySource bool + SourcePath string + BlockedLocales []string + DryRun bool Bundle string Locale string TargetPath string @@ -64,11 +69,12 @@ func (e *RunError) Error() string { } type preparedBundle struct { - bundle config.Bundle - format formats.Format - sourceUnits []formats.Unit - sourceKeys map[string]string - sourceData []byte + sourceErrors []string + bundle config.Bundle + format formats.Format + sourceUnits []formats.Unit + sourceKeys map[string]string + sourceData []byte } type job struct { @@ -131,6 +137,13 @@ func Run(ctx context.Context, cfg *config.Config, provider llm.Provider, opts Op if err != nil { return nil, err } + for i := range bundles { + for _, unit := range bundles[i].sourceUnits { + for _, finding := range validation.SyntaxSourceFindings(unit.ID, unit.Value, cfg.SourceLocale, unit.Syntax) { + bundles[i].sourceErrors = append(bundles[i].sourceErrors, fmt.Sprintf("source %q: %s", unit.ID, finding.Message)) + } + } + } memory, err := tm.Load(cfg.TMPath) if err != nil { return nil, fmt.Errorf("loading TM: %w", err) @@ -181,7 +194,16 @@ enqueue: workers.Wait() results := make([]Result, 0, next) + reportedSource := make(map[string]bool) for i := 0; i < next; i++ { + if outputs[i].result.BlockedBySource { + if reportedSource[outputs[i].result.Bundle] { + outputs[i].result.Errors = nil + } else { + outputs[i].result.BlockedLocales = locales + reportedSource[outputs[i].result.Bundle] = true + } + } results = append(results, outputs[i].result) if !opts.DryRun { for _, update := range outputs[i].updates { @@ -201,7 +223,7 @@ enqueue: failed := 0 for _, result := range results { - if len(result.Errors) > 0 { + if result.BlockedBySource || len(result.Errors) > 0 { failed++ } } @@ -228,7 +250,7 @@ func prepareBundles(bundles []config.Bundle) ([]preparedBundle, error) { if err != nil { return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err) } - units, err := formats.ParseUnits(format, data) + units, err := formats.ParseSourceUnits(format, data, bundle.MessageSyntax) if err != nil { return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err) } @@ -250,23 +272,28 @@ func translateLocale( ) jobOutput { sourceKeys := bundle.sourceKeys var optionalPluralKeys map[string]struct{} - if cfg.Validation.PluralStyle == "i18next-v4" { + if bundle.bundle.PluralStyle(cfg.Validation.PluralStyle) == "i18next-v4" { sourceKeys, _, optionalPluralKeys = validation.ExpandI18nextV4Source(bundle.sourceKeys, cfg.SourceLocale, locale) for key := range optionalPluralKeys { delete(sourceKeys, key) } } - result := Result{Bundle: bundle.bundle.ID, Locale: locale, KeysTotal: len(sourceKeys)} - if bundle.format.Name() != "markdown" { - for _, key := range sortedKeys(sourceKeys) { - if findings := validation.ICUSourceFindings(key, sourceKeys[key], cfg.SourceLocale); len(findings) > 0 { - result.Errors = append(result.Errors, fmt.Sprintf("source %q: %s", key, findings[0].Message)) - } - } - } - if len(result.Errors) > 0 { + result := Result{Bundle: bundle.bundle.ID, Locale: locale, KeysTotal: len(sourceKeys), DryRun: opts.DryRun} + if len(bundle.sourceErrors) > 0 { + result.BlockedBySource = true + result.SourcePath = bundle.bundle.Source + result.Errors = bundle.sourceErrors return jobOutput{result: result} } + syntaxes := make(map[string]message.Syntax, len(sourceKeys)) + for _, unit := range bundle.sourceUnits { + syntaxes[unit.ID] = unit.Syntax + } + for key, value := range sourceKeys { + if syntaxes[key] == "" { + syntaxes[key] = message.ResolveSyntax(bundle.format.Name(), bundle.bundle.MessageSyntax, value) + } + } targetPath, err := bundle.bundle.TargetPath(locale) if err != nil { result.Errors = append(result.Errors, err.Error()) @@ -274,6 +301,7 @@ func translateLocale( } result.TargetPath = targetPath validationContext := valueValidationContext{ + syntaxes: syntaxes, document: bundle.format.Name() == "markdown", sourcePath: bundle.bundle.Source, targetPath: targetPath, @@ -289,7 +317,7 @@ func translateLocale( result.Errors = append(result.Errors, fmt.Sprintf("style guide: %v", err)) return jobOutput{result: result} } - translationPolicy, err := policy.Resolve(cfg, locale, bundle.format.Name(), guide, terms) + translationPolicy, err := policy.Resolve(cfg, locale, bundle.format.Name(), guide, terms, bundle.bundle.MessageSyntax) if err != nil { result.Errors = append(result.Errors, fmt.Sprintf("hashing translation policy: %v", err)) return jobOutput{result: result} @@ -694,6 +722,16 @@ func FormatResults(results []Result, elapsed time.Duration) string { } summary := fmt.Sprintf("\nTranslated %d keys across %d bundle/locale jobs (%d from cache) in %s\n", translated, len(results), cached, elapsed.Round(time.Millisecond)) + if len(results) > 0 && results[0].DryRun { + planned, blocked := 0, 0 + for _, result := range results { + planned += result.KeysSkipped + if result.BlockedBySource || len(result.Errors) > 0 { + blocked++ + } + } + summary = fmt.Sprintf("\nWould translate %d keys across %d bundle/locale jobs (%d blocked jobs); dry-run, no changes\n", planned, len(results), blocked) + } summary += fmt.Sprintf("Observed before run: %d missing, %d source-stale, %d policy-stale, %d manual edits, %d untracked\n", missing, sourceStale, policyStale, manual, untracked) if tokensIn > 0 || tokensOut > 0 { summary += fmt.Sprintf("Tokens: %d input, %d output\n", tokensIn, tokensOut) @@ -711,6 +749,9 @@ func FormatResults(results []Result, elapsed time.Duration) string { if hasErrors { summary += "\nErrors:\n" for _, result := range results { + if len(result.BlockedLocales) > 0 { + summary += fmt.Sprintf(" Source %s blocks %d locales: %s\n", result.SourcePath, len(result.BlockedLocales), strings.Join(result.BlockedLocales, ", ")) + } for _, runErr := range result.Errors { summary += fmt.Sprintf(" [%s/%s] %s\n", result.Bundle, result.Locale, runErr) } diff --git a/internal/validate/protected.go b/internal/validate/protected.go index 4415797..926ad95 100644 --- a/internal/validate/protected.go +++ b/internal/validate/protected.go @@ -8,6 +8,7 @@ import ( "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern" "github.com/Tom-R-Main/Internationalizer/internal/message" + "github.com/Tom-R-Main/Internationalizer/internal/protectedtext" ) var ( @@ -21,28 +22,36 @@ var ( // must preserve exactly. Multiple damaged structures may yield multiple // findings with the same stable code and distinct messages. func ProtectedFindings(key, source, target, targetLocale string) []Finding { - return protectedFindings(key, source, target, targetLocale, "", "") + return ProtectedSyntaxFindings(key, source, target, targetLocale, message.ResolveSyntax("", message.Auto, source)) +} + +func ProtectedSyntaxFindings(key, source, target, targetLocale string, syntax message.Syntax) []Finding { + return protectedFindings(key, source, target, targetLocale, "", "", syntax) } // ProtectedDocumentFindings compares Markdown document structure while // resolving relative links from each document's own directory. A localized // document may add one link back to its source. -func ProtectedDocumentFindings(key, source, target, targetLocale, sourcePath, targetPath string) []Finding { - return protectedFindings(key, source, target, targetLocale, sourcePath, targetPath) +func ProtectedDocumentFindings(key, source, target, targetLocale, sourcePath, targetPath string, syntaxes ...message.Syntax) []Finding { + syntax := message.Legacy + if len(syntaxes) > 0 { + syntax = syntaxes[0] + } + return protectedFindings(key, source, target, targetLocale, sourcePath, targetPath, syntax) } -func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath string) []Finding { +func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath string, syntax message.Syntax) []Finding { var findings []Finding document := sourcePath != "" && targetPath != "" icu := false if !document { - icu = usesICUValidation(source, target) + icu = syntax == message.ICU && usesICUValidation(source, target) } if !icu { - if mismatch := InterpolationMismatch(key, source, target); mismatch != nil { + if mismatch := SyntaxInterpolationMismatch(key, source, target, syntax); mismatch != nil { findings = append(findings, protectedFinding(key, "interpolation variables", mismatch.SourceVars, mismatch.TargetVars)) } - if fluentpattern.LooksLike(source) || fluentpattern.LooksLike(target) { + if syntax == message.Fluent { expected, actual, preserved, err := fluentpattern.Compare(source, target) if err != nil { actual = []string{err.Error()} @@ -58,6 +67,7 @@ func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath extractTarget func(string) []string }{ {"HTML structure", extractHTMLTags, extractHTMLTags}, + {"HTML code", protectedtext.HTMLCode, protectedtext.HTMLCode}, {"fenced code", extractFencedCode, extractFencedCode}, {"inline code", extractInlineCode, extractInlineCode}, {"markdown link destinations", extractLinkDestinations, extractLinkDestinations}, @@ -65,8 +75,8 @@ func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath if document { checks[0].extractSource = func(input string) []string { return extractDocumentHTMLTags(input, sourcePath) } checks[0].extractTarget = func(input string) []string { return extractDocumentHTMLTags(input, targetPath) } - checks[3].extractSource = func(input string) []string { return extractDocumentLinkDestinations(input, sourcePath) } - checks[3].extractTarget = func(input string) []string { + checks[4].extractSource = func(input string) []string { return extractDocumentLinkDestinations(input, sourcePath) } + checks[4].extractTarget = func(input string) []string { tokens := extractDocumentLinkDestinations(input, targetPath) return removeSourceBacklink(tokens, sourcePath) } diff --git a/internal/validate/syntax.go b/internal/validate/syntax.go new file mode 100644 index 0000000..e6a6cf2 --- /dev/null +++ b/internal/validate/syntax.go @@ -0,0 +1,33 @@ +package validate + +import "github.com/Tom-R-Main/Internationalizer/internal/message" + +// SyntaxSourceFindings checks a source with its already resolved runtime grammar. +func SyntaxSourceFindings(key, source, sourceLocale string, syntax message.Syntax) []Finding { + if syntax == message.ICU { + return messageFindings(key, message.CompareICU(source, source, sourceLocale)) + } + return nil +} + +func SyntaxFindings(key, source, target, targetLocale string, syntax message.Syntax) []Finding { + if syntax == message.ICU { + return messageFindings(key, message.CompareICU(source, target, targetLocale)) + } + return nil +} + +func SyntaxInterpolationMismatch(key, source, target string, syntax message.Syntax) *Mismatch { + switch syntax { + case message.Plain, message.ICU, message.Fluent: + return nil + case message.I18next: + expected, actual := message.I18nextTokens(source), message.I18nextTokens(target) + if !sameVars(expected, actual) { + return &Mismatch{Key: key, SourceVars: expected, TargetVars: actual} + } + return nil + default: + return InterpolationMismatch(key, source, target) + } +} diff --git a/internal/validate/syntax_test.go b/internal/validate/syntax_test.go new file mode 100644 index 0000000..e0aad5d --- /dev/null +++ b/internal/validate/syntax_test.go @@ -0,0 +1,68 @@ +package validate + +import ( + "strings" + "testing" + + "github.com/Tom-R-Main/Internationalizer/internal/message" + + "gopkg.in/yaml.v3" +) + +func TestI18nextLiteralShellBraces(t *testing.T) { + const code = `<root>/{.sift,.claude,.codex,.agents}/skills` + cfg := validationConfig(t, map[string]string{"docs": "Read " + code + " for {{user.name}}"}, map[string]string{"docs": "Lire " + code + " pour {{user.name}}"}) + if err := yaml.Unmarshal([]byte("message_syntax: i18next\n"), cfg); err != nil { + t.Fatal(err) + } + reports, err := ValidateWithOptions(cfg, Options{Strict: true}) + if err != nil || HasFailures(reports) { + t.Fatalf("literal shell braces rejected: err=%v reports=%+v", err, reports) + } +} + +func TestSyntaxValidationContracts(t *testing.T) { + for _, test := range []struct { + name, source, target string + syntax message.Syntax + fail bool + }{ + {"i18next literal braces", "Run {one,two}", "Faire {one,two}", message.I18next, false}, + {"plain literal braces", "Read {draft", "Lire {brouillon", message.Plain, false}, + {"plain braces are not variables", "Read {{draft}}", "Lire {{brouillon}}", message.Plain, false}, + {"nested variable lost", "Hello {{user.name}}", "Bonjour {{name}}", message.I18next, true}, + {"escaping changed", "Hello {{- name}}", "Bonjour {{name}}", message.I18next, true}, + {"formatter lost", "Price {{amount, number}}", "Prix {{amount}}", message.I18next, true}, + {"repeated variable lost", "{{name}} / {{name}}", "{{name}}", message.I18next, true}, + {"code changed", "Read {foo,bar}", "Lire {foo,baz}", message.I18next, true}, + {"ICU unshaped source", "broken {", "cassé", message.ICU, true}, + {"ICU unshaped target", "Hello", "Bonjour }", message.ICU, true}, + {"ICU valid", "Hi {name}", "Salut {name}", message.ICU, false}, + {"ICU malformed", "{n, plural, one {One}}", "{n, plural, one {Un}}", message.ICU, true}, + } { + t.Run(test.name, func(t *testing.T) { + cfg := validationConfig(t, map[string]string{"key": test.source}, map[string]string{"key": test.target}) + cfg.MessageSyntax = test.syntax + reports, err := Validate(cfg) + if err != nil { + t.Fatal(err) + } + if HasFailures(reports) != test.fail { + t.Fatalf("failure=%v want=%v: %+v", HasFailures(reports), test.fail, reports) + } + }) + } +} + +func TestI18nextSyntaxRequiresLocalePluralKeys(t *testing.T) { + cfg := validationConfig(t, map[string]string{"items_one": "One item", "items_other": "{{count}} items"}, map[string]string{"items_one": "Un article", "items_other": "{{count}} articles"}) + cfg.MessageSyntax = message.I18next + cfg.TargetLocales = []string{"ru"} + reports, err := Validate(cfg) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(strings.Join(reports[0].Missing, ","), "items_few") { + t.Fatalf("missing plural forms not detected: %+v", reports) + } +} diff --git a/internal/validate/validate.go b/internal/validate/validate.go index 57c9910..97fbc69 100644 --- a/internal/validate/validate.go +++ b/internal/validate/validate.go @@ -28,6 +28,9 @@ type Options struct { // Report holds validation results for a single locale. Coverage remains a // compatibility alias for StructuralCoverage. type Report struct { + SourcePath string `json:"source_path,omitempty"` + BlockedBySource bool `json:"blocked_by_source,omitempty"` + BlockedLocales []string `json:"blocked_locales,omitempty"` Bundle string `json:"bundle"` Locale string `json:"locale"` TargetPath string `json:"target_path"` @@ -86,12 +89,18 @@ func ValidateWithOptions(cfg *config.Config, opts Options) ([]Report, error) { if err != nil { return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err) } - sourceUnits, err := formats.ParseUnits(format, sourceData) + sourceUnits, err := formats.ParseSourceUnits(format, sourceData, bundle.MessageSyntax) if err != nil { return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err) } sourceKeys := formats.UnitValues(sourceUnits) - for _, locale := range cfg.TargetLocales { + sourceFindings := make(map[string][]Finding) + for _, unit := range sourceUnits { + if findings := SyntaxSourceFindings(unit.ID, unit.Value, cfg.SourceLocale, unit.Syntax); len(findings) > 0 { + sourceFindings[unit.ID] = findings + } + } + for localeIndex, locale := range cfg.TargetLocales { targetPath, err := bundle.TargetPath(locale) if err != nil { return nil, err @@ -111,14 +120,26 @@ func ValidateWithOptions(cfg *config.Config, opts Options) ([]Report, error) { if loadErr != nil { return nil, loadErr } - resolved, resolveErr := policy.Resolve(cfg, locale, format.Name(), guide, terms) + resolved, resolveErr := policy.Resolve(cfg, locale, format.Name(), guide, terms, bundle.MessageSyntax) if resolveErr != nil { return nil, fmt.Errorf("hashing translation policy: %w", resolveErr) } policyHash = resolved.Hash } - report := validateLocale(bundle.ID, cfg.SourceLocale, locale, bundle.Source, sourceData, sourceUnits, sourceKeys, targetPath, format, terms, manifest, policyHash, cfg.Validation.PluralStyle, opts) + pluralStyle := bundle.PluralStyle(cfg.Validation.PluralStyle) + report := validateLocale(bundle.ID, cfg.SourceLocale, locale, bundle.Source, sourceData, sourceUnits, sourceKeys, targetPath, format, terms, manifest, policyHash, pluralStyle, opts, bundle.MessageSyntax, sourceFindings) + if len(sourceFindings) > 0 { + report.SourcePath = bundle.Source + report.BlockedBySource = true + if localeIndex == 0 { + report.BlockedLocales = cfg.TargetLocales + for _, findings := range sourceFindings { + report.Findings = append(report.Findings, findings...) + } + sortFindings(report.Findings) + } + } reports = append(reports, report) } } @@ -132,7 +153,7 @@ func formatForBundle(bundle config.Bundle) (formats.Format, error) { return formats.FormatForFile(bundle.Source) } -func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData []byte, sourceUnits []formats.Unit, sourceKeys map[string]string, targetPath string, format formats.Format, terms []glossary.Term, manifest *state.Manifest, policyHash, pluralStyle string, opts Options) Report { +func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData []byte, sourceUnits []formats.Unit, sourceKeys map[string]string, targetPath string, format formats.Format, terms []glossary.Term, manifest *state.Manifest, policyHash, pluralStyle string, opts Options, syntaxPolicy message.Syntax, sourceFindings map[string][]Finding) Report { report := Report{Bundle: bundle, Locale: locale, TargetPath: targetPath} if opts.Strict { translated := 0.0 @@ -142,19 +163,22 @@ func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData validationKeys := sourceKeys requiredPluralKeys := make(map[string]struct{}) optionalPluralKeys := make(map[string]struct{}) - if (opts.Strict || opts.RequireState) && pluralStyle == "i18next-v4" { + if (opts.Strict || opts.RequireState || syntaxPolicy == message.I18next) && pluralStyle == "i18next-v4" { validationKeys, requiredPluralKeys, optionalPluralKeys = ExpandI18nextV4Source(sourceKeys, sourceLocale, locale) } sourceICUValid := make(map[string]bool, len(validationKeys)) + syntaxes := make(map[string]message.Syntax, len(validationKeys)) + for _, unit := range sourceUnits { + syntaxes[unit.ID] = unit.Syntax + } document := format.Name() == "markdown" for _, key := range allKeys(validationKeys) { sourceValue := validationKeys[key] - var findings []Finding - if !document { - findings = ICUSourceFindings(key, sourceValue, sourceLocale) + if syntaxes[key] == "" { + syntaxes[key] = message.ResolveSyntax(format.Name(), syntaxPolicy, sourceValue) } + findings := sourceFindings[key] sourceICUValid[key] = len(findings) == 0 - report.Findings = append(report.Findings, findings...) } targetData, err := os.ReadFile(targetPath) @@ -217,20 +241,20 @@ func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData report.Findings = append(report.Findings, Finding{Code: CodeSourceIdentical, Severity: SeverityError, Key: key, Message: "target is identical to the source without an exact glossary exemption"}) } - if !usesICUValidation(sourceValue, targetValue) { - if mismatch := InterpolationMismatch(key, sourceValue, targetValue); mismatch != nil { - report.Mismatches = append(report.Mismatches, *mismatch) - } + if mismatch := SyntaxInterpolationMismatch(key, sourceValue, targetValue, syntaxes[key]); mismatch != nil { + report.Mismatches = append(report.Mismatches, *mismatch) } if sourceICUValid[key] && !document { - report.Findings = append(report.Findings, ICUFindings(key, sourceValue, targetValue, locale)...) + report.Findings = append(report.Findings, SyntaxFindings(key, sourceValue, targetValue, locale, syntaxes[key])...) } - if opts.Strict { + if opts.Strict || opts.RequireState || (syntaxPolicy != "" && syntaxPolicy != message.Auto) { if document { - report.Findings = append(report.Findings, ProtectedDocumentFindings(key, sourceValue, targetValue, locale, sourcePath, targetPath)...) + report.Findings = append(report.Findings, ProtectedDocumentFindings(key, sourceValue, targetValue, locale, sourcePath, targetPath, syntaxes[key])...) } else { - report.Findings = append(report.Findings, ProtectedFindings(key, sourceValue, targetValue, locale)...) + report.Findings = append(report.Findings, ProtectedSyntaxFindings(key, sourceValue, targetValue, locale, syntaxes[key])...) } + } + if opts.Strict { report.Findings = append(report.Findings, glossaryFindings(key, sourceValue, targetValue, terms)...) } if opts.RequireState { @@ -418,7 +442,7 @@ func InterpolationMismatch(key, source, target string) *Mismatch { // HasFailures reports whether validation should return a non-zero exit status. func HasFailures(reports []Report) bool { for _, report := range reports { - if len(report.Missing) > 0 || len(report.Mismatches) > 0 || len(report.Errors) > 0 { + if report.BlockedBySource || len(report.Missing) > 0 || len(report.Mismatches) > 0 || len(report.Errors) > 0 { return true } for _, finding := range report.Findings { @@ -503,6 +527,11 @@ func FormatHuman(reports []Report) string { _, _ = fmt.Fprintf(&b, ", %d errors", len(report.Errors)) } b.WriteByte('\n') + if len(report.BlockedLocales) > 0 { + _, _ = fmt.Fprintf(&b, " Source %s blocks %d locales: %s\n", report.SourcePath, len(report.BlockedLocales), strings.Join(report.BlockedLocales, ", ")) + } else if report.BlockedBySource { + _, _ = fmt.Fprintf(&b, " Blocked by source errors in %s (reported above)\n", report.SourcePath) + } for _, finding := range report.Findings { _, _ = fmt.Fprintf(&b, " - %s: %s", finding.Code, finding.Message) if finding.Key != "" { diff --git a/test/acceptance/syntax_test.go b/test/acceptance/syntax_test.go new file mode 100644 index 0000000..b798d20 --- /dev/null +++ b/test/acceptance/syntax_test.go @@ -0,0 +1,89 @@ +package acceptance_test + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/Tom-R-Main/Internationalizer/internal/state" +) + +func TestExecuFunctionSyntaxLifecycle(t *testing.T) { + dir := copyFixture(t, "execufunction-syntax") + targetPath := filepath.Join(dir, "web", "fr.json") + original := mustReadFile(t, targetPath) + runCLI(t, dir, nil, "validate", "--strict", "--json").requireSuccess(t) + dry := runCLI(t, dir, nil, "translate", "--dry-run") + dry.requireSuccess(t) + if !strings.Contains(dry.stdout, "Would translate 0 keys") { + t.Fatal(dry.stdout) + } + if !bytes.Equal(original, mustReadFile(t, targetPath)) { + t.Fatal("dry-run modified target") + } + if _, err := os.Stat(filepath.Join(dir, ".internationalizer.lock")); !os.IsNotExist(err) { + t.Fatalf("dry-run wrote state: %v", err) + } + for _, strategy := range []string{"accented", "bidi"} { + result := runCLI(t, dir, nil, "pseudo", "--strategy", strategy, "--dry-run") + result.requireSuccess(t) + runCLI(t, dir, nil, "pseudo", "--strategy", strategy).requireSuccess(t) + locale := "en-XA" + if strategy == "bidi" { + locale = "ar-XB" + } + var generated map[string]string + if err := json.Unmarshal(mustReadFile(t, filepath.Join(dir, "web", locale+".json")), &generated); err != nil { + t.Fatal(err) + } + for _, literal := range []string{`<root>/{.sift,.claude,.codex,.agents}/skills`, "{{user.name}}"} { + if !strings.Contains(generated["docs.tui.skills.desc"], literal) { + t.Fatalf("%s changed %s: %s", strategy, literal, generated["docs.tui.skills.desc"]) + } + } + if !strings.Contains(generated["price"], "{{amount, number}}") || !strings.Contains(generated["price"], "{{- customer.name}}") { + t.Fatalf("%s damaged interpolation: %s", strategy, generated["price"]) + } + } + runCLI(t, dir, nil, "translate", "--adopt-existing").requireSuccess(t) + manifest, err := state.Load(filepath.Join(dir, ".internationalizer.lock")) + if err != nil { + t.Fatal(err) + } + for _, entry := range manifest.Translations { + if entry.ReviewStatus != state.ReviewNeedsReview { + t.Fatal("adoption/pseudo implied approval") + } + } + runCLI(t, dir, nil, "review", "approve", "--locale", "fr", "--all").requireSuccess(t) + runCLI(t, dir, nil, "validate", "--require-approved").requireSuccess(t) + configPath := filepath.Join(dir, ".internationalizer.yml") + config := string(mustReadFile(t, configPath)) + mustWriteFile(t, configPath, strings.Replace(config, "message_syntax: i18next", "message_syntax: plain", 1)) + stale := runCLI(t, dir, nil, "validate", "--require-approved", "--json") + if stale.exitCode == 0 || !strings.Contains(stale.stdout, "policy_stale") { + t.Fatalf("policy change retained approval: %+v", stale) + } + if result := runCLI(t, dir, nil, "review", "approve", "--locale", "fr", "--all"); result.exitCode == 0 { + t.Fatal("approved stale syntax policy") + } +} + +func TestSourceErrorsReportedOnceAcrossLocales(t *testing.T) { + dir := copyFixture(t, "execufunction-syntax") + mustWriteFile(t, filepath.Join(dir, "icu", "en.json"), `{"items":"{count, plural, one {One}}"}`) + configPath := filepath.Join(dir, ".internationalizer.yml") + mustWriteFile(t, configPath, strings.Replace(string(mustReadFile(t, configPath)), "[fr]", "[fr, de, es]", 1)) + for _, args := range [][]string{{"validate", "--json"}, {"translate", "--dry-run"}} { + result := runCLI(t, dir, nil, args...) + if result.exitCode == 0 || strings.Count(result.stdout, "source ICU message:") != 1 { + t.Fatalf("%v did not report source once: %+v", args, result) + } + if !strings.Contains(result.stdout, "icu/en.json") && !strings.Contains(result.stdout, `icu\\en.json`) && !strings.Contains(result.stdout, `icu\en.json`) { + t.Fatalf("missing source path: %s", result.stdout) + } + } +} diff --git a/test/acceptance/testdata/execufunction-syntax/.internationalizer.yml b/test/acceptance/testdata/execufunction-syntax/.internationalizer.yml new file mode 100644 index 0000000..1f6f37b --- /dev/null +++ b/test/acceptance/testdata/execufunction-syntax/.internationalizer.yml @@ -0,0 +1,11 @@ +source_locale: en +target_locales: [fr] +message_syntax: i18next +bundles: + - id: web + source: web/en.json + target: web/{locale}.json + - id: icu + source: icu/en.json + target: icu/{locale}.json + message_syntax: icu diff --git a/test/acceptance/testdata/execufunction-syntax/icu/en.json b/test/acceptance/testdata/execufunction-syntax/icu/en.json new file mode 100644 index 0000000..f1f0394 --- /dev/null +++ b/test/acceptance/testdata/execufunction-syntax/icu/en.json @@ -0,0 +1 @@ +{"items": "{count, plural, one {One item for {name}} other {# items for {name}}}"} diff --git a/test/acceptance/testdata/execufunction-syntax/icu/fr.json b/test/acceptance/testdata/execufunction-syntax/icu/fr.json new file mode 100644 index 0000000..ca5c3aa --- /dev/null +++ b/test/acceptance/testdata/execufunction-syntax/icu/fr.json @@ -0,0 +1 @@ +{"items": "{count, plural, one {Un article pour {name}} other {# articles pour {name}}}"} diff --git a/test/acceptance/testdata/execufunction-syntax/web/en.json b/test/acceptance/testdata/execufunction-syntax/web/en.json new file mode 100644 index 0000000..2098cec --- /dev/null +++ b/test/acceptance/testdata/execufunction-syntax/web/en.json @@ -0,0 +1,5 @@ +{ + "docs.tui.skills.desc": "Read <root>/{.sift,.claude,.codex,.agents}/skills for {{user.name}}.", + "price": "Price {{amount, number}} for {{- customer.name}}", + "example": "Run `{.sift,.claude}/skills` to continue." +} diff --git a/test/acceptance/testdata/execufunction-syntax/web/fr.json b/test/acceptance/testdata/execufunction-syntax/web/fr.json new file mode 100644 index 0000000..0df1a93 --- /dev/null +++ b/test/acceptance/testdata/execufunction-syntax/web/fr.json @@ -0,0 +1,5 @@ +{ + "docs.tui.skills.desc": "Lire <root>/{.sift,.claude,.codex,.agents}/skills pour {{user.name}}.", + "price": "Prix {{amount, number}} pour {{- customer.name}}", + "example": "Exécutez `{.sift,.claude}/skills` pour continuer." +}