Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,15 +261,18 @@ target_locales: [fr, de, es, ja, yue, zh-CN, zh-TW, ar]

# One or more source-to-target mappings (required).
# {locale} is replaced with each configured target locale.
message_syntax: auto # default; overridden per bundle below
bundles:
- id: app
source: locales/en.json
target: locales/{locale}.json
format: json
message_syntax: i18next
- id: docs
source: README.md
target: docs/i18n/{locale}.md
format: markdown
message_syntax: plain
- id: browser
source: browser/locales/en-US/browser.ftl
target: browser/locales/{locale}/browser.ftl
Expand Down Expand Up @@ -348,6 +351,35 @@ and locale-specific provider overrides match canonical-equivalent spelling.
In the example above, locales without an override—including Japanese—inherit
the global Gemini configuration.

File format and message syntax are separate settings. `message_syntax` accepts
`auto` (the default), `i18next`, `icu`, or `plain`, globally and per bundle:

- `i18next` protects `{{name}}`, nested paths such as `{{user.name}}`, escaping
modifiers such as `{{- name}}`, formatting modifiers, and repeated placeholders.
It enables i18next v4 locale-specific plural keys. Other braces are literal;
`{.sift,.claude,.codex,.agents}` is not parsed as ICU. Custom interpolation
delimiters and nesting expressions are not part of this profile.
- `icu` always parses the message as ICU, including malformed input that cannot
be recognized by automatic detection. Parsing errors never fall back to text.
- `plain` treats braces as text and does not impose an interpolation grammar.
- `auto` infers each message's grammar from its source. Select an explicit mode
for catalogs mixing prose and code. Fluent resources require `auto` and use
their own grammar; Markdown documents cannot select `icu`.

Validation, provider and TM output checks, adoption, pseudolocalization, and
review approval share the selected syntax. HTML `<code>` contents and Markdown
code spans are preserved exactly. Explicit syntax profiles enforce protected
content even without `--strict`; `--strict` additionally checks translation
quality and glossary rules. Source errors are reported once per bundle with
`source_path`, `blocked_by_source`, and `blocked_locales` in JSON reports.

The syntax setting is part of the policy hash. This prompt-contract update
makes previously recorded policies stale, including those using `auto`.
Use `translate --dry-run` to inspect them, `translate --refresh-policy` to
regenerate, or `translate --adopt-existing` to validate and record existing
values under the new policy. Adoption leaves entries needing explicit review.
Dry-run reports “Would translate” with planned keys and blocked job counts.

ICU MessageFormat values are parsed structurally. Simple arguments, `select`,
`plural`, `selectordinal`, `number`, `date`, and `time` are supported, including
nested messages, plural offsets, exact-number selectors, and `#`. Validation
Expand Down
43 changes: 43 additions & 0 deletions docs/execufunction-message-syntax.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Using Internationalizer with ExecuFunction

The 0.1.2 regression was a grammar mismatch: the marketing catalog contains
literal shell brace syntax inside HTML code elements. Automatic ICU inference
rejected `{.sift,.claude,.codex,.agents}` before any translation could run.

Configure the web catalog independently from marketing. From ExecuFunction's
repository root, the web mapping is:

```yaml
source_locale: en
# Populate this from the web application's supported locale manifest.
target_locales: [fr]
bundles:
- id: web
source: exf-app/web/src/i18n/locales/en.json
target: exf-app/web/src/i18n/locales/{locale}.json
format: json
message_syntax: i18next
```

This profile uses i18next's default double-brace interpolation, including nested
paths and escaping modifiers, documented in the
[i18next interpolation reference](https://www.i18next.com/translation-function/interpolation).
Internationalizer also preserves formatter modifiers and checks v4 plural-key
families against the target locale.

The existing marketing config points at `tmp/english-keys.json` and sibling
locale files. Give that bundle its own explicit syntax, chosen for the runtime
that consumes those files: `plain` for literal strings or `i18next` if double-brace
interpolation is used. Coverage for these temporary catalogs measures those
catalogs only; it does not establish coverage of the web app or of the marketing
pipeline's authoritative inputs. Do not add a guessed production path. Verify
the marketing build's actual input and output mapping before adopting it.

Run `validate --json` and `translate --dry-run` first. Syntax selection removes
the ICU false positive; it can reveal actual translated code or missing
placeholders, which remain errors. Neither dry-run nor validation writes files.

The acceptance fixture `test/acceptance/testdata/execufunction-syntax` exercises
the exact shell-brace pattern alongside i18next interpolation and an explicit
ICU bundle. Its lifecycle covers both pseudo strategies, adoption without
approval, explicit approval, and invalidation after a syntax-policy change.
10 changes: 9 additions & 1 deletion docs/localization-core-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ framework runtime, translation CDN, or source-rewriting setup command.

## Message semantics

- Resource format and runtime grammar are distinct. `message_syntax` defaults
to `auto` and supports bundle overrides for `i18next`, `icu`, and `plain`.
Source units carry the resolved grammar; targets never select their grammar.
- Explicit ICU always parses, without a parse-error fallback. i18next protects
double-brace interpolation and v4 plural-key families; plain text treats
braces literally. Fluent resources own their grammar. Markdown is not ICU.
- ICU messages are parsed structurally rather than treated as brace-shaped
interpolation strings.
- Supported arguments are simple interpolation, `select`, `plural`,
Expand Down Expand Up @@ -54,14 +60,16 @@ inserted.
artifacts without a provider or translation-memory lookup.
- ICU and Fluent runtime expressions remain intact while linguistic text is
transformed.
- HTML code element contents and Markdown code spans are protected from
translation and pseudolocalization. Approval rechecks protected content.
- `data-l10n-name` identifies semantic rich-text slots. Translators may reorder
named slots, but element identity, protected attributes, nesting, and
contained markup must remain compatible.

## Policy identity

The manifest records style-guide, glossary, and prompt-contract components
separately. The combined policy hash includes those components and the provider
separately. The combined policy hash includes those components, message syntax, and the provider
settings, but not the rendered prompt text. Refactoring prompt construction
therefore leaves current translations alone; a semantic prompt change requires
an explicit prompt-contract version bump. Style guides are read-only inputs and
Expand Down
82 changes: 62 additions & 20 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"strings"

localeid "github.com/Tom-R-Main/Internationalizer/internal/locale"
"github.com/Tom-R-Main/Internationalizer/internal/message"
"gopkg.in/yaml.v3"
)

Expand All @@ -22,19 +23,20 @@ const (
)

type Config struct {
SourceLocale string `yaml:"source_locale"`
TargetLocales []string `yaml:"target_locales"`
SourcePath string `yaml:"source_path"`
Bundles []Bundle `yaml:"bundles"`
LLM LLM `yaml:"llm"`
BatchSize int `yaml:"batch_size"`
Concurrency int `yaml:"concurrency"`
StyleGuidesDir string `yaml:"style_guides_dir"`
GlossaryDir string `yaml:"glossary_dir"`
TMPath string `yaml:"tm_path"`
ManifestPath string `yaml:"manifest_path"`
Formats []string `yaml:"formats"`
Validation Validation `yaml:"validation"`
MessageSyntax message.Syntax `yaml:"message_syntax"`
SourceLocale string `yaml:"source_locale"`
TargetLocales []string `yaml:"target_locales"`
SourcePath string `yaml:"source_path"`
Bundles []Bundle `yaml:"bundles"`
LLM LLM `yaml:"llm"`
BatchSize int `yaml:"batch_size"`
Concurrency int `yaml:"concurrency"`
StyleGuidesDir string `yaml:"style_guides_dir"`
GlossaryDir string `yaml:"glossary_dir"`
TMPath string `yaml:"tm_path"`
ManifestPath string `yaml:"manifest_path"`
Formats []string `yaml:"formats"`
Validation Validation `yaml:"validation"`
}

// Validation configures optional project-specific validation rules.
Expand All @@ -45,10 +47,11 @@ type Validation struct {
// Bundle maps one source file to a locale-specific target path.
// Target must contain the literal {locale} placeholder.
type Bundle struct {
ID string `yaml:"id"` // required stable identity for explicit bundles
Source string `yaml:"source"`
Target string `yaml:"target"`
Format string `yaml:"format"`
MessageSyntax message.Syntax `yaml:"message_syntax"`
ID string `yaml:"id"` // required stable identity for explicit bundles
Source string `yaml:"source"`
Target string `yaml:"target"`
Format string `yaml:"format"`
}

type LLM struct {
Expand All @@ -71,6 +74,9 @@ type LLMOverride struct {
}

func (c *Config) ApplyDefaults() {
if c.MessageSyntax == "" {
c.MessageSyntax = message.Auto
}
if c.SourceLocale == "" {
c.SourceLocale = "en"
}
Expand Down Expand Up @@ -237,12 +243,29 @@ func (c *Config) ValidateProject() error {
}
}
bundles := c.EffectiveBundles()
if err := message.ValidateSyntax(c.MessageSyntax); err != nil {
return err
}
if len(bundles) == 0 {
return fmt.Errorf("source_path or bundles is required")
}
seen := make(map[string]struct{}, len(bundles))
targets := make(map[string]string, len(bundles)*len(c.TargetLocales))
for _, bundle := range bundles {
if err := message.ValidateSyntax(bundle.MessageSyntax); err != nil {
return fmt.Errorf("bundle %q: %w", bundle.ID, err)
}
format := strings.ToLower(bundle.Format)
extension := strings.ToLower(filepath.Ext(bundle.Source))
if format == "" && extension == ".ftl" {
format = "fluent"
}
if format == "fluent" && bundle.MessageSyntax != "" && bundle.MessageSyntax != message.Auto {
return fmt.Errorf("bundle %q: Fluent resources require message_syntax: auto", bundle.ID)
}
if (format == "markdown" || (format == "" && (extension == ".md" || extension == ".mdx"))) && bundle.MessageSyntax == message.ICU {
return fmt.Errorf("bundle %q: Markdown documents cannot use message_syntax: icu", bundle.ID)
}
if bundle.ID == "" {
return fmt.Errorf("explicit bundle id is required")
}
Expand Down Expand Up @@ -320,15 +343,21 @@ func (c *Config) EffectiveBundles() []Bundle {
if len(c.Bundles) > 0 {
bundles := make([]Bundle, len(c.Bundles))
copy(bundles, c.Bundles)
for i := range bundles {
if bundles[i].MessageSyntax == "" {
bundles[i].MessageSyntax = c.MessageSyntax
}
}
return bundles
}
if c.SourcePath == "" {
return nil
}
return []Bundle{{
ID: "default",
Source: c.SourcePath,
Target: filepath.Join(filepath.Dir(c.SourcePath), "{locale}"+filepath.Ext(c.SourcePath)),
MessageSyntax: c.MessageSyntax,
ID: "default",
Source: c.SourcePath,
Target: filepath.Join(filepath.Dir(c.SourcePath), "{locale}"+filepath.Ext(c.SourcePath)),
}}
}

Expand All @@ -343,6 +372,19 @@ func (b Bundle) TargetPath(locale string) (string, error) {
return filepath.Clean(strings.ReplaceAll(b.Target, "{locale}", locale)), nil
}

// PluralStyle uses v4 key families for i18next; explicit non-i18next grammars
// must not reinterpret keys just because they end in a plural suffix.
func (b Bundle) PluralStyle(legacyStyle string) string {
switch b.MessageSyntax {
case message.I18next:
return "i18next-v4"
case "", message.Auto:
return legacyStyle
default:
return ""
}
}

func localeOverride(overrides map[string]LLMOverride, requested string) (LLMOverride, bool) {
if override, ok := overrides[requested]; ok {
return override, true
Expand Down
57 changes: 57 additions & 0 deletions internal/config/syntax_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package config

import (
"testing"

"github.com/Tom-R-Main/Internationalizer/internal/message"
)

func TestMessageSyntaxInheritanceAndValidation(t *testing.T) {
cfg := Config{SourceLocale: "en", TargetLocales: []string{"fr"}, MessageSyntax: message.I18next, Bundles: []Bundle{
{ID: "web", Source: "web/en.json", Target: "web/{locale}.json"},
{ID: "icu", Source: "icu/en.json", Target: "icu/{locale}.json", MessageSyntax: message.ICU},
{ID: "fluent", Source: "en.ftl", Target: "{locale}.ftl", MessageSyntax: message.Auto},
}}
cfg.ApplyDefaults()
if err := cfg.ValidateProject(); err != nil {
t.Fatal(err)
}
got := cfg.EffectiveBundles()
if got[0].MessageSyntax != message.I18next || got[1].MessageSyntax != message.ICU || got[2].MessageSyntax != message.Auto {
t.Fatalf("inheritance: %+v", got)
}
if cfg.Bundles[0].MessageSyntax != "" {
t.Fatal("resolution mutated input config")
}
if got[1].PluralStyle("i18next-v4") != "" {
t.Fatal("ICU inherited i18next plural keys")
}
for _, syntax := range []message.Syntax{"typo", "fluent", message.Legacy} {
cfg.MessageSyntax = syntax
if err := cfg.ValidateProject(); err == nil {
t.Fatalf("accepted invalid default %q", syntax)
}
}
cfg.MessageSyntax = message.Auto
cfg.Bundles[0].MessageSyntax = "typo"
if err := cfg.ValidateProject(); err == nil {
t.Fatal("accepted invalid override")
}
cfg.Bundles[0].MessageSyntax = message.Auto
cfg.Bundles[2].MessageSyntax = message.I18next
if err := cfg.ValidateProject(); err == nil {
t.Fatal("accepted non-Fluent resource grammar")
}
cfg.Bundles[2] = Bundle{ID: "doc", Source: "README.md", Target: "{locale}.md", MessageSyntax: message.ICU}
if err := cfg.ValidateProject(); err == nil {
t.Fatal("accepted ICU document")
}
cfg.Bundles[2].Source = "README.MDX"
if err := cfg.ValidateProject(); err == nil {
t.Fatal("accepted ICU MDX document")
}
cfg.Bundles[2] = Bundle{ID: "fluent", Source: "EN.FTL", Target: "{locale}.ftl", MessageSyntax: message.Plain}
if err := cfg.ValidateProject(); err == nil {
t.Fatal("accepted plain Fluent resource with uppercase extension")
}
}
25 changes: 20 additions & 5 deletions internal/formats/units.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package formats
import (
"fmt"
"sort"

"github.com/Tom-R-Main/Internationalizer/internal/message"
)

// UnitKind describes the semantic role of one independently translatable unit.
Expand All @@ -21,11 +23,24 @@ const (
// Context is translator-facing information; Structure is a deterministic
// adapter-owned signature used to detect semantic message changes.
type Unit struct {
ID string `json:"id"`
Value string `json:"value"`
Kind UnitKind `json:"kind"`
Context string `json:"context,omitempty"`
Structure string `json:"structure,omitempty"`
Syntax message.Syntax `json:"syntax,omitempty"`
ID string `json:"id"`
Value string `json:"value"`
Kind UnitKind `json:"kind"`
Context string `json:"context,omitempty"`
Structure string `json:"structure,omitempty"`
}

// ParseSourceUnits attaches the resolved source grammar for downstream checks.
func ParseSourceUnits(format Format, data []byte, syntax message.Syntax) ([]Unit, error) {
units, err := ParseUnits(format, data)
if err != nil {
return nil, err
}
for i := range units {
units[i].Syntax = message.ResolveSyntax(format.Name(), syntax, units[i].Value)
}
return units, nil
}

// UnitFormat is implemented by formats with richer semantics than a flat
Expand Down
Loading