diff --git a/README.md b/README.md
index 96d1c4f..3bba438 100644
--- a/README.md
+++ b/README.md
@@ -29,6 +29,8 @@ Internationalizer is different. It's a **CLI pipeline** that combines LLM transl
- **Per-language style guides** — control tone, formality, pluralization, and typography
- **Translation memory** — skip unchanged strings, save money on API calls
- **Deterministic validation** — catch missing or extra keys, protected-structure drift, glossary issues, and plural or ICU errors before they ship
+- **Explicit approval** — keep provider or adopted provenance separate from human review
+- **Fluent and pseudolocales** — preserve translator context and exercise accented or bidirectional layouts without an API call
## Installation
@@ -105,10 +107,13 @@ internationalizer translate --dry-run
internationalizer translate
```
-5. Validate all locales:
+5. Validate and approve the exact generated artifacts:
```bash
internationalizer validate
+internationalizer review list --status needs_review
+internationalizer review approve --locale fr --all
+internationalizer validate --require-approved
```
## Commands
@@ -131,9 +136,39 @@ Translation state independently reports missing, source-stale, policy-stale,
current, and manually edited conditions, so a manual edit cannot conceal a
source or policy change. Policy-stale values are reported but only retranslated
with `--refresh-policy`. Manually edited values are never overwritten
-automatically. Use `--adopt-existing` when introducing the manifest to reviewed
-translations or when explicitly accepting a reviewed manual edit as the new
-baseline.
+automatically. Use `--adopt-existing` when introducing the manifest to existing
+translations or when explicitly accepting a manual edit as the new provenance
+baseline. Adoption does not imply human approval; use `review approve` for that
+separate decision.
+
+### `pseudo`
+
+Generate deterministic test locales without a provider or translation-memory
+lookup. Accented output defaults to `en-XA`; bidirectional output defaults to
+`ar-XB`. ICU and Fluent runtime syntax, code, links, and markup are preserved.
+
+```bash
+internationalizer pseudo # accented en-XA
+internationalizer pseudo --strategy bidi # bidi ar-XB
+internationalizer pseudo --dry-run # show planned artifacts only
+internationalizer pseudo --locale qps-ploc # choose another valid locale tag
+```
+
+The generator refreshes only artifacts it previously recorded as pseudo
+output. Use `--force` to replace any other existing target deliberately.
+
+### `review`
+
+Inspect and approve the exact target content currently bound to its source and
+translation policy. Generated, cached, and adopted values all begin in
+`needs_review`; approval is invalidated by later source, policy, or target
+changes. Pseudolocales are tracked separately as test artifacts.
+
+```bash
+internationalizer review list --status needs_review
+internationalizer review approve --locale fr --bundle app --key common.save
+internationalizer review approve --locale fr --all
+```
### `validate`
@@ -148,6 +183,7 @@ internationalizer validate --json # machine-readable JSON
internationalizer validate -q # exit code only
internationalizer validate --strict # enforce translation quality rules
internationalizer validate --require-state # require current manifest provenance
+internationalizer validate --require-approved # also require explicit approval
```
`--strict` also reports translated coverage. A linguistic value identical to
@@ -159,7 +195,9 @@ Markdown-link structure, glossary violations, and configured plural forms.
`--require-state` verifies each target against `.internationalizer.lock`. It
fails when a key is untracked, or when its recorded source, translation policy,
-or target hash is stale. It can be combined with `--strict`.
+or target hash is stale. `--require-approved` implies `--require-state` and also
+fails if the exact current artifact has not been approved. Both can be combined
+with `--strict`.
Human and JSON reports use stable finding codes:
@@ -178,6 +216,7 @@ Human and JSON reports use stable finding codes:
| `source_stale` | Source content changed after the recorded translation |
| `policy_stale` | The generated prompt or model settings changed |
| `target_modified` | Target content differs from the manifest record |
+| `needs_review` | Current provenance exists, but the exact target is not approved |
### `detect`
@@ -231,6 +270,10 @@ bundles:
source: README.md
target: docs/i18n/{locale}.md
format: markdown
+ - id: browser
+ source: browser/locales/en-US/browser.ftl
+ target: browser/locales/{locale}/browser.ftl
+ format: fluent
# Backward compatibility: source_path still maps targets to sibling files
# such as locales/fr.json. Prefer bundles for new projects.
@@ -313,6 +356,14 @@ branch identity, and target-locale CLDR plural categories. Provider output that
breaks these invariants is rejected before a locale file or translation-memory
record is written.
+Fluent (`.ftl`) resources are handled as semantic source documents rather than
+flattened maps. Message values, terms, and attributes become independent units;
+comments are passed to the provider as developer context and included in source
+provenance. Serialization preserves resource comments and ordering. Validation
+protects variables, references, functions, selector defaults and branches, and
+`data-l10n-name` markup slots while allowing target-locale selector variants and
+natural reordering of named rich-text elements.
+
With `i18next-v4`, recognized source plural families are expanded during
translation to the target locale's CLDR categories. A target-only category uses
the source family's `_other` value as its translation template. Strict
@@ -386,6 +437,8 @@ from the cache without calling the LLM. The default path is under the ignored
`.internationalizer/` directory, so it remains a local cache. Set `tm_path` to a
tracked location if your project intentionally shares translation memory. The
reviewable `.internationalizer.lock` manifest is versioned separately.
+Manifest schema v2 records provenance origin and review status independently,
+so “generated successfully” and “approved by a person” cannot be conflated.
## Supported Formats
@@ -394,6 +447,7 @@ reviewable `.internationalizer.lock` manifest is versioned separately.
| JSON | `.json` | Key-value (nested, dot-notation flattened) |
| YAML | `.yml`, `.yaml` | Key-value (preserves comments and ordering) |
| Markdown | `.md`, `.mdx` | Preamble and H2-level sections |
+| Fluent | `.ftl` | Semantic messages, terms, attributes, comments, and selectors |
Markdown targets contain invisible `internationalizer:unit` comments before
H2 sections. These stable markers let Internationalizer add, move, or edit one
@@ -415,7 +469,8 @@ cmd/internationalizer/ CLI entry point and command definitions
internal/
config/ YAML config loading with defaults
detect/ Project type auto-detection
- formats/ Format parsers (JSON, YAML, Markdown)
+ fluentpattern/ Fluent pattern validation and safe text transforms
+ formats/ Format adapters (JSON, YAML, Markdown, Fluent)
glossary/ Per-locale glossary management
llm/ LLM provider interface + implementations
anthropic.go Anthropic Claude backend
@@ -425,6 +480,8 @@ internal/
locale/ BCP 47 identity and CLDR plural categories
message/ ICU MessageFormat parser and structural comparison
policy/ Stable translation-policy hashing
+ pseudo/ Provider-free accented and bidi test locales
+ review/ Explicit artifact approval workflow
state/ Versioned translation manifest
styleguide/ Style guide loader
tm/ JSONL translation memory
diff --git a/cmd/internationalizer/main.go b/cmd/internationalizer/main.go
index 21ab076..b9ac8da 100644
--- a/cmd/internationalizer/main.go
+++ b/cmd/internationalizer/main.go
@@ -26,6 +26,8 @@ func main() {
newGlossaryCmd(),
newTmCmd(),
newValidateCmd(),
+ newReviewCmd(),
+ newPseudoCmd(),
)
if err := rootCmd.Execute(); err != nil {
diff --git a/cmd/internationalizer/pseudo.go b/cmd/internationalizer/pseudo.go
new file mode 100644
index 0000000..0a6c847
--- /dev/null
+++ b/cmd/internationalizer/pseudo.go
@@ -0,0 +1,52 @@
+package main
+
+import (
+ "fmt"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ "github.com/Tom-R-Main/Internationalizer/internal/pseudo"
+ "github.com/spf13/cobra"
+)
+
+func newPseudoCmd() *cobra.Command {
+ cmd := &cobra.Command{
+ Use: "pseudo",
+ Short: "Generate deterministic accented or bidi pseudolocales",
+ RunE: func(cmd *cobra.Command, args []string) error {
+ cfgPath, _ := cmd.Flags().GetString("config")
+ cfg, err := config.Load(cfgPath)
+ if err != nil {
+ return err
+ }
+ strategyValue, _ := cmd.Flags().GetString("strategy")
+ locale, _ := cmd.Flags().GetString("locale")
+ force, _ := cmd.Flags().GetBool("force")
+ dryRun, _ := cmd.Flags().GetBool("dry-run")
+ results, err := pseudo.Generate(cfg, pseudo.GenerateOptions{
+ Strategy: pseudo.Strategy(strategyValue),
+ Locale: locale,
+ Force: force,
+ DryRun: dryRun,
+ })
+ if err != nil {
+ return err
+ }
+ for _, result := range results {
+ action := "generated"
+ if dryRun {
+ action = "would generate"
+ }
+ if _, err := fmt.Fprintf(cmd.OutOrStdout(), "%s %s/%s: %d units -> %s\n", action, result.Bundle, result.Locale, result.Units, result.TargetPath); err != nil {
+ return err
+ }
+ }
+ return nil
+ },
+ }
+ cmd.Flags().StringP("config", "c", "", "path to config file (default: .internationalizer.yml)")
+ cmd.Flags().String("strategy", string(pseudo.Accented), "pseudo strategy (accented or bidi)")
+ cmd.Flags().StringP("locale", "l", "", "output locale (defaults to en-XA or ar-XB)")
+ cmd.Flags().Bool("force", false, "overwrite an existing artifact not owned by the pseudo generator")
+ cmd.Flags().Bool("dry-run", false, "show outputs without writing files or manifest state")
+ return cmd
+}
diff --git a/cmd/internationalizer/review.go b/cmd/internationalizer/review.go
new file mode 100644
index 0000000..013db03
--- /dev/null
+++ b/cmd/internationalizer/review.go
@@ -0,0 +1,100 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "time"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ "github.com/Tom-R-Main/Internationalizer/internal/review"
+ "github.com/Tom-R-Main/Internationalizer/internal/state"
+ "github.com/spf13/cobra"
+)
+
+func newReviewCmd() *cobra.Command {
+ cmd := &cobra.Command{
+ Use: "review",
+ Short: "Inspect and approve exact translation artifacts",
+ }
+ cmd.AddCommand(newReviewListCmd(), newReviewApproveCmd())
+ return cmd
+}
+
+func newReviewListCmd() *cobra.Command {
+ cmd := &cobra.Command{
+ Use: "list",
+ Short: "List tracked translations and review status",
+ RunE: func(cmd *cobra.Command, args []string) error {
+ cfgPath, _ := cmd.Flags().GetString("config")
+ cfg, err := config.Load(cfgPath)
+ if err != nil {
+ return err
+ }
+ manifest, err := state.Load(cfg.ManifestPath)
+ if err != nil {
+ return err
+ }
+ locale, _ := cmd.Flags().GetString("locale")
+ bundle, _ := cmd.Flags().GetString("bundle")
+ statusValue, _ := cmd.Flags().GetString("status")
+ entries, err := review.List(manifest, review.Filter{Locale: locale, Bundle: bundle, Status: state.ReviewStatus(statusValue)})
+ if err != nil {
+ return err
+ }
+ asJSON, _ := cmd.Flags().GetBool("json")
+ if asJSON {
+ encoder := json.NewEncoder(cmd.OutOrStdout())
+ encoder.SetIndent("", " ")
+ return encoder.Encode(entries)
+ }
+ for _, entry := range entries {
+ if _, err := fmt.Fprintf(cmd.OutOrStdout(), "%s\t%s\t%s\t%s\t%s\n", entry.ReviewStatus, entry.Locale, entry.Bundle, entry.Key, entry.Origin); err != nil {
+ return err
+ }
+ }
+ return nil
+ },
+ }
+ addReviewConfigFlag(cmd)
+ cmd.Flags().StringP("locale", "l", "", "filter by target locale")
+ cmd.Flags().String("bundle", "", "filter by bundle ID")
+ cmd.Flags().String("status", "", "filter by review status (needs_review or approved)")
+ cmd.Flags().Bool("json", false, "output entries as JSON")
+ return cmd
+}
+
+func newReviewApproveCmd() *cobra.Command {
+ cmd := &cobra.Command{
+ Use: "approve",
+ Short: "Approve current translations after validation",
+ RunE: func(cmd *cobra.Command, args []string) error {
+ cfgPath, _ := cmd.Flags().GetString("config")
+ cfg, err := config.Load(cfgPath)
+ if err != nil {
+ return err
+ }
+ locale, _ := cmd.Flags().GetString("locale")
+ bundle, _ := cmd.Flags().GetString("bundle")
+ keys, _ := cmd.Flags().GetStringSlice("key")
+ all, _ := cmd.Flags().GetBool("all")
+ approved, err := review.Approve(cfg, review.Filter{Locale: locale, Bundle: bundle, Keys: keys, All: all}, time.Now())
+ if err != nil {
+ return err
+ }
+ _, err = fmt.Fprintf(cmd.OutOrStdout(), "Approved %d translation(s) for %s.\n", len(approved), locale)
+ return err
+ },
+ }
+ addReviewConfigFlag(cmd)
+ cmd.Flags().StringP("locale", "l", "", "target locale to approve")
+ cmd.Flags().String("bundle", "", "bundle ID (required with --key)")
+ cmd.Flags().StringSlice("key", nil, "individual translation key(s) to approve")
+ cmd.Flags().Bool("all", false, "approve every matching current translation")
+ _ = cmd.MarkFlagRequired("locale")
+ cmd.MarkFlagsMutuallyExclusive("key", "all")
+ return cmd
+}
+
+func addReviewConfigFlag(cmd *cobra.Command) {
+ cmd.Flags().StringP("config", "c", "", "path to config file (default: .internationalizer.yml)")
+}
diff --git a/cmd/internationalizer/validate.go b/cmd/internationalizer/validate.go
index b3e9987..d22d04d 100644
--- a/cmd/internationalizer/validate.go
+++ b/cmd/internationalizer/validate.go
@@ -20,7 +20,8 @@ func newValidateCmd() *cobra.Command {
Use --strict to require translated values and enforce extra-key, protected
structure, glossary, and configured plural rules. Use --require-state to verify
-that source, policy, and target content still match the translation manifest.`,
+that source, policy, and target content still match the translation manifest.
+Use --require-approved to additionally require explicit human approval.`,
RunE: func(cmd *cobra.Command, args []string) error {
cfgPath, _ := cmd.Flags().GetString("config")
cfg, err := config.Load(cfgPath)
@@ -30,9 +31,11 @@ that source, policy, and target content still match the translation manifest.`,
strict, _ := cmd.Flags().GetBool("strict")
requireState, _ := cmd.Flags().GetBool("require-state")
+ requireApproved, _ := cmd.Flags().GetBool("require-approved")
reports, err := validate.ValidateWithOptions(cfg, validate.Options{
- Strict: strict,
- RequireState: requireState,
+ Strict: strict,
+ RequireState: requireState,
+ RequireApproved: requireApproved,
})
if err != nil {
return err
@@ -65,6 +68,7 @@ that source, policy, and target content still match the translation manifest.`,
cmd.Flags().BoolP("quiet", "q", false, "exit code only, no output")
cmd.Flags().Bool("strict", false, "fail on untranslated values and strict policy findings")
cmd.Flags().Bool("require-state", false, "fail when translation manifest state is missing or stale")
+ cmd.Flags().Bool("require-approved", false, "fail unless current translations have explicit human approval")
return cmd
}
diff --git a/docs/localization-core-v2.md b/docs/localization-core-v2.md
index 1a3a816..848c7dc 100644
--- a/docs/localization-core-v2.md
+++ b/docs/localization-core-v2.md
@@ -31,16 +31,32 @@ framework runtime, translation CDN, or source-rewriting setup command.
## Review transfer
-Only an exact source-and-policy match may inherit review state. Fuzzy
-candidates, changed policy, or changed message structure require review.
+Manifest v2 records provenance origin and human review independently. Provider,
+translation-memory, adoption, and pseudo output begin in `needs_review`; only an
+exact current source, policy, and target artifact may be approved. Changed
+source context, policy, message structure, or target content requires review.
## Source units
-Structured documents and source adapters identify a unit by adapter,
-normalized project-relative path, semantic location, and source structure.
-Formatting offsets alone are not stable identity. Markdown uses the preamble
-and each H2 section as independent units, with invisible target-side markers to
-preserve identity when sections move or are inserted.
+Structured documents pass through a format-neutral semantic-unit boundary.
+Each unit has a stable semantic ID, kind, translator context, value, and
+adapter-owned structure signature; formatting offsets are not identity. Fluent
+resources use this boundary for message values, terms, and attributes while
+preserving comments and ordering during serialization.
+
+Markdown uses the preamble and each H2 section as independent units, with
+invisible target-side markers that preserve identity when sections move or are
+inserted.
+
+## Test locales and rich text
+
+- `pseudo` creates deterministic accented (`en-XA`) and bidirectional (`ar-XB`)
+ artifacts without a provider or translation-memory lookup.
+- ICU and Fluent runtime expressions remain intact while linguistic text is
+ transformed.
+- `data-l10n-name` identifies semantic rich-text slots. Translators may reorder
+ named slots, but element identity, protected attributes, nesting, and
+ contained markup must remain compatible.
## Policy identity
@@ -54,6 +70,6 @@ are never generated or rewritten by a translation run.
## Compatibility boundary
Existing JSON/YAML key workflows and i18next-v4 plural suffixes remain
-supported. Canonical locale comparison may reject configurations that currently
-spell the same locale more than once. ICU structural failures are deterministic
+supported. Canonical locale comparison may reject configurations that spell the
+same locale more than once. ICU or Fluent structural failures are deterministic
validation errors and block provider output from being written.
diff --git a/internal/fluentpattern/pattern.go b/internal/fluentpattern/pattern.go
new file mode 100644
index 0000000..ad46dc3
--- /dev/null
+++ b/internal/fluentpattern/pattern.go
@@ -0,0 +1,478 @@
+// Package fluentpattern validates and compares runtime-significant Fluent
+// pattern structure without owning resource-file serialization.
+package fluentpattern
+
+import (
+ "fmt"
+ "sort"
+ "strings"
+ "unicode"
+)
+
+// Analysis is the protected structure of one Fluent pattern.
+type Analysis struct {
+ Expressions []string
+ Selectors []Selector
+}
+
+// Selector is one Fluent select expression and its locale variants.
+type Selector struct {
+ Head string
+ DefaultKey string
+ Variants map[string]Analysis
+}
+
+// LooksLike reports whether text contains syntax specific to Fluent rather
+// than a plain ICU-style argument.
+func LooksLike(value string) bool {
+ for offset := 0; offset < len(value); {
+ start := strings.IndexByte(value[offset:], '{')
+ if start < 0 {
+ return false
+ }
+ start += offset
+ end, err := matchingBrace(value, start)
+ if err != nil {
+ return true
+ }
+ expression := strings.TrimSpace(value[start+1 : end])
+ if strings.Contains(expression, "->") || strings.ContainsAny(expression, "$.") || strings.HasPrefix(expression, "-") || strings.HasPrefix(expression, "\"") || strings.HasPrefix(expression, "'") {
+ return true
+ }
+ offset = end + 1
+ }
+ return false
+}
+
+// Analyze validates braces and select-expression structure and returns a
+// canonical representation of all non-linguistic expressions.
+func Analyze(value string) (Analysis, error) {
+ analysis := Analysis{}
+ for offset := 0; offset < len(value); {
+ relative := strings.IndexAny(value[offset:], "{}")
+ if relative < 0 {
+ break
+ }
+ start := offset + relative
+ if value[start] == '}' {
+ return Analysis{}, fmt.Errorf("unexpected closing brace at byte %d", start)
+ }
+ end, err := matchingBrace(value, start)
+ if err != nil {
+ return Analysis{}, err
+ }
+ expression := strings.TrimSpace(value[start+1 : end])
+ if expression == "" {
+ return Analysis{}, fmt.Errorf("empty Fluent placeable at byte %d", start)
+ }
+ arrow := topLevelArrow(expression)
+ if arrow < 0 {
+ analysis.Expressions = append(analysis.Expressions, canonicalExpression(expression))
+ } else {
+ selector, err := parseSelector(expression[:arrow], expression[arrow+2:])
+ if err != nil {
+ return Analysis{}, fmt.Errorf("fluent selector at byte %d: %w", start, err)
+ }
+ analysis.Selectors = append(analysis.Selectors, selector)
+ }
+ offset = end + 1
+ }
+ sort.Strings(analysis.Expressions)
+ sort.Slice(analysis.Selectors, func(left, right int) bool {
+ return selectorSignature(analysis.Selectors[left]) < selectorSignature(analysis.Selectors[right])
+ })
+ return analysis, nil
+}
+
+// Compare checks that target preserves source expressions and selectors.
+// Target-only locale variants are allowed when they preserve the source
+// selector's default-branch structure.
+func Compare(source, target string) (expected, actual []string, preserved bool, err error) {
+ sourceAnalysis, err := Analyze(source)
+ if err != nil {
+ return nil, nil, false, fmt.Errorf("source pattern: %w", err)
+ }
+ targetAnalysis, err := Analyze(target)
+ if err != nil {
+ return sourceAnalysis.Signatures(), nil, false, fmt.Errorf("target pattern: %w", err)
+ }
+ return sourceAnalysis.Signatures(), targetAnalysis.Signatures(), compareAnalysis(sourceAnalysis, targetAnalysis), nil
+}
+
+// TransformText applies transform to linguistic text while retaining Fluent
+// placeables, selector heads, variant keys, and default markers byte-for-byte.
+func TransformText(value string, transform func(string) string) (string, error) {
+ var result strings.Builder
+ offset := 0
+ for offset < len(value) {
+ relative := strings.IndexAny(value[offset:], "{}")
+ if relative < 0 {
+ result.WriteString(transform(value[offset:]))
+ break
+ }
+ start := offset + relative
+ if value[start] == '}' {
+ return "", fmt.Errorf("unexpected closing brace at byte %d", start)
+ }
+ result.WriteString(transform(value[offset:start]))
+ end, err := matchingBrace(value, start)
+ if err != nil {
+ return "", err
+ }
+ expression := value[start+1 : end]
+ arrow := topLevelArrow(expression)
+ if arrow < 0 {
+ result.WriteString(value[start : end+1])
+ } else {
+ body := expression[arrow+2:]
+ headers, err := variantHeaders(body)
+ if err != nil || len(headers) == 0 {
+ if err == nil {
+ err = fmt.Errorf("selector has no variants")
+ }
+ return "", err
+ }
+ result.WriteByte('{')
+ result.WriteString(expression[:arrow+2])
+ result.WriteString(body[:headers[0].end])
+ for index, header := range headers {
+ branchEnd := len(body)
+ if index+1 < len(headers) {
+ branchEnd = headers[index+1].start
+ }
+ transformed, err := TransformText(body[header.end:branchEnd], transform)
+ if err != nil {
+ return "", err
+ }
+ result.WriteString(transformed)
+ if index+1 < len(headers) {
+ result.WriteString(body[headers[index+1].start:headers[index+1].end])
+ }
+ }
+ result.WriteByte('}')
+ }
+ offset = end + 1
+ }
+ return result.String(), nil
+}
+
+// Signatures returns a deterministic diagnostic view of an analysis.
+func (analysis Analysis) Signatures() []string {
+ result := make([]string, 0, len(analysis.Expressions)+len(analysis.Selectors))
+ for _, expression := range analysis.Expressions {
+ result = append(result, "expression:"+expression)
+ }
+ for _, selector := range analysis.Selectors {
+ result = append(result, selectorSignature(selector))
+ }
+ sort.Strings(result)
+ return result
+}
+
+func compareAnalysis(source, target Analysis) bool {
+ if !equalList(source.Expressions, target.Expressions) || len(source.Selectors) != len(target.Selectors) {
+ return false
+ }
+ used := make([]bool, len(target.Selectors))
+ for _, sourceSelector := range source.Selectors {
+ matched := false
+ for index, targetSelector := range target.Selectors {
+ if used[index] || sourceSelector.Head != targetSelector.Head || !compareSelector(sourceSelector, targetSelector) {
+ continue
+ }
+ used[index] = true
+ matched = true
+ break
+ }
+ if !matched {
+ return false
+ }
+ }
+ return true
+}
+
+func compareSelector(source, target Selector) bool {
+ if source.DefaultKey != target.DefaultKey {
+ return false
+ }
+ fallback, ok := source.Variants[source.DefaultKey]
+ if !ok {
+ return false
+ }
+ for key, sourceBranch := range source.Variants {
+ targetBranch, exists := target.Variants[key]
+ if !exists || !compareAnalysis(sourceBranch, targetBranch) {
+ return false
+ }
+ }
+ for key, targetBranch := range target.Variants {
+ if _, exists := source.Variants[key]; exists {
+ continue
+ }
+ if !compareAnalysis(fallback, targetBranch) {
+ return false
+ }
+ }
+ return true
+}
+
+func parseSelector(head, body string) (Selector, error) {
+ head = canonicalExpression(strings.TrimSpace(head))
+ if head == "" {
+ return Selector{}, fmt.Errorf("selector expression is empty")
+ }
+ headers, err := variantHeaders(body)
+ if err != nil {
+ return Selector{}, err
+ }
+ if len(headers) == 0 {
+ return Selector{}, fmt.Errorf("selector has no variants")
+ }
+ selector := Selector{Head: head, Variants: make(map[string]Analysis)}
+ for index, header := range headers {
+ end := len(body)
+ if index+1 < len(headers) {
+ end = headers[index+1].start
+ }
+ branch, err := Analyze(body[header.end:end])
+ if err != nil {
+ return Selector{}, fmt.Errorf("variant %q: %w", header.key, err)
+ }
+ if _, duplicate := selector.Variants[header.key]; duplicate {
+ return Selector{}, fmt.Errorf("duplicate variant %q", header.key)
+ }
+ selector.Variants[header.key] = branch
+ if header.defaultVariant {
+ if selector.DefaultKey != "" {
+ return Selector{}, fmt.Errorf("selector has multiple default variants")
+ }
+ selector.DefaultKey = header.key
+ }
+ }
+ if selector.DefaultKey == "" {
+ return Selector{}, fmt.Errorf("selector has no default variant")
+ }
+ return selector, nil
+}
+
+type variantHeader struct {
+ start int
+ end int
+ key string
+ defaultVariant bool
+}
+
+func variantHeaders(body string) ([]variantHeader, error) {
+ var headers []variantHeader
+ depth := 0
+ var quote byte
+ escaped := false
+ lineStart := true
+ for index := 0; index < len(body); index++ {
+ character := body[index]
+ if quote != 0 {
+ if escaped {
+ escaped = false
+ continue
+ }
+ switch character {
+ case '\\':
+ escaped = true
+ case quote:
+ quote = 0
+ }
+ continue
+ }
+ if character == '\'' || character == '"' {
+ quote = character
+ continue
+ }
+ switch character {
+ case '{':
+ depth++
+ case '}':
+ if depth == 0 {
+ return nil, fmt.Errorf("unexpected closing brace in selector")
+ }
+ depth--
+ case '\n', '\r':
+ lineStart = true
+ continue
+ }
+ if depth != 0 {
+ continue
+ }
+ if lineStart && (character == ' ' || character == '\t') {
+ continue
+ }
+ if !lineStart {
+ continue
+ }
+ defaultVariant := false
+ start := index
+ if character == '*' {
+ defaultVariant = true
+ index++
+ if index >= len(body) {
+ return nil, fmt.Errorf("incomplete default variant")
+ }
+ character = body[index]
+ }
+ if character != '[' {
+ lineStart = false
+ continue
+ }
+ close := strings.IndexByte(body[index+1:], ']')
+ if close < 0 {
+ return nil, fmt.Errorf("unterminated variant key")
+ }
+ close += index + 1
+ key := strings.TrimSpace(body[index+1 : close])
+ if key == "" || strings.ContainsAny(key, "[]{}") {
+ return nil, fmt.Errorf("invalid variant key %q", key)
+ }
+ headers = append(headers, variantHeader{start: start, end: close + 1, key: key, defaultVariant: defaultVariant})
+ index = close
+ lineStart = false
+ }
+ if depth != 0 || quote != 0 {
+ return nil, fmt.Errorf("unterminated selector expression")
+ }
+ if len(headers) > 0 && strings.TrimSpace(body[:headers[0].start]) != "" {
+ return nil, fmt.Errorf("content appears before the first variant")
+ }
+ return headers, nil
+}
+
+func matchingBrace(value string, start int) (int, error) {
+ depth := 0
+ var quote byte
+ escaped := false
+ for index := start; index < len(value); index++ {
+ character := value[index]
+ if quote != 0 {
+ if escaped {
+ escaped = false
+ continue
+ }
+ switch character {
+ case '\\':
+ escaped = true
+ case quote:
+ quote = 0
+ }
+ continue
+ }
+ if character == '\'' || character == '"' {
+ quote = character
+ continue
+ }
+ switch character {
+ case '{':
+ depth++
+ case '}':
+ depth--
+ if depth == 0 {
+ return index, nil
+ }
+ }
+ }
+ return 0, fmt.Errorf("unterminated Fluent placeable at byte %d", start)
+}
+
+func topLevelArrow(expression string) int {
+ depth := 0
+ var quote byte
+ escaped := false
+ for index := 0; index+1 < len(expression); index++ {
+ character := expression[index]
+ if quote != 0 {
+ if escaped {
+ escaped = false
+ } else if character == '\\' {
+ escaped = true
+ } else if character == quote {
+ quote = 0
+ }
+ continue
+ }
+ if character == '\'' || character == '"' {
+ quote = character
+ continue
+ }
+ switch character {
+ case '{', '(':
+ depth++
+ case '}', ')':
+ if depth > 0 {
+ depth--
+ }
+ case '-':
+ if depth == 0 && expression[index+1] == '>' {
+ return index
+ }
+ }
+ }
+ return -1
+}
+
+func canonicalExpression(expression string) string {
+ var result strings.Builder
+ var quote rune
+ escaped := false
+ for _, character := range strings.TrimSpace(expression) {
+ if quote != 0 {
+ result.WriteRune(character)
+ if escaped {
+ escaped = false
+ } else if character == '\\' {
+ escaped = true
+ } else if character == quote {
+ quote = 0
+ }
+ continue
+ }
+ if character == '\'' || character == '"' {
+ quote = character
+ result.WriteRune(character)
+ continue
+ }
+ if !unicode.IsSpace(character) {
+ result.WriteRune(character)
+ }
+ }
+ return result.String()
+}
+
+func selectorSignature(selector Selector) string {
+ keys := make([]string, 0, len(selector.Variants))
+ for key := range selector.Variants {
+ keys = append(keys, key)
+ }
+ sort.Strings(keys)
+ var result strings.Builder
+ result.WriteString("selector:")
+ result.WriteString(selector.Head)
+ result.WriteString(":default=")
+ result.WriteString(selector.DefaultKey)
+ for _, key := range keys {
+ result.WriteString(":variant=")
+ result.WriteString(key)
+ result.WriteByte('[')
+ result.WriteString(strings.Join(selector.Variants[key].Signatures(), ","))
+ result.WriteByte(']')
+ }
+ return result.String()
+}
+
+func equalList(left, right []string) bool {
+ if len(left) != len(right) {
+ return false
+ }
+ for index := range left {
+ if left[index] != right[index] {
+ return false
+ }
+ }
+ return true
+}
diff --git a/internal/fluentpattern/pattern_test.go b/internal/fluentpattern/pattern_test.go
new file mode 100644
index 0000000..b78d4a5
--- /dev/null
+++ b/internal/fluentpattern/pattern_test.go
@@ -0,0 +1,88 @@
+package fluentpattern
+
+import (
+ "strings"
+ "testing"
+)
+
+func TestCompareAllowsLocaleSpecificVariantsWithDefaultStructure(t *testing.T) {
+ source := `{ $count ->
+ [one] One item for { $name }
+ *[other] { $count } items for { $name }
+}`
+ target := `{ $count ->
+ [one] Один элемент для { $name }
+ [few] { $count } элемента для { $name }
+ [many] { $count } элементов для { $name }
+ *[other] { $count } элемента для { $name }
+}`
+ _, _, preserved, err := Compare(source, target)
+ if err != nil || !preserved {
+ t.Fatalf("Compare() = preserved %v, error %v", preserved, err)
+ }
+}
+
+func TestCompareRejectsDamagedFluentRuntimeStructure(t *testing.T) {
+ source := `{ PLATFORM() ->
+ [windows] Open { -brand-short-name } for { $user }
+ *[other] Open { -brand-short-name } for { $user }
+}`
+ tests := map[string]string{
+ "selector": `{ $platform ->
+ [windows] Ouvrir { -brand-short-name } pour { $user }
+ *[other] Ouvrir { -brand-short-name } pour { $user }
+}`,
+ "reference": `{ PLATFORM() ->
+ [windows] Ouvrir { -brand-full-name } pour { $user }
+ *[other] Ouvrir { -brand-short-name } pour { $user }
+}`,
+ "variable": `{ PLATFORM() ->
+ [windows] Ouvrir { -brand-short-name } pour { $account }
+ *[other] Ouvrir { -brand-short-name } pour { $user }
+}`,
+ "default": `{ PLATFORM() ->
+ *[windows] Ouvrir { -brand-short-name } pour { $user }
+ [other] Ouvrir { -brand-short-name } pour { $user }
+}`,
+ }
+ for name, target := range tests {
+ t.Run(name, func(t *testing.T) {
+ _, _, preserved, err := Compare(source, target)
+ if err == nil && preserved {
+ t.Fatal("damaged Fluent structure was accepted")
+ }
+ })
+ }
+}
+
+func TestAnalyzeRejectsMalformedPatterns(t *testing.T) {
+ for _, pattern := range []string{
+ `Hello { $name`,
+ `Hello }`,
+ `{ $count -> [one] One }`,
+ "{ $count ->\n *[one] One\n *[other] Other\n}",
+ } {
+ if _, err := Analyze(pattern); err == nil {
+ t.Fatalf("Analyze(%q) accepted malformed pattern", pattern)
+ }
+ }
+}
+
+func TestTransformTextPreservesFluentSyntaxInsideSelectors(t *testing.T) {
+ input := `{ $count ->
+ [one] One item for { -brand-short-name }
+ *[other] { $count } items for { $user }
+}`
+ output, err := TransformText(input, strings.ToUpper)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, syntax := range []string{"{ $count ->", "[one]", "*[other]", "{ -brand-short-name }", "{ $count }", "{ $user }"} {
+ if !strings.Contains(output, syntax) {
+ t.Fatalf("transformed pattern lost %q: %s", syntax, output)
+ }
+ }
+ if !strings.Contains(output, "ONE ITEM FOR") || !strings.Contains(output, "ITEMS FOR ") {
+ t.Fatalf("literal text was not transformed: %s", output)
+ }
+}
diff --git a/internal/formats/fluent.go b/internal/formats/fluent.go
new file mode 100644
index 0000000..77b49aa
--- /dev/null
+++ b/internal/formats/fluent.go
@@ -0,0 +1,496 @@
+package formats
+
+import (
+ "bytes"
+ "fmt"
+ "sort"
+ "strings"
+ "unicode"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern"
+)
+
+// FluentFormat is a lossless adapter for Fluent Translation List resources.
+// It exposes message values, terms, and attributes as independent semantic
+// units while retaining comments and resource ordering in the source file.
+type FluentFormat struct{}
+
+func (f *FluentFormat) Name() string { return "fluent" }
+func (f *FluentFormat) Extensions() []string { return []string{".ftl"} }
+
+func (f *FluentFormat) Parse(data []byte) (map[string]string, error) {
+ units, err := f.ParseUnits(data)
+ if err != nil {
+ return nil, err
+ }
+ return UnitValues(units), nil
+}
+
+func (f *FluentFormat) Serialize(entries map[string]string, original []byte) ([]byte, error) {
+ units := make([]Unit, 0, len(entries))
+ keys := make([]string, 0, len(entries))
+ for key := range entries {
+ keys = append(keys, key)
+ }
+ sort.Strings(keys)
+ for _, key := range keys {
+ kind := UnitMessage
+ if strings.HasPrefix(key, "-") {
+ kind = UnitTerm
+ }
+ if strings.Contains(key, ".") {
+ kind = UnitAttribute
+ }
+ units = append(units, Unit{ID: key, Value: entries[key], Kind: kind})
+ }
+ return f.SerializeUnits(units, original)
+}
+
+func (f *FluentFormat) ParseUnits(data []byte) ([]Unit, error) {
+ document, err := parseFluentDocument(data)
+ if err != nil {
+ return nil, err
+ }
+ units := make([]Unit, 0, len(document.assignments))
+ for _, assignment := range document.assignments {
+ if assignment.value == "" {
+ continue
+ }
+ if _, err := fluentpattern.Analyze(assignment.value); err != nil {
+ return nil, fmt.Errorf("fluent parse: pattern %q: %w", assignment.id, err)
+ }
+ units = append(units, Unit{
+ ID: assignment.id,
+ Value: assignment.value,
+ Kind: assignment.kind,
+ Context: assignment.context,
+ Structure: "fluent-pattern-v1",
+ })
+ }
+ return units, nil
+}
+
+func (f *FluentFormat) SerializeUnits(units []Unit, original []byte) ([]byte, error) {
+ if err := ValidateUnits(units); err != nil {
+ return nil, err
+ }
+ document, err := parseFluentDocument(original)
+ if err != nil {
+ return nil, err
+ }
+
+ edits := make([]fluentEdit, 0)
+ missing := make([]Unit, 0)
+ for _, unit := range units {
+ assignment, ok := document.byID[unit.ID]
+ if !ok {
+ missing = append(missing, unit)
+ continue
+ }
+ if assignment.value == unit.Value {
+ continue
+ }
+ edits = append(edits, fluentEdit{
+ start: assignment.start,
+ end: assignment.end,
+ text: renderFluentAssignment(assignment, unit.Value, document.newline),
+ })
+ }
+
+ entryInsertions := make(map[string][]Unit)
+ newEntryOrder := make([]string, 0)
+ newEntries := make(map[string][]Unit)
+ for _, unit := range missing {
+ parent, _, attribute := fluentUnitIdentity(unit)
+ if attribute {
+ if _, exists := document.entries[parent]; exists {
+ entryInsertions[parent] = append(entryInsertions[parent], unit)
+ continue
+ }
+ } else if entry, exists := document.entries[parent]; exists {
+ // The resource has an attribute-only message. Its empty value
+ // assignment is addressable for serialization but not translation.
+ assignment := entry.main
+ edits = append(edits, fluentEdit{
+ start: assignment.start,
+ end: assignment.end,
+ text: renderFluentAssignment(assignment, unit.Value, document.newline),
+ })
+ continue
+ }
+ if _, exists := newEntries[parent]; !exists {
+ newEntryOrder = append(newEntryOrder, parent)
+ }
+ newEntries[parent] = append(newEntries[parent], unit)
+ }
+
+ for parent, inserted := range entryInsertions {
+ entry := document.entries[parent]
+ var text strings.Builder
+ if entry.end > 0 && original[entry.end-1] != '\n' {
+ text.WriteString(document.newline)
+ }
+ for _, unit := range inserted {
+ _, attribute, _ := fluentUnitIdentity(unit)
+ text.WriteString(renderNewFluentAssignment(entry.attributeIndent+"."+attribute+" =", unit.Value, entry.attributeIndent+" ", document.newline))
+ }
+ edits = append(edits, fluentEdit{start: entry.end, end: entry.end, text: text.String()})
+ }
+
+ if len(newEntryOrder) > 0 {
+ var appended strings.Builder
+ if len(original) > 0 {
+ if original[len(original)-1] != '\n' {
+ appended.WriteString(document.newline)
+ }
+ if !bytes.HasSuffix(original, []byte(document.newline+document.newline)) {
+ appended.WriteString(document.newline)
+ }
+ }
+ for index, parent := range newEntryOrder {
+ if index > 0 {
+ appended.WriteString(document.newline)
+ }
+ group := newEntries[parent]
+ context := ""
+ for _, unit := range group {
+ if unit.Context != "" {
+ context = unit.Context
+ break
+ }
+ }
+ if context != "" {
+ appended.WriteString(strings.TrimRight(context, "\r\n"))
+ appended.WriteString(document.newline)
+ }
+ var main *Unit
+ attributes := make([]Unit, 0)
+ for unitIndex := range group {
+ _, _, attribute := fluentUnitIdentity(group[unitIndex])
+ if attribute {
+ attributes = append(attributes, group[unitIndex])
+ } else {
+ main = &group[unitIndex]
+ }
+ }
+ if main != nil {
+ appended.WriteString(renderNewFluentAssignment(parent+" =", main.Value, " ", document.newline))
+ } else {
+ appended.WriteString(parent + " =" + document.newline)
+ }
+ for _, attributeUnit := range attributes {
+ _, attribute, _ := fluentUnitIdentity(attributeUnit)
+ appended.WriteString(renderNewFluentAssignment(" ."+attribute+" =", attributeUnit.Value, " ", document.newline))
+ }
+ }
+ edits = append(edits, fluentEdit{start: len(original), end: len(original), text: appended.String()})
+ }
+
+ sort.SliceStable(edits, func(left, right int) bool {
+ if edits[left].start != edits[right].start {
+ return edits[left].start > edits[right].start
+ }
+ return edits[left].end > edits[right].end
+ })
+ result := append([]byte(nil), original...)
+ for _, edit := range edits {
+ result = append(result[:edit.start], append([]byte(edit.text), result[edit.end:]...)...)
+ }
+ return result, nil
+}
+
+type fluentDocument struct {
+ assignments []fluentAssignment
+ byID map[string]fluentAssignment
+ entries map[string]fluentEntry
+ newline string
+}
+
+type fluentEntry struct {
+ id string
+ end int
+ attributeIndent string
+ main fluentAssignment
+}
+
+type fluentAssignment struct {
+ id string
+ parent string
+ attribute string
+ kind UnitKind
+ context string
+ value string
+ header string
+ indent string
+ start int
+ end int
+}
+
+type fluentLine struct {
+ start int
+ end int
+ content string
+}
+
+type fluentEdit struct {
+ start int
+ end int
+ text string
+}
+
+func parseFluentDocument(data []byte) (fluentDocument, error) {
+ document := fluentDocument{
+ byID: make(map[string]fluentAssignment),
+ entries: make(map[string]fluentEntry),
+ newline: "\n",
+ }
+ if bytes.Contains(data, []byte("\r\n")) {
+ document.newline = "\r\n"
+ }
+ lines := splitFluentLines(data)
+ for index := 0; index < len(lines); {
+ parent, equals, ok := parseFluentMessageHeader(lines[index].content)
+ if !ok {
+ index++
+ continue
+ }
+ if _, duplicate := document.entries[parent]; duplicate {
+ return fluentDocument{}, fmt.Errorf("fluent parse: duplicate message or term %q", parent)
+ }
+ entryEndLine := index + 1
+ for entryEndLine < len(lines) && isFluentContinuation(lines[entryEndLine].content) {
+ entryEndLine++
+ }
+ context := fluentCommentContext(lines, index, document.newline)
+ attributeStarts := make([]int, 0)
+ for candidate := index + 1; candidate < entryEndLine; candidate++ {
+ if _, _, _, attribute := parseFluentAttributeHeader(lines[candidate].content); attribute {
+ attributeStarts = append(attributeStarts, candidate)
+ }
+ }
+ boundaries := append([]int{index}, attributeStarts...)
+ boundaries = append(boundaries, entryEndLine)
+ entry := fluentEntry{id: parent, end: lines[entryEndLine-1].end, attributeIndent: " "}
+ for boundary := 0; boundary < len(boundaries)-1; boundary++ {
+ startLine := boundaries[boundary]
+ endLine := boundaries[boundary+1]
+ assignmentParent := parent
+ attribute := ""
+ assignmentEquals := equals
+ kind := UnitMessage
+ if startLine != index {
+ indent, name, attrEquals, _ := parseFluentAttributeHeader(lines[startLine].content)
+ attribute = name
+ assignmentEquals = attrEquals
+ entry.attributeIndent = indent
+ kind = UnitAttribute
+ }
+ if strings.HasPrefix(parent, "-") && attribute == "" {
+ kind = UnitTerm
+ }
+ id := parent
+ if attribute != "" {
+ id += "." + attribute
+ }
+ if _, duplicate := document.byID[id]; duplicate {
+ return fluentDocument{}, fmt.Errorf("fluent parse: duplicate translation unit %q", id)
+ }
+ assignment := buildFluentAssignment(lines, startLine, endLine, assignmentEquals, id, assignmentParent, attribute, kind, context)
+ document.assignments = append(document.assignments, assignment)
+ document.byID[id] = assignment
+ if attribute == "" {
+ entry.main = assignment
+ }
+ }
+ document.entries[parent] = entry
+ index = entryEndLine
+ }
+ return document, nil
+}
+
+func buildFluentAssignment(lines []fluentLine, startLine, endLine, equals int, id, parent, attribute string, kind UnitKind, context string) fluentAssignment {
+ line := lines[startLine]
+ header := line.content[:equals+1]
+ first := strings.TrimLeftFunc(line.content[equals+1:], unicode.IsSpace)
+ continuations := make([]string, 0, endLine-startLine-1)
+ minimumIndent := -1
+ for index := startLine + 1; index < endLine; index++ {
+ content := lines[index].content
+ if strings.TrimSpace(content) == "" {
+ continuations = append(continuations, "")
+ continue
+ }
+ indent := leadingWhitespaceLength(content)
+ if indent > 0 && (minimumIndent < 0 || indent < minimumIndent) {
+ minimumIndent = indent
+ }
+ continuations = append(continuations, content)
+ }
+ if minimumIndent < 0 {
+ minimumIndent = 4
+ if attribute != "" {
+ minimumIndent = leadingWhitespaceLength(line.content) + 4
+ }
+ }
+ for index, continuation := range continuations {
+ if leadingWhitespaceLength(continuation) >= minimumIndent {
+ continuations[index] = continuation[minimumIndent:]
+ }
+ }
+ parts := continuations
+ if first != "" {
+ parts = append([]string{first}, continuations...)
+ }
+ value := strings.TrimRight(strings.Join(parts, "\n"), "\n")
+ indent := strings.Repeat(" ", minimumIndent)
+ if startLine+1 < endLine {
+ candidate := lines[startLine+1].content
+ if len(candidate) >= minimumIndent {
+ indent = candidate[:minimumIndent]
+ }
+ }
+ return fluentAssignment{
+ id: id, parent: parent, attribute: attribute, kind: kind, context: context,
+ value: value, header: header, indent: indent, start: line.start,
+ end: lines[endLine-1].end,
+ }
+}
+
+func renderFluentAssignment(assignment fluentAssignment, value, newline string) string {
+ return renderNewFluentAssignment(assignment.header, value, assignment.indent, newline)
+}
+
+func renderNewFluentAssignment(header, value, indent, newline string) string {
+ if !strings.Contains(value, "\n") && value != "" {
+ return header + " " + value + newline
+ }
+ var result strings.Builder
+ result.WriteString(header)
+ result.WriteString(newline)
+ for _, line := range strings.Split(value, "\n") {
+ result.WriteString(indent)
+ result.WriteString(line)
+ result.WriteString(newline)
+ }
+ return result.String()
+}
+
+func splitFluentLines(data []byte) []fluentLine {
+ if len(data) == 0 {
+ return nil
+ }
+ lines := make([]fluentLine, 0, bytes.Count(data, []byte{'\n'})+1)
+ for start := 0; start < len(data); {
+ end := bytes.IndexByte(data[start:], '\n')
+ if end < 0 {
+ end = len(data)
+ } else {
+ end += start + 1
+ }
+ contentEnd := end
+ if contentEnd > start && data[contentEnd-1] == '\n' {
+ contentEnd--
+ }
+ if contentEnd > start && data[contentEnd-1] == '\r' {
+ contentEnd--
+ }
+ lines = append(lines, fluentLine{start: start, end: end, content: string(data[start:contentEnd])})
+ start = end
+ }
+ return lines
+}
+
+func parseFluentMessageHeader(line string) (string, int, bool) {
+ if line == "" || unicode.IsSpace(rune(line[0])) {
+ return "", 0, false
+ }
+ equals := strings.IndexByte(line, '=')
+ if equals < 0 {
+ return "", 0, false
+ }
+ name := strings.TrimSpace(line[:equals])
+ if !validFluentIdentifier(name, true) {
+ return "", 0, false
+ }
+ return name, equals, true
+}
+
+func parseFluentAttributeHeader(line string) (string, string, int, bool) {
+ indentLength := leadingWhitespaceLength(line)
+ if indentLength == 0 || indentLength >= len(line) || line[indentLength] != '.' {
+ return "", "", 0, false
+ }
+ equals := strings.IndexByte(line[indentLength+1:], '=')
+ if equals < 0 {
+ return "", "", 0, false
+ }
+ equals += indentLength + 1
+ name := strings.TrimSpace(line[indentLength+1 : equals])
+ if !validFluentIdentifier(name, false) {
+ return "", "", 0, false
+ }
+ return line[:indentLength], name, equals, true
+}
+
+func validFluentIdentifier(value string, allowTerm bool) bool {
+ if allowTerm && strings.HasPrefix(value, "-") {
+ value = value[1:]
+ }
+ if value == "" || !isASCIILetter(value[0]) {
+ return false
+ }
+ for index := 1; index < len(value); index++ {
+ if !isASCIILetter(value[index]) && !isASCIIDigit(value[index]) && value[index] != '-' && value[index] != '_' {
+ return false
+ }
+ }
+ return true
+}
+
+func isFluentContinuation(line string) bool {
+ return line != "" && (unicode.IsSpace(rune(line[0])) || strings.TrimSpace(line) == "}")
+}
+
+func leadingWhitespaceLength(value string) int {
+ for index, character := range value {
+ if !unicode.IsSpace(character) {
+ return index
+ }
+ }
+ return len(value)
+}
+
+func fluentCommentContext(lines []fluentLine, entryIndex int, newline string) string {
+ start := entryIndex
+ for start > 0 {
+ candidate := strings.TrimSpace(lines[start-1].content)
+ if !strings.HasPrefix(candidate, "#") {
+ break
+ }
+ start--
+ }
+ if start == entryIndex {
+ return ""
+ }
+ comments := make([]string, 0, entryIndex-start)
+ for index := start; index < entryIndex; index++ {
+ comments = append(comments, lines[index].content)
+ }
+ return strings.Join(comments, newline)
+}
+
+func fluentUnitIdentity(unit Unit) (parent, attribute string, isAttribute bool) {
+ if unit.Kind != UnitAttribute {
+ return unit.ID, "", false
+ }
+ separator := strings.LastIndexByte(unit.ID, '.')
+ if separator < 1 || separator == len(unit.ID)-1 {
+ return unit.ID, "", false
+ }
+ return unit.ID[:separator], unit.ID[separator+1:], true
+}
+
+func isASCIILetter(value byte) bool {
+ return value >= 'A' && value <= 'Z' || value >= 'a' && value <= 'z'
+}
+func isASCIIDigit(value byte) bool { return value >= '0' && value <= '9' }
diff --git a/internal/formats/fluent_test.go b/internal/formats/fluent_test.go
new file mode 100644
index 0000000..8437bc2
--- /dev/null
+++ b/internal/formats/fluent_test.go
@@ -0,0 +1,208 @@
+package formats
+
+import (
+ "os"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "testing"
+)
+
+const fluentFixture = `## Account panel
+
+# Shown when the user has unread messages.
+inbox-status =
+ { $count ->
+ [one] One unread message
+ *[other] { $count } unread messages
+ }
+ .aria-label = Inbox status
+
+save-button =
+ .label = Save
+ .accesskey = S
+
+-brand-short-name = Acme
+welcome = Welcome to { -brand-short-name }.
+`
+
+func TestFluentParseUnitsPreservesSemanticKindsAndContext(t *testing.T) {
+ units, err := ParseUnits(&FluentFormat{}, []byte(fluentFixture))
+ if err != nil {
+ t.Fatal(err)
+ }
+ wantIDs := []string{"inbox-status", "inbox-status.aria-label", "save-button.label", "save-button.accesskey", "-brand-short-name", "welcome"}
+ if len(units) != len(wantIDs) {
+ t.Fatalf("units = %#v", units)
+ }
+ for index, want := range wantIDs {
+ if units[index].ID != want || units[index].Structure != "fluent-pattern-v1" {
+ t.Fatalf("unit %d = %#v, want ID %q", index, units[index], want)
+ }
+ }
+ if units[0].Kind != UnitMessage || units[1].Kind != UnitAttribute || units[4].Kind != UnitTerm {
+ t.Fatalf("unexpected semantic kinds: %#v", units)
+ }
+ if units[0].Context != "# Shown when the user has unread messages." {
+ t.Fatalf("context = %q", units[0].Context)
+ }
+ if !strings.Contains(units[0].Value, "{ $count ->") || !strings.Contains(units[0].Value, "*[other]") {
+ t.Fatalf("selector pattern = %q", units[0].Value)
+ }
+ if units[2].ID != "save-button.label" {
+ t.Fatalf("attribute-only message produced unexpected units: %#v", units)
+ }
+}
+
+func TestFluentUnchangedSerializationIsLossless(t *testing.T) {
+ format := &FluentFormat{}
+ units, err := format.ParseUnits([]byte(fluentFixture))
+ if err != nil {
+ t.Fatal(err)
+ }
+ output, err := format.SerializeUnits(units, []byte(fluentFixture))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if string(output) != fluentFixture {
+ t.Fatalf("unchanged round trip drifted:\n%s", output)
+ }
+}
+
+func TestFluentSerializationChangesValuesWithoutLosingResourceStructure(t *testing.T) {
+ format := &FluentFormat{}
+ units, err := format.ParseUnits([]byte(fluentFixture))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for index := range units {
+ switch units[index].ID {
+ case "save-button.label":
+ units[index].Value = "Enregistrer"
+ case "inbox-status":
+ units[index].Value = "{ $count ->\n [one] Un message non lu\n *[other] { $count } messages non lus\n}"
+ }
+ }
+ output, err := format.SerializeUnits(units, []byte(fluentFixture))
+ if err != nil {
+ t.Fatal(err)
+ }
+ text := string(output)
+ for _, preserved := range []string{"## Account panel", "# Shown when", ".accesskey = S", "-brand-short-name = Acme"} {
+ if !strings.Contains(text, preserved) {
+ t.Fatalf("serialized resource lost %q:\n%s", preserved, text)
+ }
+ }
+ reparsed, err := format.ParseUnits(output)
+ if err != nil {
+ t.Fatal(err)
+ }
+ values := UnitValues(reparsed)
+ if values["save-button.label"] != "Enregistrer" || !strings.Contains(values["inbox-status"], "Un message non lu") {
+ t.Fatalf("reparsed values = %#v", values)
+ }
+}
+
+func TestFluentSerializationAppendsNewSourceEntriesInSourceOrder(t *testing.T) {
+ format := &FluentFormat{}
+ target := []byte("existing = Existant\n")
+ baseline, err := format.ParseUnits(target)
+ if err != nil {
+ t.Fatal(err)
+ }
+ source, err := format.ParseUnits([]byte("existing = Existing\n\n# First note\nfirst = First\nsecond =\n .label = Second\n"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ values := map[string]string{"existing": "Existant", "first": "Premier", "second.label": "Deuxième"}
+ merged := MergeUnitValues(baseline, source, values)
+ if got := []string{merged[0].ID, merged[1].ID, merged[2].ID}; !reflect.DeepEqual(got, []string{"existing", "first", "second.label"}) {
+ t.Fatalf("merged order = %v", got)
+ }
+ output, err := format.SerializeUnits(merged, target)
+ if err != nil {
+ t.Fatal(err)
+ }
+ text := string(output)
+ if strings.Index(text, "first = Premier") > strings.Index(text, "second =") || !strings.Contains(text, "# First note") {
+ t.Fatalf("new entries were not appended in source order with context:\n%s", text)
+ }
+ parsed, err := format.ParseUnits(output)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(UnitValues(parsed), values) {
+ t.Fatalf("parsed appended values = %#v, want %#v", UnitValues(parsed), values)
+ }
+}
+
+func TestFluentRejectsDuplicateUnitIdentity(t *testing.T) {
+ _, err := (&FluentFormat{}).ParseUnits([]byte("same = One\nsame = Two\n"))
+ if err == nil || !strings.Contains(err.Error(), "duplicate") {
+ t.Fatalf("ParseUnits() error = %v", err)
+ }
+}
+
+func TestFluentFormatIsRegistered(t *testing.T) {
+ format, err := FormatForFile("browser.ftl")
+ if err != nil || format.Name() != "fluent" {
+ t.Fatalf("FormatForFile() = %v, %v", format, err)
+ }
+}
+
+func TestFluentOptionalCorpusRoundTrip(t *testing.T) {
+ root := os.Getenv("INTERNATIONALIZER_FLUENT_CORPUS")
+ if root == "" {
+ t.Skip("set INTERNATIONALIZER_FLUENT_CORPUS to a directory of .ftl resources")
+ }
+ format := &FluentFormat{}
+ files := 0
+ err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ if entry.IsDir() || filepath.Ext(path) != ".ftl" {
+ return nil
+ }
+ clean := filepath.ToSlash(path)
+ if strings.Contains(clean, "/test/") || strings.Contains(clean, "/tests/") {
+ return nil
+ }
+ files++
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ units, err := format.ParseUnits(data)
+ if err != nil {
+ return &fluentCorpusError{path: path, err: err}
+ }
+ output, err := format.SerializeUnits(units, data)
+ if err != nil {
+ return &fluentCorpusError{path: path, err: err}
+ }
+ if !reflect.DeepEqual(output, data) {
+ return &fluentCorpusError{path: path, err: errFluentRoundTripDrift}
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if files == 0 {
+ t.Fatalf("no .ftl files found under %s", root)
+ }
+}
+
+type fluentCorpusError struct {
+ path string
+ err error
+}
+
+func (err *fluentCorpusError) Error() string { return err.path + ": " + err.err.Error() }
+
+var errFluentRoundTripDrift = &fluentRoundTripError{}
+
+type fluentRoundTripError struct{}
+
+func (*fluentRoundTripError) Error() string { return "unchanged serialization drifted" }
diff --git a/internal/formats/formats.go b/internal/formats/formats.go
index a69f4ba..32f6c1e 100644
--- a/internal/formats/formats.go
+++ b/internal/formats/formats.go
@@ -36,6 +36,7 @@ var registry = []Format{
&JSONFormat{},
&YAMLFormat{},
&MarkdownFormat{},
+ &FluentFormat{},
}
// FormatForFile returns the appropriate format handler for a file path.
diff --git a/internal/formats/units.go b/internal/formats/units.go
new file mode 100644
index 0000000..73a8924
--- /dev/null
+++ b/internal/formats/units.go
@@ -0,0 +1,142 @@
+package formats
+
+import (
+ "fmt"
+ "sort"
+)
+
+// UnitKind describes the semantic role of one independently translatable unit.
+// Adapters may add format-specific structure while the translation pipeline
+// continues to address units by stable ID.
+type UnitKind string
+
+const (
+ UnitMessage UnitKind = "message"
+ UnitDocument UnitKind = "document"
+ UnitAttribute UnitKind = "attribute"
+ UnitTerm UnitKind = "term"
+)
+
+// Unit is the format-neutral boundary between source adapters and translation.
+// Context is translator-facing information; Structure is a deterministic
+// adapter-owned signature used to detect semantic message changes.
+type Unit struct {
+ ID string `json:"id"`
+ Value string `json:"value"`
+ Kind UnitKind `json:"kind"`
+ Context string `json:"context,omitempty"`
+ Structure string `json:"structure,omitempty"`
+}
+
+// UnitFormat is implemented by formats with richer semantics than a flat
+// key/value catalog. Existing formats automatically receive a compatibility
+// adapter through ParseUnits and SerializeUnits.
+type UnitFormat interface {
+ ParseUnits(data []byte) ([]Unit, error)
+ SerializeUnits(units []Unit, original []byte) ([]byte, error)
+}
+
+// ParseUnits parses a document into stable semantic translation units.
+func ParseUnits(format Format, data []byte) ([]Unit, error) {
+ if adapter, ok := format.(UnitFormat); ok {
+ units, err := adapter.ParseUnits(data)
+ if err != nil {
+ return nil, err
+ }
+ if err := ValidateUnits(units); err != nil {
+ return nil, err
+ }
+ return units, nil
+ }
+ entries, err := format.Parse(data)
+ if err != nil {
+ return nil, err
+ }
+ keys := make([]string, 0, len(entries))
+ for key := range entries {
+ keys = append(keys, key)
+ }
+ sort.Strings(keys)
+ units := make([]Unit, 0, len(keys))
+ for _, key := range keys {
+ units = append(units, Unit{ID: key, Value: entries[key], Kind: UnitMessage})
+ }
+ return units, nil
+}
+
+// SerializeUnits writes semantic units through a rich adapter or the legacy
+// key/value serializer when the format has no additional structure.
+func SerializeUnits(format Format, units []Unit, original []byte) ([]byte, error) {
+ if err := ValidateUnits(units); err != nil {
+ return nil, err
+ }
+ if adapter, ok := format.(UnitFormat); ok {
+ return adapter.SerializeUnits(units, original)
+ }
+ return format.Serialize(UnitValues(units), original)
+}
+
+// UnitValues returns the compatibility key/value view of semantic units.
+func UnitValues(units []Unit) map[string]string {
+ values := make(map[string]string, len(units))
+ for _, unit := range units {
+ values[unit.ID] = unit.Value
+ }
+ return values
+}
+
+// MergeUnitValues preserves baseline unit metadata while replacing values and
+// appending source units absent from a new target document.
+func MergeUnitValues(baseline, source []Unit, values map[string]string) []Unit {
+ merged := make([]Unit, 0, len(values))
+ seen := make(map[string]struct{}, len(values))
+ for _, unit := range baseline {
+ value, ok := values[unit.ID]
+ if !ok {
+ continue
+ }
+ unit.Value = value
+ merged = append(merged, unit)
+ seen[unit.ID] = struct{}{}
+ }
+ for _, sourceUnit := range source {
+ if _, ok := seen[sourceUnit.ID]; ok {
+ continue
+ }
+ value, ok := values[sourceUnit.ID]
+ if !ok {
+ continue
+ }
+ sourceUnit.Value = value
+ merged = append(merged, sourceUnit)
+ seen[sourceUnit.ID] = struct{}{}
+ }
+ missing := make([]string, 0)
+ for id := range values {
+ if _, ok := seen[id]; !ok {
+ missing = append(missing, id)
+ }
+ }
+ sort.Strings(missing)
+ for _, id := range missing {
+ unit := Unit{ID: id, Kind: UnitMessage}
+ unit.Value = values[id]
+ merged = append(merged, unit)
+ }
+ return merged
+}
+
+// ValidateUnits enforces stable non-empty unique unit identities.
+func ValidateUnits(units []Unit) error {
+ seen := make(map[string]struct{}, len(units))
+ for _, unit := range units {
+ if unit.ID == "" {
+ return fmt.Errorf("translation unit ID must not be empty")
+ }
+ if _, duplicate := seen[unit.ID]; duplicate {
+ return fmt.Errorf("duplicate translation unit ID %q", unit.ID)
+ }
+ seen[unit.ID] = struct{}{}
+ }
+ return nil
+}
diff --git a/internal/formats/units_test.go b/internal/formats/units_test.go
new file mode 100644
index 0000000..3e46e60
--- /dev/null
+++ b/internal/formats/units_test.go
@@ -0,0 +1,48 @@
+package formats
+
+import (
+ "reflect"
+ "testing"
+)
+
+func TestParseUnitsAdaptsLegacyFormatsDeterministically(t *testing.T) {
+ units, err := ParseUnits(&JSONFormat{}, []byte(`{"z":"Last","a":"First"}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []Unit{
+ {ID: "a", Value: "First", Kind: UnitMessage},
+ {ID: "z", Value: "Last", Kind: UnitMessage},
+ }
+ if !reflect.DeepEqual(units, want) {
+ t.Fatalf("units = %#v, want %#v", units, want)
+ }
+}
+
+func TestMarkdownUsesStableSectionUnits(t *testing.T) {
+ units, err := ParseUnits(&MarkdownFormat{}, []byte("# Hello\n"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(units) != 1 || units[0].ID != markdownPreambleKey || units[0].Kind != UnitMessage {
+ t.Fatalf("units = %#v", units)
+ }
+}
+
+func TestMergeUnitValuesPreservesMetadataAndSourceUnits(t *testing.T) {
+ baseline := []Unit{{ID: "title", Value: "Ancien", Kind: UnitAttribute, Context: "window title"}}
+ source := []Unit{
+ {ID: "title", Value: "Title", Kind: UnitAttribute, Context: "window title"},
+ {ID: "body", Value: "Body", Kind: UnitMessage, Structure: "text"},
+ }
+ merged := MergeUnitValues(baseline, source, map[string]string{"title": "Nouveau", "body": "Corps"})
+ if len(merged) != 2 || merged[0].Value != "Nouveau" || merged[0].Context != "window title" || merged[1].ID != "body" || merged[1].Structure != "text" {
+ t.Fatalf("merged = %#v", merged)
+ }
+}
+
+func TestValidateUnitsRejectsDuplicateIdentity(t *testing.T) {
+ if err := ValidateUnits([]Unit{{ID: "same"}, {ID: "same"}}); err == nil {
+ t.Fatal("ValidateUnits accepted duplicate identity")
+ }
+}
diff --git a/internal/llm/anthropic.go b/internal/llm/anthropic.go
index 876cbc3..880f7bc 100644
--- a/internal/llm/anthropic.go
+++ b/internal/llm/anthropic.go
@@ -34,11 +34,7 @@ func (a *Anthropic) Name() string { return "anthropic" }
func (a *Anthropic) Translate(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) {
// Build the user message as a JSON object of keys to translate.
- input := make(map[string]string, len(req.Entries))
- for _, e := range req.Entries {
- input[e.Key] = e.Value
- }
- inputJSON, err := json.Marshal(input)
+ inputJSON, err := marshalEntries(req.Entries)
if err != nil {
return nil, fmt.Errorf("marshal input: %w", err)
}
diff --git a/internal/llm/gemini.go b/internal/llm/gemini.go
index 965a5e8..5186481 100644
--- a/internal/llm/gemini.go
+++ b/internal/llm/gemini.go
@@ -33,11 +33,7 @@ func NewGemini(apiKey, model string) *Gemini {
func (g *Gemini) Name() string { return "gemini" }
func (g *Gemini) Translate(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) {
- input := make(map[string]string, len(req.Entries))
- for _, e := range req.Entries {
- input[e.Key] = e.Value
- }
- inputJSON, err := json.Marshal(input)
+ inputJSON, err := marshalEntries(req.Entries)
if err != nil {
return nil, fmt.Errorf("marshal input: %w", err)
}
diff --git a/internal/llm/openai.go b/internal/llm/openai.go
index aac037f..088f148 100644
--- a/internal/llm/openai.go
+++ b/internal/llm/openai.go
@@ -64,11 +64,7 @@ func NewOpenRouter(apiKey, model string) *OpenAI {
func (o *OpenAI) Name() string { return o.providerName }
func (o *OpenAI) Translate(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) {
- input := make(map[string]string, len(req.Entries))
- for _, e := range req.Entries {
- input[e.Key] = e.Value
- }
- inputJSON, err := json.Marshal(input)
+ inputJSON, err := marshalEntries(req.Entries)
if err != nil {
return nil, fmt.Errorf("marshal input: %w", err)
}
diff --git a/internal/llm/provider.go b/internal/llm/provider.go
index 7cf1225..3586a2b 100644
--- a/internal/llm/provider.go
+++ b/internal/llm/provider.go
@@ -2,6 +2,7 @@ package llm
import (
"context"
+ "encoding/json"
"fmt"
"strings"
@@ -11,8 +12,9 @@ import (
// Entry is a source key-value pair to be translated.
type Entry struct {
- Key string
- Value string
+ Key string
+ Value string
+ Context string
}
// TranslateRequest is the input to a translation call.
@@ -127,6 +129,34 @@ func BuildSystemPrompt(sourceLocale, targetLocale, styleGuide string, terms []gl
return b.String()
}
+// BuildFluentPrompt constructs the translation contract for semantic units
+// extracted from a Fluent resource.
+func BuildFluentPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term) string {
+ prompt := BuildSystemPrompt(sourceLocale, targetLocale, styleGuide, terms)
+ prompt += "\n## Fluent rules\n"
+ prompt += "- Input values with developer context are objects; translate only their value and still return a string.\n"
+ prompt += "- Developer context is authoritative translator guidance, not text to translate.\n"
+ prompt += "- Preserve Fluent variables, terms, message references, functions, and named markup slots exactly.\n"
+ prompt += "- Preserve every source selector variant and its default marker; add target-locale plural variants when required.\n"
+ prompt += "- Translate only the natural-language content inside each pattern.\n"
+ return prompt
+}
+
+func marshalEntries(entries []Entry) ([]byte, error) {
+ input := make(map[string]interface{}, len(entries))
+ for _, entry := range entries {
+ if entry.Context == "" {
+ input[entry.Key] = entry.Value
+ continue
+ }
+ input[entry.Key] = struct {
+ Value string `json:"value"`
+ Context string `json:"context"`
+ }{Value: entry.Value, Context: entry.Context}
+ }
+ return json.Marshal(input)
+}
+
// BuildDocumentPrompt constructs a prompt for whole-document translation (e.g. Markdown).
func BuildDocumentPrompt(sourceLocale, targetLocale, styleGuide string, terms []glossary.Term) string {
var b strings.Builder
diff --git a/internal/llm/provider_test.go b/internal/llm/provider_test.go
index 46abcbd..e042c93 100644
--- a/internal/llm/provider_test.go
+++ b/internal/llm/provider_test.go
@@ -1,6 +1,7 @@
package llm
import (
+ "encoding/json"
"strings"
"testing"
@@ -14,6 +15,36 @@ func TestBuildDocumentPromptKeepsProviderJSONContract(t *testing.T) {
}
}
+func TestBuildFluentPromptDefinesSemanticRuntimeContract(t *testing.T) {
+ prompt := BuildFluentPrompt("en", "fr", "", nil)
+ for _, rule := range []string{"developer context", "Fluent variables", "target-locale plural variants"} {
+ if !strings.Contains(prompt, rule) {
+ t.Fatalf("Fluent prompt lacks %q: %s", rule, prompt)
+ }
+ }
+}
+
+func TestMarshalEntriesIncludesDeveloperContextWithoutChangingOutputKey(t *testing.T) {
+ data, err := marshalEntries([]Entry{
+ {Key: "plain", Value: "Save"},
+ {Key: "guided", Value: "Open", Context: "Verb used on a button."},
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ var decoded map[string]interface{}
+ if err := json.Unmarshal(data, &decoded); err != nil {
+ t.Fatal(err)
+ }
+ if decoded["plain"] != "Save" {
+ t.Fatalf("plain entry = %#v", decoded["plain"])
+ }
+ guided, ok := decoded["guided"].(map[string]interface{})
+ if !ok || guided["value"] != "Open" || guided["context"] != "Verb used on a button." {
+ t.Fatalf("guided entry = %#v", decoded["guided"])
+ }
+}
+
func TestEffectiveReasoningEffort(t *testing.T) {
tests := []struct {
name string
diff --git a/internal/message/message.go b/internal/message/message.go
index 5eeeeca..2a90d9e 100644
--- a/internal/message/message.go
+++ b/internal/message/message.go
@@ -940,6 +940,31 @@ func (message *Message) String() string {
return output.String()
}
+// TransformText applies transform only to literal text while preserving ICU
+// arguments, selectors, formatter styles, plural offsets, and pound tokens.
+func TransformText(input string, transform func(string) string) (string, error) {
+ parsed, err := Parse(input)
+ if err != nil {
+ return "", err
+ }
+ transformMessageText(parsed, transform)
+ return parsed.String(), nil
+}
+
+func transformMessageText(message *Message, transform func(string) string) {
+ for index := range message.elements {
+ element := &message.elements[index]
+ switch element.kind {
+ case elementText:
+ element.text = transform(element.text)
+ case elementArgument:
+ for optionIndex := range element.argument.Options {
+ transformMessageText(element.argument.Options[optionIndex].Message, transform)
+ }
+ }
+ }
+}
+
func (message *Message) writeTo(output *strings.Builder, pluralContext bool) {
for _, element := range message.elements {
switch element.kind {
diff --git a/internal/policy/policy.go b/internal/policy/policy.go
index 3f30c67..be7a0ed 100644
--- a/internal/policy/policy.go
+++ b/internal/policy/policy.go
@@ -41,8 +41,11 @@ func Resolve(cfg *config.Config, targetLocale, format, styleGuide string, terms
effectiveLLM := cfg.LLMForLocale(canonicalTarget)
prompt := llm.BuildSystemPrompt(canonicalSource, canonicalTarget, styleGuide, terms)
- if format == "markdown" {
+ switch format {
+ case "markdown":
prompt = llm.BuildDocumentPrompt(canonicalSource, canonicalTarget, styleGuide, terms)
+ case "fluent":
+ prompt = llm.BuildFluentPrompt(canonicalSource, canonicalTarget, styleGuide, terms)
}
guideHash, err := state.HashValue(styleGuide)
diff --git a/internal/pseudo/generate.go b/internal/pseudo/generate.go
new file mode 100644
index 0000000..49fa1e5
--- /dev/null
+++ b/internal/pseudo/generate.go
@@ -0,0 +1,257 @@
+package pseudo
+
+import (
+ "bytes"
+ "fmt"
+ "os"
+ "strings"
+ "time"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ "github.com/Tom-R-Main/Internationalizer/internal/formats"
+ localeid "github.com/Tom-R-Main/Internationalizer/internal/locale"
+ "github.com/Tom-R-Main/Internationalizer/internal/state"
+)
+
+// GenerateOptions configures pseudolocale generation.
+type GenerateOptions struct {
+ Strategy Strategy
+ Locale string
+ Force bool
+ DryRun bool
+}
+
+// GenerateResult describes one generated bundle artifact.
+type GenerateResult struct {
+ Bundle string
+ Locale string
+ TargetPath string
+ Units int
+ Written bool
+}
+
+// Generate creates deterministic pseudolocale artifacts without an LLM or TM.
+func Generate(cfg *config.Config, opts GenerateOptions) ([]GenerateResult, error) {
+ effectiveConfig := *cfg
+ effectiveConfig.ApplyDefaults()
+ cfg = &effectiveConfig
+ if err := cfg.ValidateProject(); err != nil {
+ return nil, err
+ }
+ if _, err := DefaultLocale(opts.Strategy); err != nil {
+ return nil, err
+ }
+ locale := opts.Locale
+ if locale == "" {
+ var err error
+ locale, err = DefaultLocale(opts.Strategy)
+ if err != nil {
+ return nil, err
+ }
+ }
+ canonicalLocale, err := localeid.Canonical(locale)
+ if err != nil {
+ return nil, fmt.Errorf("invalid pseudo locale %q: %w", locale, err)
+ }
+
+ manifest, err := state.Load(cfg.ManifestPath)
+ if err != nil {
+ return nil, err
+ }
+ policyHash, err := state.HashValue(struct {
+ Version int `json:"version"`
+ Strategy Strategy `json:"strategy"`
+ }{Version: 1, Strategy: opts.Strategy})
+ if err != nil {
+ return nil, err
+ }
+
+ results := make([]GenerateResult, 0, len(cfg.EffectiveBundles()))
+ now := time.Now().UTC()
+ for _, bundle := range cfg.EffectiveBundles() {
+ format, err := formatForBundle(bundle)
+ if err != nil {
+ return nil, fmt.Errorf("bundle %q format: %w", bundle.ID, err)
+ }
+ sourceData, err := os.ReadFile(bundle.Source)
+ if err != nil {
+ return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err)
+ }
+ sourceUnits, err := formats.ParseUnits(format, sourceData)
+ if err != nil {
+ return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err)
+ }
+ targetPath, err := bundle.TargetPath(locale)
+ if err != nil {
+ return nil, err
+ }
+ if existing, readErr := os.ReadFile(targetPath); readErr == nil {
+ if !opts.Force && !pseudoOwnsArtifact(manifest, bundle.ID, canonicalLocale, format, sourceData, existing) {
+ return nil, fmt.Errorf("refusing to overwrite %s without --force because it is not a tracked pseudo artifact", targetPath)
+ }
+ } else if !os.IsNotExist(readErr) {
+ return nil, fmt.Errorf("reading pseudo target %s: %w", targetPath, readErr)
+ }
+
+ pseudoUnits := make([]formats.Unit, len(sourceUnits))
+ for index, unit := range sourceUnits {
+ transformed, err := transformUnit(format, unit, opts.Strategy)
+ if err != nil {
+ return nil, fmt.Errorf("pseudolocalizing bundle %q unit %q: %w", bundle.ID, unit.ID, err)
+ }
+ unit.Value = transformed
+ pseudoUnits[index] = unit
+ }
+ var output []byte
+ if paired, ok := format.(formats.PairedFormat); ok {
+ output, err = paired.SerializeTarget(formats.UnitValues(pseudoUnits), sourceData, sourceData)
+ } else {
+ output, err = formats.SerializeUnits(format, pseudoUnits, sourceData)
+ }
+ if err != nil {
+ return nil, fmt.Errorf("serializing pseudo target %s: %w", targetPath, err)
+ }
+ output = appendOneNewline(output)
+ serializedValues, err := parseTargetValues(format, sourceData, output)
+ if err != nil {
+ return nil, fmt.Errorf("validating pseudo target %s: %w", targetPath, err)
+ }
+ result := GenerateResult{Bundle: bundle.ID, Locale: canonicalLocale, TargetPath: targetPath, Units: len(pseudoUnits), Written: !opts.DryRun}
+ results = append(results, result)
+ if opts.DryRun {
+ continue
+ }
+ if err := state.WriteFileAtomic(targetPath, output, 0o644); err != nil {
+ return nil, err
+ }
+ for _, sourceUnit := range sourceUnits {
+ targetValue, ok := serializedValues[sourceUnit.ID]
+ if !ok {
+ return nil, fmt.Errorf("validating pseudo target %s: unit %q is missing after serialization", targetPath, sourceUnit.ID)
+ }
+ manifest.Set(state.Entry{
+ Bundle: bundle.ID,
+ Key: sourceUnit.ID,
+ Locale: canonicalLocale,
+ SourceHash: state.SourceUnitHash(format.Name(), sourceUnit.Value, sourceUnit.Context, sourceUnit.Structure),
+ PolicyHash: policyHash,
+ TargetHash: state.TargetHash(targetValue),
+ Origin: "pseudo",
+ ReviewStatus: state.ReviewNeedsReview,
+ UpdatedAt: now,
+ })
+ }
+ }
+ if !opts.DryRun {
+ if err := manifest.Save(cfg.ManifestPath); err != nil {
+ return nil, err
+ }
+ }
+ return results, nil
+}
+
+func formatForBundle(bundle config.Bundle) (formats.Format, error) {
+ if bundle.Format != "" {
+ return formats.FormatByName(bundle.Format)
+ }
+ return formats.FormatForFile(bundle.Source)
+}
+
+func transformUnit(format formats.Format, unit formats.Unit, strategy Strategy) (string, error) {
+ if format.Name() != "markdown" {
+ return Transform(unit.Value, strategy)
+ }
+ lineEnd := strings.IndexByte(unit.Value, '\n')
+ if lineEnd < 0 {
+ lineEnd = len(unit.Value)
+ }
+ line := strings.TrimSuffix(unit.Value[:lineEnd], "\r")
+ prefixEnd := 0
+ for prefixEnd < len(line) && (line[prefixEnd] == ' ' || line[prefixEnd] == '\t') {
+ prefixEnd++
+ }
+ headingStart := prefixEnd
+ for prefixEnd < len(line) && line[prefixEnd] == '#' && prefixEnd-headingStart < 6 {
+ prefixEnd++
+ }
+ if prefixEnd == headingStart || prefixEnd >= len(line) || (line[prefixEnd] != ' ' && line[prefixEnd] != '\t') {
+ return Transform(unit.Value, strategy)
+ }
+ for prefixEnd < len(line) && (line[prefixEnd] == ' ' || line[prefixEnd] == '\t') {
+ prefixEnd++
+ }
+ heading, err := Transform(line[prefixEnd:], strategy)
+ if err != nil {
+ return "", err
+ }
+ newline := ""
+ body := ""
+ if lineEnd < len(unit.Value) {
+ newline = "\n"
+ if lineEnd > 0 && unit.Value[lineEnd-1] == '\r' {
+ newline = "\r\n"
+ }
+ body = unit.Value[lineEnd+1:]
+ }
+ transformedBody, err := transformPadded(body, strategy)
+ if err != nil {
+ return "", err
+ }
+ return line[:prefixEnd] + heading + newline + transformedBody, nil
+}
+
+func transformPadded(value string, strategy Strategy) (string, error) {
+ start := 0
+ for start < len(value) && strings.ContainsRune(" \t\r\n", rune(value[start])) {
+ start++
+ }
+ end := len(value)
+ for end > start && strings.ContainsRune(" \t\r\n", rune(value[end-1])) {
+ end--
+ }
+ if start == end {
+ return value, nil
+ }
+ transformed, err := Transform(value[start:end], strategy)
+ if err != nil {
+ return "", err
+ }
+ return value[:start] + transformed + value[end:], nil
+}
+
+func pseudoOwnsArtifact(manifest *state.Manifest, bundle, locale string, format formats.Format, source, target []byte) bool {
+ sourceUnits, err := formats.ParseUnits(format, source)
+ if err != nil || len(sourceUnits) == 0 {
+ return false
+ }
+ targetValues, err := parseTargetValues(format, source, target)
+ if err != nil || len(targetValues) != len(sourceUnits) {
+ return false
+ }
+ for _, sourceUnit := range sourceUnits {
+ targetValue, ok := targetValues[sourceUnit.ID]
+ if !ok {
+ return false
+ }
+ entry, ok := manifest.Get(bundle, sourceUnit.ID, locale)
+ if !ok || entry.Origin != "pseudo" || entry.TargetHash != state.TargetHash(targetValue) {
+ return false
+ }
+ }
+ return true
+}
+
+func parseTargetValues(format formats.Format, source, target []byte) (map[string]string, error) {
+ if paired, ok := format.(formats.PairedFormat); ok {
+ return paired.ParseTarget(source, target)
+ }
+ targetUnits, err := formats.ParseUnits(format, target)
+ if err != nil {
+ return nil, err
+ }
+ return formats.UnitValues(targetUnits), nil
+}
+
+func appendOneNewline(data []byte) []byte {
+ return append(bytes.TrimRight(data, "\n"), '\n')
+}
diff --git a/internal/pseudo/generate_test.go b/internal/pseudo/generate_test.go
new file mode 100644
index 0000000..4f74991
--- /dev/null
+++ b/internal/pseudo/generate_test.go
@@ -0,0 +1,143 @@
+package pseudo
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ "github.com/Tom-R-Main/Internationalizer/internal/state"
+)
+
+func TestGenerateWritesTrackedPseudoArtifactAndCanRefreshIt(t *testing.T) {
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "locales", "en.json")
+ targetTemplate := filepath.Join(dir, "locales", "{locale}.json")
+ manifestPath := filepath.Join(dir, ".internationalizer.lock")
+ writeTestFile(t, sourcePath, `{"greeting":"Hello {name}","link":"Read docs"}`)
+ cfg := pseudoTestConfig(sourcePath, targetTemplate, manifestPath)
+
+ results, err := Generate(cfg, GenerateOptions{Strategy: Accented})
+ if err != nil {
+ t.Fatalf("Generate() error = %v", err)
+ }
+ if len(results) != 1 || !results[0].Written || results[0].Locale != "en-XA" || results[0].Units != 2 {
+ t.Fatalf("Generate() results = %#v", results)
+ }
+ targetPath := filepath.Join(dir, "locales", "en-XA.json")
+ data, err := os.ReadFile(targetPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ output := string(data)
+ if !strings.Contains(output, "{name}") || !strings.Contains(output, ``) || !strings.Contains(output, "[!!") {
+ t.Fatalf("pseudo output did not preserve runtime syntax: %s", output)
+ }
+ manifest, err := state.Load(manifestPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ entry, ok := manifest.Get("app", "greeting", "en-XA")
+ if !ok || entry.Origin != "pseudo" || entry.ReviewStatus != state.ReviewNeedsReview {
+ t.Fatalf("pseudo manifest entry = %#v, present = %v", entry, ok)
+ }
+
+ // A tracked, byte-identical pseudo artifact is safe to regenerate.
+ if _, err := Generate(cfg, GenerateOptions{Strategy: Accented}); err != nil {
+ t.Fatalf("second Generate() error = %v", err)
+ }
+}
+
+func TestGenerateRefusesUntrackedTargetUnlessForced(t *testing.T) {
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "en.json")
+ targetTemplate := filepath.Join(dir, "{locale}.json")
+ writeTestFile(t, sourcePath, `{"greeting":"Hello"}`)
+ writeTestFile(t, filepath.Join(dir, "en-XA.json"), `{"greeting":"Human text"}`)
+ cfg := pseudoTestConfig(sourcePath, targetTemplate, filepath.Join(dir, "manifest.json"))
+
+ _, err := Generate(cfg, GenerateOptions{Strategy: Accented})
+ if err == nil || !strings.Contains(err.Error(), "refusing to overwrite") {
+ t.Fatalf("Generate() error = %v, want overwrite refusal", err)
+ }
+ if _, err := Generate(cfg, GenerateOptions{Strategy: Accented, Force: true}); err != nil {
+ t.Fatalf("forced Generate() error = %v", err)
+ }
+}
+
+func TestGenerateDryRunDoesNotWriteArtifactOrManifest(t *testing.T) {
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "en.json")
+ targetTemplate := filepath.Join(dir, "{locale}.json")
+ manifestPath := filepath.Join(dir, "manifest.json")
+ writeTestFile(t, sourcePath, `{"greeting":"Hello"}`)
+ cfg := pseudoTestConfig(sourcePath, targetTemplate, manifestPath)
+
+ results, err := Generate(cfg, GenerateOptions{Strategy: Bidi, DryRun: true})
+ if err != nil {
+ t.Fatalf("Generate() error = %v", err)
+ }
+ if len(results) != 1 || results[0].Written || results[0].Locale != "ar-XB" {
+ t.Fatalf("Generate() results = %#v", results)
+ }
+ for _, path := range []string{filepath.Join(dir, "ar-XB.json"), manifestPath} {
+ if _, err := os.Stat(path); !os.IsNotExist(err) {
+ t.Fatalf("dry run unexpectedly wrote %s", path)
+ }
+ }
+}
+
+func TestGenerateWritesRefreshableMarkdownPseudoArtifact(t *testing.T) {
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "README.md")
+ targetTemplate := filepath.Join(dir, "docs", "{locale}.md")
+ manifestPath := filepath.Join(dir, "manifest.json")
+ writeTestFile(t, sourcePath, "# Project\n\n## Install\n\nRead [the guide](docs/guide.md).\n\n## Usage\n\nRun `internationalizer`.\n")
+ cfg := pseudoTestConfig(sourcePath, targetTemplate, manifestPath)
+ cfg.Bundles[0].Format = "markdown"
+
+ if _, err := Generate(cfg, GenerateOptions{Strategy: Accented}); err != nil {
+ t.Fatalf("first Generate() error = %v", err)
+ }
+ if _, err := Generate(cfg, GenerateOptions{Strategy: Accented}); err != nil {
+ t.Fatalf("refresh Generate() error = %v", err)
+ }
+ target, err := os.ReadFile(filepath.Join(dir, "docs", "en-XA.md"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ text := string(target)
+ if !strings.HasPrefix(text, "# [!! ") {
+ t.Fatalf("Markdown pseudo output did not preserve the document heading:\n%s", text)
+ }
+ for _, marker := range []string{"", ""} {
+ if !strings.Contains(text, marker) {
+ t.Fatalf("Markdown pseudo output lacks %q:\n%s", marker, text)
+ }
+ }
+}
+
+func pseudoTestConfig(source, target, manifest string) *config.Config {
+ return &config.Config{
+ SourceLocale: "en",
+ TargetLocales: []string{"fr"},
+ ManifestPath: manifest,
+ Bundles: []config.Bundle{{
+ ID: "app",
+ Source: source,
+ Target: target,
+ Format: "json",
+ }},
+ }
+}
+
+func writeTestFile(t *testing.T, path, contents string) {
+ t.Helper()
+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
+ t.Fatal(err)
+ }
+}
diff --git a/internal/pseudo/transform.go b/internal/pseudo/transform.go
new file mode 100644
index 0000000..c4279d9
--- /dev/null
+++ b/internal/pseudo/transform.go
@@ -0,0 +1,202 @@
+// Package pseudo generates deterministic accented and bidirectional test text.
+package pseudo
+
+import (
+ "fmt"
+ "regexp"
+ "sort"
+ "strings"
+ "unicode"
+ "unicode/utf8"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern"
+ "github.com/Tom-R-Main/Internationalizer/internal/message"
+)
+
+// Strategy selects a pseudolocalization transformation.
+type Strategy string
+
+const (
+ Accented Strategy = "accented"
+ Bidi Strategy = "bidi"
+)
+
+var protectedTokenRe = regexp.MustCompile("(?s)|```.*?```|~~~.*?~~~|`+[^`\\n]*`+|]*>|\\{\\{[A-Za-z0-9_.-]+\\}\\}|%\\{[A-Za-z0-9_.-]+\\}|\\{[A-Za-z0-9_.-]+\\}")
+
+type span struct {
+ start int
+ end int
+}
+
+// Transform transforms linguistic text while preserving runtime syntax.
+func Transform(input string, strategy Strategy) (string, error) {
+ if strategy != Accented && strategy != Bidi {
+ return "", fmt.Errorf("unsupported pseudolocalization strategy %q", strategy)
+ }
+ transformLiteral := func(value string) string {
+ return transformPreserving(value, strategy)
+ }
+ output := ""
+ var err error
+ if fluentpattern.LooksLike(input) {
+ output, err = fluentpattern.TransformText(input, transformLiteral)
+ if err != nil {
+ return "", fmt.Errorf("pseudolocalizing Fluent pattern: %w", err)
+ }
+ } else if message.LooksLike(input) {
+ output, err = message.TransformText(input, transformLiteral)
+ if err != nil {
+ return "", fmt.Errorf("pseudolocalizing ICU message: %w", err)
+ }
+ } else {
+ output = transformLiteral(input)
+ }
+ if strategy == Accented {
+ return "[!! " + output + " !!]", nil
+ }
+ return "\u2067" + output + "\u2069", nil
+}
+
+// DefaultLocale returns the conventional Unicode pseudo-locale for strategy.
+func DefaultLocale(strategy Strategy) (string, error) {
+ switch strategy {
+ case Accented:
+ return "en-XA", nil
+ case Bidi:
+ return "ar-XB", nil
+ default:
+ return "", fmt.Errorf("unsupported pseudolocalization strategy %q", strategy)
+ }
+}
+
+func transformPreserving(input string, strategy Strategy) string {
+ spans := protectedSpans(input)
+ var output strings.Builder
+ position := 0
+ for _, protected := range spans {
+ if protected.start > position {
+ output.WriteString(transformText(input[position:protected.start], strategy))
+ }
+ output.WriteString(input[protected.start:protected.end])
+ position = protected.end
+ }
+ output.WriteString(transformText(input[position:], strategy))
+ return output.String()
+}
+
+func protectedSpans(input string) []span {
+ spans := make([]span, 0)
+ for _, location := range protectedTokenRe.FindAllStringIndex(input, -1) {
+ spans = append(spans, span{start: location[0], end: location[1]})
+ }
+ spans = append(spans, markdownDestinationSpans(input)...)
+ sort.Slice(spans, func(i, j int) bool {
+ if spans[i].start != spans[j].start {
+ return spans[i].start < spans[j].start
+ }
+ return spans[i].end > spans[j].end
+ })
+ merged := spans[:0]
+ for _, candidate := range spans {
+ if len(merged) == 0 || candidate.start > merged[len(merged)-1].end {
+ merged = append(merged, candidate)
+ continue
+ }
+ if candidate.end > merged[len(merged)-1].end {
+ merged[len(merged)-1].end = candidate.end
+ }
+ }
+ return merged
+}
+
+func markdownDestinationSpans(input string) []span {
+ var spans []span
+ for offset := 0; offset+1 < len(input); {
+ relative := strings.Index(input[offset:], "](")
+ if relative < 0 {
+ break
+ }
+ start := offset + relative + 2
+ depth := 1
+ escaped := false
+ end := start
+ for ; end < len(input); end++ {
+ character := input[end]
+ if escaped {
+ escaped = false
+ continue
+ }
+ if character == '\\' {
+ escaped = true
+ continue
+ }
+ switch character {
+ case '(':
+ depth++
+ case ')':
+ depth--
+ }
+ if depth == 0 {
+ break
+ }
+ }
+ if depth != 0 {
+ break
+ }
+ spans = append(spans, span{start: start, end: end})
+ offset = end + 1
+ }
+ return spans
+}
+
+func transformText(input string, strategy Strategy) string {
+ if strategy == Bidi {
+ return mapRunes(input, bidiRune)
+ }
+ mapped := mapRunes(input, accentedRune)
+ var expanded strings.Builder
+ for _, character := range mapped {
+ expanded.WriteRune(character)
+ if isVowel(character) {
+ expanded.WriteRune(character)
+ }
+ }
+ return expanded.String()
+}
+
+func mapRunes(input string, transform func(rune) rune) string {
+ return strings.Map(transform, input)
+}
+
+func accentedRune(character rune) rune {
+ if replacement, ok := accentedCharacters[character]; ok {
+ return replacement
+ }
+ return character
+}
+
+func bidiRune(character rune) rune {
+ if replacement, ok := bidiCharacters[character]; ok {
+ return replacement
+ }
+ return character
+}
+
+func isVowel(character rune) bool {
+ decomposed := unicode.ToLower(character)
+ return strings.ContainsRune("aàáâãäåæeèéêëiìíîïoòóôõöøuùúûüyýÿȧḗī", decomposed) && utf8.RuneLen(character) > 0
+}
+
+var accentedCharacters = map[rune]rune{
+ 'a': 'ȧ', 'b': 'ƀ', 'c': 'ƈ', 'd': 'ḓ', 'e': 'ḗ', 'f': 'ƒ', 'g': 'ɠ', 'h': 'ħ', 'i': 'ī', 'j': 'ĵ', 'k': 'ķ', 'l': 'ŀ', 'm': 'ḿ',
+ 'n': 'ƞ', 'o': 'ǿ', 'p': 'ƥ', 'q': 'ɋ', 'r': 'ř', 's': 'ş', 't': 'ŧ', 'u': 'ŭ', 'v': 'ṽ', 'w': 'ẇ', 'x': 'ẋ', 'y': 'ẏ', 'z': 'ž',
+ 'A': 'Ȧ', 'B': 'Ɓ', 'C': 'Ƈ', 'D': 'Ḓ', 'E': 'Ḗ', 'F': 'Ƒ', 'G': 'Ɠ', 'H': 'Ħ', 'I': 'Ī', 'J': 'Ĵ', 'K': 'Ķ', 'L': 'Ŀ', 'M': 'Ḿ',
+ 'N': 'Ƞ', 'O': 'Ø', 'P': 'Ƥ', 'Q': 'Ɋ', 'R': 'Ř', 'S': 'Ş', 'T': 'Ŧ', 'U': 'Ŭ', 'V': 'Ṽ', 'W': 'Ẇ', 'X': 'Ẋ', 'Y': 'Ẏ', 'Z': 'Ž',
+}
+
+var bidiCharacters = map[rune]rune{
+ 'a': 'ɐ', 'b': 'q', 'c': 'ɔ', 'd': 'p', 'e': 'ǝ', 'f': 'ɟ', 'g': 'ƃ', 'h': 'ɥ', 'i': 'ı', 'j': 'ɾ', 'k': 'ʞ', 'l': 'ן', 'm': 'ɯ',
+ 'n': 'u', 'o': 'o', 'p': 'd', 'q': 'b', 'r': 'ɹ', 's': 's', 't': 'ʇ', 'u': 'n', 'v': 'ʌ', 'w': 'ʍ', 'x': 'x', 'y': 'ʎ', 'z': 'z',
+ 'A': '∀', 'B': 'ꓭ', 'C': 'Ɔ', 'D': '◖', 'E': 'Ǝ', 'F': 'Ⅎ', 'G': '⅁', 'H': 'H', 'I': 'I', 'J': 'ſ', 'K': 'ꓘ', 'L': '˥', 'M': 'W',
+ 'N': 'N', 'O': 'O', 'P': 'Ԁ', 'Q': 'Ό', 'R': 'ꓤ', 'S': 'S', 'T': '⊥', 'U': '∩', 'V': 'Λ', 'W': 'M', 'X': 'X', 'Y': '⅄', 'Z': 'Z',
+}
diff --git a/internal/pseudo/transform_test.go b/internal/pseudo/transform_test.go
new file mode 100644
index 0000000..a74e851
--- /dev/null
+++ b/internal/pseudo/transform_test.go
@@ -0,0 +1,134 @@
+package pseudo
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern"
+ "github.com/Tom-R-Main/Internationalizer/internal/formats"
+ "github.com/Tom-R-Main/Internationalizer/internal/message"
+ "github.com/Tom-R-Main/Internationalizer/internal/validate"
+)
+
+func TestAccentedPreservesProtectedSyntax(t *testing.T) {
+ source := "Save {{name}} in Account; read [guide](https://example.com/a_(b)) and run `go test`."
+ target, err := Transform(source, Accented)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if target == source || !strings.HasPrefix(target, "[!! ") {
+ t.Fatalf("target = %q", target)
+ }
+ if findings := validate.ProtectedFindings("message", source, target, "en-XA"); len(findings) != 0 {
+ t.Fatalf("protected findings = %#v; target = %q", findings, target)
+ }
+}
+
+func TestTransformsOnlyICULiteralText(t *testing.T) {
+ source := `{count, plural, one {One item for {name}} other {# items for {name}}}`
+ target, err := Transform(source, Accented)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if issues := message.Compare(source, target, "en-XA"); len(issues) != 0 {
+ t.Fatalf("ICU issues = %#v; target = %q", issues, target)
+ }
+ if findings := validate.ProtectedFindings("items", source, target, "en-XA"); len(findings) != 0 {
+ t.Fatalf("protected findings = %#v; target = %q", findings, target)
+ }
+}
+
+func TestBidiUsesIsolateAndPreservesInterpolation(t *testing.T) {
+ source := "Hello {name}; read [guide](https://example.com) in Account."
+ target, err := Transform(source, Bidi)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.HasPrefix(target, "\u2067") || !strings.HasSuffix(target, "\u2069") || !strings.Contains(target, "{name}") {
+ t.Fatalf("target = %q", target)
+ }
+ if findings := validate.ProtectedFindings("message", source, target, "ar-XB"); len(findings) != 0 {
+ t.Fatalf("protected findings = %#v; target = %q", findings, target)
+ }
+}
+
+func TestRejectsUnknownStrategy(t *testing.T) {
+ if _, err := Transform("Hello", Strategy("unknown")); err == nil {
+ t.Fatal("Transform accepted unknown strategy")
+ }
+}
+
+func TestTransformPreservesFluentSelectorStructure(t *testing.T) {
+ source := `{ $count ->
+ [one] One item for { -brand-short-name }
+ *[other] { $count } items for { $user }
+}`
+ target, err := Transform(source, Accented)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, syntax := range []string{"{ $count ->", "[one]", "*[other]", "{ -brand-short-name }", "{ $count }", "{ $user }"} {
+ if !strings.Contains(target, syntax) {
+ t.Fatalf("pseudo target lost %q: %s", syntax, target)
+ }
+ }
+}
+
+func TestPseudoOptionalFluentCorpus(t *testing.T) {
+ root := os.Getenv("INTERNATIONALIZER_FLUENT_CORPUS")
+ if root == "" {
+ t.Skip("set INTERNATIONALIZER_FLUENT_CORPUS to a directory of .ftl resources")
+ }
+ files := 0
+ err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ clean := filepath.ToSlash(path)
+ if entry.IsDir() || filepath.Ext(path) != ".ftl" || strings.Contains(clean, "/test/") || strings.Contains(clean, "/tests/") {
+ return nil
+ }
+ files++
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ units, err := formats.ParseUnits(&formats.FluentFormat{}, data)
+ if err != nil {
+ return err
+ }
+ for _, unit := range units {
+ target, err := Transform(unit.Value, Accented)
+ if err != nil {
+ return err
+ }
+ _, _, preserved, err := fluentpattern.Compare(unit.Value, target)
+ if err != nil || !preserved {
+ return &fluentPseudoCorpusError{path: path, unit: unit.ID, err: err}
+ }
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if files == 0 {
+ t.Fatalf("no .ftl files found under %s", root)
+ }
+}
+
+type fluentPseudoCorpusError struct {
+ path string
+ unit string
+ err error
+}
+
+func (err *fluentPseudoCorpusError) Error() string {
+ detail := "structure mismatch"
+ if err.err != nil {
+ detail = err.err.Error()
+ }
+ return err.path + ": " + err.unit + ": " + detail
+}
diff --git a/internal/review/review.go b/internal/review/review.go
new file mode 100644
index 0000000..278a03b
--- /dev/null
+++ b/internal/review/review.go
@@ -0,0 +1,176 @@
+// Package review manages explicit human approval of current translations.
+package review
+
+import (
+ "fmt"
+ "sort"
+ "time"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ localeid "github.com/Tom-R-Main/Internationalizer/internal/locale"
+ "github.com/Tom-R-Main/Internationalizer/internal/state"
+ "github.com/Tom-R-Main/Internationalizer/internal/validate"
+)
+
+// Filter selects manifest entries for listing or approval.
+type Filter struct {
+ Locale string
+ Bundle string
+ Keys []string
+ Status state.ReviewStatus
+ All bool
+}
+
+// List returns deterministically ordered manifest entries matching filter.
+func List(manifest *state.Manifest, filter Filter) ([]state.Entry, error) {
+ canonicalLocale := ""
+ var err error
+ if filter.Locale != "" {
+ canonicalLocale, err = localeid.Canonical(filter.Locale)
+ if err != nil {
+ return nil, err
+ }
+ }
+ if filter.Status != "" && filter.Status != state.ReviewNeedsReview && filter.Status != state.ReviewApproved {
+ return nil, fmt.Errorf("unsupported review status %q", filter.Status)
+ }
+ keySet := make(map[string]struct{}, len(filter.Keys))
+ for _, key := range filter.Keys {
+ keySet[key] = struct{}{}
+ }
+
+ entries := make([]state.Entry, 0, len(manifest.Translations))
+ for _, entry := range manifest.Translations {
+ if canonicalLocale != "" {
+ entryLocale, err := localeid.Canonical(entry.Locale)
+ if err != nil || entryLocale != canonicalLocale {
+ continue
+ }
+ }
+ if filter.Bundle != "" && entry.Bundle != filter.Bundle {
+ continue
+ }
+ if filter.Status != "" && entry.ReviewStatus != filter.Status {
+ continue
+ }
+ if len(keySet) > 0 {
+ if _, ok := keySet[entry.Key]; !ok {
+ continue
+ }
+ }
+ entries = append(entries, entry)
+ }
+ sort.Slice(entries, func(i, j int) bool {
+ if entries[i].Locale != entries[j].Locale {
+ return entries[i].Locale < entries[j].Locale
+ }
+ if entries[i].Bundle != entries[j].Bundle {
+ return entries[i].Bundle < entries[j].Bundle
+ }
+ return entries[i].Key < entries[j].Key
+ })
+ return entries, nil
+}
+
+// Approve verifies selected entries against source, policy, and target state,
+// then records explicit approval. It never approves a stale or invalid value.
+func Approve(cfg *config.Config, filter Filter, reviewedAt time.Time) ([]state.Entry, error) {
+ effectiveConfig := *cfg
+ effectiveConfig.ApplyDefaults()
+ cfg = &effectiveConfig
+ if err := cfg.ValidateProject(); err != nil {
+ return nil, err
+ }
+ if filter.Locale == "" {
+ return nil, fmt.Errorf("locale is required")
+ }
+ configuredLocale, ok := cfg.ConfiguredTargetLocale(filter.Locale)
+ if !ok {
+ return nil, fmt.Errorf("locale %q is not in target_locales", filter.Locale)
+ }
+ if filter.All == (len(filter.Keys) > 0) {
+ return nil, fmt.Errorf("choose exactly one of all entries or one or more keys")
+ }
+ if len(filter.Keys) > 0 && filter.Bundle == "" {
+ return nil, fmt.Errorf("bundle is required when approving individual keys")
+ }
+
+ manifest, err := state.Load(cfg.ManifestPath)
+ if err != nil {
+ return nil, err
+ }
+ selection := filter
+ selection.Locale = configuredLocale
+ selection.Status = ""
+ entries, err := List(manifest, selection)
+ if err != nil {
+ return nil, err
+ }
+ if len(entries) == 0 {
+ return nil, fmt.Errorf("no tracked translations match the approval selection")
+ }
+ if len(filter.Keys) > 0 {
+ found := make(map[string]struct{}, len(entries))
+ for _, entry := range entries {
+ found[entry.Key] = struct{}{}
+ }
+ for _, key := range filter.Keys {
+ if _, ok := found[key]; !ok {
+ return nil, fmt.Errorf("translation %s/%s/%s is not tracked", filter.Bundle, key, configuredLocale)
+ }
+ }
+ }
+
+ reports, err := validate.ValidateWithOptions(cfg, validate.Options{RequireState: true})
+ if err != nil {
+ return nil, err
+ }
+ for _, entry := range entries {
+ if err := entryValidationError(entry, reports); err != nil {
+ return nil, err
+ }
+ }
+
+ approved := make([]state.Entry, 0, len(entries))
+ for _, entry := range entries {
+ updated, err := manifest.Approve(entry.Bundle, entry.Key, entry.Locale, reviewedAt)
+ if err != nil {
+ return nil, err
+ }
+ approved = append(approved, updated)
+ }
+ if err := manifest.Save(cfg.ManifestPath); err != nil {
+ return nil, err
+ }
+ return approved, nil
+}
+
+func entryValidationError(entry state.Entry, reports []validate.Report) error {
+ for _, report := range reports {
+ reportLocale, reportLocaleErr := localeid.Canonical(report.Locale)
+ entryLocale, entryLocaleErr := localeid.Canonical(entry.Locale)
+ if report.Bundle != entry.Bundle || reportLocaleErr != nil || entryLocaleErr != nil || reportLocale != entryLocale {
+ continue
+ }
+ if len(report.Errors) > 0 {
+ return fmt.Errorf("cannot approve %s/%s/%s: %s", entry.Bundle, entry.Key, entry.Locale, report.Errors[0])
+ }
+ for _, missing := range report.Missing {
+ if missing == entry.Key {
+ return fmt.Errorf("cannot approve %s/%s/%s: target key is missing", entry.Bundle, entry.Key, entry.Locale)
+ }
+ }
+ for _, mismatch := range report.Mismatches {
+ if mismatch.Key == entry.Key {
+ return fmt.Errorf("cannot approve %s/%s/%s: interpolation mismatch", entry.Bundle, entry.Key, entry.Locale)
+ }
+ }
+ for _, finding := range report.Findings {
+ if finding.Key == entry.Key && finding.Severity == validate.SeverityError {
+ return fmt.Errorf("cannot approve %s/%s/%s: %s", entry.Bundle, entry.Key, entry.Locale, finding.Message)
+ }
+ }
+ return nil
+ }
+ return fmt.Errorf("cannot approve %s/%s/%s: no configured bundle/locale report", entry.Bundle, entry.Key, entry.Locale)
+}
diff --git a/internal/review/review_test.go b/internal/review/review_test.go
new file mode 100644
index 0000000..736316c
--- /dev/null
+++ b/internal/review/review_test.go
@@ -0,0 +1,104 @@
+package review
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/Tom-R-Main/Internationalizer/internal/config"
+ "github.com/Tom-R-Main/Internationalizer/internal/policy"
+ "github.com/Tom-R-Main/Internationalizer/internal/state"
+)
+
+func TestApproveVerifiesCurrentArtifactAndPreservesOrigin(t *testing.T) {
+ cfg := reviewFixture(t, "Enregistrer")
+ reviewedAt := time.Date(2026, time.September, 4, 16, 0, 0, 0, time.UTC)
+
+ approved, err := Approve(cfg, Filter{Locale: "fr", Bundle: "default", Keys: []string{"save"}}, reviewedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(approved) != 1 || approved[0].Origin != "provider" || approved[0].ReviewStatus != state.ReviewApproved {
+ t.Fatalf("approved = %#v", approved)
+ }
+
+ loaded, err := state.Load(cfg.ManifestPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ entry, ok := loaded.Get("default", "save", "fr")
+ if !ok || entry.ReviewedAt == nil || !entry.ReviewedAt.Equal(reviewedAt) {
+ t.Fatalf("saved approval = %#v, found = %v", entry, ok)
+ }
+}
+
+func TestApproveRejectsModifiedTarget(t *testing.T) {
+ cfg := reviewFixture(t, "Enregistrer")
+ targetPath := filepath.Join(filepath.Dir(cfg.SourcePath), "fr.json")
+ if err := os.WriteFile(targetPath, []byte(`{"save":"Sauvegarder"}`), 0o644); err != nil {
+ t.Fatal(err)
+ }
+
+ _, err := Approve(cfg, Filter{Locale: "fr", All: true}, time.Now())
+ if err == nil || !strings.Contains(err.Error(), "target changed") {
+ t.Fatalf("Approve error = %v, want target-changed rejection", err)
+ }
+}
+
+func TestListFiltersAndSortsReviewState(t *testing.T) {
+ manifest := state.New()
+ manifest.Set(state.Entry{Bundle: "b", Key: "z", Locale: "fr", ReviewStatus: state.ReviewNeedsReview})
+ manifest.Set(state.Entry{Bundle: "a", Key: "a", Locale: "fr", ReviewStatus: state.ReviewNeedsReview})
+ manifest.Set(state.Entry{Bundle: "a", Key: "a", Locale: "de", ReviewStatus: state.ReviewNeedsReview})
+
+ entries, err := List(manifest, Filter{Locale: "fr", Status: state.ReviewNeedsReview})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(entries) != 2 || entries[0].Bundle != "a" || entries[1].Bundle != "b" {
+ t.Fatalf("entries = %#v", entries)
+ }
+}
+
+func reviewFixture(t *testing.T, target string) *config.Config {
+ t.Helper()
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "en.json")
+ targetPath := filepath.Join(dir, "fr.json")
+ if err := os.WriteFile(sourcePath, []byte(`{"save":"Save"}`), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(targetPath, []byte(`{"save":"`+target+`"}`), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ cfg := &config.Config{
+ SourceLocale: "en",
+ TargetLocales: []string{"fr"},
+ SourcePath: sourcePath,
+ ManifestPath: filepath.Join(dir, "manifest.json"),
+ LLM: config.LLM{Provider: "gemini", Model: "test-model"},
+ }
+ cfg.ApplyDefaults()
+ resolved, err := policy.Resolve(cfg, "fr", "json", "", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ manifest := state.New()
+ manifest.Set(state.Entry{
+ Bundle: "default",
+ Key: "save",
+ Locale: "fr",
+ SourceHash: state.SourceHash("json", "Save"),
+ PolicyHash: resolved.Hash,
+ TargetHash: state.TargetHash(target),
+ Origin: "provider",
+ ReviewStatus: state.ReviewNeedsReview,
+ UpdatedAt: time.Now().UTC(),
+ })
+ if err := manifest.Save(cfg.ManifestPath); err != nil {
+ t.Fatal(err)
+ }
+ return cfg
+}
diff --git a/internal/state/manifest.go b/internal/state/manifest.go
index ff89270..db504ed 100644
--- a/internal/state/manifest.go
+++ b/internal/state/manifest.go
@@ -13,7 +13,20 @@ import (
"github.com/Tom-R-Main/Internationalizer/internal/locale"
)
-const SchemaVersion = 1
+const (
+ SchemaVersion = 2
+ legacySchemaVersion = 1
+)
+
+// ReviewStatus records human approval independently from translation origin.
+// Provider, TM, adoption, and manual provenance answer how a value arrived;
+// review status answers whether a person approved that exact artifact.
+type ReviewStatus string
+
+const (
+ ReviewNeedsReview ReviewStatus = "needs_review"
+ ReviewApproved ReviewStatus = "approved"
+)
// Manifest is the versioned on-disk translation state.
type Manifest struct {
@@ -23,19 +36,21 @@ type Manifest struct {
// Entry records the inputs and output for one bundle key and target locale.
type Entry struct {
- Bundle string `json:"bundle"`
- Key string `json:"key"`
- Locale string `json:"locale"`
- SourceHash string `json:"source_hash"`
- PolicyHash string `json:"policy_hash"`
- GuideHash string `json:"guide_hash,omitempty"`
- GlossaryHash string `json:"glossary_hash,omitempty"`
- PromptVersion int `json:"prompt_version,omitempty"`
- TargetHash string `json:"target_hash"`
- Origin string `json:"origin,omitempty"`
- Provider string `json:"provider,omitempty"`
- Model string `json:"model,omitempty"`
- UpdatedAt time.Time `json:"updated_at"`
+ Bundle string `json:"bundle"`
+ Key string `json:"key"`
+ Locale string `json:"locale"`
+ SourceHash string `json:"source_hash"`
+ PolicyHash string `json:"policy_hash"`
+ GuideHash string `json:"guide_hash,omitempty"`
+ GlossaryHash string `json:"glossary_hash,omitempty"`
+ PromptVersion int `json:"prompt_version,omitempty"`
+ TargetHash string `json:"target_hash"`
+ Origin string `json:"origin,omitempty"`
+ Provider string `json:"provider,omitempty"`
+ Model string `json:"model,omitempty"`
+ ReviewStatus ReviewStatus `json:"review_status"`
+ ReviewedAt *time.Time `json:"reviewed_at,omitempty"`
+ UpdatedAt time.Time `json:"updated_at"`
}
// New returns an empty manifest using the current schema.
@@ -61,15 +76,24 @@ func Load(path string) (*Manifest, error) {
if err := json.Unmarshal(data, &manifest); err != nil {
return nil, fmt.Errorf("parsing manifest %s: %w", path, err)
}
- if manifest.SchemaVersion != SchemaVersion {
+ if manifest.SchemaVersion != SchemaVersion && manifest.SchemaVersion != legacySchemaVersion {
return nil, fmt.Errorf("manifest %s uses schema version %d; supported version is %d", path, manifest.SchemaVersion, SchemaVersion)
}
+ legacy := manifest.SchemaVersion == legacySchemaVersion
+ manifest.SchemaVersion = SchemaVersion
if manifest.Translations == nil {
manifest.Translations = make(map[string]Entry)
}
canonicalTranslations := make(map[string]Entry, len(manifest.Translations))
for _, entry := range manifest.Translations {
entry.Locale = canonicalLocaleOrOriginal(entry.Locale)
+ if legacy || entry.ReviewStatus == "" {
+ entry.ReviewStatus = ReviewNeedsReview
+ entry.ReviewedAt = nil
+ }
+ if err := validateReviewState(entry); err != nil {
+ return nil, fmt.Errorf("manifest %s contains invalid review state for %s/%s/%s: %w", path, entry.Bundle, entry.Key, entry.Locale, err)
+ }
identity := Identity(entry.Bundle, entry.Key, entry.Locale)
if existing, duplicate := canonicalTranslations[identity]; duplicate && existing != entry {
return nil, fmt.Errorf("manifest %s contains conflicting entries for %s/%s/%s", path, entry.Bundle, entry.Key, entry.Locale)
@@ -82,6 +106,12 @@ func Load(path string) (*Manifest, error) {
// Save replaces the manifest only after the complete new file is durable.
func (m *Manifest) Save(path string) error {
+ m.SchemaVersion = SchemaVersion
+ for _, entry := range m.Translations {
+ if err := validateReviewState(entry); err != nil {
+ return fmt.Errorf("saving manifest %s with invalid review state for %s/%s/%s: %w", path, entry.Bundle, entry.Key, entry.Locale, err)
+ }
+ }
data, err := json.MarshalIndent(m, "", " ")
if err != nil {
return fmt.Errorf("encoding manifest: %w", err)
@@ -101,9 +131,42 @@ func (m *Manifest) Set(entry Entry) {
m.Translations = make(map[string]Entry)
}
entry.Locale = canonicalLocaleOrOriginal(entry.Locale)
+ if entry.ReviewStatus == "" {
+ entry.ReviewStatus = ReviewNeedsReview
+ entry.ReviewedAt = nil
+ }
m.Translations[Identity(entry.Bundle, entry.Key, entry.Locale)] = entry
}
+// Approve marks the exact recorded artifact approved at reviewedAt.
+func (m *Manifest) Approve(bundle, key, locale string, reviewedAt time.Time) (Entry, error) {
+ entry, ok := m.Get(bundle, key, locale)
+ if !ok {
+ return Entry{}, fmt.Errorf("translation %s/%s/%s is not tracked", bundle, key, locale)
+ }
+ stamp := reviewedAt.UTC()
+ entry.ReviewStatus = ReviewApproved
+ entry.ReviewedAt = &stamp
+ m.Set(entry)
+ return entry, nil
+}
+
+func validateReviewState(entry Entry) error {
+ switch entry.ReviewStatus {
+ case ReviewNeedsReview:
+ if entry.ReviewedAt != nil {
+ return fmt.Errorf("needs_review entry has reviewed_at")
+ }
+ case ReviewApproved:
+ if entry.ReviewedAt == nil {
+ return fmt.Errorf("approved entry lacks reviewed_at")
+ }
+ default:
+ return fmt.Errorf("unknown review_status %q", entry.ReviewStatus)
+ }
+ return nil
+}
+
// Identity returns a stable full SHA-256 identifier for a logical translation.
func Identity(bundle, key, locale string) string {
return hashBytes([]byte(bundle + "\x00" + key + "\x00" + canonicalLocaleOrOriginal(locale)))
@@ -123,6 +186,16 @@ func SourceHash(format, source string) string {
return hashBytes([]byte(canonical))
}
+// SourceUnitHash also binds translator context and adapter-owned structure to
+// provenance. Flat legacy units retain their existing SourceHash identity.
+func SourceUnitHash(format, value, context, structure string) string {
+ if context == "" && structure == "" {
+ return SourceHash(format, value)
+ }
+ canonical := fmt.Sprintf("2:%d:%s:%d:%s:%d:%s:%d:%s", len(format), format, len(value), value, len(context), context, len(structure), structure)
+ return hashBytes([]byte(canonical))
+}
+
// TargetHash records the exact translated value last applied or adopted.
func TargetHash(target string) string {
return hashBytes([]byte(target))
diff --git a/internal/state/manifest_test.go b/internal/state/manifest_test.go
index eafc7b9..69b290f 100644
--- a/internal/state/manifest_test.go
+++ b/internal/state/manifest_test.go
@@ -11,16 +11,17 @@ func TestManifestRoundTrip(t *testing.T) {
path := filepath.Join(t.TempDir(), "state", "manifest.json")
manifest := New()
entry := Entry{
- Bundle: "app",
- Key: "common.save",
- Locale: "fr",
- SourceHash: SourceHash("json", "Save"),
- PolicyHash: mustHashValue(t, "policy"),
- TargetHash: TargetHash("Enregistrer"),
- Origin: "provider",
- Provider: "openai",
- Model: "model",
- UpdatedAt: time.Now().UTC().Truncate(time.Second),
+ Bundle: "app",
+ Key: "common.save",
+ Locale: "fr",
+ SourceHash: SourceHash("json", "Save"),
+ PolicyHash: mustHashValue(t, "policy"),
+ TargetHash: TargetHash("Enregistrer"),
+ Origin: "provider",
+ Provider: "openai",
+ Model: "model",
+ ReviewStatus: ReviewNeedsReview,
+ UpdatedAt: time.Now().UTC().Truncate(time.Second),
}
manifest.Set(entry)
if err := manifest.Save(path); err != nil {
@@ -64,6 +65,34 @@ func TestLoadMigratesLegacyLocaleIdentityInMemory(t *testing.T) {
if !ok || entry.Locale != "pt-BR" {
t.Fatalf("legacy entry = %#v, %v; want canonicalized lookup", entry, ok)
}
+ if manifest.SchemaVersion != SchemaVersion || entry.ReviewStatus != ReviewNeedsReview || entry.ReviewedAt != nil {
+ t.Fatalf("legacy review state = %#v, schema = %d; want needs_review schema v2", entry, manifest.SchemaVersion)
+ }
+}
+
+func TestApproveRecordsExplicitReviewSeparatelyFromOrigin(t *testing.T) {
+ manifest := New()
+ manifest.Set(Entry{Bundle: "app", Key: "save", Locale: "fr", Origin: "provider"})
+ reviewedAt := time.Date(2026, time.September, 4, 12, 0, 0, 0, time.FixedZone("EDT", -4*60*60))
+
+ approved, err := manifest.Approve("app", "save", "fr", reviewedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if approved.Origin != "provider" || approved.ReviewStatus != ReviewApproved || approved.ReviewedAt == nil || !approved.ReviewedAt.Equal(reviewedAt.UTC()) {
+ t.Fatalf("approved entry = %#v", approved)
+ }
+}
+
+func TestLoadRejectsInvalidApprovedState(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "manifest.json")
+ data := []byte(`{"schema_version":2,"translations":{"entry":{"bundle":"app","key":"save","locale":"fr","review_status":"approved","updated_at":"2026-01-01T00:00:00Z"}}}`)
+ if err := os.WriteFile(path, data, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := Load(path); err == nil {
+ t.Fatal("Load accepted approved state without reviewed_at")
+ }
}
func TestLoadRejectsConflictingCanonicalLocaleIdentities(t *testing.T) {
@@ -86,6 +115,18 @@ func TestHashesSeparateSourceFormatAndPolicy(t *testing.T) {
}
}
+func TestSourceUnitHashTracksTranslatorContextWithoutChangingFlatIdentity(t *testing.T) {
+ legacy := SourceHash("json", "Save")
+ if got := SourceUnitHash("json", "Save", "", ""); got != legacy {
+ t.Fatalf("flat unit hash = %q, want legacy %q", got, legacy)
+ }
+ first := SourceUnitHash("fluent", "Open", "Verb used on a button.", "fluent-pattern-v1")
+ second := SourceUnitHash("fluent", "Open", "Noun shown in a menu.", "fluent-pattern-v1")
+ if first == second {
+ t.Fatal("translator context did not affect source unit hash")
+ }
+}
+
func TestManifestIdentityCanonicalizesLocale(t *testing.T) {
manifest := New()
entry := Entry{Bundle: "app", Key: "save", Locale: "pt-br"}
diff --git a/internal/translate/translate.go b/internal/translate/translate.go
index 5c8dd90..54aab98 100644
--- a/internal/translate/translate.go
+++ b/internal/translate/translate.go
@@ -64,10 +64,11 @@ func (e *RunError) Error() string {
}
type preparedBundle struct {
- bundle config.Bundle
- format formats.Format
- sourceKeys map[string]string
- sourceData []byte
+ bundle config.Bundle
+ format formats.Format
+ sourceUnits []formats.Unit
+ sourceKeys map[string]string
+ sourceData []byte
}
type job struct {
@@ -227,11 +228,11 @@ func prepareBundles(bundles []config.Bundle) ([]preparedBundle, error) {
if err != nil {
return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err)
}
- keys, err := format.Parse(data)
+ units, err := formats.ParseUnits(format, data)
if err != nil {
return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err)
}
- prepared = append(prepared, preparedBundle{bundle: bundle, format: format, sourceKeys: keys, sourceData: data})
+ prepared = append(prepared, preparedBundle{bundle: bundle, format: format, sourceUnits: units, sourceKeys: formats.UnitValues(units), sourceData: data})
}
return prepared, nil
}
@@ -298,6 +299,7 @@ func translateLocale(
policyHash := translationPolicy.Hash
targetKeys := make(map[string]string)
+ var targetUnits []formats.Unit
var targetData []byte
targetExists := false
data, err := os.ReadFile(targetPath)
@@ -305,7 +307,12 @@ func translateLocale(
case err == nil:
targetExists = true
targetData = data
- targetKeys, err = parseTarget(bundle.format, bundle.sourceData, data)
+ if paired, ok := bundle.format.(formats.PairedFormat); ok {
+ targetKeys, err = paired.ParseTarget(bundle.sourceData, data)
+ } else {
+ targetUnits, err = formats.ParseUnits(bundle.format, data)
+ targetKeys = formats.UnitValues(targetUnits)
+ }
if err != nil {
result.Errors = append(result.Errors, fmt.Sprintf("parsing target %s: %v", targetPath, err))
return jobOutput{result: result}
@@ -317,14 +324,19 @@ func translateLocale(
}
keys := sortedKeys(sourceKeys)
+ sourceUnits := make(map[string]formats.Unit, len(bundle.sourceUnits))
+ for _, unit := range bundle.sourceUnits {
+ sourceUnits[unit.ID] = unit
+ }
plans := make([]plannedEntry, 0, len(keys))
for _, key := range keys {
sourceValue := sourceKeys[key]
- sourceHash := state.SourceHash(bundle.format.Name(), sourceValue)
+ sourceUnit := sourceUnits[key]
+ sourceHash := state.SourceUnitHash(bundle.format.Name(), sourceValue, sourceUnit.Context, sourceUnit.Structure)
targetValue, exists := targetKeys[key]
recorded, recordedOK := manifest.Get(bundle.bundle.ID, key, locale)
entryState := classify(exists, targetValue, sourceHash, policyHash, recorded, recordedOK)
- plans = append(plans, plannedEntry{key: key, source: sourceValue, sourceHash: sourceHash, state: entryState})
+ plans = append(plans, plannedEntry{key: key, source: sourceValue, context: sourceUnit.Context, sourceHash: sourceHash, state: entryState})
result.addState(entryState)
}
if opts.AdoptExisting {
@@ -387,7 +399,7 @@ func translateLocale(
batchPlans := toTranslate[i:end]
entries := make([]llm.Entry, len(batchPlans))
for j, plan := range batchPlans {
- entries[j] = llm.Entry{Key: plan.key, Value: plan.source}
+ entries[j] = llm.Entry{Key: plan.key, Value: plan.source, Context: plan.context}
}
response, err := provider.Translate(ctx, llm.TranslateRequest{
@@ -452,7 +464,17 @@ func translateLocale(
}
}
}
- output, err := serializeTarget(bundle.format, staged, bundle.sourceData, serializationBaseline)
+ var output []byte
+ if paired, ok := bundle.format.(formats.PairedFormat); ok {
+ output, err = paired.SerializeTarget(staged, bundle.sourceData, serializationBaseline)
+ } else {
+ unitBaseline := targetUnits
+ if !targetExists {
+ unitBaseline = bundle.sourceUnits
+ }
+ stagedUnits := formats.MergeUnitValues(unitBaseline, bundle.sourceUnits, staged)
+ output, err = formats.SerializeUnits(bundle.format, stagedUnits, serializationBaseline)
+ }
if err != nil {
result.Errors = append(result.Errors, fmt.Sprintf("serializing target %s: %v", targetPath, err))
return jobOutput{result: result}
@@ -510,6 +532,7 @@ func translateLocale(
Origin: origin.kind,
Provider: origin.provider,
Model: origin.model,
+ ReviewStatus: state.ReviewNeedsReview,
UpdatedAt: now,
})
}
@@ -524,13 +547,6 @@ func parseTarget(format formats.Format, source, target []byte) (map[string]strin
return format.Parse(target)
}
-func serializeTarget(format formats.Format, entries map[string]string, source, target []byte) ([]byte, error) {
- if paired, ok := format.(formats.PairedFormat); ok {
- return paired.SerializeTarget(entries, source, target)
- }
- return format.Serialize(entries, target)
-}
-
type entryState struct {
missing bool
sourceStale bool
@@ -553,6 +569,7 @@ type translationOrigin struct {
type plannedEntry struct {
key string
source string
+ context string
sourceHash string
state entryState
}
diff --git a/internal/translate/translate_test.go b/internal/translate/translate_test.go
index 100280d..0195ba0 100644
--- a/internal/translate/translate_test.go
+++ b/internal/translate/translate_test.go
@@ -5,6 +5,7 @@ import (
"encoding/json"
"os"
"path/filepath"
+ "strings"
"testing"
"time"
@@ -36,6 +37,67 @@ func (p *fakeProvider) Translate(_ context.Context, request llm.TranslateRequest
return p.response, nil
}
+func TestRunTranslatesFluentUnitsWithDeveloperContext(t *testing.T) {
+ dir := t.TempDir()
+ sourcePath := filepath.Join(dir, "en.ftl")
+ source := "# A command, not a noun.\nopen-button = Open { -brand-short-name }\n .aria-label = Open the application\n\n-brand-short-name = Acme\n"
+ if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ cfg := testConfig(dir, sourcePath)
+ provider := &fakeProvider{response: &llm.TranslateResponse{Translations: map[string]string{
+ "open-button": "Ouvrir { -brand-short-name }",
+ "open-button.aria-label": "Ouvrir l’application",
+ "-brand-short-name": "Acme",
+ }}}
+
+ if _, err := Run(context.Background(), cfg, provider, Options{}); err != nil {
+ t.Fatal(err)
+ }
+ if len(provider.requests) != 1 {
+ t.Fatalf("provider requests = %#v, want one request", provider.requests)
+ }
+ context := ""
+ for _, entry := range provider.requests[0].Entries {
+ if entry.Key == "open-button" {
+ context = entry.Context
+ }
+ }
+ if len(provider.requests[0].Entries) != 3 || context != "# A command, not a noun." {
+ t.Fatalf("provider requests = %#v", provider.requests)
+ }
+ if !strings.Contains(provider.requests[0].SystemPrompt, "Fluent variables") {
+ t.Fatalf("provider did not receive Fluent prompt: %s", provider.requests[0].SystemPrompt)
+ }
+ output, err := os.ReadFile(filepath.Join(dir, "fr.ftl"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ text := string(output)
+ for _, expected := range []string{"# A command, not a noun.", "open-button = Ouvrir { -brand-short-name }", ".aria-label = Ouvrir l’application", "-brand-short-name = Acme"} {
+ if !strings.Contains(text, expected) {
+ t.Fatalf("Fluent output lacks %q:\n%s", expected, text)
+ }
+ }
+ changedContext := strings.Replace(source, "# A command, not a noun.", "# A noun shown in the File menu.", 1)
+ if err := os.WriteFile(sourcePath, []byte(changedContext), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ reports, err := validation.ValidateWithOptions(cfg, validation.Options{RequireState: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ foundStale := false
+ for _, finding := range reports[0].Findings {
+ if finding.Key == "open-button" && finding.Code == validation.CodeSourceStale {
+ foundStale = true
+ }
+ }
+ if !foundStale {
+ t.Fatalf("developer-context change did not stale provenance: %#v", reports[0].Findings)
+ }
+}
+
func TestRunRejectsIncompleteProviderResponseWithoutWritingTarget(t *testing.T) {
dir := t.TempDir()
sourcePath := filepath.Join(dir, "en.json")
diff --git a/internal/validate/findings.go b/internal/validate/findings.go
index c446101..1be3c32 100644
--- a/internal/validate/findings.go
+++ b/internal/validate/findings.go
@@ -20,6 +20,7 @@ const (
CodeSourceStale FindingCode = "source_stale"
CodePolicyStale FindingCode = "policy_stale"
CodeTargetModified FindingCode = "target_modified"
+ CodeNeedsReview FindingCode = "needs_review"
)
// Severity determines whether a finding fails validation.
diff --git a/internal/validate/markup.go b/internal/validate/markup.go
new file mode 100644
index 0000000..b76376c
--- /dev/null
+++ b/internal/validate/markup.go
@@ -0,0 +1,326 @@
+package validate
+
+import (
+ "fmt"
+ "sort"
+ "strings"
+ "unicode"
+)
+
+const localizationNameAttribute = "data-l10n-name"
+
+type markupToken struct {
+ raw string
+ name string
+ attributes []markupAttribute
+ closing bool
+ selfClosing bool
+ comment bool
+ valid bool
+}
+
+type markupAttribute struct {
+ name string
+ value string
+ hasValue bool
+}
+
+type markupFrame struct {
+ tag string
+ slot string
+ owner string
+}
+
+// extractHTMLStructure preserves ordinary HTML structure in order while
+// comparing data-l10n-name elements by semantic identity. This permits a
+// translator to move named rich-text slots without changing their element,
+// protected attributes, nesting, or contained markup.
+func extractHTMLStructure(input string) []string {
+ tokens := scanMarkup(input)
+ root := make([]string, 0, len(tokens))
+ named := make([]string, 0, len(tokens))
+ bodies := make(map[string][]string)
+ stack := make([]markupFrame, 0)
+
+ appendOwned := func(owner, value string) {
+ if owner == "" {
+ root = append(root, value)
+ return
+ }
+ bodies[owner] = append(bodies[owner], value)
+ }
+ currentOwner := func() string {
+ for index := len(stack) - 1; index >= 0; index-- {
+ if stack[index].slot != "" {
+ return stack[index].slot
+ }
+ }
+ return ""
+ }
+
+ for _, token := range tokens {
+ owner := currentOwner()
+ if token.comment {
+ appendOwned(owner, "comment:"+token.raw)
+ continue
+ }
+ if !token.valid {
+ appendOwned(owner, "invalid:"+token.raw)
+ continue
+ }
+ if token.closing {
+ if len(stack) == 0 || stack[len(stack)-1].tag != token.name {
+ appendOwned(owner, "unmatched-close:"+token.canonical())
+ continue
+ }
+ frame := stack[len(stack)-1]
+ stack = stack[:len(stack)-1]
+ if frame.slot != "" {
+ named = append(named, fmt.Sprintf("slot:%s:parent:%s:close:%s", frame.slot, frame.owner, token.name))
+ } else {
+ appendOwned(owner, "close:"+token.name)
+ }
+ continue
+ }
+
+ slot := token.attribute(localizationNameAttribute)
+ if slot != "" {
+ named = append(named, fmt.Sprintf("slot:%s:parent:%s:open:%s", slot, owner, token.canonical()))
+ if owner != "" {
+ appendOwned(owner, "child-slot:"+slot)
+ }
+ if !token.selfClosing && !isVoidElement(token.name) {
+ stack = append(stack, markupFrame{tag: token.name, slot: slot, owner: owner})
+ }
+ continue
+ }
+
+ appendOwned(owner, "open:"+token.canonical())
+ if !token.selfClosing && !isVoidElement(token.name) {
+ stack = append(stack, markupFrame{tag: token.name, owner: owner})
+ }
+ }
+
+ for len(stack) > 0 {
+ frame := stack[len(stack)-1]
+ stack = stack[:len(stack)-1]
+ owner := frame.owner
+ if frame.slot != "" {
+ owner = frame.slot
+ }
+ appendOwned(owner, "unclosed:"+frame.tag)
+ }
+
+ slotNames := make([]string, 0, len(bodies))
+ for slot := range bodies {
+ slotNames = append(slotNames, slot)
+ }
+ sort.Strings(slotNames)
+ structure := append([]string(nil), root...)
+ for _, slot := range slotNames {
+ for index, value := range bodies[slot] {
+ structure = append(structure, fmt.Sprintf("slot-body:%s:%08d:%s", slot, index, value))
+ }
+ }
+ sort.Strings(named)
+ return append(structure, named...)
+}
+
+func scanMarkup(input string) []markupToken {
+ var tokens []markupToken
+ for offset := 0; offset < len(input); {
+ relative := strings.IndexByte(input[offset:], '<')
+ if relative < 0 {
+ break
+ }
+ start := offset + relative
+ if strings.HasPrefix(input[start:], "")
+ if end < 0 {
+ break
+ }
+ end += start + 7
+ tokens = append(tokens, markupToken{raw: input[start:end], comment: true, valid: true})
+ offset = end
+ continue
+ }
+ cursor := start + 1
+ if cursor < len(input) && input[cursor] == '/' {
+ cursor++
+ }
+ if cursor >= len(input) || !isMarkupNameStart(rune(input[cursor])) {
+ offset = start + 1
+ continue
+ }
+ end := markupTagEnd(input, cursor)
+ if end < 0 {
+ break
+ }
+ raw := input[start : end+1]
+ token := parseMarkupToken(raw)
+ token.raw = raw
+ tokens = append(tokens, token)
+ offset = end + 1
+ }
+ return tokens
+}
+
+func markupTagEnd(input string, start int) int {
+ var quote byte
+ for index := start; index < len(input); index++ {
+ character := input[index]
+ if quote != 0 {
+ if character == quote {
+ quote = 0
+ }
+ continue
+ }
+ if character == '\'' || character == '"' {
+ quote = character
+ continue
+ }
+ if character == '>' {
+ return index
+ }
+ }
+ return -1
+}
+
+func parseMarkupToken(raw string) markupToken {
+ inside := strings.TrimSpace(strings.TrimSuffix(strings.TrimPrefix(raw, "<"), ">"))
+ token := markupToken{valid: true}
+ if strings.HasPrefix(inside, "/") {
+ token.closing = true
+ inside = strings.TrimSpace(inside[1:])
+ }
+ if strings.HasSuffix(inside, "/") {
+ token.selfClosing = true
+ inside = strings.TrimSpace(inside[:len(inside)-1])
+ }
+ nameEnd := 0
+ for nameEnd < len(inside) && isMarkupNameRune(rune(inside[nameEnd])) {
+ nameEnd++
+ }
+ if nameEnd == 0 {
+ token.valid = false
+ return token
+ }
+ token.name = strings.ToLower(inside[:nameEnd])
+ remainder := inside[nameEnd:]
+ if token.closing {
+ token.valid = strings.TrimSpace(remainder) == "" && !token.selfClosing
+ return token
+ }
+
+ seen := make(map[string]struct{})
+ for position := 0; ; {
+ position = skipMarkupSpace(remainder, position)
+ if position >= len(remainder) {
+ break
+ }
+ start := position
+ for position < len(remainder) && isMarkupAttributeRune(rune(remainder[position])) {
+ position++
+ }
+ if position == start {
+ token.valid = false
+ return token
+ }
+ attribute := markupAttribute{name: strings.ToLower(remainder[start:position])}
+ position = skipMarkupSpace(remainder, position)
+ if position < len(remainder) && remainder[position] == '=' {
+ attribute.hasValue = true
+ position++
+ position = skipMarkupSpace(remainder, position)
+ if position >= len(remainder) {
+ token.valid = false
+ return token
+ }
+ if remainder[position] == '\'' || remainder[position] == '"' {
+ quote := remainder[position]
+ position++
+ valueStart := position
+ for position < len(remainder) && remainder[position] != quote {
+ position++
+ }
+ if position >= len(remainder) {
+ token.valid = false
+ return token
+ }
+ attribute.value = remainder[valueStart:position]
+ position++
+ } else {
+ valueStart := position
+ for position < len(remainder) && !unicode.IsSpace(rune(remainder[position])) {
+ position++
+ }
+ attribute.value = remainder[valueStart:position]
+ }
+ }
+ if _, duplicate := seen[attribute.name]; duplicate {
+ token.valid = false
+ return token
+ }
+ seen[attribute.name] = struct{}{}
+ token.attributes = append(token.attributes, attribute)
+ }
+ return token
+}
+
+func (token markupToken) attribute(name string) string {
+ for _, attribute := range token.attributes {
+ if attribute.name == name && attribute.hasValue {
+ return attribute.value
+ }
+ }
+ return ""
+}
+
+func (token markupToken) canonical() string {
+ if token.closing {
+ return "/" + token.name
+ }
+ attributes := append([]markupAttribute(nil), token.attributes...)
+ sort.Slice(attributes, func(left, right int) bool { return attributes[left].name < attributes[right].name })
+ var result strings.Builder
+ result.WriteString(token.name)
+ for _, attribute := range attributes {
+ result.WriteByte('|')
+ result.WriteString(attribute.name)
+ if attribute.hasValue {
+ fmt.Fprintf(&result, "=%d:%s", len(attribute.value), attribute.value)
+ }
+ }
+ if token.selfClosing {
+ result.WriteString("|/")
+ }
+ return result.String()
+}
+
+func skipMarkupSpace(input string, position int) int {
+ for position < len(input) && unicode.IsSpace(rune(input[position])) {
+ position++
+ }
+ return position
+}
+
+func isMarkupNameStart(character rune) bool {
+ return unicode.IsLetter(character)
+}
+
+func isMarkupNameRune(character rune) bool {
+ return unicode.IsLetter(character) || unicode.IsDigit(character) || character == ':' || character == '-'
+}
+
+func isMarkupAttributeRune(character rune) bool {
+ return isMarkupNameRune(character) || character == '_' || character == '.'
+}
+
+func isVoidElement(name string) bool {
+ switch name {
+ case "area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr":
+ return true
+ default:
+ return false
+ }
+}
diff --git a/internal/validate/protected.go b/internal/validate/protected.go
index 1a20495..4415797 100644
--- a/internal/validate/protected.go
+++ b/internal/validate/protected.go
@@ -6,6 +6,7 @@ import (
"regexp"
"strings"
+ "github.com/Tom-R-Main/Internationalizer/internal/fluentpattern"
"github.com/Tom-R-Main/Internationalizer/internal/message"
)
@@ -41,6 +42,15 @@ func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath
if mismatch := InterpolationMismatch(key, source, target); mismatch != nil {
findings = append(findings, protectedFinding(key, "interpolation variables", mismatch.SourceVars, mismatch.TargetVars))
}
+ if fluentpattern.LooksLike(source) || fluentpattern.LooksLike(target) {
+ expected, actual, preserved, err := fluentpattern.Compare(source, target)
+ if err != nil {
+ actual = []string{err.Error()}
+ }
+ if err != nil || !preserved {
+ findings = append(findings, protectedFinding(key, "Fluent pattern", expected, actual))
+ }
+ }
}
checks := []struct {
name string
@@ -88,7 +98,7 @@ func protectedFindings(key, source, target, targetLocale, sourcePath, targetPath
}
func extractDocumentHTMLTags(input, documentPath string) []string {
- tags := extractHTMLTags(input)
+ tags := htmlTagRe.FindAllString(input, -1)
for index, tag := range tags {
tags[index] = htmlPathAttrRe.ReplaceAllStringFunc(tag, func(attribute string) string {
match := htmlPathAttrRe.FindStringSubmatch(attribute)
@@ -144,7 +154,7 @@ func resolveDocumentDestination(destination, documentPath string) string {
}
func extractHTMLTags(input string) []string {
- return htmlTagRe.FindAllString(input, -1)
+ return extractHTMLStructure(input)
}
func extractInlineCode(input string) []string {
diff --git a/internal/validate/strict_test.go b/internal/validate/strict_test.go
index 7a61fc8..7939f82 100644
--- a/internal/validate/strict_test.go
+++ b/internal/validate/strict_test.go
@@ -6,6 +6,7 @@ import (
"path/filepath"
"reflect"
"sort"
+ "strings"
"testing"
"time"
@@ -318,6 +319,49 @@ func TestRequireStateReportsAndClearsBoundProvenanceFindings(t *testing.T) {
assertFindingCodes(t, reports[0], CodePolicyStale, CodeSourceStale, CodeTargetModified)
}
+func TestRequireApprovedRequiresExplicitReview(t *testing.T) {
+ cfg := validationConfig(t, map[string]string{"save": "Save"}, map[string]string{"save": "Enregistrer"})
+ resolved, err := policy.Resolve(cfg, "fr", "json", "", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ manifest := state.New()
+ manifest.Set(state.Entry{
+ Bundle: "default",
+ Key: "save",
+ Locale: "fr",
+ SourceHash: state.SourceHash("json", "Save"),
+ PolicyHash: resolved.Hash,
+ TargetHash: state.TargetHash("Enregistrer"),
+ Origin: "provider",
+ ReviewStatus: state.ReviewNeedsReview,
+ UpdatedAt: time.Now().UTC(),
+ })
+ if err := manifest.Save(cfg.ManifestPath); err != nil {
+ t.Fatal(err)
+ }
+
+ reports, err := ValidateWithOptions(cfg, Options{RequireApproved: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ assertFindingCodes(t, reports[0], CodeNeedsReview)
+
+ if _, err := manifest.Approve("default", "save", "fr", time.Now()); err != nil {
+ t.Fatal(err)
+ }
+ if err := manifest.Save(cfg.ManifestPath); err != nil {
+ t.Fatal(err)
+ }
+ reports, err = ValidateWithOptions(cfg, Options{RequireApproved: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if HasFailures(reports) {
+ t.Fatalf("approved current translation failed: %#v", reports[0])
+ }
+}
+
func TestRequireStateRejectsMalformedManifest(t *testing.T) {
cfg := validationConfig(t, map[string]string{"save": "Save"}, map[string]string{"save": "Enregistrer"})
if err := os.WriteFile(cfg.ManifestPath, []byte("{"), 0o644); err != nil {
@@ -422,6 +466,51 @@ func TestProtectedFindingsCarryContextIntoNestedICUBranches(t *testing.T) {
}
}
+func TestProtectedFindingsAllowNamedMarkupSlotsToMove(t *testing.T) {
+ source := `Read the guide or .`
+ target := `Vous pouvez ou lire le guide.`
+ if findings := ProtectedFindings("actions", source, target, "fr"); len(findings) != 0 {
+ t.Fatalf("valid named markup reordering produced findings: %#v", findings)
+ }
+}
+
+func TestProtectedFindingsRejectDamageToNamedMarkupSlots(t *testing.T) {
+ source := `Read the guide.`
+ tests := map[string]string{
+ "changed protected attribute": `Lire le guide.`,
+ "changed element": `Lire .`,
+ "changed contained markup": `Lire le guide.`,
+ "changed slot name": `Lire le guide.`,
+ "unbalanced markup": `Lire le guide.`,
+ }
+ for name, target := range tests {
+ t.Run(name, func(t *testing.T) {
+ if findings := ProtectedFindings("docs", source, target, "fr"); len(findings) == 0 {
+ t.Fatal("damaged named markup was accepted")
+ }
+ })
+ }
+}
+
+func TestProtectedFindingsValidateFluentPatterns(t *testing.T) {
+ source := `{ $count ->
+ [one] One message for { $user }
+ *[other] { $count } messages for { $user }
+}`
+ valid := `{ $count ->
+ [one] Un message pour { $user }
+ [many] { $count } messages pour { $user }
+ *[other] { $count } messages pour { $user }
+}`
+ if findings := ProtectedFindings("messages", source, valid, "fr"); len(findings) != 0 {
+ t.Fatalf("valid Fluent target produced findings: %#v", findings)
+ }
+ damaged := strings.Replace(valid, "{ $user }", "{ $account }", 1)
+ if findings := ProtectedFindings("messages", source, damaged, "fr"); len(findings) == 0 {
+ t.Fatal("damaged Fluent target was accepted")
+ }
+}
+
func TestEvaluationCorpusSchemaAndIDsAreStable(t *testing.T) {
data, err := os.ReadFile(filepath.Join("..", "..", "test", "evaluation", "v1", "cases.json"))
if err != nil {
diff --git a/internal/validate/validate.go b/internal/validate/validate.go
index 471d002..57c9910 100644
--- a/internal/validate/validate.go
+++ b/internal/validate/validate.go
@@ -20,8 +20,9 @@ import (
// Options enables validation contracts that are intentionally opt-in during
// their compatibility period.
type Options struct {
- Strict bool
- RequireState bool
+ Strict bool
+ RequireState bool
+ RequireApproved bool
}
// Report holds validation results for a single locale. Coverage remains a
@@ -56,6 +57,9 @@ func Validate(cfg *config.Config) ([]Report, error) {
// ValidateWithOptions checks all target locales against the source locale.
func ValidateWithOptions(cfg *config.Config, opts Options) ([]Report, error) {
+ if opts.RequireApproved {
+ opts.RequireState = true
+ }
effectiveConfig := *cfg
effectiveConfig.ApplyDefaults()
cfg = &effectiveConfig
@@ -82,10 +86,11 @@ func ValidateWithOptions(cfg *config.Config, opts Options) ([]Report, error) {
if err != nil {
return nil, fmt.Errorf("reading bundle %q source %s: %w", bundle.ID, bundle.Source, err)
}
- sourceKeys, err := format.Parse(sourceData)
+ sourceUnits, err := formats.ParseUnits(format, sourceData)
if err != nil {
return nil, fmt.Errorf("parsing bundle %q source: %w", bundle.ID, err)
}
+ sourceKeys := formats.UnitValues(sourceUnits)
for _, locale := range cfg.TargetLocales {
targetPath, err := bundle.TargetPath(locale)
if err != nil {
@@ -113,7 +118,7 @@ func ValidateWithOptions(cfg *config.Config, opts Options) ([]Report, error) {
policyHash = resolved.Hash
}
- report := validateLocale(bundle.ID, cfg.SourceLocale, locale, bundle.Source, sourceData, sourceKeys, targetPath, format, terms, manifest, policyHash, cfg.Validation.PluralStyle, opts)
+ report := validateLocale(bundle.ID, cfg.SourceLocale, locale, bundle.Source, sourceData, sourceUnits, sourceKeys, targetPath, format, terms, manifest, policyHash, cfg.Validation.PluralStyle, opts)
reports = append(reports, report)
}
}
@@ -127,7 +132,7 @@ func formatForBundle(bundle config.Bundle) (formats.Format, error) {
return formats.FormatForFile(bundle.Source)
}
-func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData []byte, sourceKeys map[string]string, targetPath string, format formats.Format, terms []glossary.Term, manifest *state.Manifest, policyHash, pluralStyle string, opts Options) Report {
+func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData []byte, sourceUnits []formats.Unit, sourceKeys map[string]string, targetPath string, format formats.Format, terms []glossary.Term, manifest *state.Manifest, policyHash, pluralStyle string, opts Options) Report {
report := Report{Bundle: bundle, Locale: locale, TargetPath: targetPath}
if opts.Strict {
translated := 0.0
@@ -172,6 +177,10 @@ func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData
sortFindings(report.Findings)
return report
}
+ sourceUnitsByID := make(map[string]formats.Unit, len(sourceUnits))
+ for _, unit := range sourceUnits {
+ sourceUnitsByID[unit.ID] = unit
+ }
present := 0
translated := 0
@@ -225,7 +234,9 @@ func validateLocale(bundle, sourceLocale, locale, sourcePath string, sourceData
report.Findings = append(report.Findings, glossaryFindings(key, sourceValue, targetValue, terms)...)
}
if opts.RequireState {
- report.Findings = append(report.Findings, provenanceFindings(bundle, key, locale, format.Name(), sourceValue, targetValue, policyHash, manifest)...)
+ sourceUnit := sourceUnitsByID[key]
+ sourceUnit.Value = sourceValue
+ report.Findings = append(report.Findings, provenanceFindings(bundle, key, locale, format.Name(), sourceUnit, targetValue, policyHash, manifest, opts.RequireApproved)...)
}
}
@@ -304,13 +315,13 @@ func missingFinding(key string, requiredPluralKeys map[string]struct{}) Finding
return Finding{Code: CodeMissingKey, Severity: SeverityError, Key: key, Message: "target key is missing"}
}
-func provenanceFindings(bundle, key, locale, format, source, target, policyHash string, manifest *state.Manifest) []Finding {
+func provenanceFindings(bundle, key, locale, format string, source formats.Unit, target, policyHash string, manifest *state.Manifest, requireApproved bool) []Finding {
recorded, ok := manifest.Get(bundle, key, locale)
if !ok {
return []Finding{{Code: CodeUntracked, Severity: SeverityError, Key: key, Message: "target has no manifest provenance"}}
}
var findings []Finding
- if recorded.SourceHash != state.SourceHash(format, source) {
+ if recorded.SourceHash != state.SourceUnitHash(format, source.Value, source.Context, source.Structure) {
findings = append(findings, Finding{Code: CodeSourceStale, Severity: SeverityError, Key: key, Message: "source changed after the recorded translation"})
}
if recorded.PolicyHash != policyHash {
@@ -319,6 +330,9 @@ func provenanceFindings(bundle, key, locale, format, source, target, policyHash
if recorded.TargetHash != state.TargetHash(target) {
findings = append(findings, Finding{Code: CodeTargetModified, Severity: SeverityError, Key: key, Message: "target changed after the recorded translation"})
}
+ if requireApproved && recorded.ReviewStatus != state.ReviewApproved {
+ findings = append(findings, Finding{Code: CodeNeedsReview, Severity: SeverityError, Key: key, Message: "translation has not been explicitly approved"})
+ }
return findings
}
diff --git a/package.json b/package.json
index 39f7e1e..6ccee5a 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "AI-native internationalization CLI for software projects",
"license": "AGPL-3.0-only",
"bin": {
@@ -16,11 +16,11 @@
"node": ">=18"
},
"optionalDependencies": {
- "internationalizer-darwin-arm64": "0.1.1",
- "internationalizer-darwin-x64": "0.1.1",
- "internationalizer-linux-arm64": "0.1.1",
- "internationalizer-linux-x64": "0.1.1",
- "internationalizer-win32-x64": "0.1.1"
+ "internationalizer-darwin-arm64": "0.1.2",
+ "internationalizer-darwin-x64": "0.1.2",
+ "internationalizer-linux-arm64": "0.1.2",
+ "internationalizer-linux-x64": "0.1.2",
+ "internationalizer-win32-x64": "0.1.2"
},
"scripts": {
"test:npm-wrapper": "node --test test/npm-wrapper.test.cjs",
diff --git a/packages/darwin-arm64/package.json b/packages/darwin-arm64/package.json
index 48a01ae..9901c0a 100644
--- a/packages/darwin-arm64/package.json
+++ b/packages/darwin-arm64/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer-darwin-arm64",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "darwin arm64 binary for the internationalizer CLI",
"license": "AGPL-3.0-only",
"os": [
diff --git a/packages/darwin-x64/package.json b/packages/darwin-x64/package.json
index 97a8bd0..fe1d72e 100644
--- a/packages/darwin-x64/package.json
+++ b/packages/darwin-x64/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer-darwin-x64",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "darwin x64 binary for the internationalizer CLI",
"license": "AGPL-3.0-only",
"os": [
diff --git a/packages/linux-arm64/package.json b/packages/linux-arm64/package.json
index d7db2a7..48a85ee 100644
--- a/packages/linux-arm64/package.json
+++ b/packages/linux-arm64/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer-linux-arm64",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "linux arm64 binary for the internationalizer CLI",
"license": "AGPL-3.0-only",
"os": [
diff --git a/packages/linux-x64/package.json b/packages/linux-x64/package.json
index 33a011a..9728090 100644
--- a/packages/linux-x64/package.json
+++ b/packages/linux-x64/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer-linux-x64",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "linux x64 binary for the internationalizer CLI",
"license": "AGPL-3.0-only",
"os": [
diff --git a/packages/win32-x64/package.json b/packages/win32-x64/package.json
index 973d4c6..105b62d 100644
--- a/packages/win32-x64/package.json
+++ b/packages/win32-x64/package.json
@@ -1,6 +1,6 @@
{
"name": "internationalizer-win32-x64",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "win32 x64 binary for the internationalizer CLI",
"license": "AGPL-3.0-only",
"os": [
diff --git a/test/acceptance/acceptance_test.go b/test/acceptance/acceptance_test.go
index b2a0be1..0c6f069 100644
--- a/test/acceptance/acceptance_test.go
+++ b/test/acceptance/acceptance_test.go
@@ -202,6 +202,81 @@ manifest_path: .internationalizer.lock
}
}
+func TestFluentPseudoAndExplicitReviewLifecycle(t *testing.T) {
+ projectDir := t.TempDir()
+ mustWriteFile(t, filepath.Join(projectDir, "locales", "en.ftl"), `# Link shown below the item count.
+items =
+ { $count ->
+ [one] One item
+ *[other] { $count } items
+ }
+learn-more = See the documentation.
+save-button =
+ .label = Save
+`)
+ mustWriteFile(t, filepath.Join(projectDir, "locales", "fr.ftl"), `# Link shown below the item count.
+items =
+ { $count ->
+ [one] Un élément
+ *[other] { $count } éléments
+ }
+learn-more = Consultez la documentation.
+save-button =
+ .label = Enregistrer
+`)
+ mustWriteFile(t, filepath.Join(projectDir, ".internationalizer.yml"), `source_locale: en
+target_locales: [fr]
+bundles:
+ - id: browser
+ source: locales/en.ftl
+ target: locales/{locale}.ftl
+ format: fluent
+manifest_path: .internationalizer.lock
+`)
+
+ adopt := runCLI(t, projectDir, nil, "translate", "--adopt-existing")
+ adopt.requireSuccess(t)
+ listed := runCLI(t, projectDir, nil, "review", "list", "--status", "needs_review")
+ listed.requireSuccess(t)
+ if got := strings.Count(listed.stdout, "needs_review"); got != 3 {
+ t.Fatalf("needs-review entries = %d, want 3:\n%s", got, listed.stdout)
+ }
+
+ required := runCLI(t, projectDir, nil, "validate", "--require-approved", "--json")
+ if required.exitCode != 1 || !strings.Contains(required.stdout, `"code": "needs_review"`) {
+ t.Fatalf("unapproved validation = exit %d:\n%s\n%s", required.exitCode, required.stdout, required.stderr)
+ }
+ approve := runCLI(t, projectDir, nil, "review", "approve", "--locale", "fr", "--all")
+ approve.requireSuccess(t)
+ approved := runCLI(t, projectDir, nil, "validate", "--require-approved", "--json")
+ approved.requireSuccess(t)
+
+ pseudo := runCLI(t, projectDir, nil, "pseudo", "--strategy", "accented")
+ pseudo.requireSuccess(t)
+ pseudoOutput := string(mustReadFile(t, filepath.Join(projectDir, "locales", "en-XA.ftl")))
+ for _, protected := range []string{"{ $count ->", "*[other]", "{ $count }", `data-l10n-name="docs"`, ".label ="} {
+ if !strings.Contains(pseudoOutput, protected) {
+ t.Fatalf("pseudo Fluent output lost %q:\n%s", protected, pseudoOutput)
+ }
+ }
+ manifest, err := state.Load(filepath.Join(projectDir, ".internationalizer.lock"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, entry := range manifest.Translations {
+ switch entry.Locale {
+ case "fr":
+ if entry.ReviewStatus != state.ReviewApproved {
+ t.Fatalf("French entry was not approved: %#v", entry)
+ }
+ case "en-XA":
+ if entry.Origin != "pseudo" || entry.ReviewStatus != state.ReviewNeedsReview {
+ t.Fatalf("pseudo entry provenance = %#v", entry)
+ }
+ }
+ }
+}
+
func TestOpenAITranslationExercisesResponsesContractAndPreservesSourceShape(t *testing.T) {
var requests atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {