Skip to content

Hi en fixes - #476

Open
mayuris-00 wants to merge 8 commits into
NVIDIA:staging/hi_en_itn_codeswitchedfrom
mayuris-00:hi-en-fixes
Open

Hi en fixes#476
mayuris-00 wants to merge 8 commits into
NVIDIA:staging/hi_en_itn_codeswitchedfrom
mayuris-00:hi-en-fixes

Conversation

@mayuris-00

@mayuris-00 mayuris-00 commented Aug 20, 2026

Copy link
Copy Markdown

What does this PR do ?

Three small fixes to the Hindi ITN grammar, plus hi_en test coverage.

  • measurements.tsv: 8 rows for per-unit measures (m/s, km/s, m/min, kg/h, L/s, ns).
    The table already had मील प्रति सेकंड and मील प्रति घंटा but not the metre,
    kilogram or litre equivalents, so the unit converted while प्रति X stayed as words.
  • whitelist.tsv: 4 honorifics (प्रो., इंजी., पं., स्व.). The table held 7 entries.
  • time.py: accept शून्य as an hour. Hours were composed through range(1, 25) and
    graph_digit covers only १ to ९, so शून्य बजकर तीस मिनट could not parse as a time
    and fell to the measure grammar as ० बजकर ३० m. Now returns ०:३०.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

mgrafu and others added 7 commits August 20, 2026 13:20
* Add Portuguese (PT) text normalization: cardinal, ordinal, decimal, fraction (NVIDIA#403)

* Add Portuguese (PT) text normalization: cardinal, ordinal, decimal, fraction

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Mai Anh <palasek182@gmail.com>

* date and time semiotic classese

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Mai Anh <palasek182@gmail.com>

* update sh files

Signed-off-by: Mai Anh <palasek182@gmail.com>

* Update Portuguese text normalization tutorial with enhanced examples and outputs

- Changed the language parameter in the Normalizer instance from 'en' to 'pt'.
- Added detailed output examples for the normalizer's methods, including documentation for `__doc__` and `normalize()`.
- Updated example input string to reflect a more complex Portuguese sentence for normalization.
- Adjusted execution counts for code cells to ensure proper order of execution.

This update aims to improve the clarity and usability of the tutorial for Portuguese text normalization.

Signed-off-by: Mai Anh <palasek182@gmail.com>

* remove current unuse file

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update minor update and punct

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mai Anh <palasek182@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* PT TN: money, measure, telephone, electronic (NVIDIA#416)

* PT TN: money, measure, telephone, electronic

Adds semiotic classes and tests on top of staging/pt-br_tn; includes
cardinal fix for X00 + 01–09 and Sparrowhawk script updates.

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bugs based on review cardinal, fraction, money, measure

Signed-off-by: Mai Anh <palasek182@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* modify test case time

Signed-off-by: Mai Anh <palasek182@gmail.com>

* modify with mariana's review

Signed-off-by: Mai Anh <palasek182@gmail.com>

---------

Signed-off-by: Mai Anh <palasek182@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update jenkins cache

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* add init to whitelist

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

---------

Signed-off-by: Mai Anh <palasek182@gmail.com>
Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Mai Anh <palasek182@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Add hi_en Code Switched (NVIDIA#415)

* Add hi_en Code Switched

Signed-off-by: RajanPutty <rputty@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address PR NVIDIA#415 review: restore ko, dedupe whitelists, expand hi_en tests, add hi_en CI

Signed-off-by: Rajan Putty <rputty@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: RajanPutty <rputty@nvidia.com>
Signed-off-by: Rajan Putty <rputty@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update jenkins

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* Add hi_en Code Switched (NVIDIA#415)

* Add hi_en Code Switched

Signed-off-by: RajanPutty <rputty@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address PR NVIDIA#415 review: restore ko, dedupe whitelists, expand hi_en tests, add hi_en CI

Signed-off-by: Rajan Putty <rputty@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: RajanPutty <rputty@nvidia.com>
Signed-off-by: Rajan Putty <rputty@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* update jenkins

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* fix jenkins bug

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* separate cache dirs

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* refresh cache dirs

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* refresh cache dir ko

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

---------

Signed-off-by: RajanPutty <rputty@nvidia.com>
Signed-off-by: Rajan Putty <rputty@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>
Co-authored-by: RajanPutty <rputty@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* ar money bugfix and sh tests

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* address review

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

---------

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* change normalization of alphanum terms

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address review

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Hi TN: fraction class improvements with lexical mappings for special cases (NVIDIA#420)

* Fix: fraction class improvements with lexical mappings

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move common fraction mappings to TSV and add ASCII digit support

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins date conflict resolved

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN date class accuracy improvement (NVIDIA#418)

* date class accuracy improvement

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Jenkins file date update for Hi TN

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Reviewer feedback regarding date tagger, tsv files and inclusion of more test cases

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* hi-tn-date: minor formatting fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date tagger modification according to feedback removed support for d-mm-yyyy, dd-m-yyyy and mm-yyyy

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* replace open()+union() with string_map for suffix and prefix unions

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN money class bug fix (NVIDIA#424)

* Hi TN money class bug fix

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added test cases with decimal portion having more than two digits for Indian Currency

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths (NVIDIA#425)

* Fix partial TSV matching, added support for chemical formulas, and relative file paths

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor electronic class with dynamic symbol matrix and chemical rules. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic.

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Remove single capital letter units from Measure class (NVIDIA#429)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Implement Serial tagger (NVIDIA#440)

* Hi TN Serial: Implement SerialFst tagger for Devanagari-numeric mixtures, number chains, and mathematical powers

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address feedback for Hindi Serial Tagger PR

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi TN: Implement Roman semiotic class (NVIDIA#442)

* Hi TN Roman class implementation

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Refactor Roman FST to utilize shared serial Devanagari chars

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Add __init__.py for serial and roman data folders

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update roman_graph weight to 1.07 and other minor changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* reducing uncommon entries in electronic class tsv data files (NVIDIA#458)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* Hi TN: Address, Electronic, Serial, and Cardinal FST Optimization and Bug Fix (NVIDIA#461)

* Hi TN Address, Electronic, Serial, and Cardinal FST Optimizations and Bug Fixes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* review feedback changes

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feedback review changes for address and electronic

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Hi tn v3: staging to main error fixed (NVIDIA#466)

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>

---------

Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Shreyas Pawar <shrpawar@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Ko TN: Add serial grammar and review fixes (NVIDIA#457)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* KO TN: Update Jenkins cache (NVIDIA#467)

* Add Korean TN post-processing rules for particle agreement and month handling

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN fraction test cases for particle agreement

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix date and fraction normalization issues based on review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Korean TN serial and review fixes

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address Korean TN review feedback

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolve Korean telephone conflict and remove serial references

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address serial and telephone review comments

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update Korean TN Jenkins cache

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Signed-off-by: Jinwoo Bae <bbae7050@gmail.com>
Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Co-authored-by: Jinwoo Bae <bbae7050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Mariana <47233618+mgrafu@users.noreply.github.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: mayuris-00 <mayuris@nvidia.com>
@mayuris-00
mayuris-00 marked this pull request as ready for review August 20, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants