Hi en fixes - #476
Open
mayuris-00 wants to merge 8 commits into
Open
Conversation
* Add Portuguese (PT) text normalization: cardinal, ordinal, decimal, fraction (NVIDIA#403) * Add Portuguese (PT) text normalization: cardinal, ordinal, decimal, fraction Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Mai Anh <palasek182@gmail.com> * date and time semiotic classese Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Mai Anh <palasek182@gmail.com> * update sh files Signed-off-by: Mai Anh <palasek182@gmail.com> * Update Portuguese text normalization tutorial with enhanced examples and outputs - Changed the language parameter in the Normalizer instance from 'en' to 'pt'. - Added detailed output examples for the normalizer's methods, including documentation for `__doc__` and `normalize()`. - Updated example input string to reflect a more complex Portuguese sentence for normalization. - Adjusted execution counts for code cells to ensure proper order of execution. This update aims to improve the clarity and usability of the tutorial for Portuguese text normalization. Signed-off-by: Mai Anh <palasek182@gmail.com> * remove current unuse file Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update minor update and punct Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Mai Anh <palasek182@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * PT TN: money, measure, telephone, electronic (NVIDIA#416) * PT TN: money, measure, telephone, electronic Adds semiotic classes and tests on top of staging/pt-br_tn; includes cardinal fix for X00 + 01–09 and Sparrowhawk script updates. Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bugs based on review cardinal, fraction, money, measure Signed-off-by: Mai Anh <palasek182@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * modify test case time Signed-off-by: Mai Anh <palasek182@gmail.com> * modify with mariana's review Signed-off-by: Mai Anh <palasek182@gmail.com> --------- Signed-off-by: Mai Anh <palasek182@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update jenkins cache Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * add init to whitelist Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> --------- Signed-off-by: Mai Anh <palasek182@gmail.com> Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com> Co-authored-by: Mai Anh <palasek182@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Add hi_en Code Switched (NVIDIA#415) * Add hi_en Code Switched Signed-off-by: RajanPutty <rputty@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address PR NVIDIA#415 review: restore ko, dedupe whitelists, expand hi_en tests, add hi_en CI Signed-off-by: Rajan Putty <rputty@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: RajanPutty <rputty@nvidia.com> Signed-off-by: Rajan Putty <rputty@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update jenkins Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * Add hi_en Code Switched (NVIDIA#415) * Add hi_en Code Switched Signed-off-by: RajanPutty <rputty@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address PR NVIDIA#415 review: restore ko, dedupe whitelists, expand hi_en tests, add hi_en CI Signed-off-by: Rajan Putty <rputty@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: RajanPutty <rputty@nvidia.com> Signed-off-by: Rajan Putty <rputty@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update jenkins Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * fix jenkins bug Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * separate cache dirs Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * refresh cache dirs Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * refresh cache dir ko Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> --------- Signed-off-by: RajanPutty <rputty@nvidia.com> Signed-off-by: Rajan Putty <rputty@nvidia.com> Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com> Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> Co-authored-by: RajanPutty <rputty@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
* ar money bugfix and sh tests Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * address review Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> --------- Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
* change normalization of alphanum terms Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address review Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Mariana Graterol Fuenmayor <marianag@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Hi TN: fraction class improvements with lexical mappings for special cases (NVIDIA#420) * Fix: fraction class improvements with lexical mappings Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * move common fraction mappings to TSV and add ASCII digit support Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Jenkins date conflict resolved Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Hi TN date class accuracy improvement (NVIDIA#418) * date class accuracy improvement Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Jenkins file date update for Hi TN Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Reviewer feedback regarding date tagger, tsv files and inclusion of more test cases Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * hi-tn-date: minor formatting fix Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * date tagger modification according to feedback removed support for d-mm-yyyy, dd-m-yyyy and mm-yyyy Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * replace open()+union() with string_map for suffix and prefix unions Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Hi TN money class bug fix (NVIDIA#424) * Hi TN money class bug fix Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * added test cases with decimal portion having more than two digits for Indian Currency Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Hi TN Electronic: Fix partial TSV matching, chemical formula logic, and relative file paths (NVIDIA#425) * Fix partial TSV matching, added support for chemical formulas, and relative file paths Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor electronic class with dynamic symbol matrix and chemical rules. Replaced hardcoded TSVs and Python symbol rules with a data-driven symbol_classes.tsv and dynamic elements.tsv. Fixed greedy over-tagging of standard English words and added Unicode support for complex chemical ions without breaking URL hyphen logic. Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Remove single capital letter units from Measure class (NVIDIA#429) Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Hi TN: Implement Serial tagger (NVIDIA#440) * Hi TN Serial: Implement SerialFst tagger for Devanagari-numeric mixtures, number chains, and mathematical powers Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address feedback for Hindi Serial Tagger PR Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Hi TN: Implement Roman semiotic class (NVIDIA#442) * Hi TN Roman class implementation Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Refactor Roman FST to utilize shared serial Devanagari chars Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Add __init__.py for serial and roman data folders Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update roman_graph weight to 1.07 and other minor changes Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Feedback changes Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * reducing uncommon entries in electronic class tsv data files (NVIDIA#458) Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * Hi TN: Address, Electronic, Serial, and Cardinal FST Optimization and Bug Fix (NVIDIA#461) * Hi TN Address, Electronic, Serial, and Cardinal FST Optimizations and Bug Fixes Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * review feedback changes Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feedback review changes for address and electronic Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Hi tn v3: staging to main error fixed (NVIDIA#466) Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> --------- Signed-off-by: Shreyas Pawar <shrpawar@nvidia.com> Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com> Co-authored-by: Shreyas Pawar <shrpawar@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
* Ko TN: Add serial grammar and review fixes (NVIDIA#457) * Add Korean TN post-processing rules for particle agreement and month handling Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Korean TN fraction test cases for particle agreement Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix date and fraction normalization issues based on review feedback Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Korean TN serial and review fixes Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Korean TN review feedback Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve Korean telephone conflict and remove serial references Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address serial and telephone review comments Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * KO TN: Update Jenkins cache (NVIDIA#467) * Add Korean TN post-processing rules for particle agreement and month handling Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Korean TN fraction test cases for particle agreement Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * Fix Korean fraction verbalization with particle-aware handling and remove post_processing dependency Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix date and fraction normalization issues based on review feedback Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Korean TN serial and review fixes Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Korean TN review feedback Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve Korean telephone conflict and remove serial references Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address serial and telephone review comments Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Korean TN Jenkins cache Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> --------- Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --------- Signed-off-by: Jinwoo Bae <bbae7050@gmail.com> Signed-off-by: Mariana <47233618+mgrafu@users.noreply.github.com> Co-authored-by: Jinwoo Bae <bbae7050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Mariana <47233618+mgrafu@users.noreply.github.com> Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
mayuris-00
force-pushed
the
hi-en-fixes
branch
from
August 20, 2026 07:51
56c241e to
66b9c1f
Compare
Signed-off-by: mayuris-00 <mayuris@nvidia.com>
mayuris-00
marked this pull request as ready for review
August 20, 2026 07:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Three small fixes to the Hindi ITN grammar, plus hi_en test coverage.
The table already had मील प्रति सेकंड and मील प्रति घंटा but not the metre,
kilogram or litre equivalents, so the unit converted while प्रति X stayed as words.
graph_digit covers only १ to ९, so शून्य बजकर तीस मिनट could not parse as a time
and fell to the measure grammar as ० बजकर ३० m. Now returns ०:३०.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.