Repository navigation
Conversation
Add Romanian (`ro`) as a self-contained language: - RomanianNumberNormalizer: spelled-out cardinals to digits (units, teens, tens, "și" compounds, hundreds, mie/milion/miliard with the "de" Romanian uses from 20 up), the colloquial contracted forms (cinșpe = 15, douăj = 20), ordinals after "al"/"a" and digit ordinals (al 3-lea, a 3-a). "un"/"o" count as 1 only before sută/mie/milion/miliard, so articles stay words. Words match with or without diacritics, and the legacy cedilla letters ş/ţ are read as ș/ț. - LanguageConfig: decimal comma with "virgula" (folded, since the decimal-word steps run after remove_diacritics), thousands dot, symbols and currencies, digit and number words. - Filler words only where their diacritic-folded form cannot be a Romanian word: "ăă" becomes "aa" and is removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept. - Replacements: okay/oke/okei/okey → ok. - Unit tests for the number normalizer and operators, a gladia-3 e2e CSV, README row.
- Drop the "de" that links a number of at least 20 (or ending in 00) to its noun, so
"20°", "20 de grade" and "douăzeci de grade" all normalize to "20 grade" (and
"300 de lei" / "trei sute de lei" to "300 lei"). "de" after smaller numbers and in
"de la" ("from") is kept.
- Recognize the informal teens of everyday speech ("cincișpe"/"cincispe" = 15) next to
the standard and most reduced forms.
- Read "și" compounds written as one word ("douăzecișicinci" = 25).
- E2e cases rewritten as natural sentences (battery level, discounts, prices, temperature).
"1.200 €", "1.200€" and "€1.200" become "1200 euro", matching the spoken "o mie două sute de euro" (the shared pipeline drops a symbol written after the amount with a space). As in the Danish normalizer, the Romanian number normalizer rewrites the config's currency symbols itself.
- Spoken clock times after "la"/"ora"/"e" become H:MM before the time-colon protection, so they match written times: "la opt și jumătate" / "la opt jumate" / "la ora 8.30" → 8:30, "opt și(-un) sfert" → 8:15, "opt fără un sfert" and "șapte patruzeci și cinci" → 7:45, "opt fără zece" → 7:50 (time_normalizer.py, via normalize_numeric_time_formats). - A half after an amount is its decimal: "trei kile jumate", "trei kilograme și jumătate" and "3,5 kg" all become "3 virgula 5 kg"; "jumătate de ..." is left alone. - Units: kil/kile/kilogram(e) → kg, kilometru/kilometri → km. - Currency codes and acronyms stay words (expand_all_caps_letter_by_letter=False): "RON" is "ron", not "r o n", and stays distinct from "lei". - Informal numbers: douășpe/doușpe (12, feminine), fused tens-and-five (douășcinci, treișcinci, patrușcinci).
|
Thanks a lot for this contribution @eek, the Romanian support looks great and is really appreciated! One thing is blocking the merge for now: the Lint commit messages check fails on the last commit ( Our commitlint config requires the subject to start with a lowercase letter. Since it's the latest commit, an amend is enough, for example: git commit --amend -m "feat: romanian clock times, halves, units and currency codes" -m "$(git log -1 --format=%b)"
git push --force-with-leaseOnce the check is green we can move forward. Thanks again! |
| def at(k: int) -> str: | ||
| return rest[k] if k < len(rest) else "" | ||
|
|
||
| if at(0) in _HALF: |
There was a problem hiding this comment.
False positive: percentages and amounts are read as clock times
After la / e, any X și jumătate / X și un sfert becomes H:MM, even when an amount follows:
Inflația a crescut la trei și jumătate la sută. → inflatia a crescut la 3:30 la suta
Inflația e trei și jumătate la sută. → inflatia e 3:30 la suta
Rata a scăzut la doi și un sfert la sută. → rata a scazut la 2:15 la suta
Inflația a urcat la 3,5%. → inflatia a urcat la 3 virgula 5 la suta
So a spelled-out reference and a digit hypothesis (or the reverse) no longer match, and this pattern is frequent in news-style text such as FLEURS. Could you skip the time reading when the phrase is followed by la sută / %, a unit, or mii / milioane, and add these sentences to ro.csv?
| for before, after, word in self._currency: | ||
| text = before.sub(rf"\1 {word}", text) | ||
| text = after.sub(rf"\1 {word}", text) |
There was a problem hiding this comment.
Currency symbol binds to the next number
The symbol-before rule runs first, so a symbol written after an amount gets attached to the following number instead, and the digits then collapse:
Am cumpărat 5 € 2 kilograme de mere. → am cumparat 52 euro kg de mere
Since Romanian writes the symbol after the amount, applying the after pattern first (or not matching before when the symbol directly follows a number) should fix it. A test row for this case would be great too.
karamouche
left a comment
karamouche
left a comment
There was a problem hiding this comment.
Thanks for your contribution! Here is some feedback flagged by AI. I don't speak Romanian, so I don't know if it's all relevant, but I'm flagging it so you can help.
Details are in the inline comments, plus the commitlint fix mentioned above.
This branch has not been deployed
2 participants
What does this PR do?
Adds Romanian (
ro) normalization: spelled-out numbers to digits, Romanian fillers, symbols,currencies, clock times and decimal conventions, so Romanian STT output and references compare fairly.
Type of change
Checklist
New language
normalization/languages/romanian/withoperators.py,replacements.py,__init__.pyreplacements.py(not hardcoded inoperators.py)LanguageConfigis filled in with the language's data (separators, currency words, digit words, …)LanguageOperators— onlyexpand_written_numbersandget_word_replacementsare overridden@register_languageand imported innormalization/languages/__init__.pytests/unit/languages/tests/e2e/files/gladia-3/ro.csvNotes
text2numhas no Romanian, sonumber_normalizer.pyparses cardinals: units, teens, tens,"și" compounds (douăzeci și cinci = 25), hundreds, mie/milion/miliard with the "de" Romanian
uses from 20 up (douăzeci de mii = 20000), colloquial contracted forms (cinșpe = 15,
douăj = 20), and ordinals after "al"/"a" (al treilea = al 3-lea = al 3).
removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept.
decimal_wordis the folded "virgula", because the decimal-word steps run afterremove_diacritics; this makes "3,5" and "trei virgulă cinci" agree."o sută de lei"); written forms ("20°", "20 grade") have none, so that "de" is dropped after
such numbers. It is kept after smaller numbers and in "de la" ("from").
forms ("cinșpe") are recognized, as are "și" compounds written as one word ("douăzecișicinci").
matching "o mie două sute de euro". The shared pipeline drops a symbol written after the
amount with a space, so, as in the Danish normalizer, the Romanian number normalizer
rewrites the config's currency symbols itself.
"opt fără un sfert", "șapte patruzeci și cinci", and "8.30" written with a dot) become H:MM.
They are converted in
normalize_numeric_time_formatsbecause it runs before the time-colonprotection; a dedicated hook for spoken times may be preferable in a future preset.
"3,5 kg"); kilogram/kilometre spellings map to kg/km.
expand_all_caps_letter_by_letter=False): "RON" is not "lei".How was this tested?