--- title: Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali date: 2026-07-26 --- # Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali # Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali Paper: [Pronunciation-Aware Syllable Tokenizer for Nepali Automatic Speech Recognition System](https://aclanthology.org/2023.icon-1.4/) (Ghimire et al., ICON 2023) Github: [Aananda-giri/nepali-tokenizer](https://github.com/Aananda-giri/nepali-tokenizer) The paper proposes a syllable-level tokenizer for Nepali that improves ASR accuracy (8.09% CER vs 9.3% for a character tokenizer) by splitting text into units that actually correspond to how Nepali is *pronounced*, instead of splitting on raw Unicode characters or BPE merges. I wanted to reproduce it, but the paper's tokenizer depends on a pre-built **syllable dataset** (a lexicon of valid Nepali syllables used for lookup), and that dataset isn't published anywhere. So instead of reproducing the lookup table, I wrote a small rule-based tokenizer that encodes the same linguistic rules directly — no lexicon required. It reproduces the paper's own worked examples exactly. Two implementations are in this directory: `nepali_syllabic_tokenizer.py` is the rule-based one, and `paper_algorithm.py` is the paper's Algorithm 1 transcribed and run against a generated lexicon, so the two can be compared directly (§5). ## 1. Why character/BPE tokenizers break on Devanagari Devanagari is an *abugida*: a consonant carries an inherent vowel sound (क = "ka"), and small marks called **vowel markers** (matras) attached above/below/beside it change that vowel (कि = "ki", के = "ke"). A **halant** (्) cancels the inherent vowel, letting consonants stack into clusters (क् + ष = क्ष). None of these marks are pronounceable on their own — they only mean something attached to a consonant. The paper's motivating example is the word **क्रान्ति** ("revolution"). A character tokenizer splits it into individual Unicode codepoints, several of which — the vowel marker ा and the halant ् — carry no sound by themselves. A BPE tokenizer does a bit better by co-occurrence, but still isn't guaranteed to respect syllable boundaries, since it optimizes for corpus statistics, not pronunciation. What you actually want is: **क्रा**, **न्**, **ति** — three phonetic chunks that map cleanly onto three spoken syllables, each alignable to a slice of audio for ASR training. ## 2. The paper's approach: sliding window + lexicon lookup The paper first builds a **syllable dataset**: every grammatically valid syllable in Nepali, generated by combining consonants, complex consonants, vowel markers, halant, and other markers according to Devanagari grammar rules. Tokenization then works like a greedy longest-match parser over that lexicon. Here is Algorithm 1 as printed in the paper: ``` Algorithm 1: Syllabic Tokenizer for Nepali Input: Sentence : S Output: Tokens : FT Data: Syllable Dataset : SD Window Size : win_size 1 FT = [ ] 2 T = [ordered list of all characters in S] 3 while current_win_pos > len(T) do 4 t_window = T[current_win_pos : (current_win_pos + win_size)] 5 ct = { all possible syllables from t_window each starting 0th position } 6 foreach ct_cur in ct do 7 if ct_cur in SD then 8 current_win_pos += len(ct_cur) 9 FT.append(ct_cur) 10 break 11 return FT ``` The sentence is first character-tokenized into an ordered list, eq. (1): ``` T = list(C₁, C₂ ... C_N) ``` Window size is 4, because "a maximum of 4 characters are used to form the complex syllables" (paper §4.2). Candidates are the prefixes of the window, longest first — eq. (2), and eq. (3) for the window that follows a 3-character match: ``` CTw₁ = { C₁C₂C₃C₄, C₁C₂C₃, C₁C₂, C₁ } CTw₂ = { C₄C₅C₆C₇, C₄C₅C₆, C₄C₅, C₄ } ``` The first candidate found in `SD` wins, and the window advances by however many characters it consumed. Figure 1 traces this for `क्षेत्रमा कसैका`, over the character list `क ् ष े त ् र म ा क स ै क ा`: | Window | Candidates `CT` | Hit | `FT` after | |---|---|---|---| | 1 | `{ क्षे, क्ष, क्, क }` | क्षे (4 chars) | `{ क्षे }` | | 2 | `{ त्रम, त्र, त्, त }` | त्र (3 chars) | `{ क्षे, त्र }` | | 3 | `{ माकस, माक, मा, म }` | मा (2 chars) | `{ क्षे, त्र, मा }` | …continuing to the final `FT = { क्षे, त्र, मा, क, सै, का }`. (The candidate sets are transcribed from the figure, where `T` silently drops the space between the two words — hence `माकस`. A working implementation has to carry the space through, since word boundaries matter downstream; both implementations here emit it as its own token, so the real third window is `{मा क, मा , मा, म}` and still hits `मा`.) Longest-first ordering is what resolves ambiguity. The paper's example is `प्रिय`, where `CTwₓ = {प्रि, प्र, प्, प}` contains three entries that are all valid Nepali syllables — `प्रि` is chosen because the scan starts from the longest. So this is maximal-munch tokenization with a dictionary — the same family of idea as WordPiece, except the "dictionary" is a hand-generated set of phonetically valid syllables rather than a corpus-derived vocabulary. **Transcribing it, three things don't run as printed.** Line 3's comparison is inverted (`>` should be `<`, or the loop never executes); `current_win_pos` is never initialized; and no branch is given for "nothing in `ct` is in `SD`" — as written the position doesn't advance and the loop hangs, so any character outside the lexicon is an infinite loop. `paper_algorithm.py` implements the algorithm with those three fixed and nothing else changed. (Figure 1 also labels its third window `CTw2`, and eq. (3) assumes the first window consumed 3 characters while the figure's consumed 4 — cosmetic, but it costs a re-read.) The real catch is the lexicon: it's what makes step 7 work, and it isn't published. Section 3 rebuilds the tokenizer without one. ## 3. Reproducing it without a lexicon Devanagari's structure means you don't actually *need* to look a syllable up to know where it ends — the character classes themselves tell you. `SyllableDataset` (in `nepali_syllabic_tokenizer.py`) defines the classes: ```python VOWELS = ['अ', 'आ', 'इ', 'ई', 'उ', 'ऊ', 'ए', 'ऐ', 'ओ', 'औ', 'ऋ', 'ॠ', 'ऌ', 'ऑ', 'ऍ'] # candra o/e, for loanwords CONSONANTS = ['क', 'ख', 'ग', ... , 'स', 'ह'] # 33 consonants HALANT = '्' # cancels the inherent vowel VOWEL_MARKERS = ['ा', 'ि', 'ी', 'ु', 'ू', 'े', 'ै', 'ो', 'ौ', 'ृ', 'ॉ', 'ॅ', 'ॄ'] # matras OTHER_MARKERS = ['ं', 'ः', 'ँ'] # anusvara, visarga, chandrabindu COMPLEX_CONSONANTS = ['क्ष', 'ज्ञ', 'द्ध', 'द्य', 'त्त', 'द्व'] + \ [c + '्' + 'र' for c in CONSONANTS] # e.g. क्र, ग्र, त्र — all length 3 # everything that attaches to the previous token instead of starting one ATTACHING_MARKERS = frozenset(VOWEL_MARKERS + [HALANT] + OTHER_MARKERS) ``` `COMPLEX_CONSONANTS` is the group that matters: a handful of consonant clusters act as a single unit even though they're three Unicode characters — six listed by hand (क्ष, ज्ञ, द्ध, द्य, त्त, द्व) plus the 33 generated *consonant + halant + र* forms (क्र, त्र, श्र, …), 39 in total. That list is the only thing that needs to be enumerated explicitly; everything else follows from the four character classes above. (The six hand-listed ones are the shakiest part of the whole scheme — see §8.) `VOWEL_MARKERS` is the one where an omission is silent *data loss* rather than mis-segmentation: anything missing falls through to the non-Devanagari branch and gets dropped (§7). `ॉ` and `ॅ` are the candra vowels that show up in loanwords (कॉलेज, डॉक्टर), and leaving them out quietly turned *kollej* into *kalej* — a different word, no error raised. `ॆ` and `ॊ` are deliberately excluded: they're Dravidian-range short e/o, not used in Nepali. ### The rule > Scan left to right. A **vowel**, **consonant**, or **complex consonant** always starts a new token. A **vowel marker**, **halant**, or **other marker** never starts a token — it always attaches to whatever token came before it. That's the entire algorithm, in `nepali_syllabic_tokenizer.py`: ```python while i < len(text): first_three_chars = text[i:i+3] char = text[i] if char.isspace(): tokens.append(" ") i += 1 continue # complex consonant (क्ष, त्र, ग्र, ...) — 3 chars, one syllable if first_three_chars in SyllableDataset.COMPLEX_CONSONANTS_SET: tokens.append(first_three_chars) i += 3 continue # vowel or plain consonant — starts a new syllable if char in SyllableDataset.VOWELS_SET or char in SyllableDataset.CONSONANTS_SET: tokens.append(char) i += 1 continue # matra / halant / anusvara-visarga-chandrabindu — glue to the last token if char in SyllableDataset.ATTACHING_MARKERS: tokens[-1] += char i += 1 continue ``` No window, no candidate generation, no dictionary lookup — just "does this character start a syllable, or does it belong to the previous one?" ### Walking through क्रान्ति | i | char | class | action | tokens so far | |---|------|-------|--------|----------------| | 0 | क्र (3-char lookahead) | complex consonant | new token | `['क्र']` | | 3 | ा | vowel marker | attach to last | `['क्रा']` | | 4 | न | consonant | new token | `['क्रा', 'न']` | | 5 | ् | halant | attach to last | `['क्रा', 'न्']` | | 6 | त | consonant | new token | `['क्रा', 'न्', 'त']` | | 7 | ि | vowel marker | attach to last | `['क्रा', 'न्', 'ति']` | Result: `['क्रा', 'न्', 'ति']` — three phonetic syllables, exactly as intended. ## 4. Does it match the paper's own examples? The paper publishes a worked example (Figure 1) and a results table (Table 3). Running this implementation on the same inputs: ```python from nepali_syllabic_tokenizer import NepaliSyllabicTokenizer t = NepaliSyllabicTokenizer() t.tokenize("क्षेत्रमा कसैका") # paper (Fig. 1): क्षे त्र मा क सै का # this impl: ['क्षे', 'त्र', 'मा', ' ', 'क', 'सै', 'का'] t.tokenize("व्यक्तित्वमा प्रभाव पर्ने") # paper (Table 3): व्, य, क्, ति, त्, व, मा, ' ', प्र, भा, व, ' ', प, र्, ने # this impl: ['व्', 'य', 'क्', 'ति', 'त्', 'व', 'मा', ' ', 'प्र', 'भा', 'व', ' ', 'प', 'र्', 'ने'] t.tokenize("घाउ लागेको क्षेत्रमा") # paper (Table 3): घा, उ, ' ', ला, गे, को, ' ', क्षे, त्र, मा # this impl: ['घा', 'उ', ' ', 'ला', 'गे', 'को', ' ', 'क्षे', 'त्र', 'मा'] ``` All three match token-for-token. That's the paper's complete set of published examples, though — three short phrases, so "matches the paper" is a weaker check than it sounds. See §8 for cases they don't cover. Tokenization is lossless *over the preprocessed text* — `"".join(tokens)` reproduces whatever came out of `_preprocess_text`, so token boundaries can be mapped back onto character spans for audio alignment. It is **not** lossless with respect to the raw input: preprocessing replaces punctuation with spaces and collapses runs of whitespace, so ```python t.decode(t.tokenize("नेपाल, एउटा देश। १२३")) # -> 'नेपाल एउटा देश ' ``` Punctuation became a space, the runs collapsed, and the digits were dropped outright (§7). Round-tripping the original string only holds for punctuation-free, digit-free, single-spaced Devanagari. ## 5. Running the paper's algorithm against the generated lexicon Since `SyllableDataset` produces a lexicon and `paper_algorithm.py` implements Algorithm 1, the two approaches can be compared directly — same lexicon, different search strategy: ```python from nepali_syllabic_tokenizer import SyllableDataset, NepaliSyllabicTokenizer from paper_algorithm import paper_tokenize SD, rule = SyllableDataset(), NepaliSyllabicTokenizer() paper_tokenize("क्रान्ति", SD) == rule.tokenize("क्रान्ति") # True ``` On every input in §4 and §8 — including the ones the rule-based version gets *wrong* — the two agree token-for-token. The sliding window buys nothing here, which is the point: given a lexicon built from these character classes, longest-match search is just a slower way of asking "does this character start a syllable?" And the failures in §8 are lexicon failures, not algorithm failures. Swapping in the paper's search doesn't fix them. **Except that window size 4 is too small for the lexicon it searches.** A complex consonant (3 chars) plus a vowel marker plus an anusvara is 5 characters — क्ष + ि + ं = `क्षिं` — and the generator emits those. So: ``` SD entries unreachable at win_size=4: 1521/4167 e.g. ['क्राँ', 'क्रां', 'क्राः', 'क्रिँ'] क्षिं paper: ['क्षि', 'ं'] rule-based: ['क्षिं'] त्रैं paper: ['त्रै', 'ं'] rule-based: ['त्रैं'] ``` **37% of the generated lexicon can never be matched**, and the leftover ं is stranded as a token with no phonetic value — exactly the failure the paper opens by criticizing character tokenizers for. The claim that "a maximum of 4 characters are used to form the complex syllables" holds for complex consonant + vowel marker, but not once a nasalization marker follows. I can't tell whether the paper's real `SD` contains these 5-character syllables, since it isn't published. If it does, `win_size` should be 5. If it doesn't, their tokenizer can't emit `क्षिं` as one unit either, and the stranding happens anyway. Either way the number in the paper's §4.2 doesn't cover the script. ## 6. What `SyllableDataset` is actually for I initially built `SyllableDataset` to hold a full generated lexicon, the same way the paper's dataset does — combining every consonant/complex-consonant with every vowel marker and marker combination gives **4,167 valid syllables**. `SyllableDataset.get_all_syllables()` will hand you that set if you want it (e.g. for validating a corpus, or building a vocabulary for a downstream model). Worth noting how far that is from what the paper actually used: their Table 4 reports a vocabulary of **650 tokens** for the syllable tokenizer. Grammatically-valid syllables vastly outnumber the ones that actually show up in a speech corpus, so the generated set is a superset to validate against, not a vocabulary to train on. But it turned out the tokenizer itself never needs to search that 4,167-entry set. It only ever consults `COMPLEX_CONSONANTS` — 39 entries — to catch the fixed-width consonant clusters, since a 1-character lookahead can't tell "त" (a plain consonant, syllable end) from "त्र" (a complex consonant, keep going) without checking ahead. Every other decision is a plain class-membership check. The lexicon-generation code stayed in the file because it's a useful standalone artifact, not because the tokenizer depends on it. ## 7. Other behavior worth knowing about **Punctuation and non-Devanagari input.** Punctuation (both ASCII and Nepali, e.g. । , ? !) is replaced with spaces before tokenization. Digits, Latin characters, and anything else outside the four Devanagari classes are dropped by default; pass `remove_non_devanagari=False` to keep them as individual tokens. ```python t.tokenize("क १२३ abc") # -> ['क', ' ', ' '] NepaliSyllabicTokenizer(remove_non_devanagari=False).tokenize("क १२३ abc") # -> ['क', ' ', '१', '२', '३', ' ', 'a', 'b', 'c'] ``` Dropping digits by default is the right call for an ASR target — `१२३` has no single spoken form to align audio against, so it needs expanding to words upstream, not tokenizing — but it does mean the tokenizer silently deletes input, which is worth knowing before pointing it at a corpus. **Provenance.** `nepali_syllabic_tokenizer.py` is flattened from the two-module version in `upstream/` — `syllable_dataset.py` plus a `syllabic_tokenizer.py` subclassing a shared `BaseNepaliTokenizer`. That copy is a snapshot of code whose real home is the ASR project it was written for; it's here to read, not to run (the base class it inherits from isn't included, so importing it fails). Flattening to one dependency-free file is what makes the examples in this post executable. The flattening costs one thing worth knowing about: `upstream/` also carries a `pre_tokenize_str` returning `(token, (start, end))` character offsets, HuggingFace pre-tokenizer style. That's the form you'd actually want for an ASR pipeline, since offsets are what let you map a token back onto a span of the source string — and, transitively, onto a slice of audio. `tokenize()` here returns bare strings, so the caller has to recover positions by walking token lengths. ## 8. What this is not This reproduces the *tokenizer* from the paper, not the ASR system built on top of it — I haven't trained the CNN+GRU/CTC model from the paper's §4.3 or measured CER/WER, since that requires the Open SLR Nepali speech corpus and a training run. What's here is the piece that the paper argues matters most: turning raw Devanagari text into pronunciation-aligned units, without needing whatever lexicon the original authors used internally. It's also not a strict superset of the paper's algorithm — the sliding-window lookup could absorb exceptions that don't fit these four character classes, if the lexicon encoded them. This rule-based version assumes the grammar is regular enough not to need exceptions. That assumption is wrong in at least two ways, both of which the paper's three published examples happen to dodge: **`consonant + ् + र` over-fires across syllable boundaries.** The rule is right for onset clusters (प्र, त्र) but wrong when र् ends one syllable and र begins the next: ```python t.tokenize("अन्तर्राष्ट्रिय") # -> ['अ', 'न्', 'त', 'र्रा', 'ष्', 'ट्रि', 'य'] ``` *an-tar-raa-ṣṭri-ya* — but `र्रा` glues the coda of one syllable to the onset of the next. **Geminates and stacked conjuncts are treated as single onsets.** त्त and द्ध are single *glyphs*, not single sounds; both span a syllable boundary: ```python t.tokenize("उत्तर") # -> ['उ', 'त्त', 'र'] (spoken ut-tar, so उत् / तर) t.tokenize("बुद्ध") # -> ['बु', 'द्ध'] (spoken bud-dha, so बुद् / ध) ``` The paper lists क्ष, त्र, ज्ञ, त्त, द्ध, श्र, द्य together as "complex characters," but that's a typographic grouping — क्ष and त्र really are single onsets, while त्त and द्ध are not. Copying the list wholesale into a phonetic rule inherits the conflation. This is exactly the gap a lexicon papers over, and the authors say as much: *"in some window segment there is a chance to get more than one valid token… sometimes the algorithm end up with error. So, we took the help of a linguist expert to appropriate the output of the tokenizer"* (paper §4.1). Their published tokenizer isn't fully automatic either — the lookup table is where the hand-tuning lives. A rule-based version just makes the exceptions visible instead of hiding them in a dictionary. ## Appendix: Full Source
nepali_syllabic_tokenizer.py ```python """ Nepali Syllabic Tokenizer V2 Implementation Rule-based approach using Devanagari character sets and linguistic rules """ import re import string from typing import Set, List class SyllableDataset: """ Syllable Dataset Generator for Nepali Devanagari Script Generates all valid syllables based on Devanagari grammar rules Generate and manage Nepali syllable dataset based on Devanagari grammar """ # Devanagari character sets VOWELS = ['अ', 'आ', 'इ', 'ई', 'उ', 'ऊ', 'ए', 'ऐ', 'ओ', 'औ', 'ऋ', 'ॠ', 'ऌ', 'ऑ', 'ऍ'] # candra o/e: loanwords (ऑफिस) CONSONANTS = [ 'क', 'ख', 'ग', 'घ', 'ङ', 'च', 'छ', 'ज', 'झ', 'ञ', 'ट', 'ठ', 'ड', 'ढ', 'ण', 'त', 'थ', 'द', 'ध', 'न', 'प', 'फ', 'ब', 'भ', 'म', 'य', 'र', 'ल', 'व', 'श', 'ष', 'स', 'ह' # क्ष = 'क' + '्' + 'ष', त्र = 'त' + '्' + 'र', ज्ञ ='ज' + '्' + 'ञ' ] HALANT = '्' # Virama/halant marker # vowel markers,HALANT, OTHER_MARKERS: should be attached to the last consonant (they should not be individual tokens) # Any matra missing from this list is silently dropped by the tokenizer, so # omissions here are data loss, not just mis-segmentation. VOWEL_MARKERS = ['ा', 'ि', 'ी', 'ु', 'ू', 'े', 'ै', 'ो', 'ौ', 'ृ', 'ॉ', 'ॅ', # candra o/e: loanwords (कॉलेज, डॉक्टर) 'ॄ'] # vocalic RR, pairs with ॠ above # Deliberately excluded: ॆ, ॊ (short e/o) -- Dravidian-range, not used in Nepali. OTHER_MARKERS = ['ं', 'ः', 'ँ'] NUMBERS = ['०', '१', '२', '३', '४', '५', '६', '७', '८', '९'] INVALID_TOKENS = ['ऌ', 'ऌं', 'ऌः', 'ऌँ', 'ॠ', 'ॠं', 'ॠः', 'ॠँ'] # Common complex consonants (NOTE: all complex consonants have len. 3) COMPLEX_CONSONANTS = [ 'क्ष', 'ज्ञ','द्ध', 'द्य', 'त्त', 'द्व' ] + [consonant + '्' + 'र' for consonant in CONSONANTS] # excluding 'त्र' (tra), 'श्र' (e.g. in shree) because they would be covered under: consonant + HALANT + 'र' # Precomputed lookup sets for the tokenizer's scan loop. The lists above stay # the public API (ordered, iterable for generation); these are for membership. VOWELS_SET = frozenset(VOWELS) CONSONANTS_SET = frozenset(CONSONANTS) COMPLEX_CONSONANTS_SET = frozenset(COMPLEX_CONSONANTS) # Everything that attaches to the preceding token instead of starting one. ATTACHING_MARKERS = frozenset(VOWEL_MARKERS + [HALANT] + OTHER_MARKERS) def __init__(self): self.syllables: Set[str] = set() self._generate_syllables() def _generate_syllables(self): """Generate all valid Nepali syllables""" # 1. Single vowels self.syllables.update(self.VOWELS) # 2. Single consonants (with inherent 'a' sound) self.syllables.update(self.CONSONANTS) # 3. Add pre-defined complex consonants self.syllables.update(self.COMPLEX_CONSONANTS) # 4. Numbers self.syllables.update(self.NUMBERS) # 5. Space and punctuation self.syllables.update([' ']) # '।', ',', '.', '?', '!', '-', '\'', '"' # 6. Consonant/complex consonant + vowel marker for c in self.CONSONANTS + self.COMPLEX_CONSONANTS: for vm in self.VOWEL_MARKERS: self.syllables.add(c + vm) # e.g. दिँदै for om in self.OTHER_MARKERS: self.syllables.add(c + vm + om) # 7. Consonant/complex consonant + halant (dead consonant) for c in self.CONSONANTS + self.COMPLEX_CONSONANTS: self.syllables.add(c + self.HALANT) # 8. Consonants/vowels/complex consonants with other markers (anusvara, visarga, chandrabindu) for c in self.CONSONANTS + self.VOWELS + self.COMPLEX_CONSONANTS: for om in self.OTHER_MARKERS: self.syllables.add(c + om) # NOTE: a "Consonant + Halanta + ra" loop used to live here. It was a # no-op -- those forms are already in COMPLEX_CONSONANTS, added in step 3. # 9. Drop the grammatically invalid combinations declared above. ऌ and ॠ # stay in VOWELS because they are real characters, but they do not form # these syllables, so remove them after generation. self.syllables.difference_update(self.INVALID_TOKENS) def contains(self, syllable: str) -> bool: """Check if a syllable exists in the dataset""" return syllable in self.syllables def get_all_syllables(self) -> Set[str]: """Return all syllables in the dataset""" return self.syllables.copy() def save_to_file(self, filepath: str): """Save syllable dataset to a file""" with open(filepath, 'w', encoding='utf-8') as f: for syllable in sorted(self.syllables): f.write(syllable + '\n') @classmethod def load_from_file(cls, filepath: str) -> 'SyllableDataset': """Load syllable dataset from a file""" dataset = cls.__new__(cls) dataset.syllables = set() with open(filepath, 'r', encoding='utf-8') as f: for line in f: syllable = line.strip() if syllable: dataset.syllables.add(syllable) return dataset def __len__(self): return len(self.syllables) def __contains__(self, item): return item in self.syllables class NepaliSyllabicTokenizer: """ Rule-based syllabic tokenizer for Nepali language using Devanagari character sets Algorithm: 1. Scan text left to right 2. Each token starts with a vowel, consonant, or complex consonant 3. Accumulate characters (vowel markers, halant, other markers) until the next vowel/consonant/complex consonant/space is reached """ def __init__(self, remove_non_devanagari: bool = True): """ Initialize the tokenizer with Devanagari character sets Args: remove_non_devanagari: If True, removes non-Devanagari characters (default: True) """ self.remove_non_devanagari = remove_non_devanagari def _preprocess_text(self, text: str) -> str: """ Preprocess text by replacing punctuations with spaces and normalizing spaces Args: text: Input text Returns: Preprocessed text with punctuations replaced by single spaces and multiple spaces collapsed to single spaces """ # Replace all standard ASCII punctuation characters with single space for punct in string.punctuation: text = text.replace(punct, ' ') # Replace Nepali punctuation marks with single space nepali_punctuation = ['।', ',', '.', '?', '!', '-', '\'', '"'] for punct in nepali_punctuation: text = text.replace(punct, ' ') # Replace multiple spaces with single space using regex text = re.sub(r'\s+', ' ', text) return text def tokenize(self, text: str) -> List[str]: """ Tokenize Devanagari text into syllabic units. Rules: - Scan left to right - Each token starts with a vowel, consonant, or complex consonant - Accumulate characters (vowel markers, halant, other markers) until the next vowel/consonant/complex consonant/space is reached Args: text: Input text string in Devanagari Returns: List of tokens """ if not text: return [] # Step 0: Preprocess text - replace punctuations with spaces and normalize spaces text = self._preprocess_text(text) tokens = [] i = 0 while i < len(text): first_three_chars = text[i:i+3] char = text[i] # 1. space if char.isspace(): tokens.append(" ") i += 1 continue # 2. Check Complex consonant if first_three_chars in SyllableDataset.COMPLEX_CONSONANTS_SET: tokens.append(first_three_chars) i += 3 continue # 3. check if char is vowel or consonant elif char in SyllableDataset.VOWELS_SET or char in SyllableDataset.CONSONANTS_SET: tokens.append(char) i += 1 continue # 4. Handle vowel markers, halant, and other markers (attach to previous token) elif char in SyllableDataset.ATTACHING_MARKERS: if tokens and not tokens[-1].isspace(): tokens[-1] += char else: # Handle case where marker appears at beginning (shouldn't happen in valid text) tokens.append(char) i += 1 continue # Handle any other characters (numbers, invalid characters etc.) # todo: convert numbers to word in pre process? `SyllableDataset.NUMBERS?` if not self.remove_non_devanagari: tokens.append(char) i += 1 return list(tokens) def encode(self, text: str) -> List[str]: """Alias for tokenize method""" return self.tokenize(text) def decode(self, tokens: List[str]) -> str: """ Decode tokens back to text Args: tokens: List of syllabic tokens Returns: Reconstructed text """ return ''.join(tokens) def tokenize_batch(self, texts: List[str]) -> List[List[str]]: """ Tokenize multiple texts Args: texts: List of input texts Returns: List of tokenized texts """ return [self.tokenize(text) for text in texts] if __name__ == "__main__": tokenizer = NepaliSyllabicTokenizer() print("व्यक्तित्वमा प्रभाव पर्ने: \t", tokenizer.tokenize("व्यक्तित्वमा प्रभाव पर्ने")) print("घाउ लागेको क्षेत्रमा: \t", tokenizer.tokenize("घाउ लागेको क्षेत्रमा")) print("क्रान्ति: \t", tokenizer.tokenize("क्रान्ति")) print("अन्तर्राष्ट्रिय: \t", tokenizer.tokenize("अन्तर्राष्ट्रिय")) """ # Output व्यक्तित्वमा प्रभाव पर्ने: ['व्', 'य', 'क्', 'ति', 'त्', 'व', 'मा', ' ', 'प्र', 'भा', 'व', ' ', 'प', 'र्', 'ने'] घाउ लागेको क्षेत्रमा: ['घा', 'उ', ' ', 'ला', 'गे', 'को', ' ', 'क्षे', 'त्र', 'मा'] क्रान्ति: ['क्रा', 'न्', 'ति'] """ ```
## References - Ghimire, R. R., Bal, B. K., Prasain, B., & Poudyal, P. (2023). [Pronunciation-Aware Syllable Tokenizer for Nepali Automatic Speech Recognition System](https://aclanthology.org/2023.icon-1.4/). ICON 2023. - (ToDo) [Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models](https://arxiv.org/abs/2512.03989v2) - (ToDo) [How efficiently do popular tokenizers handle Devanagari Nepali? A fertility benchmark](https://www.himalayaai.org/blog/nepali-tokenizer-fertility-benchmark)