Asosiy kontentga oʻtish
Maqola

Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian)

Mardiyeva Shoxzoda Totliboy QiziDepartment of Russian Language, Samarkand Branch, Kimyo International University in Tashkent, 63 Habib Abdullaev Street, Samarkand 140143, UzbekistanShakhlo DjamaldinovaHead of the Department of Uzbek Language and Literature with Russian Language, Samarkand State Medical University, 18 Amir Temur Street, Samarkand 140100, UzbekistanSabina UmirzakovaDepartment of Computer Science, Chungbuk National University, Cheongju 28644, Republic of Korea
2026en
Ochiq kirishOltin

Litsenziya: cc-by

ABI

Annotatsiya

Tokenization is the first design choice in a language model and one of the most frequently overlooked for morphologically rich languages, as it directly shapes sequence length and how related word forms are segmented into reusable sub word units. Using Russian as a case study, we train sub word tokenizers from scratch on a small, morphology-dense corpus and quantify two effects. First, a coverage penalty: a hypothetical English-centric byte-level tokenizer that has never learned Cyrillic falls back to raw UTF-8 bytes and spends 8.6× more tokens on the same Russian text than a Russian-trained Byte-Pair Encoding (BPE) tokenizer. Second, a compression morphology trade-off as vocabulary grows, BPE fertility falls toward one token per word, but it does so by memorizing complete inflected word forms rather than by isolating morphemes, so a nine-form paradigm of «дoм» (house) is stored as nine unrelated tokens with zero root sharing. A Unigram tokenizer at a comparable budget instead factors words into a shared root plus a shortening, representing the same paradigm with fewer unique token types at the price of slightly higher fertility. We release the corpus, training scripts, and an in-browser reimplementation of the trained BPE model that reproduces the reference tokenizer exactly. The numbers are illustrative, but the mechanisms are real and match findings in the recent Russian NLP literature.

Hali tarjima qilinmagan

Identifikatorlar

Iqtiboslar va manbalar