Orthographic Normalization and Tokenization Stability in Uzbek Literary Collocation Extraction
Annotatsiya
Orthographic normalization is often described as a preliminary cleaning operation, although it can determine which word combinations enter a corpus ranking and how much evidence they appear to have. The issue is especially consequential for Latin-script Uzbek, where apostrophe-like characters participate in the representation of o' and g', while hyphens may join reduplications, numeral approximations, compounds and cliticized forms. This study measures the downstream effect of these choices on adjacent lexical bigram extraction from eleven complete Uzbek literary works by eight authors. Four deterministic regimes were compared: minimal Unicode normalization, apostrophe unification, compatibility decomposition with diacritic folding, and hyphen-aware tokenization. The corpus contained 610,706 tokens under the first three regimes. Three apostrophe-like code points occurred 115,485 times, 113,585 of them between letters. Unifying them reduced the token-type inventory from 90,137 to 86,912 and the candidate-bigram inventory from 284,139 to 281,968 without changing token or candidate-instance counts. It promoted 79 bigram candidates above the frequency threshold of ten and increased the repeated-candidate set by 7.77%. Diacritic folding produced no additional change because the corpus contained neither combining marks nor decomposable precomposed Latin letters. Retaining internal hyphenated strings as single tokens reduced the token count by 17,011 and candidate instances by 6,707; the number of candidates with frequency at least ten fell from 818 to 673. Frequency rankings remained more stable than logDice rankings: 85 of the normalized top 100 frequency candidates survived in the hyphen-aware top 100, compared with only 57 logDice candidates. Forms such as qip-qizil, tez-tez and uch-to'rt ceased to behave as ordinary bigrams and became single orthographic units. The findings show that apostrophe unification is a necessary identity-preserving operation in this corpus, broad diacritic folding is empirically redundant, and hyphen handling is a linguistic segmentation decision rather than cosmetic normalization. All reported counts were calculated from the documented texts; no synthetic or simulated observations were used.
Hali tarjima qilinmagan