Study Notes

MNLP-L02 - Multilinguality and Writing Systems

34 min read

MNLP-L02: Multilinguality and Writing Systems

In short

Standard NLP pipelines split on whitespace, rely on word order, treat a word as one unit of meaning and assume text runs left to right. Those are properties of English, and each fails for a large group of languages. A second set of problems sits lower down, where text becomes numbers and bytes.

What a reader sees as one character (a grapheme) can be several Unicode codepoints, each stored as 1 to 4 bytes in UTF-8. The same visible text can have different codepoint sequences, so pipelines must normalize consistently, and the normalization form (NFC, NFD, NFKC, NFKD) decides what information survives.

No sample exam question covers this lecture. Expect statements on exact conditions: which normalization forms are lossy, which encodings are ASCII compatible, which languages break which assumption.

NLP mostly means English NLP

Most NLP resources are in English: annotated (labelled) training data, unannotated training data, toolkit coverage (dictionaries, analyzers) and benchmarks and competitions. Most publications also focus on English.

Non-English resources come about mostly in two ways, and less often in a third:

  • as a naturally occurring by-product of something else, such as translations or product reviews, which nobody made for NLP;
  • as translations of existing (annotated) English resources, so the annotation scheme and content stay English-shaped;
  • less so as a dedicated effort, the route that would actually close the gap.

The result is that standard pipelines encode assumptions that are properties of English, not of language in general: whitespace tokenization, fixed word order, closed vocabularies. Claims of language independence are usually untested outside English (or a very small set of languages). This is understandable: creating resources costs time and money.

Resource tiers follow commercial and military interest

There are over 7,000 spoken languages. Most have few speakers, but more than a hundred have over ten million each. How well a language is covered by NLP resources does not follow its speaker numbers: it depends on commercial interest and military relevance. Christof Monz (the lecturer) sorts languages into three tiers:

High resource: large labelled and unlabelled data, many tools. English and Mandarin Chinese, then a big gap, then French, Spanish, Arabic and Russian. All six are high resource, but the tier holds two very different groups.

Medium resource: limited labelled data, reasonable unlabelled data, some tools. German, Italian, Dutch, Korean, Japanese.

Low resource: no labelled data, limited unlabelled data, no tools. All of the roughly 7,000 remaining languages, which, on these tiers, includes most of the 100+ languages with over ten million speakers.

Languages differ in ways English hides

One sentence, two languages

How many words is this?

English: I don’t know where she went.

Chinese: 我不知道她去哪儿了。 Chinese has (almost) no spaces, so splitting on whitespace returns the whole sentence as one token.

我不知道她去哪儿了。
wǒbùzhīdàotāqùnǎrle
INEGknowshegowhereSFPPUNC

(NEG: negation; SFP: sentence-final particle.)

  • The verb 去 (“go”) carries no tense. English changes the verb (“go” against “went”); Chinese leaves it alone.
  • Past tense comes from the sentence-final particle 了, at the end of the clause, away from the verb. (Outside the course material: 了 marks a completed event, so L03 glosses it as perfective.)
  • Both clauses are SVO (subject, verb, object): the main clause “I NEG know” and the embedded clause “she go where”. Word order matches English here; word boundaries and the place of past-time marking do not.

Morphology: what is inside a word

Words have parts: unfriendly is un- + friend + -ly, and each part adds meaning.

Morphology

Morphology is the study of the internal structure of words and how they are formed from smaller units of meaning.

  • Morpheme: the smallest unit of meaning or grammatical function. dogs has two: dog (the base meaning) and -s (plural).
  • Root / base: the core part of a word that carries the primary meaning (friend in unfriendly).
  • Affix: a prefix (added to the front) or suffix (added to the back) that modifies a root’s meaning or grammatical role (un- and -ly in unfriendly).

Three languages show the range, from limited to very complex morphology.

English has limited morphology. From happy you get happier, happiest, unhappy, unhappier, unhappiest, happiness, unhappiness, and little more. That is why closed (fixed-size) word vocabularies seemed a reasonable design.

Russian has more complex morphology. The noun kniga (“book”) has twelve slots, six cases (grammatical roles such as subject, possessor, instrument) times singular and plural, but only nine distinct forms. So some forms fill several slots: knígi is “of the book”, “the books” and “books” as direct object, and the case cannot always be read off the string.

Finnish has very complex morphology. It is agglutinative: morphemes are glued on one after another, each adding one piece of meaning, so a single word can carry what English spreads over a phrase.

Building "even at my happiest" in Finnish

onni (“happiness”) + -llinen (“having the quality of”) gives onnellinen, “happy”; with the stem onnellis- and comparative -empi, onnellisempi is “happier”. Then onnellisimmillanikin, “even at my happiest”, is:

onnellis-  stem, "happy"
-imm-      superlative
-i-        plural oblique-case marker
-lla       adessive case, "at/on/in"
-ni        "my" (1st person possessive)
-kin       "even"

One word for four English ones.

A fixed word-level vocabulary has almost certainly never seen onnellisimmillanikin, and Finnish produces combinatorially many words like it, so more data does not fix this. That motivates subword segmentation (L04); morphology itself is L03.

The English-first assumptions

1. Words are separated by whitespace. False for Chinese, Japanese, Thai, Lao, Khmer and Burmese, which have no word delimiter. Thai does use spaces, but to end clauses or sentences (it has no dedicated full stop), so a Thai space does not mark a word boundary.

2. Grammatical relations are encoded by word order. False for languages with rich morphology, especially case marking: Russian, Finnish, Korean. Who did what to whom is marked on the nouns themselves, so word order is more flexible and a word’s position says much less about its role.

3. One word = one unit of meaning. False for agglutinative languages (Finnish, Turkish, Hungarian) and for languages with compounding (German, Dutch, Swedish). German Bundesverfassungsgericht is one token holding three concepts: federal, constitutional, court.

4. Text flows from left to right. False for Semitic languages and languages in Arabic script: Arabic, Hebrew, Urdu, Persian, which are read right to left. Unicode text in these languages is written (stored) left to right but rendered right to left: codepoints sit in the file in reading order and are displayed from the right, so the first character of the string is the rightmost on screen.

Writing systems: what a symbol stands for

Writing systems differ in what one symbol stands for: a sound, a syllable or a whole word or morpheme.

Types of writing system

  • Alphabetic: each symbol represents a consonant or vowel sound; typically a small inventory of symbols. Latin, Cyrillic, Hangul.
  • Syllabic: each symbol represents a whole syllable, a combination of consonant and vowel. Japanese Hiragana and Katakana.
  • Logographic: each symbol represents words or morphemes directly, with typically little sound information. Chinese Hanzi, Japanese Kanji.

One practical difference for NLP is inventory size: an alphabet has tens of symbols, a logographic script thousands, which a character-level vocabulary must budget for.

ScriptWordReading
Latinwaterwater
Devanagariपानीpani
Han水shui
Arabicكتبktb
Korean한글Hangul

Latin spells sounds with letters and Korean Hangul is alphabetic too (each letter a consonant or vowel), with letter shapes that encode articulatory features (how the sound is produced); Han uses one symbol per morpheme (logographic). Devanagari and Arabic fit none of the three types cleanly:

Devanagari (Hindi) is alphasyllabic: each consonant letter carries an inherent vowel. (Outside the course material: an attached vowel sign can replace it, so पानी is two units, pā + nī.)

Arabic writes the consonant root only; vowels are omitted: letters for sounds, but only the consonants. In examples from outside the course material, كتب (k-t-b, “write”) can be read kataba (“he wrote”), kutiba (“it was written”) or kutub (“books”). The writing is lossier than the language, so the model must resolve the ambiguity from context; no tokenizer can recover vowels that were never written.

A single Japanese sentence can mix three systems: in アムステルダムはオランダの首都です (“Amsterdam is the capital of the Netherlands”), the names are Katakana, 首都 (“capital”) is Kanji, the rest Hiragana.

Script is not language

One writing system, many languages:

  • Latin: English, German, French, most European languages (except some Eastern European ones), and also several African, Turkic and Asian languages
  • Arabic: Arabic, Persian, Urdu, Pashto, Sorani (Central Kurdish)
  • Cyrillic: Russian, Belarusian, Serbian
  • Devanagari: Hindi, Marathi, Nepali
  • Ethiopic (Ge’ez): Amharic, Tigrinya

Language-specific writing systems: Armenian, Georgian, Greek, Sinhala, Khmer.

One language, more than one writing system: Serbian (Cyrillic and Latin), Kurdish (Latin and Arabic), Uzbek (Latin and Cyrillic), Punjabi (Gurmukhi and Arabic).

What each direction breaks

Detecting the script does not identify the language. Latin script narrows almost nothing; Arabic script still leaves Arabic, Persian, Urdu, Pashto and Sorani.

Knowing the language does not fix the script. A Serbian corpus can contain Cyrillic and Latin text, even in the same document. A model trained on one script and tested on the other shares almost no letters with the test data, though the language is the same.

Before Unicode: ASCII and code pages

From here on, the question is how text in all these scripts becomes numbers and bytes, which is where multilingual pipelines break silently.

ASCII (1963) assigned 128 codes: enough for English letters, digits and punctuation.

Code pages extended this by reinterpretation. They use the same 128 (or 256) codes in a byte as ASCII and interpret them differently depending on the language or script: Latin-1 for Western Europe, Shift-JIS for Japanese, GBK for Chinese, KOI8-R for Russian, and many more. (Outside the course material: Shift-JIS and GBK use two bytes for most characters.)

Two disadvantages:

  • Gibberish when text encoded in one script is decoded in another. The bytes are often legal in both tables, so usually no error is raised (the result is called mojibake, a term from outside the course material).
  • No mixing of scripts in one document, since a document is read with one table.

Unicode: one number per encoded character

Unicode and codepoints

Unicode is a single, universal character set, maintained by the Unicode Consortium. Every character it encodes, in every supported script, gets exactly one unique number, its codepoint, written U+number in hexadecimal: U+0041 is Latin “A”, U+4E2D is the Han character 中. What a reader sees as one character can be a sequence of several codepoints (next section).

One code space for every script removes both code-page problems: no table to guess, and one document can mix any scripts.

  • Unicode defines meaning, not bytes. A codepoint is an abstract identifier for a character. How to store it as bytes on disk or in memory is a separate encoding question, answered by UTF-8, UTF-16 or UTF-32.
  • Unicode is a superset of ASCII: U+0000 to U+007F are identical to ASCII. So a pure ASCII file is already valid UTF-8 (UTF-8 keeps ASCII’s bytes). A Latin-1 file with accented letters is not: Latin-1 stores é as the single byte E9, which UTF-8 rejects.

Scale. The code space runs from U+0000 to U+10FFFF, over 1.1 million possible codepoints, in 17 planes of codepoints each:

  • Plane 0, the Basic Multilingual Plane (BMP): most modern languages, including the most common CJK (Chinese, Japanese, Korean) ideographs.
  • Plane 1: historic scripts, some modern scripts, symbols such as musical notation, emoji.
  • Planes 2 and 3: more CJK ideographs, mostly historical or rare, so not every CJK character is in the BMP.
  • Plane 14: tags and variation selectors; planes 15 and 16: private use (plane 0 also has a private use area); planes 4 to 13 unassigned.

Codepoint, character, glyph

Codepoint, character, glyph

  • Codepoint: one abstract number, such as U+00E9; Unicode’s unit of identity.
  • Character: what a reader perceives as one unit. It may be built from several codepoints.
  • Glyph: a stylistic rendering of a character (how a font draws it).

What a reader perceives as one character is also called a grapheme.

So a codepoint is always one number, and one perceived character is one or more codepoints: é can be the single codepoint U+00E9, or e followed by a combining accent.

Two identical-looking "café" 

café in NFC: 4 codepoints
U+0063 U+0061 U+0066 U+00E9
café in NFD: 5 codepoints
U+0063 U+0061 U+0066 U+0065 U+0301

In NFC, é is one precomposed codepoint; in NFD, e plus a combining acute accent. They render identically but compare unequal: a naive string comparison will not match them. The same holds for ã: in JavaScript, 'ã' === 'ã' is false, though both display as ã (lengths 1 and 2).

Normalization: choosing one sequence for the same text

Normalization rewrites text into a standard codepoint sequence. The four forms come from two choices: composed (C: precomposed codepoints where possible) or decomposed (D: base character plus combining marks), and canonical (merges only sequences for the same character) or compatibility (K: also merges stylistic variants, such as a ligature and its letters).

ComposedDecomposed
Canonical (lossless)NFCNFD
Compatibility (lossy)NFKCNFKD

Where compatibility forms lose information

Take long s (ſ, an old form of s) with a dot above, followed by a combining dot below: ẛ̣.

  • NFC: ẛ + combining dot below (2 codepoints: no precomposed character has both dots)
  • NFD: ſ + dot below + dot above (3 codepoints)
  • NFKC: ṩ, a plain s with both dots (1 codepoint)
  • NFKD: plain s + dot below + dot above (3 codepoints)

The canonical forms keep the long s ſ, because canonically ſ and s are different characters. The compatibility forms replace it with plain s, and nothing can bring it back.

The decomposition forms and when to use them

  • NFD (canonical decomposition): splits characters into base characters and combining marks; preserves semantic and visual equivalence. Non-lossy: you can decompose and compose again without losing styling or formatting details.
  • NFKD (compatibility decomposition): breaks characters down into their most basic parts, including splitting ligatures. Lossy: it normalizes visual styling to the base form, which cannot be reconstructed when composing.
  • Use NFD to compare rendered text.
  • Use NFKD for search indexing or NLP analysis, but not necessarily for generation.

Why: for analysis, merging variants helps (the ligature fi should match “fi” in a search index, ① should count as 1). For generation it can hurt: a model trained on NFKD targets never sees, and so cannot produce, the characters that were folded away.

Outside the course material: for a plain equality check NFC works too, because NFD and NFC agree on which strings are equivalent. The course’s rule for comparing rendered text is NFD.

Normalizing crawled data

Resources crawled from the internet can mix normalization forms even when they are all in Unicode: some pages NFC, others NFD.

Many preprocessing steps assume identical codepoint sequences: deduplication, search, string equality and prefix matching. Without proper normalization they become inaccurate, silently: deduplication keeps two copies that differ only in normalization form.

The easy fix: always perform consistent normalization as part of the pipeline, for example first decompose and then compose (canonical decomposition then composition ends in NFC). In Python, the unicodedata module can normalize a string into a given form and test whether it is already in one:

import unicodedata as ud
 
a = "caf\u00e9"    # NFC: 4 codepoints
b = "cafe\u0301"   # NFD: 5 codepoints
 
a == b                         # False
ud.is_normalized("NFD", b)     # True
ud.normalize("NFD", a) == b    # True

Text that is not in Unicode at all, but in a code page such as GB2312 or KOI8-R, is converted on the way in, for example with iconv -f GB2312 -t UTF-8. This fails if the source encoding is not correct, which is useful: a failure is better than silent mojibake. Outside the course material: iconv only fails on bytes that are invalid in the stated encoding. That is common for multi-byte GB2312, but in a single-byte code page such as KOI8-R every byte is legal, so a wrong guess converts silently into mojibake.

Storing codepoints as bytes: UTF-8, UTF-16, UTF-32

EncodingBytes per codepointASCII compatible
UTF-81 to 4yes
UTF-162 or 4no: at least 2
UTF-32exactly 4no: exactly 4

UTF-8 is by far the most common (the web, Linux, NLP tooling), and it is what people usually mean by “Unicode”. UTF-16 is used by Java, JavaScript and Windows; UTF-32 is rare, simple to index but memory-hungry.

How UTF-8 works

UTF-8 uses more bytes for larger codepoints. The number of leading 1s in the first byte gives the length of the whole sequence, the first byte included (1110xxxx starts a 3-byte sequence, so two bytes follow), and every following byte carries 6 bits of the codepoint.

The UTF-8 byte rules

  • A byte starting with 0 is a single byte, encoding one of 128 characters: ASCII.
  • A byte starting with 11 is the first byte of a multi-byte sequence; the number of leading 1s gives the length of the whole sequence.
  • A byte starting with 10 is a continuation byte inside a multi-byte sequence.
Encode codepoint c in UTF-8
───────────────────────────
c ≤ U+007F    0xxxxxxx
c ≤ U+07FF    110xxxxx 10xxxxxx
c ≤ U+FFFF    1110xxxx 10xxxxxx
              10xxxxxx
c ≤ U+10FFFF  11110xxx + three
              10xxxxxx bytes

The payload widths are 7, 11 (5+6), 16 (4+6+6) and 21 (3+6+6+6) bits. Write c in binary, pad it with zeros on the left to the payload width, and fill the x positions from left to right.

中, U+4E2D

0x4E2D in binary is 100111000101101, 15 bits. That is more than the 2-byte form’s 11, so use the 3-byte form with 16 payload bits. Pad to 0100111000101101 and split 4 + 6 + 6: 0100 | 111000 | 101101.

Byte 1 = 1110 + 0100 = 11100100 Byte 2 = 10 + 111000 = 10111000 Byte 3 = 10 + 101101 = 10101101, so 中 is E4 B8 AD.

By the same rules, A (U+0041) is one byte identical to ASCII, é (U+00E9) two bytes, most CJK characters three, and a plane-1 emoji such as U+1F600 four. Codepoints beyond plane 0 need more than 16 bits, so they take 4 bytes in both UTF-8 and UTF-16.

Since no byte is ambiguous about its role, from any point in a UTF-8 stream you find the next character start by skipping continuation bytes (outside the course material this is called self-synchronisation).

What bytes do to tokenizers: byte inflation

Byte-level tokenizers (vocabulary built over UTF-8 bytes, see L04) are affected by how characters are represented as bytes, not as codepoints.

Byte inflation

where:

  • : the length of string once encoded as UTF-8
  • : the number of characters a reader perceives

For example, “hello” is 5 bytes for 5 characters, 1.0x; “café” in NFC is 5 bytes for 4 characters (é takes two), 1.25x; “中文” is 6 bytes for 2 characters, 3.0x.

Two consequences:

  • The more bytes are needed, the more aggressively text tends to be split into tokens: more pieces per character.
  • Byte-level splitting can produce non-existent characters. Cut a 3-byte character after its second byte and neither piece is a valid character.

Outside the course material: in practice CJK text often costs more tokens for the same content, depending on how much CJK the tokenizer was trained on.

Where to break a string

Splitting a string can happen at three levels, each with a cost:

  • Byte-level breaking can split codepoints, which can consist of several bytes, producing fragments that are not characters.
  • “Smarter” codepoint breaking is not ideal either: it can break one character into pieces, for example separating a base letter from its diacritic.
  • Ideally, break at the grapheme level (what the reader perceives as a character), but grapheme boundaries are very language-specific.
"café" in NFD, split three ways
bytes:      63|61|66|65|CC|81
codepoints: c |a |f |e |◌́
graphemes:  c |a |f |é

CC and 81 on their own are not characters, and at the codepoint level the accent is stranded.

Mixing breaking strategies leads to misalignments:

  • Span annotation systems (SQuAD-style question answering stores answers as character offsets): offsets counted one way and read another give shifted answers.
  • Any counting across corpora with different breaking strategies: counts are no longer comparable.

So breaking and length computation must be consistent across documents.

The Unicode stack

A reader sees a grapheme, which is represented by one or more codepoints, each encoded in 1 to 4 bytes (UTF-8). For é in NFD:

reader sees      é               1 grapheme
Unicode numbers  U+0065 U+0301   2 codepoints
tokenizer eats   65 CC 81        3 bytes

Other pitfalls: case, sorting, punctuation

Lowercasing is language-dependent. In English, lowercase(I) = i. In Turkish, lowercase(I) = ı (dotless i), because Turkish has two pairs: dotted İ / i and dotless I / ı. Lowercasing Turkish with English rules changes the word.

Codepoint order is not alphabetical order: accented letters’ codepoints are not necessarily near their base letter’s. é is U+00E9, far after z (U+007A), so a codepoint sort puts “émile” after “zoe”.

Languages use different alphabetical orders. Swedish puts å, ä, ö after z, as separate letters. German sorts ä as ae, ö as oe, ü as ue, interleaved with the plain letters.

Punctuation. Defining a word is hard, but punctuation adjacent to a word is not part of it, so tokenizers split it off. Some also expand contractions (He’d to He would), another normalization decision and a lossy one, since He’d can also mean He had:

He'd say "I promise you!", but
then disappear.
→ He ' d say " I promise you ! " ,
  but then disappear .
→ He would say " I promise you ! " ,
  but then disappear .

Unicode gives every character a general category (letter, number, punctuation, and so on), so punctuation is matched by category rather than a hand-written list: \p{P} covers punctuation in every script, and re.sub(r'(\p{P})', r' \1 ', s) puts spaces around each mark. (Outside the course material: Python’s built-in re lacks \p{P}; the third-party regex module has it.) A class such as [.!?] misses the Japanese 。, the Bengali ।, the Amharic ።.

Punctuation normalization (“…” against «…» against 「…」) has no simple solution: the mappings must be encoded by hand. It comes up in quotes, end-of-sentence markers, question marks and single quotes.

What to be able to answer

  • The English-first assumptions and the languages that break each (Thai spaces end clauses).
  • What sets a language’s resource tier; the three tiers; why “language independent” claims are suspect.
  • Alphabetic, syllabic, logographic; the Devanagari, Arabic and Hangul cases.
  • Script and language in both directions.
  • The two disadvantages of code pages, and what Unicode fixes.
  • Codepoint, character, glyph, grapheme; why two “café” strings fail to match.
  • The four normalization forms: which are lossy, which for rendered text, which for search.
  • What breaks silently without normalization, and the fix.
  • UTF-8, UTF-16, UTF-32: bytes and ASCII compatibility; the UTF-8 byte rules; encoding by hand.
  • Byte inflation and its two consequences.
  • Byte, codepoint and grapheme breaking; where mixing them misaligns.
  • Turkish lowercasing, Swedish and German sorting, punctuation handling.

Exam questions

Flashcards

  • Course: Course overview
  • Previous: MNLP-L01 - Overview
  • Next: MNLP-L03 - Morphology and Word Formation
  • Project: MNLP - Mini Project
  • Related concept: Tokenization
  • Source: multilingual-nlp.pdf, 33 slides, Christof Monz, Informatics Institute, University of Amsterdam
  • Further reading (recommended in the lecture): Joel Spolsky (2003), The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets; Nikita Prokopov (2023), The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 (both blog posts)

Graph View

Backlinks