A modular, pip-installable Python library for the whole Turkic language family. Text and speech go in; tokens, morphemes, morphology, POS tags, dependency trees, entities, translations, embeddings and transcripts come out of one Stanza-style pipeline.
A small core, plus one extra per component group:
$ pip install turkicnlp # core
$ pip install "turkicnlp[all]" # all text components
$ pip install "turkicnlp[all,asr]" # + speech (own env)
Models are downloaded at runtime into ~/.turkicnlp, never bundled in the pip package. All extras →
Each component uses the strongest resource available for each language. Where no trained model exists, the LLM backend fills the gap.
Requires Python 3.10 – 3.12. Installation tests run on Ubuntu, macOS and Windows for every push.
| Extra | Adds |
|---|---|
turkicnlp | Tokenization, rule-based processing, transliteration, CoNLL-U I/O |
[hfst] | Apertium FST morphology (Linux, macOS) |
[stanza] | Stanza models: tokenize, POS, lemma, depparse, NER |
[transformers] | Glot500 POS / parsing / morphology models |
[translation] | NLLB-200 embeddings and machine translation |
[lid] | GlotLID language identification |
[all] | All text components above (Linux, macOS) |
[asr] | Speech recognition with Omnilingual ASR |
[dev] | pytest, black, ruff, mypy |
Text: [all], with torch 2.10, numpy 2 and transformers 5.
Speech + text: [all,asr]. omnilingual-asr pins torch 2.8, numpy 1.26 and transformers 4.57. Every text component also works with these versions.
Lock files for both are in requirements/.
[asr] to an existing text environment. It downgrades torch, numpy and huggingface_hub, which breaks transformers 5. Create a fresh environment instead.hfst has no Windows wheels, so use WSL if you need FST morphology.The model catalog knows which backend to fetch for each language and script.
import turkicnlp
turkicnlp.download("kaz")
Dependencies between processors are resolved automatically.
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "depparse"])
doc = nlp("Мен мектепке бардым.")
print([(w.text, w.upos, w.lemma) for w in doc.words])
# [('Мен', 'PRON', 'мен'), ('мектепке', 'NOUN', 'мектеп'),
# ('бардым', 'VERB', 'бар'), ('.', 'PUNCT', '.')]
print(doc.to_conllu()) # or doc.to_dict(full=True)
Pipeline("kaz") without processors= loads the language's default set: tokenizer, morphology, POS, lemma, features, parsing and NER. Large or optional components (translation, embeddings, ASR, subword tokenization, LLM tasks) load only when you ask for them.
pos_backend="multilingual_glot500_model"<processor>_<option>: translate_tgt_lang="eng"nlp.batch(texts, batch_size=32)doc._processor_log and word._provenance record which backend produced each valueJupyter notebooks for every component: turkic-nlp-code-samples.
SpeechRecognizer wraps Meta's Omnilingual ASR behind TurkicNLP conventions: ISO 639-3 codes in, transcripts in each language's own script out. Long recordings are split at pauses.
import turkicnlp
asr = turkicnlp.SpeechRecognizer(model_card="omniASR_LLM_1B")
texts = asr.transcribe(["kaz_sample.wav", "tur_sample.wav"],
lang=["kaz", "tur"], batch_size=2)
# ['мен бүгін мектепке бардым алматы қазақстанның ең үлкен қаласы',
# "bugün hava çok güzel ali dün ankara'ya gitti"]
# Ask for another script: Kazakh in the 2021 Latin alphabet
asr.transcribe("kaz_sample.wav", lang="kaz", script="Latn")
# Speech → text → analysis in one pipeline
nlp = turkicnlp.Pipeline("kaz",
processors=["asr", "tokenize", "pos", "lemma", "depparse"])
doc = nlp.from_audio("kaz_sample.wav")
print(doc.audio_segments) # time-aligned pieces
Accepts file paths, waveforms, raw bytes and Hugging Face datasets audio columns. Stereo input is mixed to mono and resampled to 16 kHz.
omniASR_LLM_300M | 6.5 GB |
omniASR_LLM_1B (default) | 9.1 GB · ≈ 6 GB RAM |
omniASR_LLM_7B | 31.2 GB · GPU |
omniASR_CTC_300M | 1.3 GB · fastest |
Five processors can run on a large language model for all 24 languages, including Khakas, Tuvan and Nogai, which have no other neural models. Point the toolkit at a local vLLM or Ollama server, OpenAI, Claude or an in-process Hugging Face model.
| Backend | What the LLM does |
|---|---|
morph=llm | Lemma, UPOS and UD features for every token |
morph=apertium_llm | Chooses one of the FST's readings for ambiguous words. It can't invent an analysis. |
ner=llm | Entity spans over the toolkit's tokens, with configurable labels |
translate=llm | Translation into any target language or script |
sentiment=llm | Positive, negative or neutral, or your own labels |
spellcheck=llm | Spelling correction, guarded against rewrites |
nlp = turkicnlp.Pipeline(
"kaz",
processors=["tokenize", "morph", "ner", "translate"],
morph_backend="llm", ner_backend="llm",
translate_backend="llm", translate_tgt_lang="eng",
llm_model="qwen2.5:7b", # local Ollama
llm_base_url="http://localhost:11434/v1",
)
# or: llm_provider="anthropic", llm_model="claude-sonnet-4-6"
# Free-form questions with the toolkit's analysis as context
turkicnlp.llm.ask("Explain the verb form.", doc=doc)
Answers are validated against the toolkit's tokens and retried once with feedback. Unusable answers are left empty rather than guessed. Prompts are versioned JSON files you can replace. Only validated answers go into a local cache.
Apertium finite-state transducers give precise analyses where the lexicon covers a word. Glot500 neural models cover the rest. The MorphemeTokenizer combines both to split words into labelled surface morphemes.
morphTransducers for 20 languages, loaded with Python-native HFST. No system Apertium install needed. Precompiled from upstream by CI, with Apertium → UD tag mapping for every language.
morph_neuralOne multilingual model gives UPOS, UD features and lemmas for 22 languages: 20 trained, plus zero-shot Karakalpak, Karachay-Balkar and Nogai.
sp_tokenizeSentencePiece over morpheme-segmented text, so subword pieces never cross morpheme boundaries. Use it to train language models on Turkic text.
from turkicnlp.processors.morpheme_tokenizer import MorphemeTokenizer
tok = MorphemeTokenizer(lang="tur"); tok.load()
tok.segment("evlerinden").labeled
# [('ev', 'STEM'), ('ler', 'PLUR'), ('in', 'POSS.2SG'), ('den', 'ABL')]
tok = MorphemeTokenizer(lang="kaz"); tok.load()
tok.segment("оқығандар").labeled
# [('оқы', 'STEM'), ('ған', 'PTCP.PST'), ('дар', 'PLUR')]
tok = MorphemeTokenizer(lang="uzb"); tok.load()
tok.segment("bolalarning").labeled
# [('bola', 'STEM'), ('lar', 'PLUR'), ('ning', 'GEN')]
Every Apertium reading is turned into a segmentation by matching its tags against language-specific suffix allomorph tables, which encode vowel harmony, consonant alternations and buffer consonants. The tokenizer keeps the reading that explains the whole word, breaking ties by agreement with the neural lemma and POS. When the transducer has no reading, it segments the neural analysis instead.
| Exact segmentation | 483 words · 20 langs |
|---|---|
| Hybrid (FST + neural) | 89% |
| Apertium only | 79% |
| Neural only | 68% |
Allomorph tables cover 23 languages. The FST path is available for the 20 with a transducer.
One shared NLLB-200 (600M) model serves translation and 1024-dimensional embeddings for 11 Turkic languages. It's downloaded once and kept in memory once.
# Turkish → English
nlp = turkicnlp.Pipeline("tur", processors=["translate"],
translate_tgt_lang="eng")
doc = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
print(doc.translation)
# Sentence embeddings for similarity and retrieval
embed = turkicnlp.Pipeline("tur", processors=["embeddings"])
doc = embed("Parkta yürüyüş çok güzeldi.")
len(doc.embedding) # 1024
translate_tgt_lang takes ISO 639-3 ("eng", "kaz") or FLORES-200 codes ("kaz_Cyrl"). For any of the 24 languages, translate_backend="llm" translates with an LLM instead.
Models and datasets: huggingface.co/turkicnlp
The GlotLID FastText model knows 1,000+ language labels. By default, LanguageDetection restricts its predictions to the Turkic languages TurkicNLP supports.
import turkicnlp
lid = turkicnlp.LanguageDetection() # Turkic label set by default
labels, probs = lid.predict("salam, hemmelere!", k=3)
# Or limit to an explicit label set
lid = turkicnlp.LanguageDetection(languages=[
"__label__eng_Latn", "__label__tur_Latn", "__label__kaz_Cyrl",
])
lid.predict("Merhaba dünya!", k=1)
Labels combine language and script (kaz_Cyrl, uig_Arab), so you get the language and the writing system in a single step. TurkicNLP maps its codes to GlotLID's where they differ: Uzbek is uzn, North Azerbaijani is azj.
Pair it with automatic script detection in the pipeline to route mixed-language collections to the right models.
Bidirectional conversion for every multi-script language, a shared Common Turkic Alphabet view for 23 languages, and Old Turkic runes.
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator
Transliterator("kaz", Script.CYRILLIC, Script.LATIN).transliterate("Қазақстан Республикасы")
# Qazaqstan Respublikasy
Transliterator("uzb", Script.LATIN, Script.CYRILLIC).transliterate("Özbekiston Respublikasi")
# Ўзбекистон Республикаси
Transliterator("uig", Script.PERSO_ARABIC, Script.LATIN).transliterate("مەكتەپ")
# mektep
Transliterator("tuk", Script.LATIN, Script.COMMON_TURKIC).transliterate("Türkmen dili baýdyr")
# Türkmen dili baydır
Transliterator("otk", Script.OLD_TURKIC_RUNIC, Script.LATIN).transliterate("𐱅𐰇𐰼𐰚")
# törk
Uzbek Latin uses ö, ğ, ş, ç throughout the toolkit. Pretrained models keep their training spelling internally, and adapters convert input and output automatically.
Request processors by name and choose a backend with <name>_backend=. Solid cards load by default. Dashed cards load only when named.
Sentences, tokens and words. Abbreviation lists, ordinals, quotes.
rule · rule_arabic · stanzaExpansion rules extracted from UD treebanks.
neural · stanzaLemma, UPOS and UD features from the FST, an LLM, or both.
apertium · llm · apertium_llmUPOS, features and lemmas for 22 languages.
multilingual_glot500_morphUPOS, XPOS and features for 15 languages.
stanza · glot500Morphological feature bundles.
multilingual_glot500_morphNeural lemmas with Turkic-aware casing.
stanza · glot500_morphBiaffine parsers that always produce valid UD trees.
stanza · glot500Starlang (Turkish) and KazNERD (Kazakh). The LLM backend covers every language.
stanza · llm1024-dim NLLB encoder vectors.
nllbTo any FLORES-200 language, or any language via an LLM.
nllb · llmSentence labels, with custom label sets.
llmCorrections with a guard against rewrites.
llmMorpheme-aware SentencePiece.
sentencepieceTranscribe audio, then analyse it with Pipeline.from_audio.
Standalone tools outside the pipeline: MorphemeTokenizer, Transliterator, LanguageDetection, SpeechRecognizer, turkicnlp.llm.ask. Full CoNLL-U reader and writer.
Grouped by branch. The LLM backends (morphology, NER, translation, sentiment, spelling) work for all 24 languages on top of everything listed.
| Language | Code | Scripts | Tokenize | Morph | POS | Lemma | DepParse | NER | Embed | MT | ASR | LLM |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oghuz | ||||||||||||
| Turkish | tur | Latn | ■● | ◆◈ | ●◇ | ●◈ | ●◇ | ● | ★ | ★ | ◎ | ✓ |
| Azerbaijani | aze | Latn, Cyrl | ■▲ | ◆◈ | ▲◇ | ▲◈ | ▲◇ | – | ★ | ★ | ◎ | ✓ |
| S. Azerbaijani | azb | Arab | ■ | – | – | – | – | – | ★ | ★ | ◎ | ✓ |
| Turkmen | tuk | Latn, Cyrl | ■▲ | ◆◈ | ▲◇ | ▲◈ | ▲◇ | – | ★ | ★ | ◎ | ✓ |
| Gagauz | gag | Latn | ■ | ◆◈ | ◈ | ◈ | – | – | – | – | ◎ | ✓ |
| Kipchak | ||||||||||||
| Kazakh | kaz | Cyrl, Latn | ■● | ◆◈ | ●◇ | ●◈ | ●◇ | ● | ★ | ★ | ◎ | ✓ |
| Kyrgyz | kir | Cyrl | ■● | ◆◈ | ●◇ | ●◈ | ●◇ | – | ★ | ★ | ◎ | ✓ |
| Tatar | tat | Cyrl, Latn | ■▲ | ◆◈ | ▲◇ | ▲◈ | ▲◇ | – | ★ | ★ | ◎ | ✓ |
| Bashkir | bak | Cyrl | ■▲ | ◆◈ | ▲◇ | ▲◈ | ▲◇ | – | ★ | ★ | ◎ | ✓ |
| Crimean Tatar | crh | Latn, Cyrl | ■ | ◆◈ | ◈ | ◈ | – | – | ★ | ★ | ◎ | ✓ |
| Karakalpak | kaa | Latn, Cyrl | ■ | ◆◈ | ◇◈ | ◈ | ◇ | – | – | – | ◎ | ✓ |
| Nogai | nog | Cyrl | ■ | ◆◈ | ◇◈ | ◈ | ◇ | – | – | – | ◎ | ✓ |
| Kumyk | kum | Cyrl | ■ | ◆◈ | ◇◈ | ◈ | ◇ | – | – | – | ◎ | ✓ |
| Karachay-Balkar | krc | Cyrl | ■ | ◆◈ | ◇◈ | ◈ | ◇ | – | – | – | ◎ | ✓ |
| Karluk | ||||||||||||
| Uzbek | uzb | Latn, Cyrl | ■▲ | ◆◈ | ▲◇ | ▲◈ | ▲◇ | – | ★ | ★ | ◎ | ✓ |
| Uyghur | uig | Arab, Latn, Cyrl | ■● | ◆◈ | ●◇ | ●◈ | ●◇ | – | ★ | ★ | ◎ | ✓ |
| Siberian | ||||||||||||
| Sakha (Yakut) | sah | Cyrl | ■ | ◆◈ | ◇◈ | ◈ | ◇ | – | – | – | ◎ | ✓ |
| Altai | alt | Cyrl | ■ | ◆◈ | ◈ | ◈ | – | – | – | – | ◎ | ✓ |
| Tuvan | tyv | Cyrl | ■ | ◆◈ | ◈ | ◈ | – | – | – | – | – | ✓ |
| Khakas | kjh | Cyrl | ■ | ◆◈ | ◈ | ◈ | – | – | – | – | ◎ | ✓ |
| Oghur | ||||||||||||
| Chuvash | chv | Cyrl | ■ | ◆◈ | ◈ | ◈ | – | – | – | – | ◎ | ✓ |
| Arghu | ||||||||||||
| Khalaj | klj | Latn | ■ | ◈ | ◈ | ◈ | – | – | – | – | – | ✓ |
| Historical | ||||||||||||
| Ottoman Turkish | ota | Arab, Latn | ■ | ◈ | ◇◈ | ◈ | ◇ | – | – | – | – | ✓ |
| Old Turkic | otk | Runic, Latn | ■ | – | – | – | – | – | – | – | – | ✓ |
Apertium quality: production for Turkish, Kazakh and Tatar. Stable for Kyrgyz and Uzbek. Beta for Azerbaijani, Turkmen, Bashkir, Uyghur, Crimean Tatar and Chuvash. Prototype for the rest. Stanza: official UD models for Turkish, Kazakh, Kyrgyz and Uyghur. Custom-trained models for Uzbek, Turkmen, Azerbaijani, Tatar and Bashkir. Glot500 zero-shot: Karakalpak, Kumyk, Sakha, Karachay-Balkar and Nogai for POS and parsing.
TurkicNLP follows Stanza's architecture: Pipeline → Document → Sentence → Token → Word, mapping one-to-one to CoNLL-U. Every processor declares what it PROVIDES and REQUIRES, and the pipeline resolves the order.
One owner per field. The last processor that provides a field owns it, for example pos owns UPOS. Values from earlier processors are kept in word._provenance, so you can compare the Apertium and Stanza analyses of the same word.
Script-aware. Models are keyed by lang/script/processor/backend. A packaged catalog says what exists, and turkicnlp.download() fetches what's missing.
# Execution order
asr → script_detect → transliterate
→ tokenize → spellcheck → mwt
→ morph → morph_neural → pos → feats
→ lemma → depparse → ner
→ embeddings → sentiment → translate
→ transliterate_back
# Model storage
~/.turkicnlp/models/
kaz/Cyrl/morph/apertium/ # GPL-3.0, separate
multilingual/ # Glot500 checkpoints
huggingface/ # Glot500 backbone, NLLB
stanza/, stanza_custom/
Natural Language Processing Across Turkic Languages: Methods and Applications is a free online book with 17 chapters. Most chapters end in a practical section built on TurkicNLP 0.5.0: scripts and transliteration, finite-state and neural morphology, UD parsing, sequence labelling, MT, speech, language models and cross-lingual transfer.
Please cite the accompanying paper:
Sherzod Hakimov. TurkicNLP: An NLP Toolkit for Turkic Languages. arXiv:2602.19174, 2026.
@misc{hakimov2026turkicnlpnlptoolkit,
title = {TurkicNLP: An NLP Toolkit
for Turkic Languages},
author = {Sherzod Hakimov},
year = {2026},
eprint = {2602.19174},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2602.19174}
}
Native speakers, linguists and developers are all welcome. The roadmap lists planned features, known issues and open tasks. Items marked S are good first issues.
Open an issue with the language, the input text and the error. Reproducible reports help the most.
Speakers can spot wrong lemmas, tags and segmentations that no metric catches. Small lexicon and rule fixes go a long way.
Have a treebank, an FST, annotated data or a benchmark? Add a catalog entry or tell us about it.
Help with NER for more languages, pretrained sp_tokenize models and LLM evaluations per language.
Development setup, adding a language or processor, tests, code style and the pull-request process.
$ git clone https://github.com/turkic-nlp/turkicnlp.git
$ cd turkicnlp
$ pip install -e ".[dev]"
$ python -m pytest
Get help with the toolkit, share your research and discuss challenges with others working on Turkic languages.
The TurkicNLP library is Apache 2.0. Use it freely in research and commercial projects.
Apertium FST data is GPL-3.0-or-later. It's downloaded separately at runtime and never bundled in the pip package.
Models are used under their own licenses. NLLB-200 is non-commercial. With the LLM backend, your provider's terms apply.
| turkicnlp (pip) | Apache 2.0 |
| Apertium FSTs | GPL-3.0-or-later |
| Stanza models | Apache 2.0 |
| Omnilingual ASR | Apache 2.0 |
| NLLB-200 | CC-BY-NC-4.0 |
If TurkicNLP saves you time or powers your research, please consider supporting its development. Contributions fund model training and new language coverage.