Toolkit · v0.5

The TurkicNLP Toolkit

A modular, pip-installable Python library for the whole Turkic language family. Text and speech go in; tokens, morphemes, morphology, POS tags, dependency trees, entities, translations, embeddings and transcripts come out of one Stanza-style pipeline.

24 languages 15 processors 7 backend families Python 3.10 – 3.12 Apache 2.0

Installation

A small core, plus one extra per component group:

$ pip install turkicnlp            # core
$ pip install "turkicnlp[all]"     # all text components
$ pip install "turkicnlp[all,asr]" # + speech (own env)

Models are downloaded at runtime into ~/.turkicnlp, never bundled in the pip package. All extras →

At a glance Install Quick start Speech LLM backend Morphology Translation Language ID Scripts Processors Language matrix Architecture Cite Contribute
At a glance

One interface, many backends, every branch of the family

Each component uses the strongest resource available for each language. Where no trained model exists, the LLM backend fills the gap.

24
Turkic languages
Oghuz, Kipchak, Karluk, Siberian, Oghur, Arghu + historical
20
Apertium FSTs
Finite-state morphology via Python-native HFST
22
Neural morphology
Glot500 UPOS, UD features & lemmas
15
POS & parsing
9 Stanza (5 custom-trained) + Glot500
20
Speech recognition
Omnilingual ASR, output in each language's script
11
MT & embeddings
NLLB-200, one shared model
89%
Exact morpheme splits
Hybrid FST + neural, 483 words, 20 languages
all
LLM backend
Morph, NER, MT, sentiment, spelling for all 24
Installation

Install only what you need

Requires Python 3.10 – 3.12. Installation tests run on Ubuntu, macOS and Windows for every push.

ExtraAdds
turkicnlpTokenization, rule-based processing, transliteration, CoNLL-U I/O
[hfst]Apertium FST morphology (Linux, macOS)
[stanza]Stanza models: tokenize, POS, lemma, depparse, NER
[transformers]Glot500 POS / parsing / morphology models
[translation]NLLB-200 embeddings and machine translation
[lid]GlotLID language identification
[all]All text components above (Linux, macOS)
[asr]Speech recognition with Omnilingual ASR
[dev]pytest, black, ruff, mypy

Two tested environments

Text: [all], with torch 2.10, numpy 2 and transformers 5.

Speech + text: [all,asr]. omnilingual-asr pins torch 2.8, numpy 1.26 and transformers 4.57. Every text component also works with these versions.

Lock files for both are in requirements/.

Don't add [asr] to an existing text environment. It downgrades torch, numpy and huggingface_hub, which breaks transformers 5. Create a fresh environment instead.
Windows: everything except Apertium FST morphology works natively. hfst has no Windows wheels, so use WSL if you need FST morphology.
Quick start

Up and running in three steps

1

Install the library

$ pip install "turkicnlp[all]"
2

Download models for your language

The model catalog knows which backend to fetch for each language and script.

import turkicnlp

turkicnlp.download("kaz")
3

Build a pipeline and annotate text

Dependencies between processors are resolved automatically.

nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "depparse"])
doc = nlp("Мен мектепке бардым.")

print([(w.text, w.upos, w.lemma) for w in doc.words])
# [('Мен', 'PRON', 'мен'), ('мектепке', 'NOUN', 'мектеп'),
#  ('бардым', 'VERB', 'бар'), ('.', 'PUNCT', '.')]

print(doc.to_conllu())          # or doc.to_dict(full=True)

Sensible defaults, full control

Pipeline("kaz") without processors= loads the language's default set: tokenizer, morphology, POS, lemma, features, parsing and NER. Large or optional components (translation, embeddings, ASR, subword tokenization, LLM tasks) load only when you ask for them.

  • Pick a backend per processor: pos_backend="multilingual_glot500_model"
  • Pass options as <processor>_<option>: translate_tgt_lang="eng"
  • Batch processing: nlp.batch(texts, batch_size=32)
  • Provenance: doc._processor_log and word._provenance record which backend produced each value
  • Input in a script without models, such as Uzbek Cyrillic or Kazakh Latin, is transliterated, and results map back to your original text and offsets

Jupyter notebooks for every component: turkic-nlp-code-samples.

Speech recognition · new

From audio to annotated text in 20 Turkic languages

SpeechRecognizer wraps Meta's Omnilingual ASR behind TurkicNLP conventions: ISO 639-3 codes in, transcripts in each language's own script out. Long recordings are split at pauses.

import turkicnlp

asr = turkicnlp.SpeechRecognizer(model_card="omniASR_LLM_1B")
texts = asr.transcribe(["kaz_sample.wav", "tur_sample.wav"],
                       lang=["kaz", "tur"], batch_size=2)
# ['мен бүгін мектепке бардым алматы қазақстанның ең үлкен қаласы',
#  "bugün hava çok güzel ali dün ankara'ya gitti"]

# Ask for another script: Kazakh in the 2021 Latin alphabet
asr.transcribe("kaz_sample.wav", lang="kaz", script="Latn")

# Speech → text → analysis in one pipeline
nlp = turkicnlp.Pipeline("kaz",
    processors=["asr", "tokenize", "pos", "lemma", "depparse"])
doc = nlp.from_audio("kaz_sample.wav")
print(doc.audio_segments)   # time-aligned pieces

Supported languages

Turkish, Azerbaijani, S. Azerbaijani, Turkmen, Gagauz, Kazakh, Kyrgyz, Tatar, Bashkir, Crimean Tatar, Karakalpak, Nogai, Kumyk, Karachay-Balkar, Uzbek, Uyghur, Sakha, Altai, Khakas, Chuvash

Accepts file paths, waveforms, raw bytes and Hugging Face datasets audio columns. Stereo input is mixed to mono and resampled to 16 kHz.

omniASR_LLM_300M6.5 GB
omniASR_LLM_1B (default)9.1 GB · ≈ 6 GB RAM
omniASR_LLM_7B31.2 GB · GPU
omniASR_CTC_300M1.3 GB · fastest
LLM backend · new

Use any LLM for languages without trained models

Five processors can run on a large language model for all 24 languages, including Khakas, Tuvan and Nogai, which have no other neural models. Point the toolkit at a local vLLM or Ollama server, OpenAI, Claude or an in-process Hugging Face model.

BackendWhat the LLM does
morph=llmLemma, UPOS and UD features for every token
morph=apertium_llmChooses one of the FST's readings for ambiguous words. It can't invent an analysis.
ner=llmEntity spans over the toolkit's tokens, with configurable labels
translate=llmTranslation into any target language or script
sentiment=llmPositive, negative or neutral, or your own labels
spellcheck=llmSpelling correction, guarded against rewrites
nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "morph", "ner", "translate"],
    morph_backend="llm", ner_backend="llm",
    translate_backend="llm", translate_tgt_lang="eng",
    llm_model="qwen2.5:7b",                      # local Ollama
    llm_base_url="http://localhost:11434/v1",
)
# or: llm_provider="anthropic", llm_model="claude-sonnet-4-6"

# Free-form questions with the toolkit's analysis as context
turkicnlp.llm.ask("Explain the verb form.", doc=doc)

Answers are validated against the toolkit's tokens and retried once with feedback. Unusable answers are left empty rather than guessed. Prompts are versioned JSON files you can replace. Only validated answers go into a local cache.

LLM quality varies a lot between languages: usually good for Turkish, Kazakh, Uzbek and Azerbaijani, often weak for Khakas, Tuvan, Nogai or Kumyk. Evaluate a model on your language first, and remember that text goes to whichever server you configure.
Morphology & segmentation

Three ways into Turkic morphology, and a hybrid of them

Apertium finite-state transducers give precise analyses where the lexicon covers a word. Glot500 neural models cover the rest. The MorphemeTokenizer combines both to split words into labelled surface morphemes.

🧬

Apertium FST · morph

Transducers for 20 languages, loaded with Python-native HFST. No system Apertium install needed. Precompiled from upstream by CI, with Apertium → UD tag mapping for every language.

🧠

Glot500 neural · morph_neural

One multilingual model gives UPOS, UD features and lemmas for 22 languages: 20 trained, plus zero-shot Karakalpak, Karachay-Balkar and Nogai.

✂️

Morpheme-aware subwords · sp_tokenize

SentencePiece over morpheme-segmented text, so subword pieces never cross morpheme boundaries. Use it to train language models on Turkic text.

from turkicnlp.processors.morpheme_tokenizer import MorphemeTokenizer

tok = MorphemeTokenizer(lang="tur"); tok.load()
tok.segment("evlerinden").labeled
# [('ev', 'STEM'), ('ler', 'PLUR'), ('in', 'POSS.2SG'), ('den', 'ABL')]

tok = MorphemeTokenizer(lang="kaz"); tok.load()
tok.segment("оқығандар").labeled
# [('оқы', 'STEM'), ('ған', 'PTCP.PST'), ('дар', 'PLUR')]

tok = MorphemeTokenizer(lang="uzb"); tok.load()
tok.segment("bolalarning").labeled
# [('bola', 'STEM'), ('lar', 'PLUR'), ('ning', 'GEN')]

How the hybrid works

Every Apertium reading is turned into a segmentation by matching its tags against language-specific suffix allomorph tables, which encode vowel harmony, consonant alternations and buffer consonants. The tokenizer keeps the reading that explains the whole word, breaking ties by agreement with the neural lemma and POS. When the transducer has no reading, it segments the neural analysis instead.

Exact segmentation483 words · 20 langs
Hybrid (FST + neural)89%
Apertium only79%
Neural only68%

Allomorph tables cover 23 languages. The FST path is available for the 20 with a transducer.

Translation & embeddings

NLLB-200 translation and sentence vectors

One shared NLLB-200 (600M) model serves translation and 1024-dimensional embeddings for 11 Turkic languages. It's downloaded once and kept in memory once.

# Turkish → English
nlp = turkicnlp.Pipeline("tur", processors=["translate"],
                         translate_tgt_lang="eng")
doc = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
print(doc.translation)

# Sentence embeddings for similarity and retrieval
embed = turkicnlp.Pipeline("tur", processors=["embeddings"])
doc = embed("Parkta yürüyüş çok güzeldi.")
len(doc.embedding)   # 1024

translate_tgt_lang takes ISO 639-3 ("eng", "kaz") or FLORES-200 codes ("kaz_Cyrl"). For any of the 24 languages, translate_backend="llm" translates with an LLM instead.

Languages with NLLB support

Turkishtur
Azerbaijaniaze
S. Azerbaijaniazb
Turkmentuk
Kazakhkaz
Kyrgyzkir
Tatartat
Bashkirbak
Crimean Tatarcrh
Uzbekuzb
Uyghuruig

Models and datasets: huggingface.co/turkicnlp

Language identification

Tell closely related languages apart

The GlotLID FastText model knows 1,000+ language labels. By default, LanguageDetection restricts its predictions to the Turkic languages TurkicNLP supports.

import turkicnlp

lid = turkicnlp.LanguageDetection()      # Turkic label set by default
labels, probs = lid.predict("salam, hemmelere!", k=3)

# Or limit to an explicit label set
lid = turkicnlp.LanguageDetection(languages=[
    "__label__eng_Latn", "__label__tur_Latn", "__label__kaz_Cyrl",
])
lid.predict("Merhaba dünya!", k=1)

Script-aware labels

Labels combine language and script (kaz_Cyrl, uig_Arab), so you get the language and the writing system in a single step. TurkicNLP maps its codes to GlotLID's where they differ: Uzbek is uzn, North Azerbaijani is azj.

Pair it with automatic script detection in the pipeline to route mixed-language collections to the right models.

Scripts & transliteration

Latin, Cyrillic, Perso-Arabic, Runic and the Common Turkic Alphabet

Bidirectional conversion for every multi-script language, a shared Common Turkic Alphabet view for 23 languages, and Old Turkic runes.

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

Transliterator("kaz", Script.CYRILLIC, Script.LATIN).transliterate("Қазақстан Республикасы")
# Qazaqstan Respublikasy

Transliterator("uzb", Script.LATIN, Script.CYRILLIC).transliterate("Özbekiston Respublikasi")
# Ўзбекистон Республикаси

Transliterator("uig", Script.PERSO_ARABIC, Script.LATIN).transliterate("مەكتەپ")
# mektep

Transliterator("tuk", Script.LATIN, Script.COMMON_TURKIC).transliterate("Türkmen dili baýdyr")
# Türkmen dili baydır

Transliterator("otk", Script.OLD_TURKIC_RUNIC, Script.LATIN).transliterate("𐱅𐰇𐰼𐰚")
# törk

Coverage

Cyrillic ↔ Latin: Kazakh (2021), Uzbek, Azerbaijani, Tatar (Zamanälif), Turkmen, Karakalpak (2016), Crimean Tatar
Perso-Arabic ↔ Latin ↔ Cyrillic: Uyghur (ULY, Uyghur Cyrillic)
Latin → Perso-Arabic: Ottoman Turkish
Runic → Latin: Old Turkic (Orkhon-Yenisei)
Native script → CTS: all languages except Old Turkic (23)

Uzbek Latin uses ö, ğ, ş, ç throughout the toolkit. Pretrained models keep their training spelling internally, and adapters convert input and output automatically.

Pipeline processors

15 processors, one backend switch each

Request processors by name and choose a backend with <name>_backend=. Solid cards load by default. Dashed cards load only when named.

tokenize

Tokenization

Sentences, tokens and words. Abbreviation lists, ordinals, quotes.

rule · rule_arabic · stanza
mwt

Multi-word tokens

Expansion rules extracted from UD treebanks.

neural · stanza
morph

Morphological analysis

Lemma, UPOS and UD features from the FST, an LLM, or both.

apertium · llm · apertium_llm
morph_neural

Neural morphology

UPOS, features and lemmas for 22 languages.

multilingual_glot500_morph
pos

POS tagging

UPOS, XPOS and features for 15 languages.

stanza · glot500
feats

UD features

Morphological feature bundles.

multilingual_glot500_morph
lemma

Lemmatization

Neural lemmas with Turkic-aware casing.

stanza · glot500_morph
depparse

Dependency parsing

Biaffine parsers that always produce valid UD trees.

stanza · glot500
ner

Named entities

Starlang (Turkish) and KazNERD (Kazakh). The LLM backend covers every language.

stanza · llm
embeddings

Sentence embeddings

1024-dim NLLB encoder vectors.

nllb
translate

Machine translation

To any FLORES-200 language, or any language via an LLM.

nllb · llm
sentiment

Sentiment

Sentence labels, with custom label sets.

llm
spellcheck

Spelling correction

Corrections with a guard against rewrites.

llm
sp_tokenize

Subword tokenization

Morpheme-aware SentencePiece.

sentencepiece
asr

Speech recognition

Transcribe audio, then analyse it with Pipeline.from_audio.

omnilingual_asr

Standalone tools outside the pipeline: MorphemeTokenizer, Transliterator, LanguageDetection, SpeechRecognizer, turkicnlp.llm.ask. Full CoNLL-U reader and writer.

Language support matrix

What works for each language, and which backend provides it

Grouped by branch. The LLM backends (morphology, NER, translation, sentiment, spelling) work for all 24 languages on top of everything listed.

■Rule-based ◆Apertium FST ●Stanza / UD ▲Custom Stanza ◇Glot500 POS/parse ◈Glot500 morph ★NLLB-200 ◎Omnilingual ASR –not yet
LanguageCodeScripts TokenizeMorphPOSLemmaDepParseNEREmbedMTASRLLM
Oghuz
TurkishturLatn■●◆◈●◇●◈●◇●★★◎✓
AzerbaijaniazeLatn, Cyrl■▲◆◈▲◇▲◈▲◇–★★◎✓
S. AzerbaijaniazbArab■–––––★★◎✓
TurkmentukLatn, Cyrl■▲◆◈▲◇▲◈▲◇–★★◎✓
GagauzgagLatn■◆◈◈◈––––◎✓
Kipchak
KazakhkazCyrl, Latn■●◆◈●◇●◈●◇●★★◎✓
KyrgyzkirCyrl■●◆◈●◇●◈●◇–★★◎✓
TatartatCyrl, Latn■▲◆◈▲◇▲◈▲◇–★★◎✓
BashkirbakCyrl■▲◆◈▲◇▲◈▲◇–★★◎✓
Crimean TatarcrhLatn, Cyrl■◆◈◈◈––★★◎✓
KarakalpakkaaLatn, Cyrl■◆◈◇◈◈◇–––◎✓
NogainogCyrl■◆◈◇◈◈◇–––◎✓
KumykkumCyrl■◆◈◇◈◈◇–––◎✓
Karachay-BalkarkrcCyrl■◆◈◇◈◈◇–––◎✓
Karluk
UzbekuzbLatn, Cyrl■▲◆◈▲◇▲◈▲◇–★★◎✓
UyghuruigArab, Latn, Cyrl■●◆◈●◇●◈●◇–★★◎✓
Siberian
Sakha (Yakut)sahCyrl■◆◈◇◈◈◇–––◎✓
AltaialtCyrl■◆◈◈◈––––◎✓
TuvantyvCyrl■◆◈◈◈–––––✓
KhakaskjhCyrl■◆◈◈◈––––◎✓
Oghur
ChuvashchvCyrl■◆◈◈◈––––◎✓
Arghu
KhalajkljLatn■◈◈◈–––––✓
Historical
Ottoman TurkishotaArab, Latn■◈◇◈◈◇––––✓
Old TurkicotkRunic, Latn■––––––––✓

Apertium quality: production for Turkish, Kazakh and Tatar. Stable for Kyrgyz and Uzbek. Beta for Azerbaijani, Turkmen, Bashkir, Uyghur, Crimean Tatar and Chuvash. Prototype for the rest. Stanza: official UD models for Turkish, Kazakh, Kyrgyz and Uyghur. Custom-trained models for Uzbek, Turkmen, Azerbaijani, Tatar and Bashkir. Glot500 zero-shot: Karakalpak, Kumyk, Sakha, Karachay-Balkar and Nogai for POS and parsing.

Design

Built to be extended

TurkicNLP follows Stanza's architecture: Pipeline → Document → Sentence → Token → Word, mapping one-to-one to CoNLL-U. Every processor declares what it PROVIDES and REQUIRES, and the pipeline resolves the order.

One owner per field. The last processor that provides a field owns it, for example pos owns UPOS. Values from earlier processors are kept in word._provenance, so you can compare the Apertium and Stanza analyses of the same word.

Script-aware. Models are keyed by lang/script/processor/backend. A packaged catalog says what exists, and turkicnlp.download() fetches what's missing.

# Execution order
asr → script_detect → transliterate
  → tokenize → spellcheck → mwt
  → morph → morph_neural → pos → feats
  → lemma → depparse → ner
  → embeddings → sentiment → translate
  → transliterate_back

# Model storage
~/.turkicnlp/models/
  kaz/Cyrl/morph/apertium/      # GPL-3.0, separate
  multilingual/                 # Glot500 checkpoints
  huggingface/                  # Glot500 backbone, NLLB
  stanza/, stanza_custom/
Cover of Natural Language Processing Across Turkic Languages
The companion book

Learn the methods behind the toolkit

Natural Language Processing Across Turkic Languages: Methods and Applications is a free online book with 17 chapters. Most chapters end in a practical section built on TurkicNLP 0.5.0: scripts and transliteration, finite-state and neural morphology, UD parsing, sequence labelling, MT, speech, language models and cross-lingual transfer.

Citation

If you use TurkicNLP in your research

Please cite the accompanying paper:

Sherzod Hakimov. TurkicNLP: An NLP Toolkit for Turkic Languages. arXiv:2602.19174, 2026.

@misc{hakimov2026turkicnlpnlptoolkit,
  title  = {TurkicNLP: An NLP Toolkit
            for Turkic Languages},
  author = {Sherzod Hakimov},
  year   = {2026},
  eprint = {2602.19174},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url    = {https://arxiv.org/abs/2602.19174}
}
Contributing

Help us cover every Turkic language

Native speakers, linguists and developers are all welcome. The roadmap lists planned features, known issues and open tasks. Items marked S are good first issues.

🐛

Report bugs

Open an issue with the language, the input text and the error. Reproducible reports help the most.

🔎

Check analyses in your language

Speakers can spot wrong lemmas, tags and segmentations that no metric catches. Small lexicon and rule fixes go a long way.

🌍

Add resources

Have a treebank, an FST, annotated data or a benchmark? Add a catalog entry or tell us about it.

🧠

Train and evaluate models

Help with NER for more languages, pretrained sp_tokenize models and LLM evaluations per language.

📘

Read CONTRIBUTING.md

Development setup, adding a language or processor, tests, code style and the pull-request process.

Development setup

$ git clone https://github.com/turkic-nlp/turkicnlp.git
$ cd turkicnlp
$ pip install -e ".[dev]"
$ python -m pytest
💬

Join the TurkicNLP community

Get help with the toolkit, share your research and discuss challenges with others working on Turkic languages.

Licensing

Open source, with clear boundaries

The TurkicNLP library is Apache 2.0. Use it freely in research and commercial projects.

Apertium FST data is GPL-3.0-or-later. It's downloaded separately at runtime and never bundled in the pip package.

Models are used under their own licenses. NLLB-200 is non-commercial. With the LLM backend, your provider's terms apply.

Quick reference

turkicnlp (pip)Apache 2.0
Apertium FSTsGPL-3.0-or-later
Stanza modelsApache 2.0
Omnilingual ASRApache 2.0
NLLB-200CC-BY-NC-4.0
Support the project

Keep Turkic NLP open & growing

If TurkicNLP saves you time or powers your research, please consider supporting its development. Contributions fund model training and new language coverage.