Open · Research-backed · Community-driven

NLP for all 24 Turkic languages.

A pip-installable Python toolkit that takes text and speech to annotated output across the whole language family, and a free online book on the methods behind it.

24 languages Morphology & parsing Speech recognition LLM backend Open book

What's new

  • bookNLP Across Turkic Languages: 17 chapters, exercises and TurkicNLP practicals, free online
  • speechSpeech recognition for 20 Turkic languages, from audio straight into the pipeline
  • llmMorphology, NER, translation, sentiment and spelling with any LLM, for all 24 languages
  • morphHybrid FST + neural morpheme tokenizer: 89% exact segmentation across 20 languages
$ pip install "turkicnlp[all]"
🔧

The Toolkit

A Stanza-style modular pipeline with seven backend families: rule-based, Apertium FSTs, Stanza, multilingual Glot500 models, NLLB-200, Omnilingual ASR, and any LLM you choose.

It handles Latin, Cyrillic, Perso-Arabic and Old Turkic runes natively. Text in a script without models is transliterated automatically and mapped back.

📖

The Book

Natural Language Processing Across Turkic Languages: Methods and Applications explains how to build and evaluate language technology for the family. It covers scripts and corpora, finite-state and neural morphology, syntax, MT, speech, language models and cross-lingual transfer.

Each chapter pairs linguistic analysis with a practical section and exercises.

Toolkit capabilities

Everything you need to build Turkic NLP systems

Fifteen pipeline processors with a backend switch on each. Pick what you need, chain it together, and export to CoNLL-U or JSON.

🔤

Tokenization & Scripts

Rule-based and Stanza tokenizers for every script, with automatic script detection and transparent transliteration for scripts that lack models.

🧬

Morphological Analysis

Apertium FSTs for 20 languages through Python-native HFST, plus Glot500 neural UPOS, features and lemmas for 22.

✂️

Morpheme Segmentation

A hybrid FST + neural tokenizer that splits words into labelled morphemes, plus morpheme-aware SentencePiece for training language models.

🏷️

POS, Parsing & NER

Neural taggers and parsers for 15 languages: official and custom-trained Stanza models, and multilingual Glot500. NER for Turkish and Kazakh, and for every language through the LLM backend.

🎙️

Speech Recognition

Omnilingual ASR for 20 Turkic languages, with transcripts in each language's script. Pipeline.from_audio runs speech → text → analysis.

🤖

LLM Backend

Run morphology, NER, translation, sentiment and spelling correction on vLLM, Ollama, OpenAI, Claude or a Hugging Face model. Answers are validated, cached and logged.

🌐

Translation & Embeddings

One NLLB-200 model gives translation and 1024-dimensional sentence embeddings for 11 Turkic languages.

🔍

Language Identification

GlotLID with script-aware labels, restricted to Turkic languages by default to separate close relatives.

🔁

Transliteration

Cyrillic ↔ Latin ↔ Perso-Arabic, Old Turkic runes, and the Common Turkic Alphabet for 23 languages.

Cover of Natural Language Processing Across Turkic Languages: Methods and Applications, by Sherzod Hakimov
The book · free online

Natural Language Processing Across Turkic Languages

The family shares much of its grammar, yet data, tools and evaluations differ greatly from one language and task to the next. This book shows how to build and assess language technology with both facts in view, around three questions: how morphology affects a task, what changes when the script changes, and what transfers between related languages.

  • 1–2The Language Family
  • 3–4Writing Systems & Corpora
  • 5–7Morphology
  • 8Syntax
  • 9–11Applications
  • 12–15Language Models
  • 16–17Cross-Lingual Methods
  • A–DAppendices & Glossary
Language coverage

24 Turkic languages and growing

From well-resourced Turkish to endangered Siberian varieties. Labels show the trained components. The LLM backend adds morphology, NER, translation, sentiment and spelling for every language.

TurkishturFull + NER
KazakhkazFull + NER
KyrgyzkirFull pipeline
UyghuruigFull pipeline
UzbekuzbFull pipeline
AzerbaijaniazeFull pipeline
TatartatFull pipeline
BashkirbakFull pipeline
TurkmentukFull pipeline
Crimean TatarcrhMorph + MT + ASR
S. AzerbaijaniazbMT + ASR
SakhasahMorph + Parse + ASR
KarakalpakkaaMorph + Parse + ASR
KumykkumMorph + Parse + ASR
NogainogMorph + Parse + ASR
Karachay-BalkarkrcMorph + Parse + ASR
GagauzgagMorph + ASR
ChuvashchvMorph + ASR
AltaialtMorph + ASR
KhakaskjhMorph + ASR
TuvantyvMorphology
KhalajkljNeural morph
Ottoman TurkishotaMorph + Parse
Old TurkicotkTransliteration

"Full pipeline" means tokenization, FST and neural morphology, POS, lemmas, dependency parsing, NLLB translation and embeddings, and speech recognition.

Join the community

Get help, share your work, and collaborate

The Turkic NLP community spans researchers, practitioners and language enthusiasts around the world. Native speakers can help most by checking analyses in their language. The roadmap lists open tasks.

Support the project

Help keep Turkic NLP open

The toolkit and the book are free and open. If this work is useful to you for research, teaching or building products, please consider supporting its development.

Funds go directly toward model training, infrastructure and new language coverage.