Appendix B — Computational Resources and Tools
This appendix catalogues resources used in the book: environments and libraries (Section B.1, Section B.2), datasets (Section B.3), pretrained models (Section B.4) and community infrastructure (Section B.5). Reproduction notes follow in Section B.6. Entries refer to the sources and revisions checked; newer and unlisted resources need their own verification.
The resources include rule-based tools from Apertium and HFST, corpora from OPUS and Common Voice, models from the Hugging Face Hub, and language-specific datasets from groups such as ISSAI and SIGTURK. This appendix gives starting points; the task chapters describe the relevant releases and limitations.
B.1 Environment Setup
B.1.1 Tested Profiles and Installation
Use separate environments for HFST/toolkit work, transformer experiments and Omnilingual ASR. The book’s runs used Python 3.11 to 3.13, with packages varying by chapter. Each run records Python and package versions and downloaded-file SHA-256 hashes. Reuse those records when reproducing a result, and check other platform or package combinations before relying on them.
For a fresh environment, choose a supported Python/runtime and the platform-specific PyTorch installation from its official instructions, then install the pinned direct dependencies in code/common/requirements-tested.txt. That file records the checked package versions; it is not a complete cross-platform lock or evidence of a clean install on every OS. Save the resolved environment for a published experiment.
Install only the toolkit extras needed by the chosen example. HFST availability and native dependencies differ by platform; a Windows environment may need WSL for those tools. The Omnilingual wrapper’s environment is separate from the Whisper recipe in Chapter 11. Do not install conflicting speech stacks into an existing working text environment.
Memory and runtime depend on the exact configuration, so measure them on the planned run; each exercise README records the runtime of the book’s run. The exercises run on a CPU; the procedure-only fine-tuning recipes are faster on a GPU.
B.2 Key Libraries and Frameworks
B.2.1 Hugging Face Ecosystem
The Hugging Face ecosystem is the primary software infrastructure used throughout this book. Four libraries are central.
transformers provides a unified API for loading, fine-tuning, and running pretrained models. Models are identified by a Hub ID string (e.g., dbmdz/bert-base-turkish-cased); the appropriate model class and input conventions depend on whether the task uses an encoder, a causal language model or a sequence-to-sequence model. Pin the revision of every checkpoint, as the exercises do; a newly added task head is random until trained.
datasets loads corpora and benchmarks from the Hugging Face Hub or from local files, with streaming for large datasets. The book’s exercises instead read most datasets from commit-pinned GitHub files and check their SHA-256 hashes (for KazNERD and KyrgyzNER, see Chapter 9), so that a rerun uses exactly the same release.
peft implements parameter-efficient fine-tuning methods including LoRA and QLoRA, which reduce the memory needed to adapt large models. Use the complete XLM-R sequence-classification recipe in Section 12.5. Target names are architecture-specific, and a new classifier head must be trainable and saved with its adapter. Copying q_proj/v_proj from a decoder into a BERT example is not a valid universal setup.
evaluate wraps standard metrics such as seqeval for BIO entity spans, sacrebleu and chrf for machine translation and rouge for summarisation. Feature and exact-bundle accuracy for morphology are computed by the book’s own code (Chapter 15); report the metric signature with every score.
The Hugging Face Hub at huggingface.co hosts models, datasets and spaces relevant to Turkic NLP. Language tags (e.g. kk or kaz for Kazakh) are self-reported metadata: a tag search can miss resources and count copies, so record the query and date, as the resource audit in Section 17.6 does.
B.2.2 TurkicNLP Toolkit
The TurkicNLP toolkit (Hakimov 2026) provides one pipeline for 24 Turkic languages, with interchangeable rule-based, finite-state (Apertium), per-language neural (Stanza), and multilingual neural (Glot500) backends, plus script detection, transliteration, Common Turkic Script (CTS) conversion, language identification (GlotLID), morpheme segmentation, and NLLB-based embeddings and translation.
- GitHub: github.com/turkic-nlp/turkicnlp
- Installation:
pip install turkicnlp(extras such asturkicnlp[hfst]add heavier backends) - Languages: 24 (coverage per component differs; see the model catalogue)
- Processors:
tokenize,mwt,morph,morph_neural,pos,feats,lemma,depparse,ner,embeddings,translate,sentiment,spellcheck,sp_tokenize,asr - Other entry points:
LanguageDetection,SpeechRecognizer,MorphemeTokenizer,Transliterator,turkicnlp.llm.ask,list_processors,list_asr_languages - LLM backends:
morph,ner,translate,sentimentandspellcheckcan use any LLM (vLLM, Ollama, OpenAI, Claude, or a local Hugging Face model) for all 24 languages - Licences: the library is Apache-2.0; Apertium transducers (GPL-3.0) and NLLB weights (CC-BY-NC-4.0) are downloaded separately
# pip install "turkicnlp[stanza]==0.5.0" # list_processors works without it; loading Stanza models needs it
import turkicnlp
print(sorted(turkicnlp.list_processors("kaz"))) # what the catalog offers for Kazakh
turkicnlp.download("kaz", processors=["tokenize", "morph"])
kz = turkicnlp.Pipeline("kaz", processors=["tokenize", "morph"])
doc = kz("Үйлерімізден өте алыс кеткенбіз.")
for word in doc.words:
print(word.text, word.lemma, word.upos, word.feats)
print(doc._processor_log) # which backend produced what
print(doc.to_conllu())Three behaviours are worth knowing. Field ownership: when several processors provide the same field, the last one in the pipeline owns it, and the others’ values are kept in word._provenance. Script bridging: input in a script without models (Kazakh Latin, Uzbek Cyrillic) is transliterated, processed, and mapped back to the original text. Batching: nlp.batch(texts, batch_size=32) hands whole batches to the neural and LLM backends.
B.2.3 Apertium
Apertium (Forcada et al. 2011; Khanna et al. 2021) is an open-source rule-based machine translation and morphological analysis platform. For Turkic NLP, its primary value is the collection of finite-state morphological analysers for 20+ Turkic languages, whose coverage and ambiguity should be evaluated against independent gold data.
- Website: apertium.org
- GitHub: github.com/apertium
- Turkic language modules: github.com/apertium/apertium-tur (Turkish), github.com/apertium/apertium-kaz (Kazakh), and equivalents for other languages
Section 10.2 gives a dated inventory of Apertium translation pairs and their development stages. Check the selected pair’s release and supported direction before use; a repository listing does not guarantee support in both directions.
B.2.4 HFST (Helsinki Finite-State Toolkit)
HFST (Lindén et al. 2011) is a toolkit for compiling and running finite-state transducers. The Apertium Turkic analysers used in this book are compiled with HFST; Apertium also supports other finite-state toolchains.
- Website: hfst.github.io
- Installation:
pip install hfst(Python bindings) or system packages
HFST supports the standard finite-state operations (composition, intersection, minimisation); Chapter 6 shows how to load and query the Apertium transducers with its Python bindings.
B.2.5 Stanza (Stanford NLP)
Stanza (Qi et al. 2020) provides neural NLP pipelines for tokenisation, POS tagging, morphological analysis, and dependency parsing. Its 1.11.0 model catalogue has models for five Turkic languages with UD data: Turkish, Kazakh, Kyrgyz, Uyghur and Ottoman Turkish (later catalogues add Uzbek; Chapter 8); TurkicNLP adds Stanza models for Azerbaijani, Uzbek, Turkmen, Tatar, and Bashkir.
TurkicNLP exposes Stanza backends through its pipeline (Section 8.7). Check the catalogue for the requested language and processor: a parsing model does not imply that a dedicated NER model is available.
Use Stanza 1.11 (pip install "turkicnlp[stanza]" enforces stanza>=1.8,<1.12): the custom Azerbaijani, Uzbek, Turkmen, Tatar and Bashkir models were trained with it, and newer Stanza versions fail to load their parsers.
The measured Stanza baselines, including the caveat on the Kazakh replay score, are in Chapter 8. For other languages, TurkicNLP’s multilingual Glot500 tagger and parser and its Apertium backend are the alternatives.
B.2.6 Zemberek-NLP (Turkish)
Zemberek (Akın and Akın 2007) is an open-source Java library for Turkish morphological analysis, disambiguation, tokenisation, spell-checking and normalisation (github.com/ahmetaa/zemberek-nlp); community Python wrappers exist. Verify the maintained implementation and evaluate it on the intended domain.
B.3 Datasets and Corpora
B.3.1 Parallel Corpora
| Corpus | Languages | Size | URL |
|---|---|---|---|
| TIL corpus | 22 Turkic languages | release-specific totals and direction counts in Chapter 10 | github.com/turkic-interlingua/til-mt |
| OPUS | Many, incl. all major Turkic | Varies by subcorpus | opus.nlpl.eu |
| FLORES-200 | 200 languages | 997 dev; 1,012 devtest sentences | huggingface.co/datasets/facebook/flores |
The TIL corpus (Mirzakhalov et al. 2021) provides parallel data for 22 Turkic languages; release-specific counts, directions and split-overlap cautions are discussed in Chapter 10. FLORES-200 provides the same evaluation sentences for the eleven Turkic varieties in NLLB-200 and is one possible MT evaluation source, provided language coverage, contamination and domain fit are checked.
B.3.2 Monolingual Corpora
| Corpus | Language | Approx. Size | URL |
|---|---|---|---|
| HPLT v3.0 | many, incl. 11 Turkic subsets | per-language sizes in Section 4.1 | hplt-project.org |
| FineWeb2 | many, incl. Turkic | per-language sizes in Section 4.1 | huggingface.co/datasets/HuggingFaceFW/fineweb-2 |
| CulturaX | 167 languages | per-language sizes in Section 4.1 | huggingface.co/datasets/uonlp/CulturaX |
| OSCAR | 160+ languages incl. Turkic | release-specific | huggingface.co/datasets/oscar-corpus/OSCAR-2301 |
| CC-100 | 100 languages | release-specific | data.statmt.org/cc-100 |
| Turkish National Corpus | tur | 50M words (2012 description) | Corpus description |
The Turkish National Corpus (TNC) (Aksan et al. 2012) was designed as a balanced, general-purpose corpus of 50 million words. This is the size reported in the 2012 description. HPLT, FineWeb2 and CulturaX are automatically labelled web collections. Their per-language sizes differ substantially; Chapter 4 compares their releases and explains how to audit a subset before use.
B.3.3 Treebanks
UD 2.17 has treebanks for Turkish, Ottoman Turkish, Kazakh, Kyrgyz, Uzbek, Uyghur, Tatar, Azerbaijani, Sakha (UD: Yakut) and Old Turkic (UD: Old Turkish), plus Turkish–German and Turkish–English code-switching treebanks (Chapter 8, Table 8.5 and Table 8.6). They provide reference annotations for tokenisation, POS tagging, morphological features and dependency structure in CoNLL-U format. Annotation methods and quality vary by treebank; check the release documentation and the examples in Chapter 8.
- Download: universaldependencies.org (all treebanks, versioned releases)
- Parallel UD-Turkic sentences: github.com/ud-turkic/parallel (Akhundjanova et al. 2025)
- Format: CoNLL-U (tab-separated syntactic-word records, with additional records for multiword tokens and empty nodes)
- Python loading:
pip install conllu
In the audited official UD 2.17 inventory, languages without a dedicated released treebank include Turkmen, Bashkir, Chuvash, Crimean Tatar, Gagauz, Karakalpak, Kumyk, Nogai, Karachay-Balkar, and all Siberian Turkic languages except Sakha.
B.3.4 NER Datasets
| Dataset | Language | Entity Types | Source | Reference |
|---|---|---|---|---|
| KazNERD | Kazakh | 25 classes; 112,702 sentences | github.com/IS2AI/KazNERD | Yeshpanov, Khassanov, and Varol (2022) |
| KyrgyzNER | Kyrgyz | 27 classes; 7,033 training and 3,867 test sentences | github.com/Akyl-AI/KyrgyzNER | Turatali et al. (2025) |
| WikiANN | 10+ Turkic | PER, ORG, LOC | Hub configuration per language | Pan et al. (2017) |
KazNERD reports 112,702 sentences from Kazakh television news, with 136,333 annotations in 25 classes (Yeshpanov, Khassanov, and Varol 2022). It supplies the NER examples in Chapter 9. WikiANN derives silver-standard annotations from Wikipedia in many languages. Its original publication and later releases differ in coverage; use the pinned configurations in that chapter and assess label quality for the chosen language and task.
B.3.5 Benchmarks
| Benchmark | Languages | Tasks | Reference |
|---|---|---|---|
| TUMLU | 9 listed; release caveat in Section 15.3 | Native MCQs | Isbarov et al. (2025) |
| Kardeş-NLU | az, kk, ky, uz, ug | Cross-lingual NLU | Senel et al. (2024) |
| Mukayese | Turkish | Multiple NLP tasks | Safaya et al. (2022) |
| KazQAD | Kazakh | QA (reading comprehension) | Yeshpanov et al. (2024) |
| X-WMT | 8 Turkic + en, ru | MT (88 directions) | Mirzakhalov et al. (2021) |
| KazMMLU | Kazakh, Russian | Native MCQs | Togmanov et al. (2025) |
| FLORES-200 / SIB-200 | 11 Turkic varieties | MT / topic classification | NLLB Team et al. (2024); Adelani et al. (2024) |
TUMLU (Isbarov et al. 2025) lists Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Kyrgyz, Tatar, Turkish, Uyghur and Uzbek, with school questions in 11 subjects; its conflicting language counts require release-specific checks (Chapter 15). Kardeş-NLU (Senel et al. 2024) evaluates cross-lingual NLU for five Turkic languages with Turkish as a high-resource relative. Mukayese (Safaya et al. 2022) is a multi-task benchmark suite for Turkish.
B.3.6 Speech Datasets
| Dataset | Language | Size | Source |
|---|---|---|---|
| KSC2 | Kazakh | >1,000 hours | issai.nu.edu.kz (Mussakhojayeva, Khassanov, and Varol 2022a) |
| KazakhTTS2 | Kazakh | ~271 hours, 5 speakers | issai.nu.edu.kz |
| FLEURS | tr, az, kk, ky, uz, … | n-way parallel read speech | google/fleurs |
| KazEmoTTS | Kazakh | Emotional speech | issai.nu.edu.kz |
| Common Voice | Multiple Turkic | Per release; validated and training hours differ | Mozilla Data Collective; see Chapter 11 |
ISSAI (Nazarbayev University) (Mussakhojayeva et al. 2023; Mussakhojayeva, Khassanov, and Varol 2022b) has produced Kazakh speech resources including the KSC2 ASR corpus and the KazakhTTS2 synthesis corpus. Common Voice has datasets for several Turkic languages whose sizes depend on volunteer activity; check the current release statistics.
B.4 Pretrained Models
B.4.1 Verified Configurations
AutoModel on the meta device, without pretrained weights; a task head changes the count. Availability is not a quality evaluation. Sources, revisions and licence metadata: tables/tbl-verified-models/data.csv.
| Checkpoint | Role | Base parameters | Loaded class |
|---|---|---|---|
| dbmdz/bert-base-turkish-cased | Turkish encoder | 110,617,344 | BertModel |
| FacebookAI/xlm-roberta-base | Multilingual encoder | 278,043,648 | XLMRobertaModel |
| google/mt5-base | Multilingual encoder-decoder | 390,315,264 | MT5Model |
| openai/whisper-small | Speech encoder-decoder | 241,734,912 | WhisperModel |
| facebook/mms-tts-chv | Chuvash TTS | 36,284,784 | VitsModel |
| facebook/mms-tts-crh | Crimean Tatar TTS | 36,285,744 | VitsModel |
The table counts base-model parameters from pinned configurations on a meta device, without downloading weights. BERTurk vocabulary variants have different embedding sizes; XLM-R base is not the 125M-parameter monolingual RoBERTa base. A task head changes the count. The CSV preserves model class, source URL, revision and licence metadata.
B.4.2 Additional Models
Glot500, NLLB, BERTbek, TURNA, KAZ-LLM and Sherkala are discussed with their publications and task results in Chapter 7, Chapter 10 and Chapter 12. Use each experiment’s pinned checkpoint and verify any additional Hub release. Choose by measured performance and model purpose: NLLB is trained for translation, while base mT5 uses span-corruption pretraining. Instruction-following chat requires further adaptation or evaluation.
B.4.3 Speech Models
| Model | Coverage | Source |
|---|---|---|
| Whisper large-v3 | 100 language tokens, including seven Turkic languages | openai/whisper-large-v3 configuration |
| TurkicASR | 10 Turkic languages | github.com/IS2AI/TurkicASR |
| TurkicTTS | Kazakh-trained, zero-shot for other Turkic languages | github.com/IS2AI/TurkicTTS |
| Söyle | 11 Turkic and other languages | github.com/IS2AI/Soyle |
| MMS | 1,100+ languages | facebook/mms-1b-all |
| Omnilingual ASR | 20 Turkic languages through TurkicNLP | via turkicnlp[asr]; see Chapter 11 |
Whisper (OpenAI) covers seven Turkic languages (tr, az, uz, kk, tt, tk, ba) but not Kyrgyz or Uyghur. TurkicASR (Mussakhojayeva et al. 2023) covers Azerbaijani, Bashkir, Chuvash, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Uyghur, and Uzbek; TurkicTTS (Yeshpanov, Mussakhojayeva, and Khassanov 2023) synthesises Turkic languages zero-shot from a Kazakh-trained model (Chapter 11).
B.5 Community and Infrastructure
The organisations behind most of these resources (Apertium, SIGTURK, ISSAI, Universal Dependencies, Turkic Interlingua, the UD-Turkic group, TurkLang and the Turkic Academy) are described in Section 1.5. Two practical points for contributors:
- Universal Dependencies publishes two releases a year; new treebanks are submitted through the UD GitHub organisation, and language-specific guidelines are in the UD documentation. For a language without a treebank, see Section 8.6.
- Apertium modules are separate GitHub repositories (
apertium-kaz,apertium-kir,apertium-nog, …) documented on wiki.apertium.org; extending an existing module is the route described in Section 17.3.
B.6 Reproducing Results from This Book
B.6.1 Local Reproduction Manifest
The code is organised by chapter: code/ch04-corpus/, code/ch05-ambiguity/, code/ch06-toy-fst/, code/ch07-neural/, code/ch09-ner/, code/ch10-mt/, code/ch11-speech/, code/ch12-classify/, code/ch13-retrieval/, code/ch14-generation/, code/ch15-tumlu/, code/ch16-transfer/ and code/ch17-resources/. Shared helpers, the mechanical checks and the tested package versions are in code/common/, whose README documents inputs, commands, outputs and limits for classification, speech, TUMLU, corpus auditing, retrieval, transfer and resource observations. The exercise notebooks are in exercises/.
Run python code/common/check.py in the recorded test environment. This checks fixtures, tiny models and artefact reload; it does not reproduce pretrained-model task accuracy. Full data/model downloads, dataset access conditions and independent human labels are separate prerequisites. Code-block extraction alone is not an adequate end-to-end reproduction test.
B.6.2 Setting Up Chapter by Chapter
The reading routes in the Preface give the order of chapters. Install only the extras a chapter uses, and keep the speech environment separate.