# Task 3: Diacritics – Automatic Restoration of Diacritics in Slovak Text *English translation by SOTA – AI Community of the Slovak original, [uloha3-diakritika.pdf](https://ai-olympiada.sk/wp-content/uploads/sites/91/2026/08/uloha3-diakritika.pdf) ("Úloha 3: Diakritika: Automatická obnova diakritiky v slovenskom texte"), from the Olympiáda v umelej inteligencii (Slovak Olympiad in Artificial Intelligence), 2025/26, home round. The original task belongs to its authors. Slovak example sentences are kept as they are, because they are the data of the task; English glosses in square brackets were added by the translator. Organisers who would like this translation removed can email sota.ai.community@gmail.com.* ## Story Morning. A bus. A mobile phone in your hand. A friend texts you: > *"Ahoj, mozes mi pozicat poznamky z matiky? Zabudol som si zosit doma a ucitelka sa bude pytat. Dakujem!"* > > [*"Hi, can you lend me your maths notes? I left my exercise book at home and the teacher is going to ask. Thanks!"*] You understand him, of course. But what if a computer received a message like this? Or what if you wanted to publish this text in the school magazine? Writing without diacritics is common in Slovakia – especially when typing quickly on a mobile phone, in chats, or when we use a keyboard without Slovak characters. The problem arises when we need to process such a text automatically, translate it, or simply make it more readable. The sentence "mama ma rada" can mean: - "mama má rada" (mum likes doing something) [*má* = "has"; *má rada* = "likes"] - "mama ma rada" (mum likes doing something with me, say "dressing" me) [*ma* = "me"] - "mamá mä rada" (well, this one is not correct at all…) Your task is therefore relatively straightforward: to create an intelligent system that can **automatically restore diacritics** in Slovak text. In doing so, you will use the power of modern language models – specifically **SlovakBERT**, the first (not all that) large language model trained specifically for Slovak. ## Task overview The task consists of **three parts** that build on one another: | Part | Name | Points | Description | |---|---|---|---| | 1 | Data preparation | 30 | Extracting and processing training data from the Slovak Wikipedia | | 2 | Zero-shot approach | 20 | Restoring diacritics without fine-tuning, using only the pretrained SlovakBERT | | 3 | Fine-tuned model | 50 | Training your own model on the data from Part 1 | **Common input:** Text in Slovak with the **diacritics removed**. **Common output:** The same text with the **diacritics correctly restored**. ### Example **Input:** ``` Slovensko je krajina v strednej Europe. Hlavne mesto je Bratislava. ``` **Expected output:** ``` Slovensko je krajina v strednej Európe. Hlavné mesto je Bratislava. ``` [*"Slovakia is a country in Central Europe. The capital is Bratislava."*] --- ## Part 1: Data preparation (30 points) ### Task Your first task is to prepare **training data** for a diacritics-restoration model. As the source you will use the **Slovak Wikipedia dumps**, which are freely available at: ``` https://dumps.wikimedia.org/skwiki/20251120/ ``` Specifically, you need the file `skwiki-latest-pages-articles.xml.bz2`. ### Requirements 1. **Download and process** the Slovak Wikipedia dump 2. **Extract clean text** from the articles (remove wiki markup, templates, references, infoboxes) 3. **Split the text into sentences** (sentence segmentation) 4. **Create pairs** (text_without_diacritics, text_with_diacritics): - Original text = text with diacritics - Input = the same text with the diacritics systematically removed 5. **Prepare a train/val split** (we recommend 90/10) ### Minimum requirements - At least **10 000 sentence pairs** - Sentences should have a reasonable length (we recommend 50-2000 characters) - The data must be **reproducible** (be sure to document your procedure) ### Removing diacritics To create the input data, remove all diacritics according to this map: ``` á → a Á → A ä → a Ä → A č → c Č → C ď → d Ď → D é → e É → E í → i Í → I ĺ → l Ĺ → L ľ → l Ľ → L ň → n Ň → N ó → o Ó → O ô → o Ô → O ŕ → r Ŕ → R š → s Š → S ť → t Ť → T ú → u Ú → U ý → y Ý → Y ž → z Ž → Z ``` ### What to submit 1. **Code** for processing the data (`.py` or `.ipynb`) 2. **Statistics** in JSON format: ```json { "total_sentence_pairs": 125000, "train_pairs": 112500, "val_pairs": 12500, "avg_sentence_length_chars": 87.3, "total_characters": 10937500, "diacritic_characters_count": 892341, "unique_words": 234567 } ``` 3. **Short documentation** (a README or comments in the code) explaining: - which steps you used to clean the data - how you dealt with problematic cases (special characters, foreign words, etc.) ### Scoring of Part 1 | Criterion | Points | |---|---| | Working extraction pipeline | 10 | | Meeting the minimum of 10k sentence pairs | 10 | | Data quality and documentation | 5 | | Reproducibility of the code | 5 | ### Tips - The `mwparserfromhell` library or the `wikiextractor` tool can help with parsing wiki markup - Watch out for: - Foreign-language quotations and names - Mathematical formulae - Tables and lists - Incomplete sentences ### Data licence The data from the Slovak Wikipedia are available under the **CC-BY-SA 3.0** licence. When they are used, the source must be credited. --- ## Part 2: Zero-shot approach (20 points) ### Task In this part you are to restore diacritics **without any fine-tuning** of the model. You will use the pretrained **SlovakBERT** and its masked language modelling (MLM) capability. ### Constraints - **You must use** the model `gerulata/slovakbert` from HuggingFace - **You must not fine-tune the model** – no changes to the weights, no further training - You may use only **inference** operations - You may implement any **heuristics and post-processing** ### Possible approaches (inspiration) SlovakBERT is a **masked language model** – it can predict masked tokens in context. You can make use of this, for example, as follows: 1. **Iterative prediction:** Mask, one after another, the positions where a diacritic could be, and let the model predict the best token 2. **Diffusion approach:** Start with the text without diacritics and iteratively refine the predictions 3. **Candidate scoring:** For each word, generate candidates with different diacritics and choose the most probable one These are only suggestions – creative solutions are welcome! ### Beware of tokenisation SlovakBERT uses **subword tokenisation**. The word "krásny" [*"beautiful"*] may be split into several tokens. You must take this into account when working with diacritics. ```python from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("gerulata/slovakbert") model = AutoModelForMaskedLM.from_pretrained("gerulata/slovakbert") # Tokenisation example tokens = tokenizer.tokenize("krásny") # Possible output: ['krá', 'sny'] ``` ### What to submit 1. **Code** of the solution (`.py` or `.ipynb`) 2. **Predictions** on the test set in the format: ``` \t 1 Slovensko je krajina v strednej Európe. 2 Hlavné mesto je Bratislava. ... ``` [*predikovaný_text* = predicted text] ### Scoring What is scored is the **accuracy on characters that may carry diacritics** (see the Evaluation section). The results will be shown on a **separate leaderboard**. ## Part 3: Fine-tuned model (50 points) **Task:** Now you can use the full power of machine learning. Your task is to **train a model** to restore diacritics using the data you prepared in Part 1. ### Constraints - **You must use** the model `gerulata/slovakbert` as the base - **You may use only the data** from Part 1 (no external corpora other than skwiki) - Computing resources: **Google Colab free tier** (T4 GPU, ~12GB RAM, ~12h session) ### Possible formulations of the problem 1. **Token classification:** For each token, predict whether it has a diacritic and which one 2. **Sequence-to-sequence:** The input is the text without diacritics, the output is the text with diacritics (requires an encoder-decoder) 3. **Character-level:** Work at the level of characters, not tokens ### Example: Token classification ```python from transformers import AutoModelForTokenClassification # Define labels for each type of diacritic # For example: O (no change), A_ACUTE (a→á), A_UMLAUT (a→ä), ... model = AutoModelForTokenClassification.from_pretrained( "gerulata/slovakbert", num_labels=NUM_LABELS ) ``` ### Recommendations - Start with a simple baseline and improve it step by step - Monitor the performance on the validation set - Experiment with: - Learning rate (we recommend 1e-5 to 5e-5) - Batch size (according to the available memory) - Number of epochs (typically 3-10) - Different formulations of the problem ### What to submit 1. **Training code** (`.py` or `.ipynb`) 2. **Predictions** on the test set (the same format as in Part 2) 3. **Short report** (optional, but recommended): - Which approach you chose and why - Which experiments you tried - Results on the validation set ### Scoring What is scored is the **accuracy on characters that may carry diacritics** (see the Evaluation section). The results will be shown on a **separate leaderboard** (separately from Part 2). ## Evaluation ### Metric The main metric is the **accuracy on positions that may carry diacritics**: $$ \text{accuracy} = \frac{\text{correctly predicted characters}}{\text{total number of positions that may carry diacritics}} $$ (In the original: `accuracy = správne_predikované_znaky / celkový_počet_pozícií_s_možnou_diakritikou`.) ### Which positions are counted? Only **characters that can carry diacritics in Slovak** are counted: ``` a, A (may be á, ä) c, C (may be č) d, D (may be ď) e, E (may be é) i, I (may be í) l, L (may be ĺ, ľ) n, N (may be ň) o, O (may be ó, ô) r, R (may be ŕ) s, S (may be š) t, T (may be ť) u, U (may be ú) y, Y (may be ý) z, Z (may be ž) ``` ### Worked example **Reference:** `Môj pes má rád kosti.` **Prediction:** `Môj pes ma rád kosti.` [*"My dog likes bones."*] Positions that may carry diacritics (lowercase for simplicity): - `o` at position 2 → ref: `ô`, pred: `ô` ✓ - `a` at position 8 → ref: `a`, pred: `a` ✓ - `a` at position 10 → ref: `á`, pred: `a` ✗ - `a` at position 13 → ref: `á`, pred: `á` ✓ - `o` at position 16 → ref: `o`, pred: `o` ✓ - `i` at position 19 → ref: `i`, pred: `i` ✓ Accuracy = 5/6 = 83.3% ### Submission format A file with the predictions in TSV (tab-separated) format: ``` id text 1 Môj pes má rád kosti. 2 Bratislava je hlavné mesto Slovenska. ``` ## Technical requirements ### Environment - **Python 3.10+** - **Google Colab** (free tier) as the reference environment ### Allowed libraries ``` torch transformers datasets numpy pandas scikit-learn tqdm mwparserfromhell (for Part 1) ``` Other standard Python libraries are allowed. If you need anything special, ask the organisers. ### Model You must use `gerulata/slovakbert`: ```python from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("gerulata/slovakbert") model = AutoModel.from_pretrained("gerulata/slovakbert") ``` Documentation: ## Submission ### What to submit | Part | Files | |---|---| | Part 1 | `data_preparation.py` (or `.ipynb`), `stats.json`, documentation | | Part 2 | `zeroshot_solution.py`, `predictions_zeroshot.tsv` | | Part 3 | `finetuning_solution.py`, `predictions_finetuned.tsv`, (optionally) a report | ### Where to submit **Edupage**, **under the corresponding task number.**\ **Create a single .zip file (no larger than 70MB) from all the files you are submitting, and submit that file.** ## Useful links - **SlovakBERT:** - **Slovak Wikipedia dumps:** - **HuggingFace Transformers documentation:** - **Wikiextractor:** ## Frequently asked questions (FAQ) **Q: May I use a model other than SlovakBERT?** A: No, you must use `gerulata/slovakbert`. **Q: In Part 3, may I use data from sources other than Wikipedia?** A: No, you may use only the data prepared in Part 1 from the Slovak Wikipedia. **Q: What if my solution needs more time/memory than Colab free offers?** A: Optimise your solution. The competition is designed to be solvable on the free tier. **Q: How is the score determined when results are tied?** A: When scores are equal, the submission time decides. ## Bonus ideas (not scored) For those who want to go further: - **Joint correction of diacritics and typos** – what if the input also contains typos? - **Multilingual version** – an extension to Czech or other languages with diacritics - **Error analysis** – which types of words/contexts are the hardest for your model? - **Comparison with ByT5** – how would a character-level approach differ? Send them directly to the organisers – they will surely be glad to discuss your solutions further! *Good luck!*