Diacritics: Automatic Restoration of Diacritics in Slovak Text

Task 3: Diacritics – Automatic Restoration of Diacritics in Slovak Text

English translation by SOTA – AI Community of the Slovak original, uloha3-diakritika.pdf ("Úloha 3: Diakritika: Automatická obnova diakritiky v slovenskom texte"), from the Olympiáda v umelej inteligencii (Slovak Olympiad in Artificial Intelligence), 2025/26, home round. The original task belongs to its authors. Slovak example sentences are kept as they are, because they are the data of the task; English glosses in square brackets were added by the translator. Organisers who would like this translation removed can email [email protected].

Story

Morning. A bus. A mobile phone in your hand. A friend texts you:

"Ahoj, mozes mi pozicat poznamky z matiky? Zabudol som si zosit doma a ucitelka sa bude pytat. Dakujem!"

["Hi, can you lend me your maths notes? I left my exercise book at home and the teacher is going to ask. Thanks!"]

You understand him, of course. But what if a computer received a message like this? Or what if you wanted to publish this text in the school magazine?

Writing without diacritics is common in Slovakia – especially when typing quickly on a mobile phone, in chats, or when we use a keyboard without Slovak characters. The problem arises when we need to process such a text automatically, translate it, or simply make it more readable.

The sentence "mama ma rada" can mean:

• "mama má rada" (mum likes doing something) [má = "has"; má rada = "likes"]
• "mama ma rada" (mum likes doing something with me, say "dressing" me) [ma = "me"]
• "mamá mä rada" (well, this one is not correct at all…)

Your task is therefore relatively straightforward: to create an intelligent system that can automatically restore diacritics in Slovak text. In doing so, you will use the power of modern language models – specifically SlovakBERT, the first (not all that) large language model trained specifically for Slovak.

Task overview

The task consists of three parts that build on one another:

Part	Name	Points	Description
1	Data preparation	30	Extracting and processing training data from the Slovak Wikipedia
2	Zero-shot approach	20	Restoring diacritics without fine-tuning, using only the pretrained SlovakBERT
3	Fine-tuned model	50	Training your own model on the data from Part 1

Common input: Text in Slovak with the diacritics removed.

Common output: The same text with the diacritics correctly restored.

Example

Input:

Slovensko je krajina v strednej Europe. Hlavne mesto je Bratislava.

Expected output:

Slovensko je krajina v strednej Európe. Hlavné mesto je Bratislava.

["Slovakia is a country in Central Europe. The capital is Bratislava."]

---

Part 1: Data preparation (30 points)

Task

Your first task is to prepare training data for a diacritics-restoration model. As the source you will use the Slovak Wikipedia dumps, which are freely available at:

https://dumps.wikimedia.org/skwiki/20251120/

Specifically, you need the file skwiki-latest-pages-articles.xml.bz2.

Requirements

1. Download and process the Slovak Wikipedia dump
2. Extract clean text from the articles (remove wiki markup, templates, references, infoboxes)
3. Split the text into sentences (sentence segmentation)
4. Create pairs (text_without_diacritics, text_with_diacritics):
   • Original text = text with diacritics
   • Input = the same text with the diacritics systematically removed
5. Prepare a train/val split (we recommend 90/10)

Minimum requirements

• At least 10 000 sentence pairs
• Sentences should have a reasonable length (we recommend 50-2000 characters)
• The data must be reproducible (be sure to document your procedure)

Removing diacritics

To create the input data, remove all diacritics according to this map:

á → a    Á → A
ä → a    Ä → A
č → c    Č → C
ď → d    Ď → D
é → e    É → E
í → i    Í → I
ĺ → l    Ĺ → L
ľ → l    Ľ → L
ň → n    Ň → N
ó → o    Ó → O
ô → o    Ô → O
ŕ → r    Ŕ → R
š → s    Š → S
ť → t    Ť → T
ú → u    Ú → U
ý → y    Ý → Y
ž → z    Ž → Z

What to submit

1. Code for processing the data (.py or .ipynb)
2. Statistics in JSON format:

{
  "total_sentence_pairs": 125000,
  "train_pairs": 112500,
  "val_pairs": 12500,
  "avg_sentence_length_chars": 87.3,
  "total_characters": 10937500,
  "diacritic_characters_count": 892341,
  "unique_words": 234567
}

3. Short documentation (a README or comments in the code) explaining:
   • which steps you used to clean the data
   • how you dealt with problematic cases (special characters, foreign words, etc.)

Scoring of Part 1

Criterion	Points
Working extraction pipeline	10
Meeting the minimum of 10k sentence pairs	10
Data quality and documentation	5
Reproducibility of the code	5

Tips

• The mwparserfromhell library or the wikiextractor tool can help with parsing wiki markup
• Watch out for:
  • Foreign-language quotations and names
  • Mathematical formulae
  • Tables and lists
  • Incomplete sentences

Data licence

The data from the Slovak Wikipedia are available under the CC-BY-SA 3.0 licence. When they are used, the source must be credited.

---

Part 2: Zero-shot approach (20 points)

Task

In this part you are to restore diacritics without any fine-tuning of the model. You will use the pretrained SlovakBERT and its masked language modelling (MLM) capability.

Constraints

• You must use the model gerulata/slovakbert from HuggingFace
• You must not fine-tune the model – no changes to the weights, no further training
• You may use only inference operations
• You may implement any heuristics and post-processing

Possible approaches (inspiration)

SlovakBERT is a masked language model – it can predict masked tokens in context. You can make use of this, for example, as follows:

1. Iterative prediction: Mask, one after another, the positions where a diacritic could be, and let the model predict the best token
2. Diffusion approach: Start with the text without diacritics and iteratively refine the predictions
3. Candidate scoring: For each word, generate candidates with different diacritics and choose the most probable one

These are only suggestions – creative solutions are welcome!

Beware of tokenisation

SlovakBERT uses subword tokenisation. The word "krásny" ["beautiful"] may be split into several tokens. You must take this into account when working with diacritics.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("gerulata/slovakbert")
model = AutoModelForMaskedLM.from_pretrained("gerulata/slovakbert")

# Tokenisation example
tokens = tokenizer.tokenize("krásny")
# Possible output: ['krá', 'sny']

What to submit

1. Code of the solution (.py or .ipynb)
2. Predictions on the test set in the format:

<id>\t<predikovaný_text>
1 Slovensko je krajina v strednej Európe.
2 Hlavné mesto je Bratislava.
...

[predikovaný_text = predicted text]

Scoring

What is scored is the accuracy on characters that may carry diacritics (see the Evaluation section).

The results will be shown on a separate leaderboard.

Part 3: Fine-tuned model (50 points)

Task: Now you can use the full power of machine learning. Your task is to train a model to restore diacritics using the data you prepared in Part 1.

Constraints

• You must use the model gerulata/slovakbert as the base
• You may use only the data from Part 1 (no external corpora other than skwiki)
• Computing resources: Google Colab free tier (T4 GPU, ~12GB RAM, ~12h session)

Possible formulations of the problem

1. Token classification: For each token, predict whether it has a diacritic and which one
2. Sequence-to-sequence: The input is the text without diacritics, the output is the text with diacritics (requires an encoder-decoder)
3. Character-level: Work at the level of characters, not tokens

Example: Token classification

from transformers import AutoModelForTokenClassification

# Define labels for each type of diacritic
# For example: O (no change), A_ACUTE (a→á), A_UMLAUT (a→ä), ...

model = AutoModelForTokenClassification.from_pretrained(
    "gerulata/slovakbert",
    num_labels=NUM_LABELS
)

Recommendations

• Start with a simple baseline and improve it step by step
• Monitor the performance on the validation set
• Experiment with:
  • Learning rate (we recommend 1e-5 to 5e-5)
  • Batch size (according to the available memory)
  • Number of epochs (typically 3-10)
  • Different formulations of the problem

What to submit

1. Training code (.py or .ipynb)
2. Predictions on the test set (the same format as in Part 2)
3. Short report (optional, but recommended):
   • Which approach you chose and why
   • Which experiments you tried
   • Results on the validation set

Scoring

What is scored is the accuracy on characters that may carry diacritics (see the Evaluation section).

The results will be shown on a separate leaderboard (separately from Part 2).

Evaluation

Metric

The main metric is the accuracy on positions that may carry diacritics:

$$\text{accuracy} = \frac{\text{correctly predicted characters}}{\text{total number of positions that may carry diacritics}}$$

(In the original: accuracy = správne_predikované_znaky / celkový_počet_pozícií_s_možnou_diakritikou.)

Which positions are counted?

Only characters that can carry diacritics in Slovak are counted:

a, A  (may be á, ä)
c, C  (may be č)
d, D  (may be ď)
e, E  (may be é)
i, I  (may be í)
l, L  (may be ĺ, ľ)
n, N  (may be ň)
o, O  (may be ó, ô)
r, R  (may be ŕ)
s, S  (may be š)
t, T  (may be ť)
u, U  (may be ú)
y, Y  (may be ý)
z, Z  (may be ž)

Worked example

Reference: Môj pes má rád kosti. Prediction: Môj pes ma rád kosti. ["My dog likes bones."]

Positions that may carry diacritics (lowercase for simplicity):

• o at position 2 → ref: ô, pred: ô ✓
• a at position 8 → ref: a, pred: a ✓
• a at position 10 → ref: á, pred: a ✗
• a at position 13 → ref: á, pred: á ✓
• o at position 16 → ref: o, pred: o ✓
• i at position 19 → ref: i, pred: i ✓

Accuracy = 5/6 = 83.3%

Submission format

A file with the predictions in TSV (tab-separated) format:

id text
1 Môj pes má rád kosti.
2 Bratislava je hlavné mesto Slovenska.

Technical requirements

Environment

• Python 3.10+
• Google Colab (free tier) as the reference environment

Allowed libraries

torch
transformers
datasets
numpy
pandas
scikit-learn
tqdm
mwparserfromhell (for Part 1)

Other standard Python libraries are allowed. If you need anything special, ask the organisers.

Model

You must use gerulata/slovakbert:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("gerulata/slovakbert")
model = AutoModel.from_pretrained("gerulata/slovakbert")

Documentation: https://huggingface.co/gerulata/slovakbert

Submission

What to submit

Part	Files
Part 1	data_preparation.py (or .ipynb), stats.json, documentation
Part 2	zeroshot_solution.py, predictions_zeroshot.tsv
Part 3	finetuning_solution.py, predictions_finetuned.tsv, (optionally) a report

Where to submit

Edupage, under the corresponding task number.
Create a single .zip file (no larger than 70MB) from all the files you are submitting, and submit that file.

Useful links

• SlovakBERT: https://huggingface.co/gerulata/slovakbert
• Slovak Wikipedia dumps: https://dumps.wikimedia.org/skwiki/latest/
• HuggingFace Transformers documentation: https://huggingface.co/docs/transformers
• Wikiextractor: https://github.com/attardi/wikiextractor

Frequently asked questions (FAQ)

Q: May I use a model other than SlovakBERT? A: No, you must use gerulata/slovakbert.

Q: In Part 3, may I use data from sources other than Wikipedia? A: No, you may use only the data prepared in Part 1 from the Slovak Wikipedia.

Q: What if my solution needs more time/memory than Colab free offers? A: Optimise your solution. The competition is designed to be solvable on the free tier.

Q: How is the score determined when results are tied? A: When scores are equal, the submission time decides.

Bonus ideas (not scored)

For those who want to go further:

• Joint correction of diacritics and typos – what if the input also contains typos?
• Multilingual version – an extension to Czech or other languages with diacritics
• Error analysis – which types of words/contexts are the hardest for your model?
• Comparison with ByT5 – how would a character-level approach differ?

Send them directly to the organisers – they will surely be glad to discuss your solutions further!

Good luck!