Pink Lemurs: Classification Metrics

Pink Lemurs: Classification Metrics

English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email [email protected].

Task 3 of the school stage (Moscow) of the All-Russian School Olympiad (VsOSh) 2025/26 in artificial intelligence, grades 9–11. Original: tasks-ai-9-11-sch-msk-25-26.pdf.

A nature reserve plans to introduce automatic monitoring of rare pink lemurs. To test the system, a group of 100 animals was selected, in which 10% are pink lemurs (class 1) and the remaining 90% are ordinary lemurs (class 0). We will call this group the test set.

For each individual in the test set, AI models made a prediction of which class (0 or 1) that lemur belongs to. To assess the quality of recognition, we introduce the following standard notation:

• $TP$. Predicted class (1), actual class (1).
• $FP$. Predicted class (1), actual class (0).
• $TN$. Predicted class (0), actual class (0).
• $FN$. Predicted class (0), actual class (1).

In total, the test set contains $P = TP + FN$ objects (lemurs) of class 1 and $N = TN + FP$ objects of class 0.

Four AI models (A, B, C and D) recognised the lemurs from the test set. Model A assigned all lemurs to class 0, and model B assigned all lemurs to class 1. Data on the performance of models C and D are given in the table below.

For models A, B, C, D, we compute the following five metrics $M_1, M_2, M_3, M_4, M_5$ (if a division by zero occurs for some model, the corresponding metric is not computed for it).

$$\begin{aligned} M_1 &= \frac{TP + TN}{P + N} && \text{(overall accuracy)},\\ M_2 &= \frac{TP}{TP + FN} && \text{(correct on class 1)},\\ M_3 &= \frac{TN}{TN + FP} && \text{(correct on class 0)},\\ M_4 &= \tfrac{1}{2}\,(M_2 + M_3) && \text{(balanced accuracy)},\\ M_5 &= \frac{TP}{TP + FP} && \text{(precision on predictions of 1)}. \end{aligned}$$

For each of the metrics $M_1, M_2, \ldots, M_5$, determine for which of the models its maximum value is attained.

Translator's note: the statement paper gives only the total for the olympiad (112 points); the official solutions give this task a maximum of 12 points.