Blind Curator

Blind Curator

English translation by SOTA – AI Community of the Hungarian original. Licensed CC BY-NC-SA 4.0, like the original. Organisers who would like this translation removed can email [email protected].

Magyar MI Diákolimpia (Hungarian AI Olympiad)
Task description · Summer National Selection (Nyári Országos Válogató)
June 2026

[Figure: see the original one-pager.]

1. Blind Curator

There are 15,000 unknown images in front of you, and you see them only through faint thumbnails and embedding vectors. The images belong to a world of fifty classes (animals, plants, vehicles and everyday objects selected from CIFAR-100), but not a single image has a label. A machine will learn these classes on the basis of your choice, and the only source of its learning is what a human expert says about the images you choose. The expert is expensive and your budget is strict: you may hand over exactly 300 images to the expert for labelling. The machine is built from the expert's answers, and its quality is measured on a test set of 5,000 items that you never see. Your score depends entirely on which 300 images you choose.

The model built on your selection is deliberately simple. For the 300 selected images, the evaluator reads out the labels, collects the corresponding embedding vectors class by class, averages them and then L2-normalises the result. Let $S$ denote the set of your 300 chosen indices, $S_c = \{i \in S : y_i = c\}$ your selections belonging to class $c$, and $\mathbf{e}_i \in \mathbb{R}^{384}$ the embedding vector of the $i$-th pool item. The 50 vectors obtained in this way are the prototype centroids of the classes:

$$\boldsymbol{\mu}_c = \frac{\sum_{i \in S_c} \mathbf{e}_i}{\left\lVert \sum_{i \in S_c} \mathbf{e}_i \right\rVert_2}, \qquad \hat{y}_t = \operatorname*{arg\,max}_{c \in \{0,\dots,49\}} \mathbf{e}_t^\top \boldsymbol{\mu}_c,$$

where $\mathbf{e}_t$ is the embedding vector of the $t$-th test image. The model is not trainable and has no hyperparameters: the centroids are worth exactly as much as your selection puts into them.

The pool is not ideal. 8% of the observed labels (those read out by the evaluator) are noisy: the image is correct, but the label assigned to it is wrong, and the evaluator does not correct these labels, so they are included in the averaging of the centroids. 4% of the pool are distractors: images whose true class is not among the visible fifty but which, in the CIFAR-100 hierarchy, are a relative within the superclass of one of the visible classes – that is, they are visually close to them (e.g. if cat and dog are among the visible classes, a distractor could be a hamster, which is also a small mammal). For this reason, the distractors cannot be filtered out simply by how far they lie from the seed centroids in the embedding space. The class distribution is asymmetric: thirty frequent and twenty rare classes.

2. What you receive

For each of the 15,000 images in the pool you receive two representations: a 384-dimensional DINOv2 embedding vector and a 32×32 thumbnail PNG. The evaluator uses only the embedding; the thumbnail is meant for you, for visual inspection. You do not receive the labels of the pool in either form.

In addition to the pool, you receive an already labelled set of 200 items (seed), with four examples per class. For these items you receive the embedding vector together with the label, and they may be used freely for analysis, clustering or training your own small model. You have no access to the test set: you see neither the 5,000 images nor their labels.

3. What you submit

A single CSV file with the header id,target and exactly 300 rows. id is a serial number increasing from 0 to 299, and target is the chosen image index, between 0 and 14,999. The 300 indices must be unique. No model needs to be submitted; the evaluator computes the score from the CSV.

4. Scoring

Let $\mathcal{T}$ denote the test set of 5,000 items, $\hat{y}_t$ the model's prediction for the $t$-th test image, and $y^*_t$ the true label. Four subscores are computed, with a maximum of 10 points per subtask:

• al_s1 (overall accuracy): $\frac{100}{|\mathcal{T}|} \sum_t \mathbb{1}[\hat{y}_t = y^*_t]$.
• al_s2 (easily confused rare classes): accuracy on the test images of the five rarest pool classes that have at least one sibling among the visible classes (another visible class in the same CIFAR-100 superclass family). The classes in this group are therefore both rare and visually easy to confuse. The contestant's task is to make sure that the selection explicitly covers these classes as well.
• al_s3 (selection purity): $\frac{100 \cdot |S \cap \mathcal{C}|}{|S|}$, where $\mathcal{C}$ is the set of non-distractor (in-distribution) pool indices. al_s3 therefore measures only the avoidance of OOD distractors; the 8% label noise is not penalised, because it cannot be identified from the embedding (it is applied to the labels uniformly, leaving the image unchanged).
• al_s4 (borderline-case slice): accuracy on the 1,000 hardest test images. The difficulty of a test image is defined by its class margin measured against the seed centroids: $\mathrm{margin}(t) = \mathbf{e}_t^\top \boldsymbol{\mu}^{\mathrm{seed}}_{y^*_t} - \max_{c \neq y^*_t} \mathbf{e}_t^\top \boldsymbol{\mu}^{\mathrm{seed}}_c$. A negative margin indicates a test image that is closer to the centroid of another class than to that of its true class. From each class, the 20 test images with the smallest margin are taken; 50 classes × 20 = 1,000 borderline samples.

Maximum score: 40 points (4 × 10).

Scoring curve. The four subscores are not computed from the raw measure (e.g. 53% accuracy = 5.3/10 points) but from a threshold-based curve: each subscore has a lower threshold, below which 0 points are awarded, and an upper threshold, on reaching which the full 10 points are awarded; between the two, the scaling is linear. The thresholds are calibrated empirically: the lower threshold is at the raw performance of random selection, and the upper one at the performance of the best known fair strategy. A random submission therefore scores close to 0/40, while the full 40/40 can be reached by optimising each of the four subtasks separately, in a targeted way (see below).

Public and private test set. The test set is divided into a public part and a closed (private) part. During the contest you only see the score measured on the public part on the leaderboard; the final ranking is decided by the result achieved on the closed part.

5. Technical information

To solve the task, the attached .ipynb notebook and the files pool_emb.npy, pool_thumbs.zip, seed_emb.npy and seed_labels.npy are available. The solution only has to produce the CSV containing the 300 indices. No GPU is needed to generate the selection; the task can be done both locally and in Google Colab.

Constraint: only the provided DINOv2 embedding vectors and the thumbnails may be used to analyse the pool. Applying external pretrained image models (e.g. CLIP, BLIP or other image classifiers) to the pool images for the purpose of class or label estimation is not allowed.

Scoring is carried out automatically, on the server side, by the DOCK platform: from the uploaded CSV it builds the prototype model and computes the four subscores; you do not need to submit a model. You may upload the CSV separately for each of the four subtasks, so you may even use a different 300-item selection for each subtask if you want to optimise for that particular criterion in a targeted way. You have at most 6 upload attempts per subtask, and the score of your best submission counts towards the final result.

6. Useful resources

• DINOv2 (ViT-S/14) model card
• NumPy documentation
• pandas documentation (writing CSV)
• Cosine similarity (scikit-learn)
• CIFAR-100 dataset