Corrupt Codex

Corrupt Codex

English translation by SOTA – AI Community of the Hungarian original. Licensed CC BY-NC-SA 4.0, like the original. Organisers who would like this translation removed can email [email protected].

Magyar MI Diákolimpia (Hungarian AI Olympiad)
Task description · Summer National Selection (Nyári Országos Válogató)
May 2026

[Figure: see the original one-pager.]

1. Corrupt Codex

A financial institution operates a fraud detection system. The heart of the system is a neural network that estimates from 12 anonymised features whether a given transaction is suspected of fraud or legitimate. The model was trained earlier and has been running in production ever since. Under the strict internal data retention and security policies, the transaction data used for training had to be deleted once the retention period expired, so only the weights of the trained model have been kept.

In the meantime, it has turned out that the labels of the data used for training are wrong in places, for unknown reasons: samples that belonged to fraudulent transactions but were recorded as legitimate made their way into the training set.

While operating the system, 5,000 samples were collected whose transactions the model classified as legitimate, but there is a suspicion that hidden frauds are among them as well. You have been asked, as an external expert, to find out – starting from an examination of the model – which samples come from fraudulent transactions. Since the original training data is not available, you have to carry out the investigation based solely on the behaviour of the trained network.

2. What you receive

The set to be examined consists of 5,000 samples. Each sample is described by 12 anonymised features, and each one corresponds to a transaction that the model classified as legitimate. Some of the samples in the set presumably belong to fraudulent transactions. Your task is to estimate which samples come from fraudulent transactions.

For 50 samples of the set to be examined, an expert manually checked the corresponding transaction and determined whether it is fraudulent or genuinely legitimate. You receive this 50-item calibration set together with the actual status (1 = belongs to a fraudulent transaction / wrong label, 0 = belongs to a legitimate transaction / correct label). The remaining 4,950 samples have no verified status. Both sets may be used freely.

The trained model is provided in a .pt file, in the form of PyTorch weights. The architecture of the model can be found in the attached notebook. The network is feed-forward and takes the 12 features directly as input.

3. What you submit

A single CSV file with the header id,score, containing exactly 5,000 rows to be evaluated. id is the serial number of the samples of the set to be examined (from 0 to 4,999), and score is a real number expressing the degree of suspicion of a wrong label: a higher value = a stronger suspicion that the sample actually belongs to a fraudulent transaction. The scale of the scores does not matter, since evaluation is based on AUC ROC, which is threshold-independent. No model needs to be submitted; the evaluator computes the score from the CSV.

4. Scoring

Let $\mathcal{T}$ denote the unlabelled test set of $N$ items, $s_t \in \mathbb{R}$ the submitted score for the $t$-th sample (a higher value = a stronger suspicion of a wrong label), and $y^*_t \in \{0, 1\}$ the true status (1 = wrong label, 0 = correct label). The performance of the model is evaluated on the basis of a single metric:

• AUC ROC: the area under the Receiver Operating Characteristic curve computed on the pairs $(s_t, y^*_t)$. Its value lies in $[0, 1]$, where 0.5 corresponds to random guessing and 1.0 to perfect separation.

Maximum score: 100 points.

Scoring curve. The score is not computed directly from the AUC value but from a piecewise linear curve that we calibrated to the difficulty of the task.

Between the anchor points, linear interpolation is used. The curve is predominantly convex: the better the AUC, the more a further improvement is worth.

Public and private evaluation. The submitted scores are evaluated on two different sets. The public evaluation is carried out on a randomly selected subset of the test set, and its result is visible on the leaderboard during the contest. The private evaluation is carried out on the full test set, and it decides the final ranking.

5. Technical information

To solve the task, the attached .ipynb notebook and the files calibration.csv, test.csv and net_weights.pt are available. The job of the solution is to produce the CSV file containing the 5,000 scores.

Scoring is carried out automatically, on the server side, by the DOCK platform; you do not need to submit a model. You have at most 15 upload attempts, and the score of your best submission counts towards the final result.

6. Useful resources

• PyTorch documentation
• NumPy documentation
• pandas documentation (writing CSV)
• AUC ROC (scikit-learn)
• ROC curve (Wikipedia)