Checklist OAI 2025 Stage I · Task 4
Noise in Data Labels
Polish title: Szum w Etykietach Danych
Write the sample-selection function used while co-training two fixed small networks on imbalanced binary image data with partly wrong training labels.
The task
Real datasets often contain wrong labels, caused by annotator subjectivity or fatigue, poor data quality, automatic labelling or samples near class boundaries, and such noise hinders training. The task is to train two neural networks to classify images into two classes correctly even though part of the training labels are noisy; the training set is also imbalanced, whereas the validation and test sets contain only correct labels.
The architecture (SmallMobileNet) and the training loop are fixed. The contestant implements your_selected_indices(targets, losses), which receives the batch labels and the per-sample losses of both models and returns a two-element list of index tensors: the samples used to train the first and the second model, respectively. Participants are invited to consider why two models are used.
The training loop trains both models for 6 epochs (batch size 128, learning rate 1e-2, weight decay 1e-3) and calls the selection function at each step.
Abridged and translated by SOTA from the official Polish materials. The official statement has the exact rules, and it wins wherever this summary differs.
In English
This task was published in Polish. SOTA translated its 2 files into English on 16 September 2026. Only the words changed in the notebooks: markdown, code comments, messages and printed output. The code, file names and paths are the original's, so a translated notebook runs with the original data.
- Task notebook Polish original of Task notebook
- Official solution Polish original of Official solution
Read the task notebook in English
Noise in Data Labels
Image generated using a generative model from OpenArt.ai.
Introduction
Data are the key to machine learning. Everything starts with them. In practice, however, data are often not perfect and contain some noise that reduces their quality. One type of such noise is noise in data labels, which means that the labels of some observations are incorrect.
The causes of label noise can vary. They often stem from the subjectivity of assessment – different experts may have different opinions, e.g. when assessing emotions in photographs or the quality of an essay. Another source of errors may be annotator fatigue, which affects their concentration and accuracy.
Ambiguities may also result from poor data quality, which makes unambiguous classification difficult (e.g. a blurred photograph of a dog that resembles a wolf). Sometimes labels are generated automatically by artificial intelligence models, which can also make mistakes.
It is worth mentioning samples that lie on the boundary between classes. Such cases, e.g. in medical data where the symptoms of different diseases are similar, also make it difficult to assign an unambiguous label.
Noisy data make it difficult to train a good-quality model, because the model may focus more on the incorrect information than on the general rules contained in the data.
Task
Your task is to train two neural networks to perform binary image classification correctly despite the partially noisy labels in the training data. The training set is imbalanced (take this into account in your solution). Both the validation set and the test set (which will be used to evaluate your final solution) contain only correct labels (without noise).
The architecture of the models is fixed and you may not change it.
Think about why we use two models rather than one (this is something of a puzzle) - it will help you to understand the task and solve it.
Your role in this task is to implement the function your_selected_indices(targets, losses), which will select the indices of the training data to be used for training the models. The function takes as input a tensor with the data labels (targets) and a tensor with the loss function values from both models (losses). The result of this function should be a two-element list whose elements are tensors containing the indices selected for training the models. One model receives one set of indices, and the other model receives the other.
Further down in the notebook you will find a cell that contains a place for your function. The cell that you should modify is clearly marked. To better understand how it works and what it is for, it is worth looking at the context and at the place in the training loop where this function will be called.
Scoring Criterion
The final score for the task will be based on the mean value of the balanced accuracy (BAC) of the two models, i.e. , where is the balanced accuracy of model , ().
You can score between 0 and 100 points for this task.
Your final score for the solution will be calculated according to the function below (the higher the value, the better), with additional rounding to integer values:
Note: Observe that to obtain the maximum number of points it is not necessary to achieve the maximum balanced accuracy value of 1. If is at least 0.8, you will receive the maximum number of points.
This criterion and all the functions mentioned above are implemented by us below.
Constraints
- Your solution will be tested on the Contest Platform without internet access and in an environment with a GPU.
- The evaluation of your final solution on the Contest Platform must not take longer than 5 minutes with a GPU.
- You may not change the architecture of the models - it must be the
SmallMobileNetdefined by us.
Submission files
This notebook, completed with your solution (see the your_selected_indices function).
Evaluation
Remember that during checking, the FINAL_EVALUATION_MODE flag will be set to True.
You can score between 0 and 100 points for this task. The number of points you receive will be calculated on the (secret) test set on the Contest Platform on the basis of the formula given above, rounded to an integer. If your solution does not meet the above criteria or does not run correctly, you will receive 0 points for the task.
Starter Code
In this section, we initialise the environment by importing the required libraries and functions. The prepared code will help you to work with the data efficiently and to build the actual solution.
# While your solution is being checked, the value of the FINAL_EVALUATION_MODE flag will be changed to True
FINAL_EVALUATION_MODE = False######################### DO NOT CHANGE THIS CELL ##########################
import os
from tqdm import tqdm
from typing import Optional, Tuple, List
import zipfile
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from PIL import Image
import torch
import torch.nn as nn
from torch.optim import AdamW
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
from torchvision.datasets.folder import VisionDataset
from sklearn.metrics import balanced_accuracy_scoreHelper Functions and Constants
######################### DO NOT CHANGE THIS CELL ##########################
SEED = 123
IMAGES_DIR = "data"
TASK_DATASET_LABELS_FILE = "dataset_labels.csv"
ROOT_DIR = os.getcwd()
TRAIN_DATASET_PATH = os.path.join(ROOT_DIR,'train')
VAL_DATASET_PATH = os.path.join(ROOT_DIR, 'val')
TRAIN_DATASET_URL = "1qmNNmDv-wUcAv5mvO6vYJV3mQ2SNIGnI"
VAL_DATASET_URL = "1YUJYD12NmKRSzFJGMrX-a61d6mnTaWbG"######################### DO NOT CHANGE THIS CELL ##########################
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
LEARNING_RATE = 1e-2
NUM_EPOCHS = 6
NUM_CLASSES = 2
BATCH_SIZE = 128
WEIGHT_DECAY = 1e-3
if not FINAL_EVALUATION_MODE:
print(f"Using {DEVICE} device")######################### DO NOT CHANGE THIS CELL ##########################
def seed_everything(seed: int) -> None:
"""
Sets the seed for reproducibility of results in Python, NumPy and PyTorch.
The function sets the seed of the random number generators in Python, NumPy and PyTorch,
and also configures PyTorch to work in deterministic mode.
Parameters:
seed (int): The seed value to set.
"""
os.environ["PYTHONHASHSEED"] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = FalseLoading the Data
The code below loads the data and prepares them appropriately.
######################### DO NOT CHANGE THIS CELL ##########################
def download_data(dataset_path, dataset_url):
"""Downloads the dataset from the Olympiad's Google Drive and saves it in a folder."""
import gdown
import shutil
# Create or reset the folder
output = dataset_path+".zip"
if os.path.exists(dataset_path):
shutil.rmtree(dataset_path)
if os.path.exists(output):
os.remove(output)
url = f'https://drive.google.com/uc?id={dataset_url}'
gdown.download(url, output, fuzzy=True)
print(f"Downloaded: {output}")
# Download the data only if you are not in FINAL_EVALUATION_MODE
if not FINAL_EVALUATION_MODE:
download_data(TRAIN_DATASET_PATH, TRAIN_DATASET_URL)
download_data(VAL_DATASET_PATH, VAL_DATASET_URL)######################### DO NOT CHANGE THIS CELL ##########################
# Dataset class
class TaskDataset(VisionDataset):
def __init__(
self,
root: str,
transform: Optional[callable] = None,
):
super().__init__(
root,
transform=transform,
)
self.root = root
if not self._check_integrity():
raise RuntimeError(
f"Dataset not found. Check whether the path {self.root} exists. It should contain the '{IMAGES_DIR}' folder and the '{TASK_DATASET_LABELS_FILE}' file"
)
self.labels_df = self._read_labels_from_file()
self.labels_header = 'label'
def _read_labels_from_file(self) -> pd.DataFrame:
df = pd.read_csv(os.path.join(self.root, TASK_DATASET_LABELS_FILE))
return df
def _check_integrity(self) -> bool:
return os.path.exists(os.path.join(self.root, IMAGES_DIR)) and os.path.exists(
os.path.join(self.root, TASK_DATASET_LABELS_FILE)
)
def __len__(self) -> int:
return len(self.labels_df)
def __getitem__(self, idx: int) -> Tuple[Image.Image, np.ndarray]:
img = self._load_image(idx)
label = self._load_label(idx)
if self.transform is not None:
img = self.transform(img)
return img, label
def _load_image(self, idx: int) -> Image.Image:
img_path = os.path.join(
self.root, IMAGES_DIR, self.labels_df.iloc[idx]['file_name']
)
img = Image.open(img_path)
return img
def _load_label(self, idx: int):
label = self.labels_df.iloc[idx][self.labels_header]
return np.array([int(label)])######################### DO NOT CHANGE THIS CELL ##########################
def unpack_data(unpack_path, dataset_name) -> None:
dataset_zip_path = os.path.join(ROOT_DIR, dataset_name+".zip")
dataset_local_dir = os.path.join(unpack_path, dataset_name)
if not os.path.exists(dataset_local_dir):
if not os.path.exists(dataset_zip_path):
raise FileNotFoundError(
f"File {dataset_zip_path} was not found in the current folder."
)
with zipfile.ZipFile(dataset_zip_path, "r") as zip_ref:
zip_ref.extractall(unpack_path)######################### DO NOT CHANGE THIS CELL ##########################
# Function that loads the training and validation data
def load_data() -> Tuple[DataLoader, DataLoader]:
"""
Function that loads the training and validation data using the TaskDataset class.
The function creates datasets for the training and validation data,
applies a basic transformation (conversion to a tensor), and then
wraps them in DataLoader objects.
Returns:
Tuple[DataLoader, DataLoader]: DataLoader objects for the training set and the validation set.
"""
base_transform = transforms.Compose([transforms.ToTensor()])
train_dataset = TaskDataset(root=TRAIN_DATASET_PATH, transform=base_transform)
val_dataset = TaskDataset(root=VAL_DATASET_PATH, transform=base_transform)
train_loader = DataLoader(
dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=False
)
val_loader = DataLoader(dataset=val_dataset, batch_size=BATCH_SIZE, shuffle=False)
return train_loader, val_loader######################### DO NOT CHANGE THIS CELL ##########################
# Let us unpack and load the data
if not FINAL_EVALUATION_MODE:
unpack_data(ROOT_DIR, "train")
unpack_data(ROOT_DIR, "val")
train_loader, val_loader = load_data()Model Architecture
######################### DO NOT CHANGE THIS CELL ##########################
class SmallMobileNet(nn.Module):
def __init__(self, num_classes=NUM_CLASSES):
super(SmallMobileNet, self).__init__()
# Main convolutional blocks
self.features = nn.Sequential(
nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1, bias=False),
nn.BatchNorm2d(32),
nn.ReLU6(inplace=True),
nn.Conv2d(
32, 32, kernel_size=3, stride=1, padding=1, groups=32, bias=False
),
nn.BatchNorm2d(32),
nn.ReLU6(inplace=True),
nn.Conv2d(32, 64, kernel_size=1, stride=1, bias=False),
nn.BatchNorm2d(64),
nn.ReLU6(inplace=True),
nn.Conv2d(
64, 64, kernel_size=3, stride=2, padding=1, groups=64, bias=False
),
nn.BatchNorm2d(64),
nn.ReLU6(inplace=True),
nn.Conv2d(64, 128, kernel_size=1, stride=1, bias=False),
nn.BatchNorm2d(128),
nn.ReLU6(inplace=True),
nn.Conv2d(
128, 128, kernel_size=3, stride=2, padding=1, groups=128, bias=False
),
nn.BatchNorm2d(128),
nn.ReLU6(inplace=True),
nn.Conv2d(128, 256, kernel_size=1, stride=1, bias=False),
nn.BatchNorm2d(256),
nn.ReLU6(inplace=True),
)
self.pool = nn.AdaptiveAvgPool2d(1)
self.classifier = nn.Sequential(
nn.Linear(256, 128),
nn.ReLU6(inplace=True),
nn.Dropout(0.5),
nn.Linear(128, num_classes),
)
def forward(self, x):
x = self.features(x)
x = self.pool(x)
x = torch.flatten(x, 1)
x = self.classifier(x)
return xCode with the Scoring Criterion
Code similar to the code below will be used to score the solution on the test set.
######################### DO NOT CHANGE THIS CELL ##########################
def predict_and_evaluate(model, val_loader, device, verbose=False):
model.eval()
all_preds, all_targets = [], []
with torch.no_grad():
for inputs, targets in val_loader:
inputs, targets = inputs.to(device), targets.to(device)
outputs = model(inputs)
preds = torch.argmax(outputs, dim=1)
all_preds.extend(preds.cpu().numpy())
all_targets.extend(targets.cpu().numpy())
balanced_accuracy = balanced_accuracy_score(all_targets, all_preds)
if verbose:
print(f"Balanced Accuracy: {balanced_accuracy}")
return balanced_accuracy######################### DO NOT CHANGE THIS CELL ##########################
def performance(bac_1: float, bac_2: float) -> None:
"""
Computes and prints the performance score based on two balanced accuracy values.
The final result is the mean of the two values, rescaled between fixed bounds,
which translates into the number of points scored.
Parameters:
bac_1 (float): The balanced accuracy value of the first model.
bac_2 (float): The balanced accuracy value of the second model.
"""
bac_mean = (bac_1 + bac_2) / 2
if bac_mean <= 0.5:
points = 0
elif 0.5 < bac_mean < 0.8:
points = (bac_mean - 0.5) / (0.8 - 0.5) * 100
points = int(round(points))
else:
points = 100
print(
f"Your solution has a mean balanced accuracy of {round(bac_mean, 5)} on the validation set, which gives {points}/100 points."
)
return pointsModel Training
######################### DO NOT CHANGE THIS CELL ##########################
# Function for training the model
def train(
model1,
model2,
optimizer1,
optimizer2,
criterion,
train_loader,
val_loader,
num_epochs,
device,
select_indices_fn,
):
verbose = False if FINAL_EVALUATION_MODE else True
# Metric history for each model
metrics = {
k: [[], []]
for k in [
"train_loss",
"val_loss",
"train_bac",
"val_bac",
]
}
epochs_range = np.arange(num_epochs) + 1
# Main training loop
for epoch in epochs_range:
print(f"Epoch {epoch}")
# Statistics history for each model
stats = {
k: [0, 0] for k in ["train_loss", "train_total", "val_loss", "val_total"]
}
preds_targets = {
k: [[], []]
for k in ["train_preds", "train_targets", "val_preds", "val_targets"]
}
model1.train(), model2.train()
for inputs, targets in tqdm(train_loader, desc=f"Epoch {epoch}/{num_epochs}"):
inputs, targets = inputs.to(device), targets.squeeze().long().to(device)
outputs = [m(inputs) for m in (model1, model2)]
losses = [criterion(out, targets) for out in outputs]
# --- THE KEY POINT OF THE TASK ---
selected_indices = select_indices_fn(targets, losses)
# ---------------------------
# Backpropagation for each model
for i, (model, optim) in enumerate(
[(model1, optimizer1), (model2, optimizer2)]
):
optim.zero_grad()
sel_idx = selected_indices[i]
loss = criterion(model(inputs[sel_idx]), targets[sel_idx]).mean()
loss.backward()
optim.step()
# Statistics history
stats["train_loss"][i] += loss.item() * len(sel_idx)
stats["train_total"][i] += len(sel_idx)
preds = outputs[i].max(1)[1]
preds_targets["train_preds"][i].extend(preds[sel_idx].cpu().numpy())
preds_targets["train_targets"][i].extend(targets[sel_idx].cpu().numpy())
# Evaluation on the validation set
if verbose:
model1.eval(), model2.eval()
with torch.no_grad():
for inputs, targets in tqdm(
val_loader, desc=f"Validation {epoch}/{num_epochs}"
):
inputs, targets = inputs.to(device), targets.squeeze().long().to(
device
)
for i, model in enumerate([model1, model2]):
outputs = model(inputs)
loss = criterion(outputs, targets).mean()
preds = outputs.max(1)[1]
stats["val_loss"][i] += loss.item() * inputs.size(0)
stats["val_total"][i] += inputs.size(0)
preds = outputs.max(1)[1]
preds_targets["val_preds"][i].extend(preds.cpu().numpy())
preds_targets["val_targets"][i].extend(targets.cpu().numpy())
# Computing the metrics
if verbose:
models = [model1, model2]
for i in range(2):
for phase in ["train", "val"]:
preds = preds_targets[f"{phase}_preds"][i]
targets = preds_targets[f"{phase}_targets"][i]
metrics[f"{phase}_loss"][i].append(
stats[f"{phase}_loss"][i] / stats[f"{phase}_total"][i]
)
metrics[f"{phase}_bac"][i].append(
balanced_accuracy_score(targets, preds)
)
print(
f"Model{i+1} - Train Loss: {metrics['train_loss'][i][-1]:.4f}, "
f"Train balanced accuracy: {metrics['train_bac'][i][-1]:.4f} --- "
f"Validation Loss: {metrics['val_loss'][i][-1]:.4f}, "
f"Validation balanced accuracy: {metrics['val_bac'][i][-1]:.4f}, "
)
# Generating the plots
if verbose:
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6))
colors = ["#fa2729", "#ac1a1c", "#1a6aff", "#144aad"]
linestyles = ["-", "--"]
for i, model_name in enumerate(["Model1", "Model2"]):
for j, phase in enumerate(["train", "val"]):
color = colors[i * 2 + j]
ax1.plot(
epochs_range,
metrics[f"{phase}_loss"][i],
color=color,
marker="o",
linestyle=linestyles[j],
label=f"{model_name} {phase.title()} Loss",
)
ax2.plot(
epochs_range,
metrics[f"{phase}_bac"][i],
color=color,
marker="o",
linestyle=linestyles[j],
label=f"{model_name} {phase.title()} balanced accuracy",
)
for ax, title in zip([ax1, ax2], ["Loss", "Balanced accuracy"]):
ax.set_title(f"Training and Validation {title}")
ax.set_xticks(epochs_range)
ax.set_xlabel("Epochs")
ax.set_ylabel(title)
ax.legend()
plt.tight_layout()
plt.show()Example Solution
Below we present a simplified solution that serves as an example demonstrating the basic functionality of the notebook. It can serve as a starting point for developing your own solution.
######################### DO NOT CHANGE THIS CELL ##########################
def default_select_indices(targets, losses):
# All indices for both models
selected_indices = [torch.arange(targets.shape[0]).to(DEVICE) for _ in range(2)]
return selected_indices######################### DO NOT CHANGE THIS CELL ##########################
# IMPORTANT: We always train two models and evaluate them
if not FINAL_EVALUATION_MODE:
seed_everything(SEED)
criterion = nn.CrossEntropyLoss(reduction="none")
model1 = SmallMobileNet(NUM_CLASSES).to(DEVICE)
model2 = SmallMobileNet(NUM_CLASSES).to(DEVICE)
optimizer1 = AdamW(model1.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)
optimizer2 = AdamW(model2.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)
seed_everything(SEED)
train(
model1,
model2,
optimizer1,
optimizer2,
criterion,
train_loader,
val_loader,
NUM_EPOCHS,
DEVICE,
select_indices_fn=default_select_indices,
)
# Evaluation of the example solution
bac_1 = predict_and_evaluate(model1, val_loader, DEVICE, verbose=True)
bac_2 = predict_and_evaluate(model2, val_loader, DEVICE, verbose=True)
performance(bac_1, bac_2)Your Solution
Place your solution in this section. Make changes only here!
The starting solution is currently the example one. Your task is to modify the inside (body) of the function. Do not use default_select_indices in your solution.
def your_select_indices(
targets: torch.Tensor, losses: List[torch.Tensor]
) -> List[torch.Tensor]:
"""
Function that selects the indices of the training set to be used when training the models.
Parameters:
targets: The labels of the training data from the given batch as a tensor. Shape: (batch_size,)
losses: A two-element list containing the tensors of loss function values for the individual models. Shape: [(batch_size,), (batch_size,)]
Returns:
A list of the indices selected for updating the weights of the models.
"""
# Your code
selected_indices = []
return default_select_indices(targets, losses)Evaluation
Running the cells below lets you check how many points your solution would score on the validation data. Before submitting, make sure that the whole notebook (also with the flag FINAL_EVALUATION_MODE = True set) runs from start to finish without errors and without any user intervention after selecting the "Run All" option.
######################### DO NOT CHANGE THIS CELL ##########################
def final_evaluate(evaluate_data_path, model1, model2):
base_transform = transforms.Compose([transforms.ToTensor()])
evaluate_dataset = TaskDataset(root=evaluate_data_path, transform=base_transform)
evaluate_loader = DataLoader(
dataset=evaluate_dataset, batch_size=BATCH_SIZE, shuffle=False
)
bac_1 = predict_and_evaluate(model1, evaluate_loader, DEVICE, verbose=True)
bac_2 = predict_and_evaluate(model2, evaluate_loader, DEVICE, verbose=True)
return performance(bac_1, bac_2)######################### DO NOT CHANGE THIS CELL ##########################
if not FINAL_EVALUATION_MODE:
seed_everything(SEED)
criterion = nn.CrossEntropyLoss(reduction="none")
model1 = SmallMobileNet(NUM_CLASSES).to(DEVICE)
model2 = SmallMobileNet(NUM_CLASSES).to(DEVICE)
optimizer1 = AdamW(model1.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)
optimizer2 = AdamW(model2.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)
seed_everything(SEED)
train(
model1,
model2,
optimizer1,
optimizer2,
criterion,
train_loader,
val_loader,
NUM_EPOCHS,
DEVICE,
select_indices_fn=your_select_indices,
)
final_evaluate(VAL_DATASET_PATH, model1, model2)Your your_select_indices function will be saved to the file your_select_indices.pkl and then used to train the models on the training set (in accordance with the code above). The final number of points will be calculated on the basis of the classification quality on the test set.
######################### DO NOT CHANGE THIS CELL ##########################
if FINAL_EVALUATION_MODE:
import cloudpickle
OUTPUT_PATH = "file_output"
FUNCTION_FILENAME = "your_select_indices.pkl"
FUNCTION_OUTPUT_PATH = os.path.join(OUTPUT_PATH, FUNCTION_FILENAME)
if not os.path.exists(OUTPUT_PATH):
os.makedirs(OUTPUT_PATH)
with open(FUNCTION_OUTPUT_PATH, "wb") as f:
cloudpickle.dump(your_select_indices, f)Translated by SOTA. The Polish original is the official version and wins wherever the two differ. The statement calls the function your_selected_indices, but the code defines your_select_indices, as in the original. The plots in the official solution come from the original run, so some of their titles are still in Polish. If you organise this olympiad and would like the translation removed, email [email protected] and we will take it down.
At a glance
- You get
- Training and validation image archives with
dataset_labels.csv, downloaded from Google Drive. - You submit
- This notebook with
your_selected_indices; the function is saved toyour_select_indices.pkland used to train the models on the training set during checking. - Scoring
- Mean balanced accuracy of the two models, BAC_mean = (BAC_1 + BAC_2)/2, on the test set. Points = 0 if BAC_mean ≤ 0.5, 100 × (BAC_mean − 0.5)/(0.8 − 0.5) if 0.5 < BAC_mean < 0.8, 100 if BAC_mean ≥ 0.8; rounded to an integer.
- Rules
- Tested without Internet access, with a GPU; evaluation must take at most 5 minutes with a GPU.
- The model architecture (SmallMobileNet) must not be changed.
default_select_indicesmust not be used in the solution.
- Format
- Stage I (online), 17 February – 22 March 2025; up to 100 points per task (500 in total). Evaluated automatically on the Competition Platform (Platforma Konkursowa) on a hidden test set; points are rounded to an integer, and a notebook that fails the requirements or does not run scores 0.