Discord

Checklist OAI 2024 Stage I · Task 5

Object Tracking

Polish title: Śledzenie obiektów

Determine the final permutation of cups in rendered "three cups" shell-game videos, first from bounding boxes, then from noisy boxes, and finally from raw frames.

  • Vision
  • Multi-object tracking (three subtasks)
  • Polish original · English translation

The task

The task consists of three subtasks, each in its own notebook, built around animations of the "three cups" game. The initial arrangement is denoted [0, 1, 2] from left to right (by x coordinate); the algorithm must output the permutation describing where the cups end up after a series of swaps (for example, moving the leftmost cup to the opposite end gives [1, 2, 0]).

Subtask 1 (level_1): for every frame the bounding boxes of the cups, as predicted by a detection model, are given, and the algorithm may use only these boxes (the frames serve for visualisation). Subtask 2 (level_2): the same, but boxes may merge when cups occlude each other, may fit the objects imprecisely and may be missing in some frames. Subtask 3 (level_3): no boxes are provided and the algorithm must work directly on the unlabelled frames.

Each subtask notebook also requires a short report (up to 300 words) describing the solution and stating the validation accuracy obtained with the provided submission_script function.

Abridged and translated by SOTA from the official Polish materials. The official statement has the exact rules, and it wins wherever this summary differs.

In English

This task was published in Polish. SOTA translated its 3 files into English on 16 September 2026. Only the words changed in the notebooks: markdown, code comments, messages and printed output. The code, file names and paths are the original's, so a translated notebook runs with the original data.

Read the task notebook, part 1 in English 528 words and 8 code cells

Object tracking

Introduction

In the digital era, faced with an avalanche-like growth in the amount of video data, the ability to recognise and interpret such data automatically is becoming crucial in many fields – from public safety to autonomous vehicles. Technologies based on deep learning are revolutionising the way we process visual information. A key challenge here is the detection and tracking of objects in video footage.

The goal of this task is to develop an algorithm that can analyse sequences of moves in the "three cups" game. Participants are asked to determine the final position of the cups after a series of moves, using the analysis of static images from each frame of the recording.

######################### DO NOT CHANGE THIS CELL WHEN SUBMITTING ##########################

# The functions below make it easier to work with the provided data
# In the following cells you will see examples of their use
from utils.utils import get_level_info, get_video_data, display_video, download_and_replace_data

FINAL_EVALUATION_MODE = False
# While checking your solution, we will change this value to True
# The value of this flag M U S T be set to False in the solution you send us!

images, coordinates, target, path_to_images = get_video_data(level=1,video_id=0,dataset="example")
display_video(images,rescale=0.7,FINAL_EVALUATION_MODE=FINAL_EVALUATION_MODE)

Task 1: The three-cup game

One possible approach to the problem of recognising objects in video is to apply, to each frame, a model dedicated to the analysis of static images. That is exactly what we will try to achieve here. For each frame in the animations, we have provided a description, predicted by a model, of where the cups are located. The recordings show how the cups are swapped. Your task will be to determine the positions in which they finally end up. We denote the initial arrangement as [0,1,2][0,1,2], counting in order from left to right (along the x coordinate). If we then move the leftmost cup to the opposite end, we obtain [1,2,0][1,2,0].

You will have access both to all the frames of the animations and to the bounding boxes we have marked, which contain the cups. Importantly, the algorithm you create must use only the information about the bounding boxes. In this task, the video frames are provided only so that you can visualise the examples and the algorithm for your own purposes.

Points for this task will be awarded for achieving the most accurate predictions possible on the test set. The criterion will be accuracy. The evaluation on the test set will be carried out by the organisers.

Submission files

Only this notebook, containing the code and a short report describing your solution (up to 300 words). You will find the place for the report at the end of this notebook.

Constraints

  • Your function should return its predictions in at most 5 minutes on Google Colab without a GPU.

Notes and hints

  • Test your solution on the set of video files level_1.
  • Model performance: test the performance of your model on the validation set using the function submission_script that we provide, and include this result in the report.

Evaluation

Remember that during checking the flag FINAL_EVALUATION_MODE will be set to True. Using the script validation_script.py, you can make sure that your solution will be executed correctly on our grading servers.

For this subtask you can score between 0 and 0.5 points. You will score 0 points if your accuracy on the test set is below 50%. If it is equal to 100%, you will receive 0.5 points. Between these values, the score increases linearly with the value of the metric.

Starter code

# The libraries below are sufficient to complete all the tasks
# If, however, you want to use others, check whether they are available on the server (requirements.txt)
import numpy as np
import os
import matplotlib.pyplot as plt
import torch
import IPython.display
import json
import PIL
import sklearn as sk
# Helper function for loading the data
images, _, _, _ = get_video_data(level=1,video_id=0,dataset="example")

with open(os.path.join(os.getcwd(),'example_tracks','tracks_1_0.json'), 'r') as f:
    tracks = json.load(f)

for key in tracks.keys():
    tracks[key] = [tuple(el) for el in tracks[key]]

# Helper function for displaying the data
display_video(images,
                tracks=tracks,
                rescale=0.7,
                FINAL_EVALUATION_MODE=FINAL_EVALUATION_MODE)
# Downloading the data for subtasks 1, 2 and 3 (about ~646Mb); the script will run for a few minutes
# It is enough to download the data only once. On the grading server the data will already be downloaded
# The file structure will be identical to the one here
if not FINAL_EVALUATION_MODE:
    download_and_replace_data()
######################### DO NOT CHANGE THIS CELL ##########################

# Helper function for testing the algorithm
def submission_script(algorithm,level,verbose=False,dataset="valid"):
    num_videos, _ = get_level_info(level=level,dataset=dataset)
    correct = []
    exception_messages = set()
    for video_number in range(num_videos):
        _, coordinates, target, _ = get_video_data(level=level,video_id=video_number,dataset=dataset)
        try:
            prediction = algorithm(coordinates)
            if tuple(target) == tuple(prediction):
                correct.append(1)
            else:
                correct.append(0)
            if verbose:
                print(f"Video: animation_{str(video_number).zfill(4)}")
                print(f"Prediction: {prediction}")
                print(f"Target:     {target}")
                print(f"Score: {tuple(target) == tuple(prediction)}", end='\n\n')
        except Exception as e:
            correct.append(0)
            exception_messages.add(str(e))
    if verbose:
        print(f"Accuracy: {np.mean(correct)}")
        print(f"Correctness: {correct}")
    return np.sum(correct) / num_videos, correct, exception_messages

Your solution

def your_algorithm_task_1(coordinates): # do not change the function name
    #TODO - implement your algorithm 

    permutation = [0,1,2]
    return permutation
# Check how your algorithm works
accuracy, correctness, _ = submission_script(
    algorithm=your_algorithm_task_1,
    level=1,
    verbose=True,
    dataset="train")
# Save your report in the variable below, so that we can later read it automatically with the checker
raport_1 = \
"""
Task report:
. 
.
.
"""

Translated by SOTA. The Polish original is the official version and wins wherever the two differ. The example animations are too large for the English copies; see them in the original notebooks. The notebooks need the utils, example_data and example_tracks folders from the original task folder. If you organise this olympiad and would like the translation removed, email [email protected] and we will take it down.

At a glance

You get
Example animations and tracks in the repository (example_data, example_tracks); training and validation data (train_data.zip and valid_data.zip, about 646 MB in total according to the notebook) downloaded from Google Drive by utils/general.py.
You submit
The three notebooks with your_algorithm_task_1(coordinates), your_algorithm_task_2(coordinates) and your_algorithm_task_3(images), each returning a permutation, plus the report strings raport_1, raport_2 and raport_3.
Scoring
Accuracy of the predicted permutation on the organisers' test set, per subtask worth 0–0.5 points. Subtask 1: 0 points below 50%, 0.5 points at 100%, linear in between. Subtask 2: 0 points below 50%, 0.5 points above 95%, linear in between (results above 80% are expected). Subtask 3: 0 points below 30%, 0.5 points at 100%, linear in between.
Rules
  • Each function must return its predictions within 5 minutes on Google Colab without a GPU.
  • In subtasks 1 and 2 the algorithm may use only the bounding-box information.
  • Other libraries must be available on the evaluation server (requirements.txt).
Format
Stage I (online, solved at home), 22 April – 27 May 2024; notebook submitted through the Olympiad's submission website and scored automatically. Worth up to 1.5 points (three subtasks of 0.5 points) of the stage total of 10.

Details

Year
2024, Online
Round
Stage I · Task 5
Language
Polish; English translation by SOTA
License
Not stated by the source