Multi-class Animal Detection in Camera-trap Photos

Multi-class Animal Detection in Camera-trap Photos

English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email [email protected].

Source: the 2024/25 task collection of the National Technology Olympiad (NTO), "Artificial Intelligence" profile, Stage 2, pp. 104–109: original PDF.

3.2. Engineering tour

The aim of the task is to identify the participants' skills in working with classification algorithms in order to develop a model capable of automatically classifying the group of animals in camera-trap photos. While completing the tasks, participants' competence in data processing and in applying machine learning algorithms is tested, as well as their ability to immerse themselves in the subject area and to use computer vision algorithms in solving problems.

The task helps to assess participants on their knowledge and skills in programming, zoology, image processing, machine learning and the application of technical solutions to solve real-world problems.

3.2.1. Individual task

The task is devoted to multi-class detection of animals.

For this, participants will receive access to unique data: images from camera traps provided by partner research organisations. Solving the task will make it possible to filter out empty images and images with people or vehicles, and to single out images with animals by group.

All materials and data for the task are available at: https://disk.yandex.ru/d/hbZfV-2Dq9qJVQ.

Task 3.2.1.1. Multi-class Animal Detection in Camera-trap Photos (100 points)

Topics: neural networks, computer science.

Problem statement

Task description

Monitoring wild animal populations is an important task for the worldwide community of wildlife researchers. It helps to find out which animal species are endangered, how they behave in different periods of their lives, where they live, and much more.

Various tools are used to monitor animals, including camera traps: special cameras that are installed in the forest and react to movement in the frame. Every year these cameras deliver hundreds of thousands of photographs, in which the animals have to be found and categorised. This is very difficult and painstaking work, because the total number of species can reach several hundred, and there may be some that are visually hard to tell apart. In addition, only part of an animal may be in the frame at the moment the camera is triggered, and because many species are active at night, some photographs may be blurred or overexposed owing to the way the cameras work at night.

Within the NTO Olympiad, participants are invited to help scientists automate the routine work of processing camera-trap data by training machine learning models for this purpose.

For the primary filtering of the raw data and for further processing (analytics and the study of each individual animal), you will need to develop a multi-class detector that sorts photographs of animals into separate groups, organised by the similarity of the species present in the data (for example, Siberian roe deer, sika deer, Manchurian wapiti and maral are assigned to the deer group).

Metric

To assess the quality of the multi-class detection model, the metric mAP 0.5–0.95 is used: the average precision (AP) is computed for each class at different values of the IoU (Intersection over Union) threshold from 0.5 to 0.95 in steps of 0.05 (the minimum IoU for a match to count as positive), and the results are then averaged over the classes.

[Fig. 3.2.1 (IoU = Area of Intersection / Area of Union): see the original statement, p. 105.]

For each class $c$, the mean value of AP over all IoU thresholds is computed:

$$\mathrm{mAP}_c = \frac{1}{N}\sum_{i=1}^{N} \mathrm{AP}_{c,\tau_i},$$

where $N = 10$ is the number of IoU threshold values (from 0.5 to 0.95 in steps of 0.05).

The mean value of AP over all classes (mAP) is computed as:

$$\mathrm{mAP} = \frac{1}{C}\sum_{j=1}^{C} \mathrm{mAP}_j,$$

where $C$ is the total number of classes.

The value of the metric ranges from 0 (in the worst case) to 1, if all bounding boxes exactly match the ground-truth ones and their classes are predicted correctly.

The metric is computed automatically on the platform when a solution is submitted.

Data

The data are available at: https://disk.yandex.ru/d/hbZfV-2Dq9qJVQ.

The dataset is a set of images and a file train.csv that maps each image to a particular group of animals and gives the coordinates of the bounding boxes for each animal in the image.

The bounding-box coordinates are the centre of the bounding box and its size (width and height) in normalised form, in the interval from 0 to 1. The width and height vary in the range from 0 to 1.

[Fig. 3.2.2 (an image with axes +X and +Y from the Origin (0,0) to (1,1), and a box marked Width = 0.69, Height = 0.71, X = 0.48, Y = 0.63): see the original statement, p. 106.]

The correspondence between the group index and its name is given below:

• 0 — hare,
• 1 — wild boar,
• 2 — cats,
• 3 — mustelids,
• 4 — bear,
• 5 — deer,
• 6 — panthers,
• 7 — bovids,
• 8 — canids,
• 9 — marmot.

Note that the dataset contains difficult examples: photos in which only a small part of an animal is in the frame, many animals overlapping one another, as well as night-time and blurred shots.

The data were labelled by data-labelling specialists together with zoologists.

The data are split into a training set and a test set in a ratio of approximately 70% / 30%.

The test part is split into a public part and a private part in a ratio of approximately 40% / 60%.

Baseline

baseline_detection.ipynb — a jupyter-notebook that takes you all the way from installing the libraries and training the model to obtaining a file with predictions, and lets you inspect the algorithm's results visually.

Input format

Solutions are accepted as a zip archive of at most 5 Gb with the following structure:

• metadata.json;
• entry_point — a Python file;
• a file with the model weights;
• any additional files needed for the solution to work (Python files, directories, models and their weights);
• submission.zip — a zip archive containing an example solution for uploading to the platform and obtaining the metric.

The root of the archive will be the current directory for running the solution and for loading the model weights.

metadata.json has the following form:

{
   "image": "odsai/nto24-baseline:1.0",
    "entry_point": "python -u run.py"
}

• image — the field with the name of the Docker image on https://hub.docker.com/ in which the solution will be run; it must contain all the required libraries;
• entry_point — the command that launches the prediction file run.py.

This file must have the following mandatory parameters:

• --img_dir — the path to the folder with the test images on the platform;
• --output_path — the path at which the checking system expects the result of the solution.

The algorithm for obtaining predictions from the model must produce a prediction file, on the basis of which the metric is computed. The file itself is a csv document with two columns, image_name and predicted_detection. The rows are pairs of an image file name and the list of predictions corresponding to it.

The list of the model's predictions has the following format:

$$label_1\ cx_1\ cy_1\ w_1\ h_1\ confidence_1; label_2\ cx_2\ cy_2\ w_2\ h_2\ confidence_2;\ \ldots\ ;$$

$$label_n\ cx_n\ cy_n\ w_n\ h_n\ confidence_n,$$

where

• $label_i$ is the identifier of the group to which the animal belongs;
• $(cx_i, cy_i)$ is the coordinate of the centre of the $i$-th box (in relative values);
• $w_i, h_i$ are the width and height of the $i$-th box (in relative values);
• $confidence_i$ is the detection model's confidence in the prediction, from 0 to 1.

If no bounding box is predicted, the prediction must be given as an empty string ("").

An example of producing a correct prediction file.

For the structure of the prediction file, see Table 3.2.1.

Table 3.2.1

image_name	predicted_detection
cc27b9b56583a615fb3.JPG	0 0.5 0.5 0.25 0.25 0.9
087872711fe672676fd.JPG	0 0.5 0.5 0.5 0.5 0.9;0 0.5 0.5 0.5 0.5 0.9
424aa1aa8eb5bbdd07.JPG	-
c5537eaa60525efd7b.JPG	0 0.5 0.5 0.25 0.25 0.9
e9f15b67ca49453e28.JPG	0 0.5 0.5 0.25 0.25 0.9

Online course on working with Docker: https://ai-academy.ru/training/courses/docker-git/.

Memory and resource limits

A participant may upload at most five solutions for evaluation per day. Only valid attempts that received a numerical score are counted.

The container with the solution is run under the following conditions:

• 96 GB of RAM;
• Nvidia v100 32 GB;
• time to run the solution: 10 min for public, 10 min for private;
• the solution has no access to internet resources;
• maximum size of the packed solution archive: 5 GB.

Use of third-party datasets

The use of open (available online) datasets with a licence permitting free non-commercial use is allowed.

Solution

You are required to develop an algorithm for detecting groups of animals in camera-trap photos and to present the result of its work as a csv file with the columns image_name, predicted_detection.

The solution to the task is a csv file with predictions; example file: sample_submission.zip.

A walkthrough of the reference solution is available at: https://gist.github.com/ntomaterials/0e543eb9c086ce419211d5c57ecbe272.