# NTO AI Final. The "Lost Items" case

*English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email sota.ai.community@gmail.com.*

By taking part in the competition, you agree to the [rules of the final of the NTO AI profile](https://disk.yandex.ru/i/DmNYduYXiFUPKA)

## 1. Case description

### 1.1. Backstory

The overnight release of a book service went wrong. During the migration of the event schema, some of the production workers began writing logs in the new schema, while the rest stayed on the old one. Because of a bug in the consumer and in deduplication, some of the positive events never reached the final storage.

As a result, the users' interaction history has become incomplete. This degrades personalisation, recommendations and product analytics.

Your role is the recommender-system team, which has to restore the most likely lost positive interactions from the observed data.

### 1.2. What the solution must do

For each user in `targets.csv`, you need to build a ranked list of **20 editions** (`edition_id`) that are most likely to be lost positive interactions.

The solution consists in building an ML pipeline.

1. Candidate generation — searching for potentially relevant editions across the entire available catalogue.
2. Ranking — selecting the top 20 most likely lost items.
3. Assembly of the final list — outputting the top 20 for each user.

A detailed description of the file structure and the fields is given in the "Data" section and in  `data_description.md` in the baseline repository.

### 1.3. What counts as a positive interaction and as a lost item

Positive events in the logs.

1. `event_type = 1` — adding to the wishlist (`wishlist`)
2. `event_type = 2` — reading (`read`)

Scoring is done at the level of a **unique pair** `(user_id, edition_id)`, not at the level of event rows.

A "lost item" is a hidden (lost) positive pair `(user_id, edition_id)` from the incident window. The solution does not need to predict the type of the positive event (`wishlist`/`read`).

### 1.4. What is known about the nature of the losses

1. The loss of positive interactions is partial: in the incident window, about **20%** of the positive pairs `(user_id, edition_id)` are hidden.
2. The losses are uneven across users and over time.
3. The event types (`wishlist` and `read`) may be affected to different degrees.
4. The exact protocol by which the hidden lost items were formed is not disclosed.

### 1.5. Time windows

The case is built on a period of 214 days, which contains an **incident window** (from 2025-10-01 00:00:00 to 2025-11-01 00:00:00) — the period in which the logging failure occurred and some of the events were lost.

Participants receive the observed log for the whole period (180 days), in which artificial losses have been applied within the incident window.

### 1.6. Task context

Incomplete logs are a realistic problem of production systems. Event losses can arise from migration errors, ETL failures, schema desynchronisation and problems in the message-delivery pipelines.

This case models the emergency recovery of signals for a recommender system. The goal is to build a reproducible ML pipeline that restores the most likely lost positive interactions from the observed data.

> Important. In real systems, recovered events are usually treated as a model layer rather than as a replacement for the primary logs.

---

## 2. Scoring

### 2.1. Main metric

The final score is computed as the mean **NDCG@20** over all test users.

$$
Score = \frac{1}{|U|}\sum_{user \in U} NDCG_{user}@20
$$

The metric rewards separately for:

1. hitting a lost item
2. correct ranking

### 2.2. The participant's prediction and the hidden data

For each `user_id` in `targets.csv`, the participant predicts 20 rows with the fields `edition_id` and `rank` (1..20).

The organisers hold a hidden labelling `is_lost ∈ {0,1}` for the pairs `(user_id, edition_id)` (`1` — the pair is a lost item, `0` — it is not).

### 2.3. Metric formulas

Definitions:

* `rel(rank) = 1` if the edition at position `rank` is a lost item for this user, otherwise `rel(rank) = 0`.

$$
discount(rank)=\frac{1}{\log_2(rank+1)}
$$

$$
DCG_{user}@20=\sum_{rank=1}^{20} rel(rank)\cdot discount(rank)
$$

Let `relevant_items` be all the hidden lost items of the user.

For `IDCG`, we assume an ideal ranking in which all the lost items occupy the first positions.

$$
IDCG_{user}@20=
\sum_{rank=1}^{\min(|relevant\_items|,20)} discount(rank)
$$

$$
NDCG_{user}@20=
\begin{cases}
\frac{DCG_{user}@20}{IDCG_{user}@20}, & IDCG_{user}@20 > 0 \\
0, & IDCG_{user}@20 = 0
\end{cases}
$$

### 2.4. Examples of computing the metric

The examples are given in a separate document: `metric_examples.md` (see the baseline repository).

### 2.5. Notes on scoring

1. Averaging is done over users.
2. The metric evaluates only the hidden lost positive interactions.
3. Scoring is done over the pairs `(user_id, edition_id)`.
4. The Public leaderboard and the Private leaderboard use the same scoring principle on different hidden subsets of users.
5. The split into Public and Private is made by users (`user_id`) in a **50/50** proportion.

---

## 3. Solution format and requirements

### 3.1. Submission format

You must submit the file `submission.csv` of the following form:

```csv
user_id,edition_id,rank
```

### 3.2. Technical requirements

1. For each `user_id` in `targets.csv` there must be **exactly 20 rows**.
2. `rank` must be an integer from `1` to `20`.
3. Within one `user_id`, the `rank` values must not repeat.
4. Within one `user_id`, the `edition_id` values must not repeat.
5. The file must contain predictions for **all** users in `targets.csv`.
6. The order of the rows in the file does not matter.

### 3.3. Example submission (`sample_submission.csv`)

An example of a valid solution file.

| Field | Description | Comment |
|-------|-------------|---------|
| `user_id` | user identifier | - |
| `edition_id` | restored edition | - |
| `rank` | position in the top 20 | values 1..20, unique within a user_id |

## Baseline

Available at https://github.com/Orange-Hack/nto-ai-25-26-final-baseline