# NTO 2025/2026 — "Artificial Intelligence" profile

*English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email sota.ai.community@gmail.com.*

## Team stage

---

## 1. Task description

**Context:** The book platform distinguishes two types of user activity: adding a book to one's "plans" (`has_read=0`) and actually reading it (`has_read=1`). These are signals of different strength: an intention to read does not always lead to a real action.

**Goal:** Develop a model that can correctly order a mixed list of candidate books according to a **three-level relevance hierarchy**:
1. **Highest value (2 points):** Books that the user actually read (`has_read=1`)
2. **Medium value (1 point):** Books that the user added to their plans but did not read (`has_read=0`)
3. **Lowest value (0 points):** "Cold" candidates — books with which the user never interacted, but which were added to the candidate pool based on the recommendations of a basic model

For each user, `candidates.csv` provides a mixed list consisting of their real interactions in the test period (both read and planned) plus "cold" candidates. The model's task is to order this list correctly, so that the books read are in the very first positions and the planned books are below the books read but above the "cold" candidates.

This is a **conversion prediction** task with elements of "cold start", disguised as Learning-to-Rank (LTR). The model must learn to separate the "gold" (books read), the "silver" (books planned) and the "waste rock" ("cold" candidates); this tests the skills of building models that can distinguish the strength of interaction signals and predict the conversion of interest into actual consumption of content.

### 1.1. Features of the task

**Temporal split:** The training set (`train.csv`) contains the history of user interactions up to a certain point in time (T_global), while the test data (the candidates to rank) belong to a later period. This makes the task realistic: participants must predict future interactions from past ones.

**Mixed candidates:** For each user, `candidates.csv` provides a mixed list consisting of:
- All books the user interacted with in the test period (both read and planned)
- "Cold" candidates — up to 15 additional books generated by a basic recommendation model (ALS), with which the user never interacted

This makes the task harder and more realistic: participants must not only tell the books read from the planned ones, but also filter out the "cold" candidates, which do not reflect the user's real interest.

**Timestamps:** The training set (`train.csv`) contains timestamps (`timestamp`), which allows participants to build temporal models and analyse interaction patterns. However, `candidates.csv` provides no timestamps, which requires other features to be used for ranking and makes the task harder.

**Data isolation:** Stage 2 users are completely isolated from Stage 1 users, which guarantees that the stages of the competition are independent.

---

## 2. Baseline solution

For a quick start, a baseline solution is provided, which can and should be used as a starting point for developing your own solution.

**The baseline is available at:** [https://github.com/Orange-Hack/nto-ai-25-26-team-baseline](https://github.com/Orange-Hack/nto-ai-25-26-team-baseline)

**Recommendation:** To get started, it is recommended to fork the repository. This will let you receive baseline updates seamlessly by syncing with the original repository.

The baseline contains:

- A fully working training and prediction pipeline

- Examples of feature engineering (aggregated features, working with text via TF-IDF and BERT)

- The project structure and the utilities needed to validate a solution

It is recommended to study the baseline before starting work on the task.

---

## 3. Data

Participants are given a set of files containing the interaction history, the candidate pools to rank and the necessary metadata.

**Main files:**
- `train.csv` — the training interaction history (with timestamps)
- `targets.csv` — the list of users to rank for
- `candidates.csv` — the candidate pools for each user (without timestamps)
- `books.csv`, `users.csv`, `genres.csv`, `book_genres.csv`, `book_descriptions.csv` — metadata

> **A detailed description** of the set of files, the table structure, the fields and the principles by which the data were formed is given in the accompanying document "Stage 2B: Description of the data and of the solution format".

---

## 4. Solution format

The solution file must be in CSV format with a comma (`,`) separator.

> **Detailed requirements** for the column structure, the data types and the constraints are described in the accompanying document "Stage 2B: Description of the data and of the solution format".

---

## 5. Evaluation metric

The final score is computed with the metric **NDCG@20 (Normalized Discounted Cumulative Gain at 20)**, a standard metric for ranking tasks with multi-level relevance.

### 5.1. Why NDCG@20?

NDCG@20 is ideally suited to our task because:

1. **It supports multi-level relevance:** The metric was designed to evaluate rankings in which the items have different "value" (in our case: 2 points for books read, 1 for planned books, 0 for "cold" candidates).

2. **It penalises the wrong order:** If you place a planned book (1 point) above a book that was read (2 points), the metric drops considerably. This ensures that the model correctly understands the value hierarchy of interactions.

3. **It takes positions into account:** The higher a relevant book is in the list, the more it contributes to the metric. This encourages correct ranking not only across levels, but also within each level.

4. **Fairness:** The metric is normalised by the ideal result, which makes it possible to compare users with different numbers of interactions.

### 5.2. NDCG@20 (Normalized Discounted Cumulative Gain at 20)

For each user, **NDCG@20** is computed on the basis of the three-level relevance:

**Step 1: Assigning relevance scores**
Each book in your ranked list is assigned a relevance score:

- rel(r_i) = 2 if the book was read by the user (present in `book_id_list_read`)
- rel(r_i) = 1 if the book was added to the plans (present in `book_id_list_planned`)
- rel(r_i) = 0 if the book is a "cold" candidate (absent from both lists)

**Step 2: Computing DCG (Discounted Cumulative Gain)**

$$
\mathrm{DCG@20}(u) = \sum_{i=1}^{\min(|R(u)|, 20)} \frac{rel(r_i)}{\log_2(i+1)}
$$

**Step 3: Computing IDCG (Ideal DCG)**
The ideal DCG is computed for the ideal ranking, in which all books are sorted by relevance in descending order:

$$
\mathrm{IDCG@20}(u) = \sum_{i=1}^{\min(|R(u)|, 20)} \frac{rel_{ideal}(r_i)}{\log_2(i+1)}
$$

where rel_ideal is the list of relevance scores sorted in descending order (first all the twos, then all the ones, then all the zeros).

**Step 4: Normalisation**

$$
\mathrm{NDCG@20}(u) = \frac{\mathrm{DCG@20}(u)}{\mathrm{IDCG@20}(u)}
$$

**The final metric is the mean over all users:**

$$
\overline{\mathrm{NDCG@20}} = \frac{1}{N} \sum_{j=1}^N \mathrm{NDCG@20}(u_j)
$$

**Edge case handling:** If all relevances are 0 (for example, the user neither read nor planned any of the candidate books), the metric equals 0.0.

**Notation:**
- $N$ — the number of users in the set
- $u$ — a particular user
- $\mathrm{rel}(r_i)$ — the relevance score of the book at position $i$ (0, 1, or 2)
- $R(u)$ — your ranked list of books for user $u$ (up to 20 books, or all available candidates if there are fewer than 20)

### 5.3. Final score (Score)

The final score equals NDCG@20:

$$
\mathrm{Score} = \overline{\mathrm{NDCG@20}}
$$

**The higher the Score, the better the result.** The leaderboard is sorted by Score in descending order.

### 5.4. Worked example

Suppose that for user $u$:
- Candidates (10 books): `[A, B, C, D, E, F, G, H, I, J]`
- Books read: `{E, F}` (2 books, relevance = 2)
- Planned books: `{A, C}` (2 books, relevance = 1)
- "Cold" candidates: `{B, D, G, H, I, J}` (6 books, relevance = 0)

**Option 1: Correct order**
Your submission: `[E, F, A, C, B, D, G, H, I, J]`

**Relevance scores:** `[2, 2, 1, 1, 0, 0, 0, 0, 0, 0]`

**DCG@20:**
- Position 1 (E): 2 / log_2(2) = 2.0
- Position 2 (F): 2 / log_2(3) = 1.261
- Position 3 (A): 1 / log_2(4) = 0.5
- Position 4 (C): 1 / log_2(5) = 0.431
- The rest: 0

**DCG = 2.0 + 1.261 + 0.5 + 0.431 = 4.192**

**IDCG@20:** For the ideal ranking (first all the twos, then the ones, then the zeros), the result is the same: **4.192**

**NDCG@20 = 4.192 / 4.192 = 1.0** ✓

**Option 2: Wrong order**
Your submission: `[A, B, C, D, E, F, G, H, I, J]` (planned books above books read!)

**Relevance scores:** `[1, 0, 1, 0, 2, 2, 0, 0, 0, 0]`

**DCG@20:**
- Position 1 (A): 1 / log_2(2) = 1.0
- Position 2 (B): 0 / log_2(3) = 0
- Position 3 (C): 1 / log_2(4) = 0.5
- Position 4 (D): 0 / log_2(5) = 0
- Position 5 (E): 2 / log_2(6) = 0.774
- Position 6 (F): 2 / log_2(7) = 0.712
- The rest: 0

**DCG = 1.0 + 0 + 0.5 + 0 + 0.774 + 0.712 = 2.986**

**IDCG@20:** Still **4.192** (ideal ranking)

**NDCG@20 = 2.986 / 4.192 = 0.712** ✗

As you can see, the wrong order (planned books above books read) **greatly lowers** the metric, even if all the books have been found.

---

## 6. Competition conditions

### 6.1. Checking and leaderboard

- **Public leaderboard (Public):** Computed on the visible part of the test data. The result is updated after each successful submission.
- **Private leaderboard (Private):** Computed on the hidden part of the test data. The final results of the competition are determined solely by this leaderboard.
- **Limits:** There is a limit on the number of submissions per day and for the whole stage. The exact values will be given on the competition page.

### 6.2. Constraints

- It is forbidden to use any external data and pre-trained models, except for publicly available ones (e.g. for text processing).
- The solution must be fully self-contained and must not require internet access while running.

For details, see the Participation Rules