Ranking Candidate Books by Three-level Relevance

Ranking Candidate Books by Three-level Relevance

English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email [email protected].

3.1.3. Team tasks

Introduction

In the team part of the second qualifying stage, the team solves a ranking task: for each user, a mixed list of candidate books must be ordered so that the books read end up above the planned ones, and the planned ones above the "cold" candidates.

Confirmation form for the final submissions: https://forms.yandex.ru/u/6912ff409029024866e7cbce/.

Task 1 (100 points)

Ranking candidate books by three-level relevance

Topics: recommender systems, ranking, training scoring models, text processing, ranking metrics.

Statement

The book platform distinguishes two types of user activity: adding a book to one's plans (has_read=0) and actually reading it (has_read=1). These are signals of different strength: an intention to read does not always lead to a real action.

For each user, a list of candidates candidates.csv is provided, which includes:

• the books the user interacted with in the test period (both read and planned);
• "cold" candidates — additional books generated by a basic model, with which the user never interacted.

The total number of candidates for a user does not exceed 20.

The list of candidate books must be ordered according to a three-level relevance hierarchy:

• 2 points: books read (has_read=1);
• 1 point: planned books (has_read=0);
• 0 points: "cold" candidates.

The data and the file formats are provided on the page: https://ods.ai/competitions/nto25-26-2-team/dataset/.

Constraints, submission format and scoring (from the collection's Solution/Answer sections)

Constraints:

• using external data not provided by the organisers is forbidden;
• the solution must be reproducible and self-contained; it must not depend on internet access when the pipeline is reproduced and the final submission is prepared.

Solution format (submission):

• a CSV file with a comma (,) separator;
• columns: user_id, book_id_list;
• book_id_list is a single CSV string cell in double quotes containing an ordered list of book_id values separated by commas (,), without duplicates (CSV escaping is standard, RFC 4180);
• list length: $K = \min(20, \text{number of candidates for the user})$;
• only book_id values from the user's own candidate pool (from candidates.csv) may be used;
• order matters: the most relevant books go at the start of the list (ideally: first the "read" books, then the "planned" ones, then the "cold" ones);
• the order of the rows in the file does not matter; a header is allowed.

Example rows in submission.csv:

• 210,"971259,1281035,1673950,2225251,2274394"
• 211,"3015694"

Evaluation metric: the final Score equals the mean NDCG@20 over all users. NDCG@20 takes into account the position of a book in the output and the three-level relevance (2/1/0).

The answer is the submission file submission.csv uploaded to the competition page, in CSV format with the columns user_id, book_id_list.

Scoring details:

• the final score is computed with the NDCG@20 metric based on the three-level relevance (2/1/0);
• the public leaderboard (Public) is computed on the visible part of the data;
• the private leaderboard (Private) is computed on the hidden part of the data; the final results are determined by it alone;
• the limits on the number of submissions per day and for the whole stage are given on the competition page;
• if all relevances for a user are 0, the metric value for that user is taken to be 0.0;
• for the result to be taken into account in the final selection of submissions, the participant must fill in the confirmation form.

The collection's solution for this task is in the separate solution file.