Semantic Equivalence of Questions

Semantic Equivalence of Questions

English translation by SOTA – AI Community of the Georgian original. Organisers who would like this translation removed can email [email protected].

Source: the Final Round of the Georgian National AI Olympiad, organised by the Georgian Artificial Intelligence Association (GAIA), 25 May 2025, hosted on the Bohrium platform: original statement. The statement file on Bohrium is named e_en.md, but its text is in Georgian.

🧠 Story:

Imagine that you work in the data science team of a large-scale digital knowledge platform. One of the platform's main challenges is managing the enormous number of questions asked by users efficiently. Users often ask questions that have already been answered, or phrase questions slightly differently while meaning the same thing. Identifying such semantically identical, i.e. equivalent questions, is critically important for improving the system's efficiency and the user experience.

To solve this problem, your team has already carried out some preparatory work and developed a basic, small BERT model (custom small BERT), which has been trained in general to understand text. Now this model needs to be fine-tuned specifically to recognise equivalent questions.

🎯 Your mission:

Your task is to fine-tune (fine-tuning) the provided custom BERT model so that it can determine whether a given pair of questions is semantically equivalent. The model must learn to establish the semantic relationship between questions and decide whether a pair is equivalent (class 1) or not (class 0).

Pay particular attention to the structure and nature of the provided data, because your success will depend greatly on how cleverly you prepare the data for effective fine-tuning of the BERT model.

⚙️ Formal task:

Fine-tune the provided, partially pre-trained, small BERT model so that it can classify pairs of questions as semantically equivalent (1 – equivalent, 0 – non-equivalent).

⚠️ Important nuance:

Note that, for preparing the training data needed for fine-tuning, besides 500 pairs confirmed to belong to the positive class (i.e. equivalent), you are given 1000 unique sentences (not pairs, but individual sentences). It must be stressed that these 1000 sentences are definitely not semantic equivalents of one another. Using this information strategically and effectively will be critical for fine-tuning your model. Your solution must run on a T4 GPU within at most 1 hour.

📁 Provided materials and data:

You are given the following files and resources:

• train_data.csv: this file contains 500 pairs of questions that are confirmed to be semantic equivalents. The file has the following columns: question1, question2,is_duplicate.
• raw_questions.csv: this file contains 1000 unique questions. Each question is given in the question column. These questions, in turn, are not equivalents of one another.
• test.csv: this file contains 1500 pairs of questions on which your fine-tuned model must make predictions. The file has the columns: question1, question2.
• custom_bert_model/: this directory contains the files of a small, custom BERT model that is ready for fine-tuning with the Hugging Face transformers library (for example, it may contain files such as config.json, pytorch_model.bin or model.safetensors, tokenizer_config.json, vocab.txt, etc.). You must use this model as the starting point.

📦 Deliverables:

• This Jupyter Notebook, with the code of your complete solution, describing the data preparation strategy you developed and the BERT model fine-tuning process. After a full run ("Run all"), the notebook must create the predictions file automatically.
• predictions.json: a JSON file containing your fine-tuned model's predictions for each question pair given in test_data.csv.
  • The file's key must be the pair_id from the test_data.csv file (as a string, e.g. "0", "1", "2", ...).
  • The file's value must be the class predicted by your model: 1 (equivalent) or 0 (non-equivalent) (as a number).
  • Example of the contents of a predictions.json file:

    {
      "0": 1,
      "1": 0,
      "2": 1,
      "3": 0,
      ...
    }
    

🏆 Evaluation:

Your solution will be evaluated with the F1-Score metric on the test_data.csv data. Note that to obtain a high score both Recall and Precision are important, which in turn requires building a well-balanced and effective classifier.

Good luck with solving the semantic equivalence of questions and fine-tuning your BERT model effectively!

Starter code

Below are the instructions for loading the initial data and creating the final output. In the last part, please change only the part where you store the answers in the Dict; do not change the saving step, and above all do not change the file name, otherwise you will receive 0 points.

import pandas as pd
from transformers import AutoModel, AutoTokenizer

# Note that here you must insert the path that you will see in the Bohrium notebook
train_data = pd.read_csv("train_data.csv")
train_data

# Note that here you must insert the path that you will see in the Bohrium notebook
raw_questions = pd.read_csv("raw_questions.csv")
raw_questions

# Note that here you must insert the path that you will see in the Bohrium notebook
test = pd.read_csv("test.csv")
test

# Note that here you must insert the path that you will see in the Bohrium notebook
model = AutoModel.from_pretrained("./my_custom_bert_model")
tokenizer = AutoTokenizer.from_pretrained("./my_custom_bert_model")


# Your code
# ...
# ...




import json


# You may also change the parameters of this function; the main thing is that at the end you #create submission.json following the approach written here
def create_predictions_json(test_data):
   """
   Creates submission.json file. Currently outputs 0 for all question couples.
   TODO: Replace hardcoded 0 with actual model predictions.
   Do not modify the JSON saving part at the end.
   """
   predictions = {}
   for idx in range(len(test_data)):
       # TODO: add prediction logic here using your model.
       final_prediction = 0  # Replace with actual prediction
      
       predictions[str(idx)] = final_prediction


   # DO NOT MODIFY
   with open('submission.json', 'w') as f:
       json.dump(predictions, f, indent=4)


   print(f"Predictions saved to submission.json with {len(predictions)} entries")


create_predictions_json(test)