Checklist IOAI 2024 Scientific Round (At-Home) · NLP task
Help BOBAI: Classify an unknown language
Build the best mBERT-based text classifier for an encrypted language mBERT never saw, using a small labelled set and raw text.
The task
Bob's start-up Bobai builds text-processing products on the multilingual encoder mBERT, which covers 101 languages. A major client, Amoira, now wants support for a new language X that is not among them.
Amoira supplies a small labelled text-classification dataset and a larger corpus of raw text in language X, both encrypted so competitors can't tell which market it is targeting. Bob shares the baseline he uses for supported languages and asks you to build the strongest classifier you can. Trying to decrypt the data is explicitly off-limits.
The classifier must be based on mBERT with no other pretrained language encoder, train in under 8 hours on an L4 GPU, and classify any 500 random samples in under 5 minutes. Teams submitted test predictions (test inputs came out 48 hours before the deadline), their best model on the Hugging Face Hub, and a Colab notebook that reproduces it.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- An encrypted labelled dataset (
InternationalOlympiadAI/NLP_problemon Hugging Face; its card lists 1,524 train and 218 dev examples) and an encrypted raw-text corpus (InternationalOlympiadAI/NLP_problem_raw; 611,245 texts per its card). The unlabelled test set (InternationalOlympiadAI/NLP_problem_test; 438 texts per its card) was released 48 hours before the deadline. - You submit
- Predictions for the test inputs saved as a text file of integer classes, one per line (
test_predictions.txtin the notebook), a link to the best trained model on the Hugging Face Hub, and working code in the Colab notebook that reproduces it. - Scoring
- Macro-averaged F1, as computed by the notebook's baseline evaluation code (evaluate 'f1' with average='macro'); the task text itself does not name the scoring metric.
- Rules
- The classifier must be based on mBERT and may not use any additional pretrained language encoder
- Training must take under 8 hours on an L4 GPU
- Inference on any random 500 samples must take under 5 minutes
- Do not try to decrypt the data
- Format
- Scientific round, at-home stage (teams had one month for the three at-home tasks)