Checklist HAIO 2024 Summer National Final · NLP-4 task
Needle in a Haystack
English title: Tű a Szénakazalban
Train RNN and CNN detectors for a binary "needle" pattern and design needle-in-a-haystack tests for Mistral models through the Mistral API.
The task
Inspired by Greg Kamradt's Needle in a Haystack (NIAH) test, the task has two independent parts. In the PyTorch part, a generator produces binary 0/1 sequences; positive examples contain an alternating 1010... "needle" at a random position, negatives are random. A two-layer RNN classifier and an ignite training loop that reaches only about 50% accuracy are provided.
PyTorch subtasks (55 points + 30 per mini-project): compute the BCE-with-logits loss of the untrained network (5); derive the probability that a random negative sequence of length k + l contains the k-long needle (10); improve the training code with minimal changes and document it (5); evaluate the trained model on longer sequences (5); compare other RNN architectures (5); visualise input gradients to show what the model attends to (10); argue whether a CNN can solve the task (5); implement and evaluate a CNN (10); and optional free mini-projects worth up to 30 points each.
Mistral API subtasks (35 points + 30 per mini-project): design (5) and implement (10) a natural-language NIAH test for Mistral 7B, design (5) and implement (10) a coding/formal-language NIAH test, compare Mistral 7B with Mixtral 8x7B (5), and optional free mini-projects worth up to 30 points each.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- A synthetic dataset generator (
get_binary_dataset), a baseline RNN classifier and ignite training code; for the API part, access to the Mistral API with the contestant's own key stored as the Colab secret MISTRAL_API_KEY. - You submit
- The completed notebook with code, figures and short written evaluations.
- Scoring
- PyTorch part: 55 points plus up to 30 points per mini-project; Mistral API part: 35 points plus up to 30 points per mini-project; graded by the organisers against stated mini-project criteria (motivation and creativity, implementation, evaluation and conclusions).
- Rules
- Using the NIAH GitHub repository as-is is not allowed.
- Use the Mistral 7B open model for the API tests; keep inputs below roughly 1000 tokens per message and use no more than about 10,000,000 tokens in total.
- Subtasks may be attempted in any order.
- Format
- Summer National Final (Nyári Országos Döntő), held on site at the ELTE Faculty of Informatics, Budapest, on 30 May 2024. Solved in Google Colab; the completed notebook is downloaded as .ipynb and uploaded, zipped together with the other solutions, to the CMS.