Checklist IOAI 2026 At-Home Round · Task 3
The Analytical Language of John Wilkins
Identify a hidden animal among roughly 1,400 candidates by asking at most 15 yes/no questions to an LLM-based oracle.
The task
Inspired by a Borges essay, the task imagines a cabinet whose drawers each hide one animal. A drawer answers only yes or no to questions drawn from a fixed pool of about 500, or accepts a guess from a pool of about 1,400 animals, and a correct guess ends the round. Questions or animals outside the pools raise an error without using up budget.
The oracle is a local language model, Qwen2.5-3B-Instruct by default, prompted at temperature 0 to answer yes or no about the hidden animal, so the same question about the same animal always gets the same answer. You may run the same model yourself to predict its replies, but its beliefs about animals are occasionally wrong.
Each animal allows at most 15 queries, and every query lowers that animal's score. Your solution is a class whose constructor runs once and whose solve method is called for each hidden animal. You tune on dev and report a test1 score, and the organisers also grade the solution on a hidden test set.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
animals_pool.txt(~1,400 animals),questions_pool.txt(~500 yes/no questions), helper code (interactor.py,evaluate.py) and dev/test1 CSV files from a Google Drive folder; the oracle runsQwen/Qwen2.5-3B-Instructlocally (about 6 GB, downloaded on first use).- You submit
- A MySolution class with __init__(
animals_pool,questions_pool) and solve(interactor). A screenshot of the final summary table (dev, test1, and a FINAL n-weighted mean over the available test splits) is submitted, and the organisers also score the submitted MySolution on a separate hidden test set. - Scoring
- Per animal: max(0, (1 if a correct guess was ever made, else 0) − 0.02 × queries used); the score is the mean across all animals in the test set.
- Rules
- At most 15 queries per hidden animal
- Questions must be lines from
questions_pool.txtand guesses must be words fromanimals_pool.txt
- Format
- At-Home Round (educational; the 2026 Contest Rules say the three problems were given to teams about one month before IOAI 2026 and do not affect final results)