Checklist EUROAI (CEOAI) 2026 Contest Day 1 · Task 1
Faithful Explanations
Produce token-level rationales for a frozen DistilBERT sentiment classifier that satisfy a faithfulness probe, in three differently scored variants.
The task
A court-ordered audit concerns a frozen distilbert-base-uncased-finetuned-sst-2-english sentiment classifier. The contestant receives 100 pre-tokenised SST-2-style sentences with the model's predicted label and confidence, and must give rationales: subsets of real token positions (never [CLS] or [SEP]) of size at most min(12, floor(0.30 × T_real)).
A rationale passes the faithfulness probe (from ERASER) when the margin with all other tokens replaced by [MASK] keeps at least half the original margin (sufficiency) and masking the rationale removes at least 0.3 of it (comprehensiveness). Subtask 1 (Witness, 30 pts) asks for one rationale per sentence, compared with the smallest passing set built by adding tokens in order of the provided occlusion deltas; Subtask 2 (Counter-Argument, 35 pts) asks for two different rationales per sentence, scored by match quality times their Jaccard distance; Subtask 3 (Journalist, 35 pts) asks for one rationale compared with a hidden 2-of-3 consensus of attention rollout, occlusion and gradient × input.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- Starter kit:
sentences.json(100 sentences withtoken_ids,attention_mask,predicted_label, confidence),classifier_info.json,occlusion_pack.json(per-token occlusion deltas and reference margins), 25 practice sentences with labelled rationales and occlusion deltas, and the notebook 'Faithful Explanations (RAW).ipynb' with the probe and baselines. - You submit
- One CSV with columns subtaskID, datapointID, answer, where answer is 1 if a token is in the rationale; datapointID = sid × 1000 + tok (subtasks 1 and 3) or sid × 2000 + slot × 1000 + tok (subtask 2).
- Scoring
- Per subtask, points = round(clamp((metric − floor)/(ceil − floor), 0, 1) × max). Subtask 1: pooled token-F1, floor 0.32, ceiling 0.90, 30 pts. Subtask 2: mean per-sentence pair score (rationale match × Jaccard distance), floor 0.25, ceiling 0.85, 35 pts. Subtask 3: token-F1 against the consensus set, floor 0.35, ceiling 0.90, 35 pts.
- Rules
- The classifier is frozen and may not be retrained.
- Nothing uploaded is executed; the CSV is compared with hidden ground truth.
- Submission limit 30, two final submissions (platform settings).
- Format
- Contest Day 1, 16 July 2026, 5 hours (10:30–15:30 local time; Nitro AI Judge 07:30–12:32 UTC), on-site at CREIC, Cluj-Napoca; three tasks, 100 points each.