Checklist USA-NA-AIO 2026 Round 2 · Task 1
Linear Attention Is All You Need
Derive linear attention with relative positional encoding, implement it in PyTorch and use it in a transformer for FordA time-series classification.
The task
Problem 1 (90 points, 'non open-ended') studies the cost of softmax self-attention, rewrites attention with a feature map φ so that it becomes linear in sequence length, explains why an exact feature map needs infinite dimension, and builds an explicit map from a second-order Taylor expansion.
It then introduces relative directional positional encodings r_{j−i}, their range and initialisation variance, and their use within linear attention. The contestant implements a multi-head module MyLinearAttentionRelativePos (feature map φ(x) = 1 + ELU(x)), counts its parameters, and builds an encoder-only transformer from scratch for the FordA dataset (automotive sensor time series of length 500, two classes).
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- FordA train/test CSV files and a starter notebook (Google Drive; data also in the usaaio-official/2026_USAAIO_Round2 dataset).
- You submit
- One notebook named Attn_LastName_FirstName_SchoolName, plus a CSV of test predictions (columns
test_sample_id, label) named FordA_LastName_FirstName_SchoolName. - Scoring
- Part 1.4 (20 points): macro F1; score = (X − X_baseline)/(X_best − X_baseline) × 100%, 0 if below the baseline (random guessing ≈ 0.5).
- Rules
- No nn.MultiheadAttention, nn.TransformerEncoder or Hugging Face models; no extra imports in Part 1.3.
- No additional positional encoding at the input.
- Training and testing on an L4 GPU; manual labelling of the test set scores 0.
- Format
- 2026 USA-NA-AIO Round 2, Day 1 (4 April 2026).