# Constrained Generation

*English translation by SOTA – AI Community of the Hungarian original. Licensed CC BY-NC-SA 4.0, like the original. Organisers who would like this translation removed can email sota.ai.community@gmail.com.*

**Magyar MI Diákolimpia** (Hungarian AI Olympiad)\
**Task description · Summer National Selection (Nyári Országos Válogató)**\
May 2026

*[Figure: see the original one-pager.]*

## 1. Situation report

In 1969, Georges Perec wrote his novel *La Disparition*: more than 300 pages without a single letter "e". Such deliberately constrained pieces of writing (lipograms) still fascinate linguists and writers' workshops today.

An experimental workshop would like to automate this. A small language model writes continuations for given prompts while never uttering a single forbidden word. Your task is to implement this constrained decoding. At every step the model proposes a next token (in the form of `logits`), and you decide which one goes into the output.

You face 20 experiments, each with a prompt on a different theme and a forbidden-word list of varying difficulty: from simple stop words (`the`, `a`, `an`) to bans on thematic word classes (in a detective text the entire vocabulary of violence is forbidden, in a programming description the trivial technical terms). The output must remain long, diverse and natural-sounding while respecting the constraint.

The organisations and events appearing in the task are fictitious (Perec and his work are not).

## 2. Constrained Generation

Causal language models produce text autoregressively: at every step they determine a probability distribution over the vocabulary (the *logits*), and then select the next token on the basis of this distribution. By modifying the logits, or by applying an alternative sampling rule or search procedure, the output of the model can be constrained without retraining.

You are given a small pretrained language model and a series of test cases. Each test case consists of an initial prompt, a list of forbidden words and the maximum number of tokens that may be generated. Your task is to determine, for each generation step, the identifier of the next token so that the final output does not contain a single forbidden word, while its length, diversity and naturalness are also preserved.

A word counts as forbidden in the output if it occurs in it as a separate word (delimited by word boundaries), without distinguishing between upper and lower case. For example, if `apple` is forbidden, then the occurrences `apple`, `Apple`, `APPLE`, `apple,` and `apple.` all violate the rule. `pineapple`, on the other hand, is allowed, since it is not a standalone occurrence of `apple`.

## 3. What you receive

A pretrained `Qwen/Qwen3-0.6B-Base` causal language model, accessed through the Hugging Face Transformers library, together with its tokenizer. During evaluation, the model is loaded with these same weights and package versions, so the output is deterministic across runs.

A function stub to be implemented, with the signature:

```
select_next_token(logits, prompt, forbidden, generated_ids) -> int
```

The meaning of the parameters:

- `logits` – the model's output logits for the last position (size: `[vocab_size]`),
- `prompt` – the initial text,
- `forbidden` – the list of forbidden words,
- `generated_ids` – the list of identifiers of the tokens selected so far.

The return value is the identifier of the next token as an integer, between 0 and `vocab_size`−1.

The attached notebook provides the complete evaluation framework (`generate_constrained`, `score_case`, `run_tests`), a `would_violate` helper function that indicates, according to the same rules as the evaluation, whether a given partial result contains a forbidden word, and a `make_submission()` helper function that produces the `submission.csv` to be submitted from the completed `select_next_token`.

The `test_cases.json` file contains the official test set (currently **20 test cases**) in the form of a list, each element of which is a test case with the fields `id`, `prompt`, `forbidden` and `max_new_tokens`. Scoring is carried out exclusively on this test set. The same file must also be used for generation.

## 4. What you submit

A single CSV file with the header `id,output,fluency,tokens`, containing exactly as many data rows as there are test cases in the `test_cases.json` file. The meaning of the columns:

- `id` – refers to the corresponding entry of `test_cases.json`.
- `output` – the generated text (without the prompt).
- `fluency` – the value $\Phi = \exp(\overline{\log p_\theta}/5)$ from the `Qwen3-0.6B-Base` model, computed on the full `(prompt + output)` text. The notebook's `score_case` function computes and fills in this value automatically.
- `tokens` – the length of `tokenizer.encode(output)`.

The file follows the standard CSV format (RFC 4180): fields containing a comma, a quotation mark or a line break must be enclosed in quotation marks, and quotation marks must be doubled. Using the attached `make_submission()` helper function for the submission is recommended. It handles the above rules automatically and writes the output with UTF-8 encoding.

The `fluency` and `tokens` fields are computed and filled in by the notebook's `make_submission()` function. They **must not be modified by hand**: the evaluator overrides the submitted values.

The output may be produced only with the `Qwen/Qwen3-0.6B-Base` model specified in the task. Using any other generative model results in immediate disqualification. Any decoding strategy of your own for the model is allowed, provided that it uses only the logits of the given model.

## 5. Scoring

Let $\mathcal{T}$ denote the test set and, for a given test case, let $p$ be the prompt, $\mathcal{F}$ the set of forbidden words, $M$ the allowed maximum number of tokens, and $y$ the generated output text. The score of a test case is determined as follows:

- if $y$ contains a forbidden word: **0 points**,
- if $y$ is empty: **0 points**,
- otherwise: $\text{score}(y) = L \cdot \sqrt{\text{distinct}_2} \cdot \Phi$, where $L$ is the length ratio, $\text{distinct}_2$ is the bigram diversity and $\Phi$ is the fluency.

**Per-test-case maximum:** 1.0 (if $L = \sqrt{\text{distinct}_2} = \Phi = 1$).

**Scoring factors.** The three factors are defined by the formulas below.

| Factor | Formula |
|:---:|:---|
| $L$ | $\displaystyle \min\!\left(1.0,\ \left(\frac{\text{number of tokens}}{M}\right)^{1.5}\right)$ |
| $\text{distinct}_2$ | $\displaystyle \frac{\lvert\{\text{unique word bigrams}\}\rvert}{\lvert\{\text{all word bigrams}\}\rvert}$ |
| $\Phi$ | $\displaystyle \exp\!\left(\frac{\overline{\log p_\theta}}{5}\right)$ |

The power of 1.5 on the length ratio penalises short outputs more strongly, while the upper bound $L \le 1.0$ prevents any reward beyond the maximum number of tokens. $\text{distinct}_2$ is computed as the proportion of unique bigrams in the lower-cased output split into words with the pattern `\w+`, so repeated word sequences lead to a deduction. Fluency is given by the model's own average token log-probability, so unnatural, improbable continuations also reduce the score.

**Final score.** The *average* of the per-test-case scores × 100 gives the final result of the submission. The scale is thus **[0, 100]** for any test-set size.

**Public and private subsets.** The 20 test cases are divided into two parts: **6 public cases** (the score visible during the contest is computed on these) and **14 private cases** (revealed only at the end of the contest; these decide the final ranking). `submission.csv` must contain the outputs of all 20 test cases; the split only affects how the score is displayed. The purpose of the split is to weaken over-tuning to the leaderboard during the contest (*leaderboard-grinding*).

## 6. Technical information

To solve the task, the attached `.ipynb` notebook and the `test_cases.json` file are available. The notebook contains the loading of the model and the tokenizer, the evaluation framework, and the `would_violate` and `make_submission` helper functions. The solver's job is to implement the `select_next_token` function, run the notebook on the `test_cases.json` test set, and submit the resulting `submission.csv`.

Only the designated cell (`select_next_token`) may be modified. Neither the model, nor the generation loop, nor any function of the scoring framework may be rewritten. The submitted notebook is subject to a post-hoc audit, and any manipulation that circumvents the scoring results in disqualification.

**A T4 GPU is recommended.** It also runs on a CPU, but considerably more slowly.

## 7. Useful resources

- [PyTorch documentation](https://pytorch.org/docs/stable/index.html)
- [Hugging Face Transformers documentation](https://huggingface.co/docs/transformers/index)
- [Qwen3-0.6B-Base model card](https://huggingface.co/Qwen/Qwen3-0.6B-Base)