Checklist ROAI 2024 National Stage · CV task
ROAI TransportationCo
Fine-tune a small text-to-image diffusion model so that it can also generate boats and aircraft, and publish it to the Hugging Face Hub.
The task
A transport company owns a generative model that creates images of its vehicles from text prompts, but the model has only seen buses, trucks and trains. After acquiring boats and aircraft, the marketing team wants the model to generate those classes from English prompts as well.
Contestants read an introductory notebook on the diffusers library, then fine-tune the UNet and/or VAE of the provided mini Stable Diffusion pipeline on the company's image–caption dataset, write an explanation of at most 300 words and push the model to the Hugging Face Hub. An evaluation notebook shows how submissions are judged with an object detector.
Abridged by SOTA from the official materials. The official statement has the exact rules, and it wins wherever this summary differs.
At a glance
- You get
- Dataset unibuc-cs/ROAItransportationCo (Hugging Face), base pipeline unibuc-cs/ROAITransportationCo_v2, and three notebooks: Intro, Task and EVAL.
- You submit
- A fine-tuned diffusion pipeline pushed to the Hugging Face Hub, plus the written explanation.
- Scoring
- Accuracy judged by an object detector on generated images (the detector should see the expected vehicle and not others): 50% of the score from the mentioned vehicle labels and 50% from other, non-vehicle objects (e.g. a football stadium, a building). The final standing uses a similar script with modifications to reduce detector imprecision.
- Rules
- The tokenizer and text encoder stay frozen; the UNet, the VAE or both may be trained
- Google Colab GPU (L4 recommended)
- Format
- ROAI 2024 national stage, 12 July 2024; alternative to the NLP problem.