# Classify Gases from Multimodal Information: Official solution summary *English translation by SOTA – AI Community of the Japanese original, shared under CC BY-NC-SA 3.0 IGO like the original. Organisers who would like this translation removed can email sota.ai.community@gmail.com.* *Translation of the article "JOAI2025 公式解法解説：上位解法に共通して見られた工夫まとめ" (JOAI2025 official solution commentary: a summary of the techniques commonly seen in the top solutions), published by the JOAI committee on ioai-japan.org on 23 December 2025.* **Contents** - Introduction - 1. Recap of the problem setting - 2. Overall strategy - 2.1 Making the local score trustworthy - 2.2 Models and pipelines used - 2.3 What the top solutions had in common - 3. Techniques by modality - 3.1 Table (sensors: MQ5/MQ8) - 3.2 Text (captions) - Pattern A: extraction with regular expressions (temperature, colour, coordinates) - Pattern B: extracting embeddings from DeBERTa / RoBERTa - 3.3 Images (infrared) - 4. Other techniques - Conclusion ## Introduction This article surveys several write-ups of top solutions submitted in JOAI 2025 and aims to organise "the techniques commonly seen among many participants". Rather than detailed tuning of individual hyperparameters and the like, it focuses on the techniques that tended to lead to winning. Note: a Playground competition page in the same format as the live JOAI2025, [[Playground] JOAI Competition 2025](https://www.kaggle.com/competitions/playground-joai-competition-2025), has been published. Note: [tutorial code for beginners](https://www.kaggle.com/competitions/playground-joai-competition-2025/code) has been published. A [recording of a tutorial session](https://www.youtube.com/watch?v=kceL-FIzcCY) based on it has also been published. --- ## 1. Recap of the problem setting Each sample is given the following, and the gas state is classified into 4 classes (NoGas / Perfume / Smoke / Mixture). - **Sensor values**: MQ5, MQ8 - **Infrared images**: thermographic images - **Text**: image captions (which tend to contain fixed expressions for temperature, colour, position, etc.) The evaluation uses the F1 score. The task is therefore designed so that not only "strengthening only the classes that are easy to predict" but also reducing misses on the difficult classes tends to improve the score. --- ## 2. Overall strategy ### 2.1 Making the local score trustworthy When performing cross-validation (CV), some participants split the data with a method called Stratified K-Fold so that the 4 classes were evenly represented. Using such a method makes the local evaluation reliable. ### 2.2 Models and pipelines used Even among the top solutions, a variety of models was seen, such as: - training and predicting with ResNet or ViT on the images alone - converting each modality into tabular features and training and predicting with models such as LightGBM - training and predicting with a single multimodal deep model built to handle all modalities Almost all solutions finally ensembled or stacked the predictions produced above: - taking the mean or weighted mean of the output probabilities of the models - stacking with a meta-model such as LightGBM ### 2.3 What the top solutions had in common Because the sensor (MQ5/MQ8), image and caption information are each important, the participants who placed near the top all combined several modalities in some way for their final predictions. Also, partly because the competition for the top scores was fierce, almost everyone used some kind of ensembling or stacking method. --- ## 3. Techniques by modality ### 3.1 Table (sensors: MQ5/MQ8) Because these were important features, almost all solutions used them. For ease of handling, many solutions fed them, together with information obtained from the other modalities, into gradient boosting models such as LightGBM, XGBoost and CatBoost. There were also solutions that trained and ran inference on them together with the information from the other modalities in a single multimodal deep model. In addition, as feature engineering, some participants fed in the following derived features: - **Raw values**: MQ5, MQ8 - **Interactions**: sum / diff / ratio / product / log-ratio - **Binning**: quantile bucket (e.g. turning values into categories with qcut) ### 3.2 Text (captions) The captions contain information such as temperature ranges, colours, positions and coordinates. One of the following was common among the top submissions. #### Pattern A: extraction with regular expressions (temperature, colour, coordinates) - extracting temperatures (min/max/diff), unifying ℃/℉ and converting them to numbers - occurrence of colour names (has_red, etc.) and the number of colours (color_count) - presence and values of coordinates (but handle these with care, as noted below) This approach used few computing resources, tended to work reliably, and was easy to add as tabular data to models such as LightGBM. "Temperature" in particular was a frequently used feature. #### Pattern B: extracting embeddings from DeBERTa / RoBERTa Many participants fed embeddings obtained from the final layer of models such as DeBERTa / RoBERTa into LightGBM or into a multimodal deep learning model. ### 3.3 Images (infrared) Approaches such as the following were seen: - fine-tuning image models such as ResNet or ViT for training and prediction - feeding embeddings extracted with ResNet, CLIP, etc., or statistics computed after a temperature-conversion step, into LightGBM or similar models as tabular data When converting the image data into tabular data, some participants also compressed the features by converting to temperature → averaging and reshaping to 4x4. --- ## 4. Other techniques - When using a single multimodal deep learning model, some participants improved performance by using a layer called [FiLM, which conditions and modulates the image with language](https://arxiv.org/abs/2212.06817). - Rule-based post-processing: texts that contain ℃ but not ℉ are assigned NoGas or Perfume. - Some participants used the similarity between the image and the caption, computed with CLIP, as a feature. --- ## Conclusion When setting the task, we designed it so that solutions combining several of the table, image and text sources of information would rank near the top. As intended, most of the top solutions used all of this information, but the contest turned out to be of a higher level than we expected. We hope this will be useful for reviewing JOAI2025 and for preparing for future JOAI competitions. For preparing for JOAI, please also see "[How to prepare and study for the Olympiad in Artificial Intelligence (IOAI/JOAI)](https://ioai-japan.org/preparation/)". *(Translator's note: section 3.2 refers to a caution about coordinates "noted below", which the original article does not give.)*