# A Beginner's Mistake: Official Solution
*English translation by SOTA – AI Community of the Russian original. Organisers who would like this translation removed can email sota.ai.community@gmail.com.*
All-Russian School Olympiad in Informatics 2025–2026, Final Stage
“Artificial Intelligence” profile, Tour 2, Moscow, 25 March 2026
Task C
*Translator's note: the official solution is the authors' Jupyter notebook exported to PDF. Code cells are marked “In [n]”, their results “Out [n]”, and printed output follows the cell. Lines that the PDF wrapped are joined again. Comments, docstrings and messages are translated; the code itself is unchanged. Figures are described in words; see the original PDF for the images.*
In [1]:
```python
import json
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from catboost import CatBoostClassifier
import warnings
from IPython.display import display
warnings.filterwarnings('ignore')
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
# Fix the seed for reproducibility of all steps
SEED = 42
np.random.seed(SEED)
# Load the available files
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
solution = pd.read_csv('ytest.csv')
with open('map.json', 'r') as f:
map_data = json.load(f)
display(train.head())
display(test.head())
```
```text
longitude latitude target habitat_quality_score biodiversity_index \
0 -68.624132 -32.868160 0 38.746018 0.869221
1 -51.964445 1.978971 0 17.456343 3.414865
2 -50.585866 -5.300425 0 43.883046 1.930266
3 -64.666741 -32.100087 0 29.736058 1.724559
4 -72.512342 -5.718456 1 66.547123 9.257867
canopy_density soil_moisture_level forest_age_years \
0 0.584679 6.031727 97.911830
1 0.327984 4.694215 169.739698
2 0.750885 7.864357 214.602112
3 0.322355 4.448480 275.153342
4 0.428812 3.822563 179.540444
tree_density_per_hectare annual_rainfall_mm habitat_fragmentation_index \
0 396.493164 1293.066100 0.507221
1 350.579890 2353.814818 0.247614
2 237.872776 1147.090078 0.574802
3 458.725205 1252.207036 0.648601
4 346.391230 604.782917 0.732562
predator_pressure_score vegetation_complexity forest_type \
0 1.040142 3.0 tropical_rainforest
1 7.522185 9.0 dry_forest
2 3.255146 4.0 dry_forest
3 6.404152 9.0 tropical_rainforest
4 4.880406 2.0 dry_forest
climate_zone soil_type disturbance_level conservation_status \
0 montane peat low unprotected
1 temperate loam minimal unprotected
2 subtropical loam moderate unprotected
3 temperate loam low buffer_zone
4 tropical clay low unprotected
dominant_tree_species water_source_type temperature_variability \
0 palm stream 4.037666
1 pine none 5.882945
2 palm seasonal 3.366330
3 mahogany stream 8.984897
4 bamboo seasonal 3.966467
elevation_range human_activity_index canopy_height_avg \
0 730.654928 0.075256 6.710190
1 316.780429 3.614124 30.770752
2 474.802210 2.251489 5.000000
3 187.292371 3.063365 16.284970
4 191.639490 3.888626 36.796504
seasonal_variation_score
0 6.874974
1 6.384739
2 5.690438
3 6.328448
4 5.780874
```
```text
longitude latitude habitat_quality_score biodiversity_index \
0 -72.102869 19.626859 45.511580 3.743753
1 -50.441167 -8.396785 34.751298 4.712511
2 -58.372430 0.312577 73.162533 2.579567
3 -76.362920 -14.279388 86.832262 4.097908
4 -67.162159 6.450007 63.631579 4.895632
canopy_density soil_moisture_level forest_age_years \
0 0.648016 3.260149 172.921350
1 0.226638 5.191137 357.568190
2 0.747227 7.533181 300.220431
3 0.602197 6.429846 325.612904
4 1.000000 4.502920 308.714905
tree_density_per_hectare annual_rainfall_mm habitat_fragmentation_index \
0 226.157888 1177.791799 0.558007
1 429.731562 1922.536202 0.332591
2 144.840477 1844.822680 0.654825
3 323.532762 1571.667498 0.383862
4 693.209698 1688.758358 0.344897
predator_pressure_score vegetation_complexity forest_type \
0 7.795448 3.0 secondary_forest
1 5.712367 7.0 dry_forest
2 2.073700 1.0 cloud_forest
3 0.798348 5.0 dry_forest
4 1.378641 1.0 tropical_rainforest
climate_zone soil_type disturbance_level conservation_status \
0 montane laterite high unprotected
1 temperate sandy moderate protected
2 subtropical loam high protected
3 tropical alluvial minimal buffer_zone
4 subtropical sandy minimal protected
dominant_tree_species water_source_type temperature_variability \
0 kapok stream 5.205539
1 palm seasonal 8.634001
2 pine none 5.382953
3 pine stream 8.282785
4 pine stream 10.690082
elevation_range human_activity_index canopy_height_avg \
0 318.967595 3.626041 23.686628
1 184.900374 1.830241 28.697284
2 536.571256 0.074063 26.366329
3 649.036995 0.000000 34.803212
4 388.522453 4.500977 45.697464
seasonal_variation_score ID
0 6.492307 0
1 4.313709 1
2 3.480140 2
3 3.417066 3
4 6.823274 4
```
## 1.4 Visualisation function
Let us create a helper function for visualising points on a map with the coastline contour (`map.json`).
In [2]:
```python
def plot_map(points, coastline, color_by=None, figsize=(5, 5)):
"""
Visualises points on a map with the coastline contour.
Parameters:
----------
points : pd.DataFrame
DataFrame with the points; must contain the columns 'longitude' and 'latitude'
coastline : list
List of coastline coordinates [[lon1, lat1], [lon2, lat2], ...]
color_by : str, optional
Name of the column used to colour the points. If None, all points have the same colour
figsize : tuple, optional
Figure size (width, height)
Example of use:
--------------------
# Simple visualisation without colouring
plot_map(train, coastline)
# Coloured by the target
plot_map(train, coastline, color_by='target')
# Changing the size
plot_map(train, coastline, color_by='target', figsize=(16, 10))
"""
fig, ax = plt.subplots(figsize=figsize)
# Draw the coastline
if coastline:
coastline_array = np.array(coastline)
# Draw all coastline points as a dense scatter with square markers
ax.scatter(coastline_array[:, 0], coastline_array[:, 1],
s=1.5, marker='s', color='#CCCCCC', alpha=0.7, label='Coastline')
# Draw the points
if color_by is None:
# No colouring: all points have the same colour
ax.scatter(points['longitude'], points['latitude'],
s=50, alpha=0.6, edgecolors='black', linewidths=0.5,
label='Points')
else:
# Coloured by the given column
if points[color_by].dtype in ['object', 'category']:
# Categorical variable
unique_vals = points[color_by].unique()
colors = plt.cm.tab10(np.linspace(0, 1, len(unique_vals)))
for i, val in enumerate(unique_vals):
mask = points[color_by] == val
ax.scatter(points[mask]['longitude'], points[mask]['latitude'],
s=50, alpha=0.6, edgecolors='black', linewidths=0.5,
label=f'{color_by}={val}', color=colors[i])
else:
# Numerical variable
scatter = ax.scatter(points['longitude'], points['latitude'],
c=points[color_by], cmap='RdYlGn',
s=50, alpha=0.6, edgecolors='black', linewidths=0.5)
plt.colorbar(scatter, ax=ax, label=color_by)
ax.set_xlabel('Longitude')
ax.set_ylabel('Latitude')
ax.set_title(f'Map of points (n={len(points)})')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
print("The plot_map() function is ready to use")
```
```text
The plot_map() function is ready to use
```
## 1.5 Demonstration of the visualisation function
Let us try the function on the loaded data.
In [3]:
```python
# Get the coastline from map.json
coastline = map_data.get('coastline', [])
# Example 1: Train without colouring
plot_map(train, coastline)
# Example 2: Train coloured by the target
plot_map(train, coastline, color_by='target')
```
*[Figure: see page 13 of the [original solutions PDF](https://vos.olimpiada.ru/upload/files/Arhive_tasks/2025-26/final/ai/sol-ai-9-11-tur2-final-25-26.pdf). A map titled "Map of points (n=502)": the grey coastline of South America (longitude from about −95 to −35, latitude from about −55 to 25) with the 502 training points drawn as blue dots; legend "Coastline", "Points"; axes Longitude and Latitude.]*
*[Figure: see page 14 of the [original solutions PDF](https://vos.olimpiada.ru/upload/files/Arhive_tasks/2025-26/final/ai/sol-ai-9-11-tur2-final-25-26.pdf). The same map, "Map of points (n=502)", with the points coloured by `target` on a red-yellow-green colour bar from 0.0 (red) to 1.0 (green); legend "Coastline".]*
## 1.6 Step 1: A first baseline approach
According to the statement, we have `train.csv` (direct observations) and `test.csv`, in which some of the points were added from satellite data.
First, we build an honest baseline on all features and look at the `F1 score` on the labels from `ytest.csv`. This is needed as a checkpoint, to understand how prone the model is to overfitting in the original setting.
In [4]:
```python
# Function for evaluating quality
def evaluate(y_pred, solution):
"""F1-score evaluation: overall, public, private"""
from sklearn.metrics import f1_score
merged = solution.copy()
merged['pred'] = np.asarray(y_pred).reshape(-1)
# Compute the metric on the already rounded predictions (0/1)
merged['pred_label'] = np.clip(np.rint(merged['pred']), 0, 1).astype(int)
public_mask = merged['Usage'] == 'public'
private_mask = merged['Usage'] == 'private'
overall_f1 = f1_score(merged['target'], merged['pred_label'])
public_f1 = f1_score(merged.loc[public_mask, 'target'], merged.loc[public_mask, 'pred_label'])
private_f1 = f1_score(merged.loc[private_mask, 'target'], merged.loc[private_mask, 'pred_label'])
print(f"F1 overall: {overall_f1:.6f}")
print(f"F1 public: {public_f1:.6f}")
print(f"F1 private: {private_f1:.6f}")
return public_f1, private_f1, overall_f1
# Select the features (all except the service columns)
feature_cols_numeric = [col for col in train.columns
if col not in ['ID', 'target', 'cluster', 'is_train']
and train[col].dtype in ['int64', 'float64']]
categorical_cols = train.select_dtypes(include=['object']).columns.tolist()
categorical_cols = [col for col in categorical_cols if col not in ['ID']]
all_features = feature_cols_numeric + categorical_cols
X_train = train[all_features]
y_train = train['target']
X_test = test[all_features]
from sklearn.model_selection import train_test_split
X_train_tr, X_train_val, y_train_tr, y_train_val = train_test_split(
X_train, y_train, test_size=0.2, random_state=SEED, stratify=y_train
)
model = CatBoostClassifier(
iterations=500,
learning_rate=0.1,
depth=12,
random_seed=SEED,
verbose=0,
)
model.fit(X_train_tr, y_train_tr, cat_features=categorical_cols)
y_pred_test_baseline = np.asarray(model.predict(X_test)).reshape(-1).astype(int)
baseline_public, baseline_private, baseline_overall = evaluate(y_pred_test_baseline, solution)
```
```text
F1 overall: 0.646440
F1 public: 0.645707
F1 private: 0.647208
```
In [5]:
```python
from sklearn.metrics import f1_score
f1_score(solution.target.values, (solution.target.values * 0 + 0.5 >= 0.5).astype(int))
```
Out [5]:
```text
0.5270917803169922
```
## 1.7 Step 2: Feature importance analysis
We relate the result of the baseline to the task statement. If the coordinates (`longitude`, `latitude`) turn out to be at the top of the importance ranking, the model may be using a geographic "hint" instead of robust ecological patterns. This is exactly the first typical beginner's mistake: implicit leakage/spurious correlation through features that encode the split too directly.
In [6]:
```python
# Feature importance
importance_df = pd.DataFrame({
'feature': all_features,
'importance': model.get_feature_importance()
}).sort_values('importance', ascending=False)
print("\nTop 10 features:")
print(importance_df.head(10).to_string(index=False))
print("This may mean that the model is overfitting to the coordinates.")
print("\nWARNING: longitude and latitude are at the top of the importance ranking!")
```
```text
Top 10 features:
feature importance
latitude 25.842854
habitat_quality_score 18.788004
biodiversity_index 12.708726
longitude 9.644228
forest_type 6.298411
disturbance_level 5.888199
climate_zone 2.864203
conservation_status 1.980591
seasonal_variation_score 1.594778
human_activity_index 1.516413
This may mean that the model is overfitting to the coordinates.
WARNING: longitude and latitude are at the top of the importance ranking!
```
## 1.8 Step 3: Visualising the coordinates
Let us look at how the classes are distributed in coordinate space.
In [7]:
```python
# Use our function for visualisation
print("Train data coloured by the target:")
plot_map(train, coastline, color_by='target', figsize=(10, 8))
# Analysis of the class distribution
print("\n" + "="*70)
print("COORDINATE ANALYSIS")
print("="*70)
# Statistics by longitude
print("\nDistribution of the classes by longitude:")
for target_val in [0, 1]:
subset = train[train['target'] == target_val]
print(f"\nClass {target_val}:")
print(f" Longitude: min={subset['longitude'].min():.2f}, max={subset['longitude'].max():.2f}, mean={subset['longitude'].mean():.2f}")
print(f" Latitude: min={subset['latitude'].min():.2f}, max={subset['latitude'].max():.2f}, mean={subset['latitude'].mean():.2f}")
print("\n" + "="*70)
print("CONCLUSION: The classes are clearly separated by the coordinates!")
print(" The model has memorised the geographic boundary between the classes.")
print(" In the real test, this pattern does not hold.")
print(" SOLUTION: longitude and latitude must be removed from the features.")
print("="*70)
```
```text
Train data coloured by the target:
```
*[Figure: see page 17 of the [original solutions PDF](https://vos.olimpiada.ru/upload/files/Arhive_tasks/2025-26/final/ai/sol-ai-9-11-tur2-final-25-26.pdf). A larger version of the map "Map of points (n=502)" with the 502 training points coloured by `target` (red = 0, green = 1; colour bar "target" from 0.0 to 1.0); legend "Coastline". Most green points lie in a band between latitudes of about 0 and −30; red points appear both inside this band and in the north and south of the continent.]*
```text
======================================================================
COORDINATE ANALYSIS
======================================================================
Distribution of the classes by longitude:
Class 0:
Longitude: min=-91.65, max=-39.05, mean=-64.11
Latitude: min=-55.12, max=22.89, mean=-10.67
Class 1:
Longitude: min=-90.20, max=-38.22, mean=-62.39
Latitude: min=-29.95, max=-0.10, mean=-15.78
======================================================================
CONCLUSION: The classes are clearly separated by the coordinates!
The model has memorised the geographic boundary between the classes.
In the real test, this pattern does not hold.
SOLUTION: longitude and latitude must be removed from the features.
======================================================================
```
## 1.9 Step 4: Removing the coordinates and retraining the model
We fix the first mistake: we remove `longitude` and `latitude` so that the model relies on meaningful environmental features. We compute the `F1 score` again and compare it with the baseline. If the quality improves, the coordinates really were hindering generalisation.
In [8]:
```python
# Remove the coordinates
features_no_coords = [f for f in all_features if f not in ['longitude', 'latitude']]
categorical_cols_filtered = [c for c in categorical_cols if c in features_no_coords]
X_train_no_coords = train[features_no_coords]
X_test_no_coords = test[features_no_coords]
X_train_tr_no_coords, X_train_val_no_coords, y_train_tr_no_coords, y_train_val_no_coords = train_test_split(
X_train_no_coords, y_train, test_size=0.2, random_state=SEED, stratify=y_train
)
model.fit(X_train_tr_no_coords, y_train_tr_no_coords, cat_features=categorical_cols_filtered)
y_pred_test_no_coords = np.asarray(model.predict(X_test_no_coords)).reshape(-1).astype(int)
nocoords_public, nocoords_private, nocoords_overall = evaluate(y_pred_test_no_coords, solution)
```
```text
F1 overall: 0.783972
F1 public: 0.798155
F1 private: 0.769591
```
## 1.10 Step 5: A train vs test detector (searching for distribution shift)
We check the second beginner's mistake from the statement: ignoring the fact that `train` and part of `test` were collected in different ways. If a model easily distinguishes `train` from `test`, this is a sign of domain shift (the distributions differ). Then, for some of the test points, a standard model may be overconfident and make systematic errors.
In [9]:
```python
from catboost import cv, Pool
# Create a dataset for classifying train vs test
X_combined = pd.concat([X_train_no_coords, X_test_no_coords], axis=0)
y_combined = np.concatenate([np.ones(len(X_train_no_coords)), np.zeros(len(X_test_no_coords))])
model_ood = CatBoostClassifier(
iterations=50,
learning_rate=0.1,
depth=6,
random_seed=SEED,
verbose=0
)
cv_data = Pool(data=X_combined, label=y_combined, cat_features=categorical_cols_filtered)
_ = cv(
pool=cv_data,
params={
'iterations': 50,
'learning_rate': 0.1,
'depth': 6,
'loss_function': 'Logloss',
'eval_metric': 'AUC',
'random_seed': SEED,
'verbose': False
},
fold_count=5,
shuffle=True,
stratified=True,
partition_random_seed=SEED
)
model_ood.fit(X_combined, y_combined, cat_features=categorical_cols_filtered)
test_similarity_scores = model_ood.predict(X_test_no_coords, prediction_type='Probability')[:, 1]
plt.figure(figsize=(10, 5))
plt.hist(test_similarity_scores, bins=100, alpha=0.7, edgecolor='black')
plt.xlabel('Probability of belonging to train')
plt.ylabel('Number of objects')
plt.title('Train/Test Classifier: distribution of the probabilities on test')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
```
```text
Training on fold [0/5]
bestTest = 0.6328465347
bestIteration = 9
Training on fold [1/5]
bestTest = 0.6250990099
bestIteration = 4
Training on fold [2/5]
bestTest = 0.6888
bestIteration = 6
Training on fold [3/5]
bestTest = 0.6909022556
bestIteration = 11
Training on fold [4/5]
bestTest = 0.6967669173
bestIteration = 16
```
*[Figure: see page 20 of the [original solutions PDF](https://vos.olimpiada.ru/upload/files/Arhive_tasks/2025-26/final/ai/sol-ai-9-11-tur2-final-25-26.pdf). A histogram (100 bins) titled "Train/Test Classifier: distribution of the probabilities on test"; x axis "Probability of belonging to train" (from 0.0 to about 0.43), y axis "Number of objects". There is a tall, narrow peak just above 0 (up to about 80 objects per bin at around 0.01–0.02), a second, broad mode between about 0.12 and 0.32 (roughly 30–50 objects per bin), and only a few objects above 0.35.]*
## 1.11 Step 6: OOD correction of the predictions
We use the train/test detector as an indicator of OOD points (how similar a test point is to train). For points with very low similarity, we lower the probability of class 1 and again compute only the `F1 score`.
In this way we address the second mistake: we take the distribution shift into account and make the predictions more robust on the "satellite" part of the test.
In [10]:
```python
print("\nCorrecting the predictions for OOD objects...")
print("="*70)
# OOD threshold: points with low similarity to train are considered shifted in distribution
ood_threshold = 0.07
y_pred_corrected = y_pred_test_no_coords.copy()
ood_mask = test_similarity_scores < ood_threshold
y_pred_corrected[ood_mask] = 0
# Print the F1-score for each approach: overall/public/private
_, _, _ = evaluate(y_pred_test_baseline, solution)
_, _, _ = evaluate(y_pred_test_no_coords, solution)
_, _, corrected_overall = evaluate(y_pred_corrected, solution)
```
```text
Correcting the predictions for OOD objects...
======================================================================
F1 overall: 0.646440
F1 public: 0.645707
F1 private: 0.647208
F1 overall: 0.783972
F1 public: 0.798155
F1 private: 0.769591
F1 overall: 0.918549
F1 public: 0.918340
F1 private: 0.918768
```
In [11]:
```python
# Save the submission with the best available predictions
final_pred = y_pred_corrected
submission = pd.DataFrame({
'ID': test['ID'].values,
'target': (final_pred > 0.5) * 1
})
submission.to_csv('author_submission.csv', index=False)
```