Trying to Teach MobileNetV4 with SigLIP 2

Open the notebook in Colab · Try the Hugging Face Space

What I wanted to try

This started as a notebook of me learning to work with Hugging Face Spaces. I wanted to build a pet breed classifier, but before worrying about the Space itself, I wanted to try something with the training.

Could I use SigLIP 2 to help train MobileNetV4? Or at least give it a go : )

The idea is knowledge distillation. I have a teacher that makes predictions, and a smaller student that learns from those predictions alongside the dataset labels. Once the student is trained, it can make predictions on its own.

I'm comparing two versions of MobileNetV4: one trained using the labels, and another using the labels plus the teacher's predictions. Both start from pretrained weights. I want to see whether the extra information actually helps.

The short answer from this run is: a little on test accuracy. The more interesting difference showed up when I looked at the probabilities.

Loading the pets

I'm using the Oxford-IIIT Pet dataset, which gives me 37 breeds of cats and dogs to work with.

I split the original training/validation data into 80% training and 20% validation, keeping the class proportions similar. That leaves 2,944 training images and 736 validation images. The official test split has 3,669 images, and I keep those separate from fitting models or choosing settings. I also check the image IDs for overlap between splits.

Everything gets resized to 224 × 224. In the full experiment, I save eight views of each training image: one plain resize and seven versions with crops and possible horizontal flips. Validation and test images only get the plain resized view.

Saving the views matters because I'm going to save the teacher's predictions too. If the teacher saw one crop and the student saw a different one, I'd be asking the student to match a prediction made from different pixels.

So each saved prediction belongs to an image ID and a view ID. During student training, I use those same IDs to retrieve it:

for imgs, labels, ids, views in loader("train", epoch=epoch):
    student_logits = model(normalize(imgs, MEAN, STD))
    teacher_targets = teacher_logit[
        ids.numpy(), views.numpy()
    ].to(DEVICE)

This is just the matching part of the training loop. Each image gets one view per epoch, cycling through the saved views as training continues. The teacher and student use their own normalization settings, but the crop and flip are the same.

Open full size ↗

Setting up the teacher

The teacher uses google/siglip2-base-patch16-224. I'm freezing its image encoder, extracting image features, and training a small linear classifier on top to predict the 37 breeds.

So when I say “SigLIP 2 teacher” here, I mean that frozen encoder plus my trained classifier. I'm not fine-tuning the whole SigLIP 2 model or asking its text side to match breed names.

I extract features for all the saved training views and the single validation view. Then I train the classifier for 100 epochs, keeping the weights with the best validation macro-F1. Macro-F1 averages the F1 scores across breeds, giving each breed equal weight in that average.

In the saved run, the teacher reached about 96.76% validation macro-F1.

Once the classifier is selected, I save its raw prediction scores, or logits, for every training image and view. That lets me use the teacher's predictions throughout student training without running the image encoder again for every batch.

I'm also saving the model revisions, settings, and data splits. There are enough moving parts here that I want to be able to check what a run used afterward.

What the student learns

The student is MobileNetV4 Conv Small, using the pretrained mobilenetv4_conv_small.e2400_r224_in1k backbone. I add dropout and a new classifier for the pet breeds, then train the student model.

For the baseline, the loss is cross-entropy against the labels, with label smoothing included in the settings. For the distilled version, I combine that with a loss that asks the student to match the teacher's probability distribution.

There are two settings I need to keep track of: alpha and temperature.

Alpha, $\alpha$, controls the balance between the label loss and the distillation loss. At $\alpha = 0$, I'm only using the labels. Increasing alpha gives the teacher-matching part more weight.

Temperature controls how much I soften the predictions before comparing them. Dividing logits by a temperature greater than one makes the resulting distribution less concentrated on its highest-scoring class. That gives the comparison more room to include the other breeds, instead of only the teacher's first choice.

For logits $z$ and temperature $T$, the softened probability of breed $c$ is:

I use this for both teacher and student. Their distributions are $p_t^{(T)}$ and $p_s^{(T)}$. Here $z_s$ holds the student's logits, and $i$ indexes the batch. With batch size $B$, labels $y$, and label smoothing $\varepsilon$, the loss is:

Here $\mathrm{CE}_{\varepsilon}$ is the batch-mean cross-entropy, mixing each label with a uniform distribution over 37 breeds by $\varepsilon$. The Kullback–Leibler divergence, $D_{\mathrm{KL}}$, measures how the distributions differ. Its direction is teacher to student. Only the student gets updated.

Open full size ↗

This is the core of the combined loss, shortened from the notebook:

ce = F.cross_entropy(
    student, labels, label_smoothing=label_smoothing
)
log_student = F.log_softmax(student / temperature, dim=-1)
log_teacher = F.log_softmax(teacher / temperature, dim=-1)
kd = F.kl_div(
    log_student, log_teacher,
    reduction="batchmean", log_target=True,
) * temperature ** 2
loss = (1.0 - alpha) * ce + alpha * kd

The KL term compares the two distributions, and the temperature-squared factor scales that term. These are excerpts from the training code, so they depend on the tensors and settings prepared elsewhere in the notebook.

Both students start from the same saved initial weights. That gives me a common starting point for comparing what happens during training.

Comparing the models

I have two comparisons in the notebook, and they answer slightly different questions.

First, there's the matched-settings comparison. I hold the other training settings fixed and train for 40 epochs with two alpha values: 0.0 for the control and about 0.7805 for distillation. Both use the same seed, initial weights, optimizer, and learning-rate schedule.

For these runs, I keep the checkpoint with the best validation accuracy. The control reached 90.35%, and the distilled model reached 91.58%. Using the unrounded scores, that's a gain of 1.22 percentage points.

I also compare which validation images each model gets right. The distilled model fixes 27 of the control's mistakes, but gets 18 images wrong that the control answered correctly. So the gain is nine more correct predictions overall. It isn't better on every example.

These fixed settings came from the distilled search result. This comparison tells me what happens when I change alpha under those settings; it doesn't tell me the best result each approach could reach.

For that, I also run separate Optuna studies. Each gets 12 trials with up to 24 epochs per trial, and less promising trials can stop early. Both tune learning rate, weight decay, dropout, and label smoothing. The distilled study also tunes alpha and temperature.

Here the selection metric is validation macro-F1. After the search, I train each selected configuration for 40 epochs, save its best validation macro-F1 checkpoint, and evaluate those checkpoints on the test set.

That means the test comparison includes differences in the selected training settings too. I can't attribute every difference between those models to distillation alone.

Open full size ↗

So, did it help?

On the 3,669 test images, the separately tuned models gave me these results:

Test accuracy improved by 0.11 percentage points, and macro-F1 improved by 0.17 points. That's a small change, especially next to the validation improvement in the matched comparison. These are different comparisons on different splits, so I need to keep them separate.

The probability metrics are more interesting to me. Negative log-likelihood, or NLL, looks at the probability assigned to the correct breed. Giving the correct answer a very low probability makes that score worse. Lower is better, and it dropped from 0.5747 to 0.4592.

For evaluation, $p_i$ uses ordinary softmax: $T=1$. Temperature softening belongs to the distillation loss. For $N$ images, $p_{i,y_i}$ is the probability assigned to image $i$'s true breed. I use the notebook's tiny probability floor:

For expected calibration error, or ECE, my code groups predictions into ten confidence bins. Within each bin, it compares average confidence with the fraction of predictions that are correct, then weights the gap by the number of examples. That score fell from 0.0781 to 0.0084.

Bin $S_m$ contains predictions with confidence in $((m-1)/10,m/10]$. Confidence is the highest breed probability, and $|S_m|$ counts the images in that bin. With $\mathrm{acc}$ and $\mathrm{conf}$ meaning the bin's average correctness and confidence, I skip empty bins:

So even though the distilled model barely changes how many images it gets right, its probability predictions look better by these measures. The ECE result depends on that binning, and a low overall score doesn't promise reliable confidence for every breed or every new photo.

This is still one seed. I'd want to repeat the experiment before making a bigger claim about how much distillation helps. For now, I've got a comparison that lets me look beyond the top prediction and ask how the model's confidence behaves too.

I finally deployed the models on Hugging Face Spaces, so you can try a pet photo and compare the baseline and distilled predictions yourself.

Open full size ↗