# Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift

Source: [Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift](<https://www.sota2.com/research/paper/68fa7a1c1e209131cea813f7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:12:23.158Z

Authors: Behraj Khan, Tahir Qasim Syed, Nouman M. Durrani, Bilal Naseem, Shabir Ahmad, Rizwan Qureshi

Published: 2025-07-12T09:39:07.000Z

## Abstract

Foundation models like CLIP and SAM have advanced computer vision and medical imaging via low-shot transfer learning, aiding CADD with limited data. However, their deployment faces two key challenges. \\textit{distribution shift} where pre-training and post-training data distributions differ (e.g., due to inter-center image acquisition) and \\textit{confidence misalignment}, which leads to overconfident errors. These issues surface differently, vision-language models (e.g., CLIP) suffer from 2D embedding shift (image-text misalignment), while medical models (e.g., SAM) encounter 3D domain shifts (e.g., scanner variation) and voxel-wise calibration need. Existing solutions are domain-specific. We propose \\textbf{StaRFM}, a fusion of Fisher information penalty (FIP) and confidence misalignment penalty (CMP) tackling both challenges. It applies FIP, extended to 3D via patch-wise regularization, to reduce embedding shift, and CMP, reformulated for voxel-level predictions, to calibrate segmentation uncertainty. We derive PAC-Bayes bounds. FIP controls generalization via the Fisher-Rao norm, and CMP reduces calibration error via Brier score minimization. StaRFM surpasses baselines by \\texttt{+}3.5\\% accuracy and 28\\% lower ECE on 19 vision datasets (e.g., ImageNet, Office-Home), achieves +4.2\\% DSC over SAM-FT and 4.8mm HD95 on medical benchmarks (e.g., BraTS, ATLAS), and reduces cross-domain gaps by up to 20\\%. The framework is plug-and-play, requiring minimal architectural changes. Code and models are available at: \\href{https://anonymous.4open.science/r/StaRFM-C0CD/}{\\textcolor{blue}{\\underline{StaRFM}}}

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.09222>)

[PDF](<https://arxiv.org/pdf/2507.09222.pdf>)
