# Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions Source: [Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions]() Collection index: [llms.txt]() Data updated: 2025-11-20T05:01:37.411Z Authors: Hubert Baniecki, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, Eyke H\\"ullermeier, Przemyslaw Biecek Published: 2025-08-07T14:18:56.000Z ## Abstract Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the insertion/deletion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-1k benchmarks validate that second-order methods, such as FIxLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIxLIP in comparing different models, e.g. CLIP vs. SigLIP-2. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Interaction Recognition on ImageNet-1k 1 object](): Interaction Recognition; ImageNet-1k 1 object; Interaction Recognition: 87. - [Interaction Recognition on ImageNet-1k 2 objects](): Interaction Recognition; ImageNet-1k 2 objects; Interaction Recognition: 87. - [Interaction Recognition on ImageNet-1k 4 objects](): Interaction Recognition; ImageNet-1k 4 objects; Interaction Recognition Accuracy: 88. - [Interaction Recognition on ImageNet 1k (3 objects)](): Interaction Recognition; ImageNet 1k (3 objects); Interaction Recognition Accuracy: 87. ## Original sources and implementation [Original paper]() [PDF]()