# Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge Source: [Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge]() Collection index: [llms.txt]() Data updated: 2026-01-02T08:33:12.301Z Authors: Fawaz Sammani, Nikos Deligiannis Published: 2024-10-16T20:18:21.000Z ## Abstract Contrastive Language-Image Pretraining (CLIP) performs zero-shot image classification by mapping images and textual class representation into a shared embedding space, then retrieving the class closest to the image. This work provides a new approach for interpreting CLIP models for image classification from the lens of mutual knowledge between the two modalities. Specifically, we ask: what concepts do both vision and language CLIP encoders learn in common that influence the joint embedding space, causing points to be closer or further apart? We answer this question via an approach of textual concept-based explanations, showing their effectiveness, and perform an analysis encompassing a pool of 13 CLIP models varying in architecture, size and pretraining datasets. We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions. Our approach demonstrates an effective and human-friendly way of understanding zero-shot classification decisions with CLIP. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on ImageNet 1k (val) (Accuracy & Accuracy Gain)](): Image Classification; ImageNet-1k (val); Accuracy: 77.64. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()