# Probing Image-Language Transformers for Verb Understanding Source: [Probing Image-Language Transformers for Verb Understanding]() Collection index: [llms.txt]() Data updated: 2025-10-25T23:17:37.192Z Authors: Lisa Anne Hendricks, Aida Nematzadeh Published: 2021-06-16T21:36:36.000Z ## Abstract Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Verb Understanding on SVO-probes Entire Dataset and Verb-specialized](): Verb Understanding; SVO-probes Entire Dataset and Verb-specialized; AP: 51.5. ## Original sources and implementation [Original paper]() [PDF]()