# ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition

Source: [ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition](<https://www.sota2.com/research/paper/68fa72db1e209131cea58efc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:35:49.064Z

Authors: Otto Brookes, Majid Mirmehdi, Hjalmar Kuhl, Tilo Burghardt

Published: 2024-04-13T09:17:51.000Z

## Abstract

We show that chimpanzee behaviour understanding from camera traps can be enhanced by providing visual architectures with access to an embedding of text descriptions that detail species behaviours. In particular, we present a vision-language model which employs multi-modal decoding of visual features extracted directly from camera trap videos to process query tokens representing behaviours and output class predictions. Query tokens are initialised using a standardised ethogram of chimpanzee behaviour, rather than using random or name-based initialisations. In addition, the effect of initialising query tokens using a masked language model fine-tuned on a text corpus of known behavioural patterns is explored. We evaluate our system on the PanAf500 and PanAf20K datasets and demonstrate the performance benefits of our multi-modal decoding approach and query initialisation strategy on multi-class and multi-label recognition tasks, respectively. Results and ablations corroborate performance improvements. We achieve state-of-the-art performance over vision and vision-language models in top-1 accuracy (+6.34%) on PanAf500 and overall (+1.1%) and tail-class (+2.26%) mean average precision on PanAf20K. We share complete source code and network weights for full reproducibility of results and easy utilisation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Primate Behavior Recognition on PanAf500 (test)](<https://www.sota2.com/research/sota/primate-behavior-recognition-on-panaf500-test.md>): Primate Behavior Recognition; PanAf500 (test); Accuracy: 84.91.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.08937>)

[PDF](<https://arxiv.org/pdf/2404.08937.pdf>)
