# Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data

Source: [Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data](<https://www.sota2.com/research/paper/68fa71fd1e209131cea53f50>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:04.092Z

Authors: Chenhui Zhang, Sherrie Wang

Published: 2024-01-31T04:57:12.000Z

## Abstract

Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Land Cover Classification on BigEarthNet 19 classes](<https://www.sota2.com/research/sota/land-cover-classification-on-bigearthnet-19-classes.md>): Land Cover Classification; BigEarthNet 19 classes; F1 Score: 40.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2401.17600>)

[PDF](<https://arxiv.org/pdf/2401.17600.pdf>)
