# The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition

Source: [The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition](<https://www.sota2.com/research/paper/68fa79041e209131cea7bb24>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-28T05:00:12.409Z

Authors: Yuwen Tan, Yuan Qing, Boqing Gong

Published: 2025-05-30T17:40:46.000Z

## Abstract

This paper reveals that many open-source large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual recognition (e.g., recognizing Anemone Fish but not Vertebrate). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect because the VQA tasks improve the LLMs' hierarchical consistency more than the vision LLMs'. We conjecture that one cannot make open-source vision LLMs understand visual concepts hierarchically until LLMs possess corresponding taxonomy knowledge.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Taxonomic Classification on iNat Plant 21](<https://www.sota2.com/research/sota/taxonomic-classification-on-inat-plant-21.md>): Taxonomic Classification; iNat Plant 21; HCA: 29.34.

- [Taxonomic Classification on iNat Animal 21](<https://www.sota2.com/research/sota/taxonomic-classification-on-inat-animal-21.md>): Taxonomic Classification; iNat Animal 21; HCA: 23.38.

- [Taxonomic Classification on CUB-200](<https://www.sota2.com/research/sota/taxonomic-classification-on-cub-200.md>): Taxonomic Classification; CUB-200; HCA: 46.17.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.24840>)

[PDF](<https://arxiv.org/pdf/2505.24840.pdf>)
