# Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation Source: [Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:23:18.913Z Authors: Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, Tinne Tuytelaars Published: 2024-09-25T19:49:39.000Z ## Abstract This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Decision Understanding on NavNuances](): Decision Understanding; NavNuances; DC: 92.68. ## Original sources and implementation [Original paper]() [PDF]()