# "Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration

Source: ["Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration](<https://www.sota2.com/research/paper/68fa74df1e209131cea645a2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:20:26.911Z

Authors: Dan Bohus, Sean Andrist, Yuwei Bao, Eric Horvitz, Ann Paradiso

Published: 2024-08-30T12:41:23.000Z

## Abstract

We report initial work towards constructing ecologically valid benchmarks to assess the capabilities of large multimodal models for engaging in situated collaboration. In contrast to existing benchmarks, in which question-answer pairs are generated post hoc over preexisting or synthetic datasets via templates, human annotators, or large language models (LLMs), we propose and investigate an interactive system-driven approach, where the questions are generated by users in context, during their interactions with an end-to-end situated AI system. We illustrate how the questions that arise are different in form and content from questions typically found in existing embodied question answering (EQA) benchmarks and discuss new real-world challenge problems brought to the fore.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.10525>)

[PDF](<https://arxiv.org/pdf/2409.10525.pdf>)
