# World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

Source: [World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models](<https://www.sota2.com/research/paper/68fa766f1e209131cea6cfa2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T08:30:29.199Z

Authors: Ziqiao Ma, Jiayi Pan, Joyce Chai

Published: 2023-06-14T18:10:05.000Z

## Abstract

The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words. While humans demonstrate fast mapping in new word learning, it remains unclear whether modern vision-language models can truly represent language with their grounded meanings and how grounding may further bootstrap new word learning. To this end, we introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in open-world language learning. As an initial attempt, we propose object-oriented BERT (OctoBERT), a novel visually-grounded language model by pre-training on image-text pairs highlighting grounding as an objective. Through extensive experiments and analysis, we demonstrate that OctoBERT is a more coherent and fast grounded word learner, and that the grounding ability acquired during pre-training helps the model to learn unseen words more rapidly and robustly. Our code is available at https://github.com/sled-group/world-to-words

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Grounded Open Vocabulary Acquisition on Flickr30k Entities 1.0 (Seen Vseen = 60)](<https://www.sota2.com/research/sota/grounded-open-vocabulary-acquisition-on-flickr30k-entities-1-0-seen-vseen-60.md>): Grounded Open Vocabulary Acquisition; Flickr30k Entities 1.0 (Seen Vseen = 60); HR@1: 66.9.

- [Grounded Open Vocabulary Acquisition on Flickr30k Entities 1.0 (Unseen words (Vunseen = 31))](<https://www.sota2.com/research/sota/grounded-open-vocabulary-acquisition-on-flickr30k-entities-1-0-unseen-words-vunseen-31.md>): Grounded Open Vocabulary Acquisition; Flickr30k Entities 1.0 (Unseen words (Vunseen = 31)); HR@1: 4.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2306.08685>)

[PDF](<https://arxiv.org/pdf/2306.08685.pdf>)

[Implementation](<https://github.com/sled-group/world-to-words>)