# Do language models have coherent mental models of everyday things?

Source: [Do language models have coherent mental models of everyday things?](<https://www.sota2.com/research/paper/68fa6fbc1e209131cea471b4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T02:58:59.010Z

Authors: Yuling Gu, Bhavana Dalvi Mishra, Peter Clark

Published: 2022-12-20T06:54:04.000Z

## Abstract

When people think of everyday things like an egg, they typically have a mental image associated with it. This allows them to correctly judge, for example, that "the yolk surrounds the shell" is a false statement. Do language models similarly have a coherent picture of such everyday things? To investigate this, we propose a benchmark dataset consisting of 100 everyday things, their parts, and the relationships between these parts, expressed as 11,720 "X relation Y?" true/false questions. Using these questions as probes, we observe that state-of-the-art pre-trained language models (LMs) like GPT-3 and Macaw have fragments of knowledge about these everyday things, but do not have fully coherent "parts mental models" (54-59% accurate, 19-43% conditional constraint violation). We propose an extension where we add a constraint satisfaction layer on top of the LM's raw predictions to apply commonsense constraints. As well as removing inconsistencies, we find that this also significantly improves accuracy (by 16-20%), suggesting how the incoherence of the LM's pictures of everyday things can be significantly reduced.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Parts mental model prediction on ParRoT](<https://www.sota2.com/research/sota/parts-mental-model-prediction-on-parrot.md>): Parts mental model prediction; ParRoT; Accuracy: 0.7928.

- [Parts Mental Model Accuracy on ParRoT++](<https://www.sota2.com/research/sota/parts-mental-model-accuracy-on-parrot-plus-plus.md>): Parts Mental Model Accuracy; ParRoT++; Base LM Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2212.10029>)

[PDF](<https://arxiv.org/pdf/2212.10029.pdf>)

[Implementation](<https://github.com/allenai/everyday-things>)