# Watch, read and lookup: learning to spot signs from multiple supervisors

Source: [Watch, read and lookup: learning to spot signs from multiple supervisors](<https://www.sota2.com/research/paper/68fa68811e209131cea1c86e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:38:59.898Z

Authors: Liliane Momeni, G\\"ul Varol, Samuel Albanie, Triantafyllos Afouras, Andrew Zisserman

Published: 2020-10-08T14:12:56.000Z

## Abstract

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sign Recognition on BSL-1K 37K (test)](<https://www.sota2.com/research/sota/sign-recognition-on-bsl-1k-37k-test.md>): Sign Recognition; BSL-1K 37K (test); Top-1 Acc (Instance): 60.9.

- [Sign Recognition on BSL-1K 2K (test)](<https://www.sota2.com/research/sota/sign-recognition-on-bsl-1k-2k-test.md>): Sign Recognition; BSL-1K 2K (test); Top-1 Accuracy (Instance): 70.8.

- [Sign recognition on BSL-1K 37K\_Rec (test)](<https://www.sota2.com/research/sota/sign-recognition-on-bsl-1k-37k_rec-test.md>): Sign Recognition; BSL-1K 37K\_Rec (test); Per-Instance Top-1 Acc: 62.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.04002>)

[PDF](<https://arxiv.org/pdf/2010.04002.pdf>)