# Learning Visual N-Grams from Web Data

Source: [Learning Visual N-Grams from Web Data](<https://www.sota2.com/research/paper/68fa638c1e209131ce9ff783>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-22T13:16:33.971Z

Authors: Ang Li, Allan Jabri, Armand Joulin, Laurens van der Maaten

Published: 2016-12-29T14:50:53.000Z

## Abstract

Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on aYahoo](<https://www.sota2.com/research/sota/image-classification-on-ayahoo.md>): Image Classification; aYahoo; Accuracy: 72.4.

- [Image Classification on SUN](<https://www.sota2.com/research/sota/image-classification-on-sun.md>): Image Classification; SUN; Accuracy: 23.

- [Text Retrieval on MS-COCO 5K (test)](<https://www.sota2.com/research/sota/text-retrieval-on-ms-coco-5k-test.md>): Text Retrieval; MS-COCO 5K (test); R@1: 8.7.

- [Image Retrieval on MS-COCO 5K (test)](<https://www.sota2.com/research/sota/image-retrieval-on-ms-coco-5k-test.md>): Image Retrieval; MS-COCO 5K (test); R@1: 5.

- [Text Retrieval on Flickr30K (test)](<https://www.sota2.com/research/sota/text-retrieval-on-flickr30k-test.md>): Text Retrieval; Flickr30k (test); R@1: 15.4.

- [Image Retrieval on Flickr30k (test)](<https://www.sota2.com/research/sota/image-retrieval-on-flickr30k-test.md>): Image Retrieval; Flickr30k (test); R@1: 8.8.

- [Image Classification on ImageNet 1k (test)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-test.md>): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 11.5.

- [Image Classification on ImageNet 1k (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: not recorded.

- [Image Classification on ImageNet](<https://www.sota2.com/research/sota/image-classification-on-imagenet.md>): Image Classification; ImageNet; Top-1 Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1612.09161>)

[PDF](<https://arxiv.org/pdf/1612.09161.pdf>)