# ConTextual Masked Auto-Encoder for Dense Passage Retrieval

Source: [ConTextual Masked Auto-Encoder for Dense Passage Retrieval](<https://www.sota2.com/research/paper/68fa6e251e209131cea3e216>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:56.488Z

Authors: Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, Songlin Hu

Published: 2022-08-16T11:17:22.000Z

## Abstract

Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent studies have explored improving pre-trained language models to boost dense retrieval performance. This paper proposes CoT-MAE (ConTextual Masked Auto-Encoder), a simple yet effective generative pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric encoder-decoder architecture that learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. Precisely, self-supervised masked auto-encoding learns to model the semantics of the tokens inside a text span, and context-supervised masked auto-encoding learns to model the semantical correlation between the text spans. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines, demonstrating the high efficiency of CoT-MAE. Our code is available at https://github.com/caskcsg/ir/tree/main/cotmae.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Retrieval on MS MARCO (dev)](<https://www.sota2.com/research/sota/retrieval-on-ms-marco-dev.md>): Retrieval; MS MARCO (dev); MRR@10: 0.399.

- [Information Retrieval on MS MARCO DL2019](<https://www.sota2.com/research/sota/information-retrieval-on-ms-marco-dl2019.md>): Information Retrieval; MS MARCO DL2019; nDCG@10: 70.

- [Information Retrieval on MS MARCO DL2020](<https://www.sota2.com/research/sota/information-retrieval-on-ms-marco-dl2020.md>): Information Retrieval; MS MARCO DL2020; NDCG@10: 67.8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2208.07670>)

[PDF](<https://arxiv.org/pdf/2208.07670.pdf>)