# InforMask: Unsupervised Informative Masking for Language Model Pretraining

Source: [InforMask: Unsupervised Informative Masking for Language Model Pretraining](<https://www.sota2.com/research/paper/68fa6dbf1e209131cea3bd40>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-08-12T05:00:13.408Z

Authors: Nafis Sadeq, Canwen Xu, Julian McAuley

Published: 2022-10-21T07:10:56.000Z

## Abstract

Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, a new unsupervised masking strategy for training masked language models. InforMask exploits Pointwise Mutual Information (PMI) to select the most informative tokens to mask. We further propose two optimizations for InforMask to improve its efficiency. With a one-off preprocessing step, InforMask outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2210.11771>)

[PDF](<https://arxiv.org/pdf/2210.11771.pdf>)
