# Unsupervised Improvement of Factual Knowledge in Language Models

Source: [Unsupervised Improvement of Factual Knowledge in Language Models](<https://www.sota2.com/research/paper/68fa6eed1e209131cea42ee2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:09:40.360Z

Authors: Nafis Sadeq, Byungkyu Kang, Prarit Lamba, Julian McAuley

Published: 2023-04-04T07:37:06.000Z

## Abstract

Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we propose an approach for influencing MLM pretraining in a way that can improve language model performance on a variety of knowledge-intensive tasks. We force the language model to prioritize informative words in a fully unsupervised way. Experiments demonstrate that the proposed approach can significantly improve the performance of pretrained language models on tasks such as factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2304.01597>)

[PDF](<https://arxiv.org/pdf/2304.01597.pdf>)
