# KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning

Source: [KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning](<https://www.sota2.com/research/paper/68fa68e11e209131cea1eeaa>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:48:52.480Z

Authors: Bin He, Xin Jiang, Jinghui Xiao, Qun Liu

Published: 2020-12-07T09:39:25.000Z

## Abstract

Recent studies on pre-trained language models have demonstrated their ability to capture factual knowledge and applications in knowledge-aware downstream tasks. In this work, we present a language model pre-training framework guided by factual knowledge completion and verification, and use the generative and discriminative approaches cooperatively to learn the model. Particularly, we investigate two learning schemes, named two-tower scheme and pipeline scheme, in training the generator and discriminator with shared parameter. Experimental results on LAMA, a set of zero-shot cloze-style question answering tasks, show that our model contains richer factual knowledge than the conventional pre-trained language models. Furthermore, when fine-tuned and evaluated on the MRQA shared tasks which consists of several machine reading comprehension datasets, our model achieves the state-of-the-art performance, and gains large improvements on NewsQA (+1.26 F1) and TriviaQA (+1.56 F1) over RoBERTa.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.03551>)

[PDF](<https://arxiv.org/pdf/2012.03551.pdf>)
