# Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks

Source: [Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks](<https://www.sota2.com/research/paper/68fa695a1e209131cea21bf9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:58:51.104Z

Authors: Tingyu Xia, Yue Wang, Yuan Tian, Yi Chang

Published: 2021-02-22T12:07:16.000Z

## Abstract

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Natural Language Understanding on MRPC, QQP, STS-B, MNLI, QNLI, RTE, SNLI, Sci, SICK, and Twi](<https://www.sota2.com/research/sota/natural-language-understanding-on-mrpc-qqp-sts-b-mnli-qnli-rte-snli-sci-sick-and-twi.md>): Natural Language Understanding; Ten Benchmark Datasets MRPC, QQP, STS-B, MNLI, QNLI, RTE, SNLI, Sci, SICK, Twi; MRPC Accuracy: 88.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2102.10934>)

[PDF](<https://arxiv.org/pdf/2102.10934.pdf>)