# How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering

Source: [How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering](<https://www.sota2.com/research/paper/68fa669b1e209131cea115dc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:54:21.040Z

Authors: Sanjay Kamath, Brigitte Grau, Yue Ma

Published: 2019-11-02T13:25:15.000Z

## Abstract

Using deep learning models on small scale datasets would result in overfitting. To overcome this problem, the process of pre-training a model and fine-tuning it to the small scale dataset has been used extensively in domains such as image processing. Similarly for question answering, pre-training and fine-tuning can be done in several ways. Commonly reading comprehension models are used for pre-training, but we show that other types of pre-training can work better. We compare two pre-training models based on reading comprehension and open domain question answering models and determine the performance when fine-tuned and tested over BIOASQ question answering dataset. We find open domain question answering model to be a better fit for this task rather than reading comprehension model.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1911.00712>)

[PDF](<https://arxiv.org/pdf/1911.00712.pdf>)