# On the importance of pre-training data volume for compact language models

Source: [On the importance of pre-training data volume for compact language models](<https://www.sota2.com/research/paper/68fa68821e209131cea1c8fe>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-14T17:37:20.532Z

Authors: Vincent Micheli, Martin d'Hoffschmidt, Fran\\c{c}ois Fleuret

Published: 2020-10-08T07:40:21.000Z

## Abstract

Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the impact of pre-training data volume on compact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an intermediate pre-training step on the task-specific corpus does not yield substantial improvements.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.03813>)

[PDF](<https://arxiv.org/pdf/2010.03813.pdf>)

[Implementation](<https://huggingface.co/illuin/lepetit>)
