# Q8BERT: Quantized 8Bit BERT

Source: [Q8BERT: Quantized 8Bit BERT](<https://www.sota2.com/research/paper/68fa6b7f1e209131cea2e9dc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-25T07:47:14.284Z

Authors: Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat

Published: 2019-10-14T14:55:19.000Z

## Abstract

Recently, pre-trained Transformer based language models such as BERT and GPT, have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 and Megatron, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by $4\\times$ with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text Embedding on MTEB (Quality and Efficiency)](<https://www.sota2.com/research/sota/text-embedding-on-mteb-quality-and-efficiency.md>): Text Embedding; MTEB; MTEB Quality Score: 65.8.

- [Natural Language Understanding on GLUE 1.0 (dev)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md>): Natural Language Understanding; GLUE 1.0 (dev); SST-2 (Acc): 92.24.

- [Natural Language Understanding on GLUE](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue.md>): Natural Language Understanding; GLUE; SST-2: 94.7.

- [Question Answering on SQuAD v1.1 (dev)](<https://www.sota2.com/research/sota/question-answering-on-squad-v1-1-dev.md>): Question Answering; SQuAD v1.1 (dev); F1 Score: 87.74.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1910.06188>)

[PDF](<https://arxiv.org/pdf/1910.06188.pdf>)

[Implementation](<https://github.com/NervanaSystems/nlp-architect>)
