Question Answering on SQuAD v1.1 (val)
96.22F1 ScoreT5-11B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| T5-11BParameters=11 billion2019.10 | 96.22 | 91.26 | — | — | |
| Previous best2019.10 | 95.5 | 90.1 | — | — | |
| T5-3BParameters=3 billion2019.10 | 94.95 | 88.53 | — | — | |
| T5-LargeParameters=770 million2019.10 | 93.79 | 86.66 | — | — | |
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=RoBERTa2022.09 | 92.25 | 85.83 | — | — | |
| T5-BaseParameters=220 million2019.10 | 92.08 | 85.44 | — | — | |
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BART2022.09 | 91.63 | 84.79 | — | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.09 | 91.57 | 84.86 | — | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=RoBERTa2022.09 | 91.48 | 84.53 | — | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BART2022.09 | 91.08 | 84.07 | — | — | |
| ReferenceSparsity=0%2021.11 | 90.93 | 83.99 | — | — | |
| MKOR-HModel=BERT-Large Uncased, # Iterations=600, Time (h)=3.10, Speedup (x)=2.572023.06 | 90.64 | — | — | — | |
| EvaModel=BERT-Large Uncased, # Iterations=1,000, Time (h)=5.24, Speedup (x)=1.522023.06 | 90.55 | — | — | — | |
| MKORModel=BERT-Large Uncased, # Iterations=1,000, Time (h)=5.25, Speedup (x)=1.512023.06 | 90.5 | — | — | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=BART2022.09 | 90.49 | 83.11 | — | — | |
| LAMBModel=BERT-Large Uncased, # Iterations=1,536, Time (h)=7.97, Speedup (x)=1.002023.06 | 90.44 | — | — | — | |
| KAISAModel=BERT-Large Uncased, # Iterations=1,000, Time (h)=5.71, Speedup (x)=1.392023.06 | 90.44 | — | — | — | |
| Prune OFASparsity=90%2021.11 | 90.2 | 83.35 | — | — | |
| Prune OFA + QATSparsity=90%, Quantization-aware training=true2021.11 | 90.02 | 83.22 | — | — | |
| Teacher (BERT)number of layers=122021.12 | 88.6 | 81.5 | — | — | |
| Dense BaselineModel=BERT, Pruning Pattern=None (Dense)2022.08 | 88.53 | — | — | — | |
| BaselineModel=BERT-base, Sparsity Scheme=None2021.02 | 88.52 | — | 0 | — | |
| ITRDstudent_architecture=T6, number of layers=62021.12 | 88.5 | 81.5 | — | — | |
| Ours 4:8-TModel=BERT-base, Sparsity Scheme=4:8-T2021.02 | 88.38 | — | 66 | — | |
| Dense BaselineModel=BERT6, Pruning Pattern=None (Dense)2022.08 | 88.33 | — | — | — | |
| Full PrecisionBits (W-E-A)=32-32-32, Model Architecture=BERT2022.09 | 88.28 | 80.82 | — | — | |
| TextBrewerstudent_architecture=T6, number of layers=62021.12 | 88.1 | 80.8 | — | — | |
| OMSEBits (W-E-A)=8-8-8, Model Architecture=BERT2022.09 | 87.9 | 80.16 | — | — | |
| Outlier SuppressionBits (W-E-A)=8-8-8, Model Architecture=BERT2022.09 | 87.6 | 79.8 | — | — | |
| T5-SmallParameters=60 million2019.10 | 87.24 | 79.1 | — | — | |
| ExactOBSModel=BERT6, Pruning Pattern=2:42022.08 | 86.97 | — | — | — | |
| DistilBERTstudent_architecture=T6, number of layers=62021.12 | 86.9 | 79.1 | — | — | |
| ExactOBSModel=BERT, Pruning Pattern=2:42022.08 | 86.77 | — | — | — | |
| ITRDstudent_architecture=T3, number of layers=32021.12 | 85.8 | 77.7 | — | — | |
| AdaPruneModel=BERT, Pruning Pattern=2:42022.08 | 85.24 | — | — | — | |
| AdaPruneModel=BERT6, Pruning Pattern=2:42022.08 | 85.02 | — | — | — | |
| TextBrewerstudent_architecture=T3, number of layers=32021.12 | 84.8 | 76.3 | — | — | |
| DistilBERTPLM=Gold Data, Setting=SUPERVISED, #Param=66M2022.02 | 84.67 | 76.28 | — | — | |
| Dense BaselineModel=BERT3, Pruning Pattern=None (Dense)2022.08 | 84.66 | — | — | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 84.48 | 75.53 | — | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 83.68 | 75.34 | — | — | |
| ExactOBSModel=BERT3, Pruning Pattern=2:42022.08 | 83.54 | — | — | — | |
| AdaPruneModel=BERT3, Pruning Pattern=2:42022.08 | 82.75 | — | — | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 82.45 | 72.87 | — | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 82.41 | 71.72 | — | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=BART2022.09 | 81.44 | 70.61 | — | — | |
| Outlier SuppressionBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 80.79 | 70.83 | — | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 80.47 | 70.08 | — | — | |
| DCN+Char+CoVeModel type=Single-model2017.08 | 79.9 | 71.3 | — | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 79.77 | 69.1 | — | — | |
| R-NETModel type=Single-model2017.08 | 79.5 | 71.1 | — | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=BERT2022.09 | 78.55 | 67.14 | — | — | |
| BiDAFModel type=Single-model2017.08 | 77.3 | 68 | — | — | |
| DCN+CharModel type=Single-model2017.08 | 75.6 | 65.4 | — | — | |
| hM-LSTM+APModel type=Single-model2017.08 | 73.9 | 64.1 | — | — | |
| DCRModel type=Single-model2017.08 | 72.1 | 62.5 | — | — | |
| OMSEBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 70.64 | 58.8 | — | — | |
| EasyQuantBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 67.85 | 55.92 | — | — | |
| PercentileBits (W-E-A)=6-6-6, Model Architecture=RoBERTa2022.09 | 67.24 | 53.28 | — | — | |
| LSTMPLM=Gold Data, Setting=SUPERVISED, #Param=~7M2022.02 | 57.22 | 41.86 | — | — | |
| LRModel type=Single-model2017.08 | 51 | 40 | — | — | |
| ZeroGenPLM=GPT2-XL, TAM=DistilBERT, Setting=ZEROGEN, #Param=66M2022.02 | 31.53 | 25.5 | — | — | |
| ZeroGenPLM=GPT2-Large, TAM=DistilBERT, Setting=ZEROGEN, #Param=66M2022.02 | 29.82 | 23.87 | — | — | |
| ZeroGenPLM=GPT2, TAM=DistilBERT, Setting=ZEROGEN, #Param=66M2022.02 | 21.83 | 16.44 | — | — | |
| ZeroGenPLM=GPT2-XL, TAM=LSTM, Setting=ZEROGEN, #Param=~7M2022.02 | 18.66 | 12.35 | — | — | |
| GPT2-XLSetting=PROMPTING, #Param=1.5B2022.02 | 13.32 | 4.61 | — | — | |
| ZeroGenPLM=GPT2-Large, TAM=LSTM, Setting=ZEROGEN, #Param=~7M2022.02 | 12.77 | 8.01 | — | — | |
| GPT2-LargeSetting=PROMPTING, #Param=762M2022.02 | 10.78 | 3.53 | — | — | |
| ZeroGenPLM=GPT2, TAM=LSTM, Setting=ZEROGEN, #Param=~7M2022.02 | 8.53 | 4.94 | — | — | |
| GPT2Setting=PROMPTING, #Param=117M2022.02 | 4.93 | 0.8 | — | — | |
| BERT_baseParams.=85M2023.08 | — | — | — | 88.4 | |
| CoFiParams.=~5.0M2023.08 | — | — | — | 82.6 | |
| SP3Params.=~5.0M, Speedup=x2.832023.08 | — | — | — | 83.2 | |
| TinyBERT_4Params.=4.7M2023.08 | — | — | — | 82.1 | |
| WIDParams.=5.0M2023.08 | — | — | — | 81.2 |