Question Answering on MRQA 2019 (dev)
92.5SQuAD ScoreSpanBERT
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SpanBERTTrain. Size=All2021.09 | 92.5 | 79.9 | 80.7 | 71.1 | 84.8 | 80.7 | 81.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SplinterTrain. Size=All2021.09 | 92.4 | 79.7 | 80.3 | 70.8 | 84 | 80.6 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Splinter*Train. Size=All, Note=Original paper result; deeper model with additional transformation layers2021.09 | 92.2 | 76.5 | 81 | 71.3 | 83 | 80.7 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaTrain. Size=All2021.09 | 92 | 78.1 | 80.6 | 71.9 | 85.2 | 79.1 | 81.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_RTrain. Size=All2021.09 | 91.4 | 78.9 | 80.8 | 71.4 | 85.3 | 80.6 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSPTTrain. Size=All2021.09 | 91.1 | 77 | 80 | 69.7 | 83.3 | 79.7 | 80.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_BTrain. Size=All2021.09 | 90.3 | 77.5 | 79.9 | 68.7 | 83.7 | 80.5 | 80.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTTrain. Size=All2021.09 | 88.8 | 73.6 | 78.7 | 67.5 | 82 | 76.2 | 77.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Splinter*Train. Size=1024, Note=Original paper result; deeper model with additional transformation layers2021.09 | 82.8 | 64.8 | 65.5 | 57.3 | 67.3 | 70.3 | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpanBERTTrain. Size=10242021.09 | 81.1 | 67 | 63.2 | 56.4 | 70 | 67.6 | 67.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SplinterTrain. Size=10242021.09 | 79.8 | 67.3 | 63.8 | 54.6 | 68.9 | 68.4 | 67.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_RTrain. Size=10242021.09 | 79.7 | 70.1 | 65 | 54.7 | 72.8 | 69.7 | 68.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaTrain. Size=10242021.09 | 77.9 | 62.2 | 60.3 | 55 | 67.5 | 63.4 | 64.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSPTTrain. Size=10242021.09 | 77.6 | 60.1 | 58.7 | 52.8 | 65.9 | 63.3 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_BTrain. Size=10242021.09 | 76.9 | 67.4 | 63.6 | 52.2 | 70.6 | 67.8 | 66.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Splinter*Train. Size=128, Note=Original paper result; deeper model with additional transformation layers2021.09 | 72.7 | 44.7 | 46.3 | 43.5 | 47.2 | 54.7 | 51.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_RTrain. Size=1282021.09 | 66.7 | 62.1 | 49.8 | 35.7 | 62.3 | 57.2 | 55.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTTrain. Size=10242021.09 | 64.1 | 41.6 | 50.1 | 43 | 53.1 | 46.5 | 49.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpanBERTTrain. Size=1282021.09 | 61.2 | 48.8 | 38.8 | 31 | 50 | 44 | 45.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_BTrain. Size=1282021.09 | 58.5 | 56.2 | 46.7 | 27.8 | 60.8 | 45.2 | 49.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SplinterTrain. Size=1282021.09 | 55 | 45.7 | 41.1 | 33.9 | 48.8 | 46.9 | 45.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Splinter*Train. Size=16, Note=Original paper result; deeper model with additional transformation layers2021.09 | 54.6 | 18.9 | 27.4 | 20.8 | 26.3 | 24 | 28.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaTrain. Size=1282021.09 | 48.8 | 36 | 36.4 | 22.8 | 41.3 | 35.2 | 36.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSPTTrain. Size=1282021.09 | 41.5 | 30.3 | 35 | 14 | 42.8 | 23.7 | 31.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_RTrain. Size=162021.09 | 41.3 | 45.5 | 33.6 | 16.2 | 45.8 | 34.1 | 36.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonBERT_BTrain. Size=162021.09 | 33.2 | 37.2 | 33.1 | 11.8 | 46.1 | 22.4 | 30.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTTrain. Size=1282021.09 | 21.5 | 23.9 | 31.7 | 11.3 | 32.6 | 14 | 22.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SplinterTrain. Size=162021.09 | 16.7 | 23.9 | 25.1 | 11.6 | 23.6 | 15.1 | 19.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SpanBERTTrain. Size=162021.09 | 15.7 | 27.4 | 24.3 | 8.1 | 24.1 | 16.3 | 19.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSPTTrain. Size=162021.09 | 10.8 | 21.2 | 23.7 | 6.5 | 25.8 | 9.1 | 16.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaTrain. Size=162021.09 | 10.3 | 21 | 22.5 | 6.7 | 23.4 | 11.2 | 15.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTTrain. Size=162021.09 | 9.9 | 15.4 | 20.5 | 6.5 | 16.8 | 9.6 | 13.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| 500xCompression ratio=15x2025.10 | — | — | — | — | — | — | — | 40.68 | 24.97 | 32.01 | 16.76 | 53.84 | 44.86 | 65.65 | 53.7 | 53.01 | 36.3 | 50.93 | 33.26 | 49.35 | 34.98 | |
| BeaconCompression ratio=15x2025.10 | — | — | — | — | — | — | — | 37.53 | 14.31 | 31.1 | 7.72 | 48.85 | 4.47 | 39.06 | 22.67 | 45.01 | 27.35 | 40.29 | 15.82 | 40.31 | 15.39 | |
| DASTCompression ratio=15x2025.10 | — | — | — | — | — | — | — | 36.33 | 22.65 | 31.55 | 16.29 | 56.92 | 48.14 | 68.07 | 56.51 | 54.02 | 37.93 | 52.1 | 35.31 | 49.83 | 36.14 | |
| EPLCompression ratio=15x2025.10 | — | — | — | — | — | — | — | 44.58 | 27.91 | 33.34 | 16.69 | 56.16 | 47.09 | 66.36 | 54.13 | 54.88 | 38.38 | 53.8 | 35.71 | 51.52 | 36.65 | |
| Full-FTFine-tuning=Full2025.10 | — | — | — | — | — | — | — | 77.69 | 59.71 | 63.5 | 46.04 | 68.8 | 60.54 | 73.25 | 62.07 | 74.78 | 59.26 | 71.01 | 53.47 | 71.51 | 56.85 | |
| ICAECompression ratio=15x2025.10 | — | — | — | — | — | — | — | 31.9 | 18.91 | 25.25 | 11.97 | 51.78 | 42.94 | 64.81 | 52.89 | 45.22 | 30.32 | 48.01 | 30.67 | 44.5 | 31.28 | |
| Lingua-2Compression ratio=5x2025.10 | — | — | — | — | — | — | — | 32.93 | 19.57 | 26.78 | 13.2 | 9.67 | 8.12 | 45.4 | 31.8 | 36.1 | 22.05 | 40.08 | 22.01 | 31.83 | 19.46 | |
| SACCompression ratio=15x2025.10 | — | — | — | — | — | — | — | 47.43 | 30.25 | 36.55 | 18.07 | 61.13 | 52.19 | 68.97 | 56.76 | 58.83 | 41.86 | 56.79 | 38.88 | 54.95 | 39.67 |