Question Answering on NewsQA (dev)
75.5F1 ScoreORACLE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ORACLEQA Model=S-Reader, Train Speedup=x18.8, Inference Speedup=x21.72018.05 | 75.5 | 59.2 | |
| ABEXtraining_samples=10002024.06 | 73.41 | — | |
| ABEX-stage-1training_samples=1000, fine-tuning=false2024.06 | 72.45 | — | |
| ABEX-stage-2training_samples=1000, fine-tuning=true2024.06 | 71.63 | — | |
| GENIUStraining_samples=1000, augmentations=52024.06 | 69.36 | — | |
| LLaMA-2 13Btraining_samples=1000, parameters=13B2024.06 | 68.97 | — | |
| Retro-Readerbackbone=ALBERT2020.01 | 68.6 | 58.5 | |
| ALBERTverifier=TAV2020.01 | 67.5 | 57.1 | |
| Retro-Readerbackbone=ELECTRA2020.01 | 67 | 56.9 | |
| ELECTRAverifier=TAV2020.01 | 66.5 | 56.3 | |
| DECAPROP2020.01 | 65.7 | 52.5 | |
| FULLQA Model=S-Reader, Train Speedup=x1.0, Inference Speedup=x1.02018.05 | 63.8 | 50.7 | |
| AMANDA2018.01 | 63.3 | 48.4 | |
| AMANDA2020.01 | 63.3 | 48.4 | |
| MINIMAL (Dyn)QA Model=S-Reader, Selection Method=Dynamic, Train Speedup=x15.0, Inference Speedup=x5.32018.05 | 63.2 | 50.1 | |
| MINIMAL (Top k)QA Model=S-Reader, Selection Method=Top k (k=3), Train Speedup=x15.0, Inference Speedup=x6.92018.05 | 62.3 | 49.3 | |
| AEDAtraining_samples=1000, augmentations=52024.06 | 61.78 | — | |
| ABEXtraining_samples=5002024.06 | 61.43 | — | |
| EDAtraining_samples=1000, augmentations=52024.06 | 61.01 | — | |
| SSMBAtraining_samples=1000, augmentations=52024.06 | 60.34 | — | |
| BackTranstraining_samples=1000, augmentations=52024.06 | 60.21 | — | |
| ABEX-Abstraining_samples=1000, stage=abstract-only2024.06 | 60.11 | — | |
| ABEX-stage-1training_samples=500, fine-tuning=false2024.06 | 59.28 | — | |
| Gold-onlytraining_samples=10002024.06 | 58.83 | — | |
| ABEX-stage-2training_samples=500, fine-tuning=true2024.06 | 57.38 | — | |
| GENIUStraining_samples=500, augmentations=52024.06 | 57.32 | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=128, Number of Parameters=406M2023.10 | 56.9 | — | |
| FewshotQANumber of training examples=128, Number of Parameters=406M2023.10 | 56.8 | — | |
| LLaMA-2 13Btraining_samples=500, parameters=13B2024.06 | 56.58 | — | |
| FastQAbeam-size=52017.03 | 56.4 | 43.7 | |
| FastQA2018.01 | 56.4 | 43.7 | |
| FastQAExtbeam-size=52017.03 | 56.1 | 43.7 | |
| FastQAQA Model=S-Reader2018.05 | 56.1 | 43.7 | |
| FastQAExt2018.01 | 56.1 | 43.7 | |
| FastQAExtTraining protocol=Supervised2020.05 | 56.1 | 43.7 | |
| ABEXtraining_samples=2002024.06 | 54.67 | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=64, Number of Parameters=406M2023.10 | 54.2 | — | |
| FewshotQANumber of training examples=64, Number of Parameters=406M2023.10 | 53.7 | — | |
| ABEX-stage-1training_samples=200, fine-tuning=false2024.06 | 52.83 | — | |
| PMRNumber of training examples=128, Number of Parameters=406M2023.10 | 52.3 | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=32, Number of Parameters=406M2023.10 | 51.8 | — | |
| LLaMA-2 13Btraining_samples=200, parameters=13B2024.06 | 50.24 | — | |
| AEDAtraining_samples=500, augmentations=52024.06 | 50.24 | — | |
| FewshotQANumber of training examples=32, Number of Parameters=406M2023.10 | 50 | — | |
| EDAtraining_samples=500, augmentations=52024.06 | 49.9 | — | |
| ABEX-stage-2training_samples=200, fine-tuning=true2024.06 | 49.82 | — | |
| BARB2017.03 | 49.6 | 36.1 | |
| Match-LSTM2018.01 | 49.6 | 34.4 | |
| BARB2018.01 | 49.6 | 36.1 | |
| mLSTMTraining protocol=Supervised, Source=Trischler et al. (2017)2020.05 | 49.6 | 34.4 | |
| BARB2020.01 | 49.6 | 36.1 | |
| mLSTM2020.01 | 49.6 | 34.4 | |
| Match-LSTM2017.03 | 48.9 | 35.2 | |
| SSMBAtraining_samples=500, augmentations=52024.06 | 47.56 | — | |
| GENIUStraining_samples=200, augmentations=52024.06 | 47.36 | — | |
| BackTranstraining_samples=500, augmentations=52024.06 | 47.21 | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=16, Number of Parameters=406M2023.10 | 46.5 | — | |
| REFQATraining protocol=Unsupervised, Iterative Data Refinement=true2020.05 | 46.3 | 33.6 | |
| ABEX-Abstraining_samples=500, stage=abstract-only2024.06 | 46.29 | — | |
| ABEXtraining_samples=1002024.06 | 45.75 | — | |
| Gold-onlytraining_samples=5002024.06 | 45.65 | — | |
| FewshotQANumber of training examples=16, Number of Parameters=406M2023.10 | 44.9 | — | |
| ABEX-stage-1training_samples=100, fine-tuning=false2024.06 | 43.65 | — | |
| SplinterNumber of training examples=128, Number of Parameters=110M2023.10 | 43.5 | — | |
| PMRNumber of training examples=64, Number of Parameters=406M2023.10 | 43.2 | — | |
| REFQATraining protocol=Unsupervised, Iterative Data Refinement=false2020.05 | 42.2 | 29 | |
| ABEX-stage-2training_samples=100, fine-tuning=true2024.06 | 41.78 | — | |
| LLaMA-2 13Btraining_samples=100, parameters=13B2024.06 | 40.86 | — | |
| Splinter w/ MINPROMPTNumber of training examples=128, Number of Parameters=110M2023.10 | 40.2 | — | |
| GENIUStraining_samples=100, augmentations=52024.06 | 38.88 | — | |
| ABEX-Abstraining_samples=200, stage=abstract-only2024.06 | 38.71 | — | |
| Neural BoW Baseline2017.03 | 37.6 | 25.8 | |
| Neural BoW Baseline2018.01 | 37.6 | 25.8 | |
| SplinterNumber of training examples=64, Number of Parameters=110M2023.10 | 37.4 | — | |
| AEDAtraining_samples=200, augmentations=52024.06 | 36.8 | — | |
| PMRNumber of training examples=32, Number of Parameters=406M2023.10 | 36.6 | — | |
| Splinter w/ MINPROMPTNumber of training examples=64, Number of Parameters=110M2023.10 | 36.1 | — | |
| EDAtraining_samples=200, augmentations=52024.06 | 35.81 | — | |
| BackTranstraining_samples=200, augmentations=52024.06 | 34.98 | — | |
| SSMBAtraining_samples=200, augmentations=52024.06 | 33.27 | — | |
| Splinter w/ MINPROMPTNumber of training examples=32, Number of Parameters=110M2023.10 | 33 | — | |
| ABEX-Abstraining_samples=100, stage=abstract-only2024.06 | 32.09 | — | |
| Splinter w/ MINPROMPTNumber of training examples=16, Number of Parameters=110M2023.10 | 31.9 | — | |
| Gold-onlytraining_samples=2002024.06 | 30.14 | — | |
| PMRNumber of training examples=16, Number of Parameters=406M2023.10 | 30.1 | — | |
| AEDAtraining_samples=100, augmentations=52024.06 | 29.87 | — | |
| SpanBERTNumber of training examples=128, Number of Parameters=110M2023.10 | 29.5 | — | |
| EDAtraining_samples=100, augmentations=52024.06 | 29.31 | — | |
| SSMBAtraining_samples=100, augmentations=52024.06 | 28.89 | — | |
| Lewis et al. (2019)Training protocol=Unsupervised, Backbone=BERT-Large (Whole Word Masking), Re-implementation=Current Paper2020.05 | 28.5 | 19.6 | |
| SplinterNumber of training examples=32, Number of Parameters=110M2023.10 | 27.5 | — | |
| BackTranstraining_samples=100, augmentations=52024.06 | 27.32 | — | |
| Gold-onlytraining_samples=1002024.06 | 22.45 | — | |
| SplinterNumber of training examples=16, Number of Parameters=110M2023.10 | 20.8 | — | |
| RoBERTaNumber of training examples=128, Number of Parameters=110M2023.10 | 16.7 | — | |
| SpanBERTNumber of training examples=64, Number of Parameters=110M2023.10 | 12.5 | — | |
| SpanBERTNumber of training examples=16, Number of Parameters=110M2023.10 | 7.6 | — | |
| SpanBERTNumber of training examples=32, Number of Parameters=110M2023.10 | 7.2 | — | |
| RoBERTaNumber of training examples=64, Number of Parameters=110M2023.10 | 4.6 | — | |
| RoBERTaNumber of training examples=32, Number of Parameters=110M2023.10 | 3.2 | — |