Reading Comprehension on DROP v1.0 (dev)
93.01EMOracle (Theoretical Upper Bound)
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oracle (Theoretical Upper Bound)2019.08 | 93.01 | — | — | 71.6 | — | 94.5 | — | 95.8 | — | 60.5 | — | |
| Giving BERT a CalculatorOperations=All, Pre-training=CoQA, Ensemble=6 models2019.08 | 78.97 | 82.56 | 93.01 | 59.7 | 67.7 | 83.9 | 84.1 | 81.1 | 85.4 | 6 | 47 | |
| Giving BERT a CalculatorOperations=All, Pre-training=CoQA2019.08 | 78.09 | 81.65 | 93.01 | 59.5 | 66.4 | 83.1 | 83.3 | 79.8 | 84.3 | 6.2 | 47 | |
| Giving BERT a CalculatorOperations=Basic + Diff100 + Sum3 + Merge2019.08 | 76.95 | 80.48 | 93.01 | 58.1 | 61.8 | 82 | 82.1 | 78.8 | 83.4 | 5.1 | 45 | |
| Giving BERT a CalculatorOperations=Basic + Diff100 + Sum32019.08 | 76.7 | 80.06 | 90.16 | 58 | 64.6 | 81.9 | 82.1 | 78.9 | 83.4 | 0 | 36 | |
| Giving BERT a CalculatorOperations=Basic + Diff1002019.08 | 75.52 | 78.82 | 88.75 | 53.6 | 61.3 | 80.3 | 80.5 | 78.4 | 82.8 | 0 | 35.8 | |
| Giving BERT a CalculatorOperations=Basic (D_direct, all S, Sum, Diff)2019.08 | 66.5 | 69.91 | 80.03 | 57 | 65.1 | 65.8 | 66.1 | 78 | 82.6 | 0 | 35.7 | |
| NAQANetModel=NAQANet2019.08 | 46.75 | 50.39 | — | 32 | 39.6 | 44.9 | 45 | 58.2 | 64.8 | 0 | 27.3 |