bAbI
Benchmarks
TaskDatasetSOTA resultTrendResults
bAbI (test)
0.032Mean Error
54
sys-bAbI original (test)
7.95Gap
22
BaBi
79.2PRR
20
bAbI (test)
24Accuracy
20
bAbI
26.8Accuracy
16
bAbI 10k (test)
0Task 1: 1 Supporting Fact Error
15
bAbI 1k (train test)
50Task: 1 Supporting Fact Acc
12
bAbI 10k 1.0 (test)
21Mean Error Rate
10
bAbI 1.0 (test)
0.4Task 1 Accuracy
10
bAbI Dialogue Task 4 OOV
100Per-response Accuracy
9
bAbI Dialogue Task 5
99.6Per-response Accuracy
9
bAbI Dialogue Task 3
96.3Accuracy (Per-response)
9
bAbI 1K examples 1.0 (test)
4.55Average Error Rate
8
bAbI dialog tasks (test)
0Issuing API calls Error Rate
8
bAbI en
26.8Few-Shot Accuracy
7
bAbi 1K (test)
90.1Maximum Accuracy
7
bAbI dialog
1.5Average Error Rate
7
bAbI English 10k (test)
0Failed Tasks Count (Error > 5%)
7
BABI 2-Choice
28.9Delta Accuracy
6
bAbI original (test)
99.88Task 17 Accuracy
6
bAbI QA 1k
5Failed Tasks Count
6
bAbI Task 1 qa1 50 samples
19Accuracy
5
bAbI (test)
96.27Acc (Single Supporting Fact)
5
bAbI 10K synthesized samples
26.8Accuracy
3
bAbi-Mix
11.8Average Error Rate
3