Question Answering on bAbI
26.8AccuracyGPT-2
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GPT-2attack ratio=0.02024.01 | 26.8 | — | — | — | — | — | |
| GPT-2attack ratio=0.012024.01 | 21.6 | — | — | — | — | — | |
| PIXARattack ratio=0.02024.01 | 19.6 | — | — | — | — | — | |
| GPT-2attack ratio=0.052024.01 | 12.1 | — | — | — | — | — | |
| PIXARattack ratio=0.012024.01 | 11.4 | — | — | — | — | — | |
| PIXARattack ratio=0.052024.01 | 7.4 | — | — | — | — | — | |
| GPT-2attack ratio=0.12024.01 | 6.2 | — | — | — | — | — | |
| PIXARattack ratio=0.12024.01 | 4.8 | — | — | — | — | — | |
| PIXARattack ratio=0.22024.01 | 1.6 | — | — | — | — | — | |
| GPT-2attack ratio=0.22024.01 | 1.3 | — | — | — | — | — | |
| PIXARattack ratio=0.32024.01 | 1 | — | — | — | — | — | |
| PIXARattack ratio=0.42024.01 | 0.3 | — | — | — | — | — | |
| GPT-2attack ratio=0.32024.01 | 0.2 | — | — | — | — | — | |
| GPT-2attack ratio=0.42024.01 | 0.1 | — | — | — | — | — | |
| PIXARattack ratio=0.52024.01 | 0.1 | — | — | — | — | — | |
| GPT-2attack ratio=0.52024.01 | 0 | — | — | — | — | — | |
| ARMT (GPT-2)context_type=Recurrent models2026.03 | — | 100 | 93.8 | 92.3 | 100 | 98.9 | |
| GPT-2-124mcontext_type=Full context models (upper bound)2026.03 | — | 100 | 100 | 99.8 | 100 | 99.4 | |
| GradMem (GPT-2, increased K)K=increased, Number of memory tokens=82026.03 | — | 100 | 93.9 | 79.3 | 100 | 99.2 | |
| GradMem (GPT-2, K = 1)K=1, Number of memory tokens=82026.03 | — | 100 | 94.2 | 80 | 100 | 99.2 | |
| Mamba-130m-hfcontext_type=Recurrent models2026.03 | — | 100 | 100 | 96.7 | 100 | 99.7 | |
| Pythia-160mcontext_type=Full context models (upper bound)2026.03 | — | 100 | 99.7 | 95.5 | 100 | 99 | |
| RMT (GPT-2)context_type=Recurrent models2026.03 | — | 100 | 93.9 | 87.9 | 100 | 93.9 |