MQMTAR on MQMTAR OOD lengths 2x 4x 16x 64x 256x 1024x
100Exact Match AccuracySoftmax
Evaluation results
| Method | Links | |
|---|---|---|
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=2x2025.06 | 100 | |
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=4x2025.06 | 100 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=2x2025.06 | 100 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=2x2025.06 | 100 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=4x2025.06 | 100 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=2x2025.06 | 100 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=4x2025.06 | 100 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=2x2025.06 | 99.9 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=4x2025.06 | 99.9 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=16x2025.06 | 99.7 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=16x2025.06 | 99.6 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=64x2025.06 | 99.6 | |
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=16x2025.06 | 99.5 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=16x2025.06 | 99.2 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=256x2025.06 | 99 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=64x2025.06 | 98.3 | |
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=64x2025.06 | 97.8 | |
| ASEntmaxLayers (L)=4, OOD Sequence Length Scale=1024x2025.06 | 95.3 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=64x2025.06 | 92.7 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=256x2025.06 | 90.6 | |
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=256x2025.06 | 80.2 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=256x2025.06 | 66.8 | |
| SSMaxLayers (L)=4, OOD Sequence Length Scale=1024x2025.06 | 26.7 | |
| EntmaxLayers (L)=4, OOD Sequence Length Scale=1024x2025.06 | 9.3 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=4x2025.06 | 6.2 | |
| SoftmaxLayers (L)=4, OOD Sequence Length Scale=1024x2025.06 | 3 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=16x2025.06 | 0 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=64x2025.06 | 0 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=256x2025.06 | 0 | |
| Top-K (K=32)Layers (L)=4, OOD Sequence Length Scale=1024x2025.06 | 0 |