Natural Language Understanding on ARC Easy
78.3AccuracyArcana
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Arcanazero-shot=true2024.10 | 78.3 | — | |
| Vicuna-v1.5zero-shot=true2024.10 | 72.8 | — | |
| NLSModel=Llama 3 8B, Precision=W8A8, Time Steps (T)=42026.05 | 72.77 | 77.9 | |
| LLaMA-2-Chatzero-shot=true2024.10 | 71.6 | — | |
| Tokenizer-basedSequence Reduction=3.7, FLOPs/byte Reduction=4.12026.05 | 71.6 | — | |
| Fixed (p = 4) + SPSequence Reduction=4.0, FLOPs/byte Reduction=2.12026.05 | 71 | — | |
| H-NetSequence Reduction=4.8, FLOPs/byte Reduction=3.42026.05 | 70.3 | — | |
| SpaceByte + SPSequence Reduction=6.3, FLOPs/byte Reduction=2.72026.05 | 69.8 | — | |
| Fixed (p = 8) + SPSequence Reduction=8.0, FLOPs/byte Reduction=2.62026.05 | 68.5 | — | |
| Fixed (p = 16) + SPSequence Reduction=16.0, FLOPs/byte Reduction=2.92026.05 | 67.5 | — | |
| Fixed (p = 4)Sequence Reduction=4.0, FLOPs/byte Reduction=3.12026.05 | 67.4 | — | |
| SpaceByteSequence Reduction=6.3, FLOPs/byte Reduction=4.02026.05 | 67.3 | — | |
| H-Net + SPSequence Reduction=5.3†, FLOPs/byte Reduction=2.42026.05 | 67 | — | |
| Byte-levelSequence Reduction=1.0, FLOPs/byte Reduction=1.02026.05 | 66.6 | — | |
| Entropy-basedSequence Reduction=6.4, FLOPs/byte Reduction=4.02026.05 | 66.1 | — | |
| Entropy-based + SPSequence Reduction=6.3†, FLOPs/byte Reduction=2.72026.05 | 66 | — | |
| Fixed (p = 8)Sequence Reduction=8.0, FLOPs/byte Reduction=4.52026.05 | 64.9 | — | |
| DuQModel=Llama 3 8B, Precision=W6A6, Time Steps (T)=-2026.05 | 62.25 | 53.07 | |
| WizardLMzero-shot=true2024.10 | 59.6 | — | |
| Fixed (p = 16)Sequence Reduction=16.0, FLOPs/byte Reduction=5.72026.05 | 56.5 | — | |
| LLaMA-2zero-shot=true2024.10 | 56.1 | — | |
| ProDiaLProjection Type=Both-Proj, Number of Parameters=2.38M2024.11 | 55.18 | — | |
| FTProjection Type=Out-Proj, Number of Parameters=28.31M2024.11 | 55.09 | — | |
| ProDiaLProjection Type=Out-Proj, Number of Parameters=0.90M2024.11 | 54.84 | — | |
| FTProjection Type=In-Proj, Number of Parameters=56.62M2024.11 | 54.59 | — | |
| ProDiaLProjection Type=In-Proj, Number of Parameters=1.25M2024.11 | 54.5 | — | |
| FTProjection Type=Both-Proj, Number of Parameters=84.94M2024.11 | 54.38 | — | |
| StrongProjection Type=Baseline, Number of Parameters=3.80M2024.11 | 54.17 | — | |
| DoRAProjection Type=Both-Proj, Number of Parameters=2.45M2024.11 | 54.12 | — | |
| DoRAProjection Type=In-Proj, Number of Parameters=1.55M2024.11 | 54.04 | — | |
| LoRAProjection Type=Both-Proj, Number of Parameters=2.36M2024.11 | 53.87 | — | |
| LoRAProjection Type=In-Proj, Number of Parameters=1.48M2024.11 | 53.87 | — | |
| LoRAProjection Type=Out-Proj, Number of Parameters=0.89M2024.11 | 53.66 | — | |
| DoRAProjection Type=Out-Proj, Number of Parameters=0.90M2024.11 | 53.66 | — | |
| Full-FTProjection Type=Baseline, Number of Parameters=130.00M2024.11 | 53.54 | — | |
| BitFitProjection Type=Baseline, Number of Parameters=0.07M2024.11 | 52.86 | — |