List your company
ResearchFormerly WizWand

Natural Language Understanding on ARC-c

65.8AccuracymPLUG-Owl2
25.323235.831646.3456.8484Nov 7, 2023Apr 7, 2024Sep 7, 2024Feb 6, 2025Jul 9, 2025Dec 8, 2025May 10, 2026
Updated 5mo ago

Evaluation results

MethodLinks
mPLUG-Owl2Shot setting=0-shot2023.1165.8
Vicuna-v1.5Shot setting=0-shot2023.1156.6
LLaMA-2-ChatShot setting=0-shot2023.1154.9
WizardLMShot setting=0-shot2023.1147.5
SpaceByte + SPSequence Reduction=6.3, FLOPs/byte Reduction=2.72026.0541.9
Fixed (p = 4) + SPSequence Reduction=4.0, FLOPs/byte Reduction=2.12026.0541.6
H-Net + SPSequence Reduction=5.3†, FLOPs/byte Reduction=2.42026.0540.9
Fixed (p = 16) + SPSequence Reduction=16.0, FLOPs/byte Reduction=2.92026.0540.8
Tokenizer-basedSequence Reduction=3.7, FLOPs/byte Reduction=4.12026.0540.7
Entropy-basedSequence Reduction=6.4, FLOPs/byte Reduction=4.02026.0540.6
Entropy-based + SPSequence Reduction=6.3†, FLOPs/byte Reduction=2.72026.0540.4
LLaMA-2Shot setting=0-shot2023.1140.3
H-NetSequence Reduction=4.8, FLOPs/byte Reduction=3.42026.0540.1
Fixed (p = 4)Sequence Reduction=4.0, FLOPs/byte Reduction=3.12026.0539.8
Fixed (p = 8) + SPSequence Reduction=8.0, FLOPs/byte Reduction=2.62026.0539.8
Fixed (p = 8)Sequence Reduction=8.0, FLOPs/byte Reduction=4.52026.0539.6
Byte-levelSequence Reduction=1.0, FLOPs/byte Reduction=1.02026.0538.7
SpaceByteSequence Reduction=6.3, FLOPs/byte Reduction=4.02026.0538.6
Fixed (p = 16)Sequence Reduction=16.0, FLOPs/byte Reduction=5.72026.0535.2
ProDiaLProjection Type=Out-Proj, Number of Parameters=0.90M2024.1130.8
FTProjection Type=In-Proj, Number of Parameters=56.62M2024.1129.61
ProDiaLProjection Type=In-Proj, Number of Parameters=1.25M2024.1129.61
FTProjection Type=Both-Proj, Number of Parameters=84.94M2024.1129.52
LoRAProjection Type=Both-Proj, Number of Parameters=2.36M2024.1129.52
FTProjection Type=Out-Proj, Number of Parameters=28.31M2024.1129.27
Full-FTProjection Type=Baseline, Number of Parameters=130.00M2024.1128.84
ProDiaLProjection Type=Both-Proj, Number of Parameters=2.38M2024.1128.84
DoRAProjection Type=Both-Proj, Number of Parameters=2.45M2024.1128.75
StrongProjection Type=Baseline, Number of Parameters=3.80M2024.1128.67
DoRAProjection Type=Out-Proj, Number of Parameters=0.90M2024.1128.67
LoRAProjection Type=Out-Proj, Number of Parameters=0.89M2024.1128.54
DoRAProjection Type=In-Proj, Number of Parameters=1.55M2024.1128.5
LoRAProjection Type=In-Proj, Number of Parameters=1.48M2024.1128.41
BitFitProjection Type=Baseline, Number of Parameters=0.07M2024.1126.88