List your company
ResearchFormerly WizWand

Natural Language Understanding on HellaSwag

85.6AccuracyNLS
33.693647.169360.64574.1207Nov 21, 2024Feb 19, 2025May 21, 2025Aug 20, 2025Nov 18, 2025Feb 17, 2026May 19, 2026
Updated 5mo ago

Evaluation results

MethodLinks
NLSModel=Llama 3 70B, Precision=W8A8, Time Steps (T)=22026.0585.6
PrefixQModel=Llama 2 7B, Precision=W8A8, Time Steps (T)=-2026.0576.44
PrefixQModel=Llama 2 7B, Precision=W6A6, Time Steps (T)=-2026.0574.06
Fixed (p = 4) + SPSequence Reduction=4.0, FLOPs/byte Reduction=2.12026.0559.7
Entropy-based + SPSequence Reduction=6.3†, FLOPs/byte Reduction=2.72026.0559.7
H-Net + SPSequence Reduction=5.3†, FLOPs/byte Reduction=2.42026.0559.6
SpaceByte + SPSequence Reduction=6.3, FLOPs/byte Reduction=2.72026.0559.5
Fixed (p = 8) + SPSequence Reduction=8.0, FLOPs/byte Reduction=2.62026.0559.2
H-NetSequence Reduction=4.8, FLOPs/byte Reduction=3.42026.0558.5
Tokenizer-basedSequence Reduction=3.7, FLOPs/byte Reduction=4.12026.0558.4
Fixed (p = 16) + SPSequence Reduction=16.0, FLOPs/byte Reduction=2.92026.0558
Fixed (p = 4)Sequence Reduction=4.0, FLOPs/byte Reduction=3.12026.0557.9
Entropy-basedSequence Reduction=6.4, FLOPs/byte Reduction=4.02026.0557.7
SpaceByteSequence Reduction=6.3, FLOPs/byte Reduction=4.02026.0557.3
Byte-levelSequence Reduction=1.0, FLOPs/byte Reduction=1.02026.0555.1
Fixed (p = 8)Sequence Reduction=8.0, FLOPs/byte Reduction=4.52026.0554.5
SYNPROModel Size=1.1B, Alpha (organic token ratio)=10%2026.0549.17
Fixed (p = 16)Sequence Reduction=16.0, FLOPs/byte Reduction=5.72026.0547.3
UniqueModel Size=400M, Alpha (organic token ratio)=10%2026.0543.03
FTProjection Type=Out-Proj, Number of Parameters=28.31M2024.1140.89
FTProjection Type=In-Proj, Number of Parameters=56.62M2024.1140.36
FTProjection Type=Both-Proj, Number of Parameters=84.94M2024.1140.18
ProDiaLProjection Type=Both-Proj, Number of Parameters=2.42M2024.1138.92
SYNPROModel Size=400M, Alpha (organic token ratio)=10%2026.0538.68
StrongProjection Type=Baseline, Number of Parameters=3.80M2024.1138.66
LoRAProjection Type=In-Proj, Number of Parameters=1.48M2024.1138.46
ProDiaLProjection Type=In-Proj, Number of Parameters=1.49M2024.1138.41
LoRAProjection Type=Both-Proj, Number of Parameters=2.36M2024.1138.33
Full-FTProjection Type=Baseline, Number of Parameters=130.00M2024.1138.23
ProDiaLProjection Type=Out-Proj, Number of Parameters=0.92M2024.1138.19
DoRAProjection Type=Both-Proj, Number of Parameters=2.45M2024.1138.13
DoRAProjection Type=In-Proj, Number of Parameters=1.55M2024.1138.08
LoRAProjection Type=Out-Proj, Number of Parameters=0.89M2024.1137.3
DoRAProjection Type=Out-Proj, Number of Parameters=0.90M2024.1137.19
BitFitProjection Type=Baseline, Number of Parameters=0.07M2024.1135.69