List your company
ResearchFormerly WizWand

Common Sense Reasoning on HellaSwag (0-shot)

84.4Accuracyfp16
35.83248.44161.0573.659Oct 30, 2025Dec 8, 2025Jan 17, 2026Feb 26, 2026Apr 6, 2026May 16, 2026Jun 25, 2026
Updated 4mo ago

Evaluation results

MethodLinks
fp16Size=70B2026.0284.4
OmniQuantSize=70B2026.0283.1
AstroSize=70B2026.0281.7
MagRSize=70B2026.0281.1
fp16Size=13B2026.0279.7
FP16q (mean bit precision)=16.000, tok/s (throughput)=103.8, Evaluation Protocol=0-shot2025.1279.2
OmniQuantSize=13B2026.0276.9
fp16Size=7B2026.0276.2
AstroSize=13B2026.0276.2
AQLM-1x16 + PV-Tuningq (mean bit precision)=2.213, tok/s (throughput)=49.0, PV-Tuning=true, Evaluation Protocol=0-shot2025.1274.33
MagRSize=13B2026.0274.2
CodeGEMM-m1v4g128 + PV-Tuningq (mean bit precision)=2.126, tok/s (throughput)=228.3, PV-Tuning=true, Evaluation Protocol=0-shot2025.1273.85
OmniQuantSize=7B2026.0273.4
CodeGEMM-m2v8g128 + PV-Tuningq (mean bit precision)=2.127, tok/s (throughput)=214.4, PV-Tuning=true, Evaluation Protocol=0-shot2025.1273.02
FP16Bits=–, Backbone=LLAMA-2-7B2026.0472.96
AQLM-2x8 + PV-Tuningq (mean bit precision)=2.005, tok/s (throughput)=124.5, PV-Tuning=true, Evaluation Protocol=0-shot2025.1272.43
AstroSize=7B2026.0272.2
AQLM-1x16q (mean bit precision)=2.213, tok/s (throughput)=49.0, Evaluation Protocol=0-shot2025.1270.21
MagRSize=7B2026.0270.2
TernaryLLM-8B+KD#Tokens=1T, Quantization=W1.58A162026.0663.9
CodeGEMM-m1v4g128q (mean bit precision)=2.126, tok/s (throughput)=228.3, Evaluation Protocol=0-shot2025.1263.07
CodeGEMM-m2v8g128q (mean bit precision)=2.127, tok/s (throughput)=214.4, Evaluation Protocol=0-shot2025.1262.84
BitNetV1-7B#Tokens=100B, Quantization=W1.58A162026.0661.49
AQLM-2x8q (mean bit precision)=2.005, tok/s (throughput)=124.5, Evaluation Protocol=0-shot2025.1261.4
Qwen3-8B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.0659.57
LBLLMBits=W(1+1)A4, Backbone=LLAMA-2-7B2026.0458.54
Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.1058.25
FBI-LLMBits=W1A16, Backbone=LLAMA-2-7B2026.0457.7
Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.1057.43
Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.1056.88
Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.1056.53
Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.1055.76
Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.1055.67
Qwen3-4B + CAT-Q#Tokens=2M, Quantization=W1.58A162026.0655.01
SambaSetting=Zero-shot, Model Size=1.3B parameters2025.1053.42
Qwen3-4B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.0653.24
MambaSetting=Zero-shot, Model Size=1.3B parameters2025.1052.91
OnebitBits=W1A16, Backbone=LLAMA-2-7B2026.0451.54
DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.1050.93
Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.1050.23
HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.1049.53
RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.1049.16
TriLM-3.9B#Tokens=300B, Quantization=W1.58A162026.0648.3
Qwen3-1.7B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.0647.77
TequilaLLM-3B#Tokens=10B, Quantization=W1.58A162026.0646.4
BitNetV1-3.9B#Tokens=100B, Quantization=W1.58A162026.0644.2
FlexRound-q2g128q (mean bit precision)=2.125, tok/s (throughput)=205.3, Evaluation Protocol=0-shot2025.1243.78
TriLM-1.5B#Tokens=300B, Quantization=W1.58A162026.0640.9
BitNetV1-1.3B#Tokens=100B, Quantization=W1.58A162026.0637.7