List your company
ResearchFormerly WizWand

Zero-shot Common Sense Reasoning on BoolQ

81.28AccuracyFP16
39.492850.341461.1972.0386Jul 30, 2025Aug 14, 2025Aug 29, 2025Sep 14, 2025Sep 29, 2025Oct 14, 2025Oct 30, 2025
Updated 5mo ago

Evaluation results

MethodLinks
FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.0781.28
OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0780.02
OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0778.67
FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.0778.59
FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.0777.39
OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0776.83
OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0776.06
OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.0775.77
OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0775.64
AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0774.59
AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0774.55
OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.0774.2
OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0774.01
OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.0773.03
QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.0773
QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0772.48
OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.0770.85
AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0768.07
AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0767.67
QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0766.57
OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0765.47
OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0763.89
Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.1063.21
SambaSetting=Zero-shot, Model Size=1.3B parameters2025.1062.11
Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.1061.55
AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0761.19
MambaSetting=Zero-shot, Model Size=1.3B parameters2025.1061.13
RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.1060.39
Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.1060.24
Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.1060.13
Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.1059.27
AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0758.73
Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.1057.09
HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.1056.66
DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.1055.29
Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.1054.68
QuaRotModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0743.82
QuaRotModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0741.1
QuaRotModel=Mistral, Precision=W4A3, Sequence Length=20482025.0741.1