Zero-shot Common Sense Reasoning on BoolQ
81.28AccuracyFP16
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.07 | 81.28 | |
| OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 80.02 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 78.67 | |
| FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.07 | 78.59 | |
| FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.07 | 77.39 | |
| OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 76.83 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 76.06 | |
| OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 75.77 | |
| OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 75.64 | |
| AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 74.59 | |
| AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 74.55 | |
| OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 74.2 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 74.01 | |
| OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 73.03 | |
| QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 73 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 72.48 | |
| OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 70.85 | |
| AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 68.07 | |
| AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 67.67 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 66.57 | |
| OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 65.47 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 63.89 | |
| Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 63.21 | |
| SambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 62.11 | |
| Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 61.55 | |
| AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 61.19 | |
| MambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 61.13 | |
| RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 60.39 | |
| Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 60.24 | |
| Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 60.13 | |
| Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 59.27 | |
| AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 58.73 | |
| Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 57.09 | |
| HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 56.66 | |
| DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 55.29 | |
| Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 54.68 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 43.82 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 41.1 | |
| QuaRotModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 41.1 |