Zero-shot Reasoning on ARC Challenge
60.41Zero-Shot AccuracyFP16
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16Backbone=Llama-3-70B, Bits=FP162026.06 | 60.41 | |
| FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.07 | 57.71 | |
| LiftQuantBackbone=Llama-3-70B, Bits=2.012026.06 | 56.14 | |
| FP16Backbone=Llama-2-70B, Bits=FP162026.06 | 54.44 | |
| FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.07 | 54.18 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.012026.06 | 53.58 | |
| OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 53.24 | |
| QTIPBackbone=Llama-2-70B, Bits=2.002026.06 | 53.05 | |
| QuIP#Backbone=Llama-2-70B, Bits=2.002026.06 | 52.65 | |
| VPTQBackbone=Llama-3-70B, Bits=2.022026.06 | 52.65 | |
| AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 52.39 | |
| OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 51.81 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.412026.06 | 51.62 | |
| OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 51.37 | |
| EQAT-g64Backbone=Llama-2-70B, Bits=2.262026.06 | 50.77 | |
| FP16Backbone=Llama-3-8B, Bits=FP162026.06 | 50.43 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 50.34 | |
| OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 49.32 | |
| EQAT-g64Backbone=Llama-3-70B, Bits=2.252026.06 | 49.06 | |
| FP16Backbone=Llama-2-13B, Bits=FP162026.06 | 48.29 | |
| FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.07 | 46.16 | |
| OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 45.65 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.422026.06 | 44.8 | |
| QTIPBackbone=Llama-2-13B, Bits=2.002026.06 | 43.94 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.022026.06 | 43.69 | |
| OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 43.68 | |
| OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 43.6 | |
| FP16Backbone=Llama-2-7B, Bits=FP162026.06 | 43.52 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 42.92 | |
| QuIP#Backbone=Llama-2-13B, Bits=2.002026.06 | 42.92 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 41.99 | |
| EQAT-g64Backbone=Llama-2-13B, Bits=2.252026.06 | 41.89 | |
| QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 41.06 | |
| LiftQuantBackbone=Llama-3-8B, Bits=2.022026.06 | 40.87 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 40.61 | |
| AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 40.52 | |
| AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 40.12 | |
| QTIPBackbone=Llama-2-7B, Bits=2.002026.06 | 39.76 | |
| OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 39.59 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.422026.06 | 39.42 | |
| AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 38.4 | |
| AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 37.98 | |
| QuIP#Backbone=Llama-2-7B, Bits=2.002026.06 | 37.88 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 37.46 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 37.37 | |
| EQAT-g64Backbone=Llama-3-8B, Bits=2.252026.06 | 37.03 | |
| VPTQBackbone=Llama-3-8B, Bits=2.072026.06 | 36.91 | |
| EQAT-g64Backbone=Llama-2-7B, Bits=2.252026.06 | 36.86 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.022026.06 | 36.77 | |
| AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 33.49 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 25.6 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 25.26 | |
| QuaRotModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 25.26 |