List your company
ResearchFormerly WizWand

Zero-shot Reasoning on ARC Challenge

60.41Zero-Shot AccuracyFP16
23.85433.344542.83552.3255Jul 30, 2025Sep 19, 2025Nov 9, 2025Dec 30, 2025Feb 19, 2026Apr 11, 2026Jun 2, 2026
Updated 5mo ago

Evaluation results

MethodLinks
FP16Backbone=Llama-3-70B, Bits=FP162026.0660.41
FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.0757.71
LiftQuantBackbone=Llama-3-70B, Bits=2.012026.0656.14
FP16Backbone=Llama-2-70B, Bits=FP162026.0654.44
FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.0754.18
LiftQuantBackbone=Llama-2-70B, Bits=2.012026.0653.58
OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.0753.24
QTIPBackbone=Llama-2-70B, Bits=2.002026.0653.05
QuIP#Backbone=Llama-2-70B, Bits=2.002026.0652.65
VPTQBackbone=Llama-3-70B, Bits=2.022026.0652.65
AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0752.39
OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.0751.81
LiftQuantBackbone=Llama-2-70B, Bits=2.412026.0651.62
OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0751.37
EQAT-g64Backbone=Llama-2-70B, Bits=2.262026.0650.77
FP16Backbone=Llama-3-8B, Bits=FP162026.0650.43
OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0750.34
OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.0749.32
EQAT-g64Backbone=Llama-3-70B, Bits=2.252026.0649.06
FP16Backbone=Llama-2-13B, Bits=FP162026.0648.29
FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.0746.16
OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0745.65
LiftQuantBackbone=Llama-2-13B, Bits=2.422026.0644.8
QTIPBackbone=Llama-2-13B, Bits=2.002026.0643.94
LiftQuantBackbone=Llama-2-13B, Bits=2.022026.0643.69
OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.0743.68
OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0743.6
FP16Backbone=Llama-2-7B, Bits=FP162026.0643.52
OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0742.92
QuIP#Backbone=Llama-2-13B, Bits=2.002026.0642.92
OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0741.99
EQAT-g64Backbone=Llama-2-13B, Bits=2.252026.0641.89
QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.0741.06
LiftQuantBackbone=Llama-3-8B, Bits=2.022026.0640.87
QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.0740.61
AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0740.52
AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0740.12
QTIPBackbone=Llama-2-7B, Bits=2.002026.0639.76
OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0739.59
LiftQuantBackbone=Llama-2-7B, Bits=2.422026.0639.42
AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.0738.4
AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0737.98
QuIP#Backbone=Llama-2-7B, Bits=2.002026.0637.88
QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.0737.46
OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0737.37
EQAT-g64Backbone=Llama-3-8B, Bits=2.252026.0637.03
VPTQBackbone=Llama-3-8B, Bits=2.072026.0636.91
EQAT-g64Backbone=Llama-2-7B, Bits=2.252026.0636.86
LiftQuantBackbone=Llama-2-7B, Bits=2.022026.0636.77
AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.0733.49
QuaRotModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.0725.6
QuaRotModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.0725.26
QuaRotModel=Mistral, Precision=W4A3, Sequence Length=20482025.0725.26