Zero-shot Reasoning on ARC-Easy
97.43Zero-shot AccuracyWeDLM-8B
Evaluation results
| Method | Links | |
|---|---|---|
| WeDLM-8BInstruct Model=true, Model Category=WeDLM, Shot Configuration=0-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 97.43 | |
| SDAR-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=0-shot, Inference Framework=JetEngine, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 97.01 | |
| Qwen3-8BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=0-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 96.17 | |
| WeDLM-7BInstruct Model=true, Model Category=WeDLM, Shot Configuration=0-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 96.09 | |
| LLaDA-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=0-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 94.32 | |
| Qwen2.5-7BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=0-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 93.27 | |
| Dream-7BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=0-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 93.27 | |
| FP16Backbone=Llama-3-70B, Bits=FP162026.06 | 86.99 | |
| LiftQuantBackbone=Llama-3-70B, Bits=2.012026.06 | 84.3 | |
| FP16Backbone=Llama-2-70B, Bits=FP162026.06 | 82.7 | |
| QuIP#Backbone=Llama-2-70B, Bits=2.002026.06 | 81.9 | |
| VPTQBackbone=Llama-3-70B, Bits=2.022026.06 | 81.86 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.412026.06 | 81.52 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.012026.06 | 81.36 | |
| QTIPBackbone=Llama-2-70B, Bits=2.002026.06 | 81.12 | |
| EQAT-g64Backbone=Llama-2-70B, Bits=2.262026.06 | 80.13 | |
| FP16Backbone=Llama-3-8B, Bits=FP162026.06 | 80.09 | |
| FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.07 | 79.42 | |
| FP16Backbone=Llama-2-13B, Bits=FP162026.06 | 79.42 | |
| OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 77.82 | |
| FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.07 | 77.62 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.422026.06 | 77.48 | |
| EQAT-g64Backbone=Llama-3-70B, Bits=2.252026.06 | 77.4 | |
| OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 77.26 | |
| QTIPBackbone=Llama-2-13B, Bits=2.002026.06 | 76.6 | |
| OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 76.3 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.022026.06 | 76.3 | |
| FP16Backbone=Llama-2-7B, Bits=FP162026.06 | 76.26 | |
| BaselineRatio=1.0, Memory Footprint (GB)=12.6GB, Zero-shot=true2026.04 | 76 | |
| QuIP#Backbone=Llama-2-13B, Bits=2.002026.06 | 75.72 | |
| EQAT-g64Backbone=Llama-2-13B, Bits=2.252026.06 | 74.83 | |
| Llama2-7BBackbone=Llama2-7B2026.02 | 74.6 | |
| FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.07 | 74.58 | |
| LiftQuantBackbone=Llama-3-8B, Bits=2.022026.06 | 74.33 | |
| OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 74.03 | |
| OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 74.01 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 73.95 | |
| QTIPBackbone=Llama-2-7B, Bits=2.002026.06 | 73.32 | |
| OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 73.06 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.422026.06 | 72.9 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.22026.02 | 72.6 | |
| OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 71.89 | |
| QuIP#Backbone=Llama-2-7B, Bits=2.002026.06 | 71.84 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 71.46 | |
| EQAT-g64Backbone=Llama-3-8B, Bits=2.252026.06 | 71.17 | |
| VPTQBackbone=Llama-3-8B, Bits=2.072026.06 | 71.03 | |
| QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 70.97 | |
| EQAT-g64Backbone=Llama-2-7B, Bits=2.252026.06 | 70.96 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.022026.06 | 70.66 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 70.22 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 69.61 | |
| AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 68.63 | |
| OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 65.99 | |
| Swift-SVD*Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 65 | |
| Swift-SVDRatio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 64 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 63.93 | |
| AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 63.26 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.42026.02 | 63.1 | |
| Dobi-SVD(w)Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 63 | |
| SVD-LLM(W)Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 62 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 60.48 | |
| QuIPWBits=42023.07 | 59.81 | |
| QuIPWBits=32023.07 | 59.81 | |
| OPTQWBits=162023.07 | 59.72 | |
| QuIPWBits=162023.07 | 59.72 | |
| OPTQWBits=42023.07 | 58.96 | |
| OPTQWBits=32023.07 | 54.38 | |
| QuIPWBits=22023.07 | 54.38 | |
| AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 54.06 | |
| ASVDRatio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 53 | |
| AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 52.99 | |
| AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 51.06 | |
| Swift-SVD*Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 51 | |
| Swift-SVDRatio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 49 | |
| AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 48.63 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.22026.02 | 47.6 | |
| Dobi-SVD(w)Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 45 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.42026.02 | 34.4 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.62026.02 | 33.8 | |
| SVD-LLM(W)Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 33 | |
| Swift-SVD*Ratio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 30 | |
| Swift-SVDRatio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 29 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.62026.02 | 28.3 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 27.99 | |
| QuaRotModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 27.99 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 26.18 | |
| OPTQWBits=22023.07 | 25.34 | |
| Dobi-SVD(w)Ratio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 25 | |
| FWSVDRatio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 11 | |
| ASVDRatio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 11 | |
| FWSVDRatio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 5 | |
| SVD-LLM(W)Ratio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 5 | |
| ASVDRatio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 4 | |
| FWSVDRatio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 2 |