Question Answering on Reasoning Suite (ArcC, ArcE, PiQA, Wino) Zero-shot
50.43Arc Challenge (C) AccuracyDense
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| DenseAvg. bits=16, PV-tuning=false2025.05 | 50.43 | 80.09 | 79.71 | 72.61 | 70.71 | |
| DBFAvg. bits=2.3, PV-tuning=false2025.05 | 45.73 | 76.93 | 77.47 | 70.32 | 67.61 | |
| DBFAvg. bits=2.3, PV-tuning=true2025.05 | 45.56 | 76.76 | 78.01 | 71.9 | 68.06 | |
| DBFAvg. bits=2, PV-tuning=true2025.05 | 44.45 | 75.33 | 76.98 | 70.08 | 66.71 | |
| QTIPAvg. bits=2, PV-tuning=false2025.05 | 44.2 | 75.2 | 77.6 | 70.7 | 66.93 | |
| AQLMAvg. bits=2.3, PV-tuning=true2025.05 | 42.32 | 75.46 | 78.45 | 71.67 | 66.98 | |
| CalderaAvg. bits=2.4, PV-tuning=false2025.05 | 42.3 | 73.6 | 76.5 | 70.3 | 65.68 | |
| DBFAvg. bits=2, PV-tuning=false2025.05 | 41.04 | 74.45 | 76.82 | 67.95 | 65.07 | |
| CalderaAvg. bits=2.1, PV-tuning=false2025.05 | 40 | 71.5 | 76 | 69.5 | 64.25 | |
| QUIP#Avg. bits=2, PV-tuning=false2025.05 | 39.2 | 72.9 | 75.6 | 68.2 | 63.98 | |
| DBFAvg. bits=1.5, PV-tuning=true2025.05 | 35.58 | 68.68 | 74.1 | 65.43 | 60.95 | |
| DBFAvg. bits=1.5, PV-tuning=false2025.05 | 33.61 | 67.59 | 73.99 | 64.64 | 59.96 | |
| AQLMAvg. bits=1.5, PV-tuning=true2025.05 | 32.68 | 65.78 | 72.63 | 64.4 | 58.87 | |
| DBFAvg. bits=1, PV-tuning=true2025.05 | 25.08 | 56.22 | 67.46 | 58.8 | 51.89 | |
| DBFAvg. bits=1, PV-tuning=false2025.05 | 21.67 | 50.25 | 65.17 | 56.43 | 48.38 | |
| BiLLMAvg. bits=1.1, PV-tuning=false2025.05 | 17.7 | 36 | 56.1 | 51 | 40.2 |