Syllogistic Reasoning on Syllogistic Reasoning (test)
89.9AccuracyContrastive Steering (Qwen 2.5)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Contrastive Steering (Qwen 2.5)Size=7b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=-1.52025.05 | 89.9 | 0.96 | 93.65 | 77.17 | 468.26 | |
| Contrastive Steering (Qwen 2.5)Size=7b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=0.92025.05 | 89.9 | 4.81 | 18.7 | 9.1 | 94.79 | |
| Qwen 2.5Size=7b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 89.8 | 9.36 | 9.6 | — | — | |
| Qwen 2.5Size=7b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 88.85 | 5.39 | 16.48 | — | — | |
| Contrastive Steering (Qwen 2.5)Size=3b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=0.62025.05 | 86.63 | 4.8 | 18.02 | 11.81 | 190.18 | |
| Qwen 2.5Size=3b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 86.44 | 13.91 | 6.21 | — | — | |
| Qwen 2.5Size=3b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 85.29 | 7.12 | 11.99 | — | — | |
| Contrastive Steering (Qwen 2.5)Size=3b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=0.02025.05 | 85.29 | 7.12 | 11.99 | 0 | 0 | |
| Contrastive Steering (Llama 3.1)Size=8b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=0.92025.05 | 85.1 | 14.04 | 6.06 | 3.52 | 138.58 | |
| Gemma 2Size=9b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 85 | 8.46 | 10.05 | — | — | |
| Gemma 2Size=9b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 84.61 | 15.25 | 5.54 | — | — | |
| Contrastive Steering (Gemma 2)Size=9b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=0.62025.05 | 83.27 | 1.92 | 43.37 | 33.32 | 331.54 | |
| Contrastive Steering (Gemma 2)Size=9b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=0.32025.05 | 80.38 | 3.33 | 24.14 | 18.6 | 335.74 | |
| Llama 3.1Size=8b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 78.27 | 30.77 | 2.54 | — | — | |
| Contrastive Steering (Llama 3.2)Size=3b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=0.32025.05 | 78.27 | 22.31 | 3.51 | 0.92 | 35.52 | |
| Llama 3.2Size=3b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 77.79 | 17.5 | 4.45 | — | — | |
| Contrastive Steering (Llama 3.2)Size=3b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=0.02025.05 | 77.79 | 17.5 | 4.45 | 0 | 0 | |
| Contrastive Steering (Qwen 2.5)Size=1.5b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=0.32025.05 | 77.79 | 12.88 | 6.04 | 0.79 | 15.05 | |
| Qwen 2.5Size=1.5b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 75.67 | 14.42 | 5.25 | — | — | |
| Contrastive Steering (Gemma 2)Size=2b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=1.82025.05 | 74.13 | 20.83 | 3.56 | 1.3 | 57.52 | |
| Contrastive Steering (Llama 3.2)Size=1b, Setting=Zero-shot, Steering=Steered Model (αbest), αbest=-0.92025.05 | 73.56 | 6.35 | 11.58 | 10.26 | 777.27 | |
| Gemma 2Size=2b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 73.27 | 32.43 | 2.26 | — | — | |
| Llama 3.2Size=3b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 72.79 | 28.14 | 2.59 | — | — | |
| Contrastive Steering (Qwen 2.5)Size=1.5b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=-1.22025.05 | 72.69 | 32.18 | 2.26 | 1.46 | 182.5 | |
| Contrastive Steering (Gemma 2)Size=2b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=-0.32025.05 | 70 | 12.31 | 5.69 | 0.81 | 16.6 | |
| Gemma 2Size=2b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 69.42 | 14.23 | 4.88 | — | — | |
| Contrastive Steering (Llama 3.2)Size=1b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=-1.52025.05 | 66.44 | 6.08 | 4.94 | 3.69 | 295.2 | |
| Llama 3.2Size=1b, Setting=Zero-shot, Steering=Base Model (α = 0)2025.05 | 58.17 | 44.04 | 1.32 | — | — | |
| Llama 3.2Size=1b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 57.21 | 45.7 | 1.25 | — | — | |
| Qwen 2.5Size=1.5b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 51.92 | 65.12 | 0.8 | — | — | |
| Llama 3.1Size=8b, Setting=Few-shot, Steering=Base Model (α = 0)2025.05 | 40.58 | 20.26 | 2 | — | — | |
| Contrastive Steering (Llama 3.1)Size=8b, Setting=Few-shot, Steering=Steered Model (αbest), αbest=0.32025.05 | 30.67 | 14.81 | 2.07 | 0.07 | 3.5 |