Cultural Reasoning on VIRAASAT (test)
91.45State MatchQwen2.5-7B-Instruct
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Qwen2.5-7B-InstructSetting=SFT (SCoM)2026.02 | 91.45 | 58.01 | 57.54 | |
| Qwen2.5-7B-InstructSetting=SFT (CoT)2026.02 | 89.11 | 37.33 | 36.7 | |
| Qwen3-4B-Thinking-2507Setting=SFT (CoT)2026.02 | 88.22 | 37.36 | 35.5 | |
| Qwen2.5-3B-InstructSetting=SFT (CoT)2026.02 | 87.87 | 34.73 | 34.26 | |
| Qwen2.5-3B-InstructSetting=SFT (SCoM)2026.02 | 87.25 | 51.48 | 50.7 | |
| Phi-4-mini-reasoningSetting=SFT (CoT)2026.02 | 84.19 | 30.08 | 28.53 | |
| gemini-2.0-flashSetting=Zero-shot2026.02 | 81.71 | 35.19 | 31.32 | |
| Phi-4-mini-reasoningSetting=SFT (SCoM)2026.02 | 81.03 | 43.55 | 43.39 | |
| GPT4.1-miniSetting=Zero-shot2026.02 | 77.36 | 34.57 | 32.56 | |
| Mistral-7B-Instruct-v0.3Setting=Zero-shot2026.02 | 57.05 | 15.04 | 12.25 | |
| Llama-3.1-8B-InstructSetting=Zero-shot2026.02 | 55.81 | 11.78 | 10.08 | |
| Qwen2.5-7B-InstructSetting=Zero-shot2026.02 | 43.88 | 3.72 | 1.71 | |
| Phi-4-mini-reasoningSetting=Zero-shot2026.02 | 35.97 | 8.84 | 6.05 | |
| Qwen2.5-3B-InstructSetting=Zero-shot2026.02 | 25.43 | 3.72 | 2.17 | |
| OpenHathi-7B-Hi-v0.1-BaseSetting=Zero-shot2026.02 | 25.27 | 4.19 | 2.17 | |
| gemma-2b-itSetting=Zero-shot2026.02 | 24.34 | 4.96 | 3.26 | |
| sarvam-1Setting=Zero-shot2026.02 | 11.78 | 7.13 | 2.17 | |
| Qwen3-4B-Thinking-2507Setting=Zero-shot2026.02 | 7.44 | 1.71 | 0.31 |