Commonsense Reasoning on IndoCulture (test)
79.3AccuracyIndoSoSci-based RAG (Sailor2-L-20B-Chat)
Evaluation results
| Method | Links | |
|---|---|---|
| IndoSoSci-based RAG (Sailor2-L-20B-Chat)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 79.3 | |
| IndoSoSci-based RAG (Sailor2-L-20B)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 75.7 | |
| IndoSoSci-based RAG (Qwen-SEA-LION-v4-32B-IT)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 75.5 | |
| Sailor2-L-20B-ChatRAG=None, Setting=Zero-shot2026.01 | 75.4 | |
| IndoSoSci-based RAG (Sailor2-L-8B)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 74.5 | |
| IndoSoSci-based RAG (Sailor2-L-8B-Chat)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 73.9 | |
| Sailor2-L-20BRAG=None, Setting=Zero-shot2026.01 | 72.1 | |
| Qwen-SEA-LION-v4-32B-ITRAG=None, Setting=Zero-shot2026.01 | 70.9 | |
| Sailor2-L-8B-ChatRAG=None, Setting=Zero-shot2026.01 | 70.5 | |
| IndoSoSci-based RAG (SEALLMs-v3-7B-Chat)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 65.3 | |
| Sailor2-L-8BRAG=None, Setting=Zero-shot2026.01 | 64.2 | |
| IndoSoSci-based RAG (SEALLMs-v3-7B)RAG=IndoSoSci facts + hypothetical document queries, D=20, Setting=Zero-shot2026.01 | 61.3 | |
| SEALLMs-v3-7B-ChatRAG=None, Setting=Zero-shot2026.01 | 60.6 | |
| SEALLMs-v3-7BRAG=None, Setting=Zero-shot2026.01 | 54.6 |