Zero-shot QA and Reasoning on Evaluation Suite (ARC, LogiQA, Wino, CSQA, BoolQ, PIQA, MMLU)
83.88ARCAvg-DeUS
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Avg-DeUSBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 83.88 | 26.42 | 75.45 | 72.89 | 83.18 | 80.47 | 63.1 | 69.34 | |
| LESABackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 83.84 | 26.57 | 75.53 | 73.05 | 83 | 80.69 | 63.57 | 69.47 | |
| OpT-DeUSBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 83.8 | 26.73 | 76.09 | 73.05 | 83.36 | 80.85 | 63.84 | 69.67 | |
| MIDUS-HMLBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 83.5 | 28.11 | 74.9 | 73.63 | 83.39 | 80.96 | 64.54 | 69.86 | |
| Llama ProBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 83.33 | 27.19 | 74.11 | 72.07 | 82.26 | 80.79 | 62.32 | 68.87 | |
| MIDUS-HMLBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 82.37 | 28.57 | 74.59 | 70.84 | 82.87 | 80.25 | 63.4 | 68.98 | |
| Avg-DeUSBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 82.15 | 27.5 | 73.48 | 71.09 | 82.17 | 80.2 | 62.11 | 68.39 | |
| LESABackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 82.07 | 27.96 | 74.11 | 72.4 | 81.93 | 80.3 | 62.63 | 68.77 | |
| OpT-DeUSBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 82.07 | 27.34 | 74.74 | 71.91 | 82.26 | 80.79 | 62.96 | 68.87 | |
| Llama ProBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 81.61 | 29.49 | 73.72 | 70.93 | 81.65 | 79.98 | 62.56 | 68.56 | |
| BaseBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 81.1 | 24.58 | 72.14 | 68.3 | 82.14 | 79.71 | 59.17 | 66.73 | |
| SOLARBackbone=Llama-3.1-8B, Training Stage=SFT, Evaluation Protocol=Zero-shot2025.12 | 80.68 | 25.19 | 71.19 | 61.81 | 81.19 | 79.16 | 55.03 | 64.8 | |
| BaseBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 79.97 | 26.88 | 72.06 | 65.19 | 81.83 | 78.84 | 58.61 | 66.2 | |
| SOLARBackbone=Llama-3.1-8B, Training Stage=CPT, Evaluation Protocol=Zero-shot2025.12 | 79.88 | 26.88 | 71.59 | 57.41 | 80.7 | 78.56 | 54.37 | 64.2 | |
| SOLARBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 69.4 | 23.04 | 59.04 | 27.52 | 60.03 | 75.14 | 30.77 | 49.28 | |
| BaseBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 68.64 | 21.35 | 60.3 | 27.52 | 63.27 | 75.24 | 30.49 | 49.55 | |
| Llama ProBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 68.1 | 23.81 | 60.54 | 41.93 | 63.7 | 75.24 | 34.03 | 52.48 | |
| Avg-DeUSBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 67.51 | 22.73 | 60.3 | 46.6 | 65.02 | 74.37 | 36.79 | 53.33 | |
| OpT-DeUSBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 67.21 | 23.81 | 61.48 | 49.14 | 63.76 | 75.19 | 37.83 | 54.06 | |
| MIDUS-HMLBase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 66.62 | 22.12 | 61.72 | 48.65 | 66.51 | 75.14 | 38.59 | 54.19 | |
| LESABase Model=Llama-3.2-1B, Pre-training=CPT-1B, SFT Dataset=Databricks-Dolly-15k, Protocol=Zero-shot Accuracy2025.12 | 66.46 | 23.66 | 60.77 | 49.06 | 64.98 | 74.86 | 38.22 | 54 |