Zero-shot Evaluation on Reasoning Tasks (PI, AE, AC, HS, WG, LB)
81.6PI Score (Zero-shot)LoLA-8B
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| LoLA-8BTokens (B)=0.04, Teacher Model=Llama-3.1-8B2025.05 | 81.6 | 82.5 | 55.4 | 79.8 | 73.6 | 74.9 | 74.6 | 1.009 | |
| Llama-3.1-8BTokens (B)=150002025.05 | 81.1 | 81.7 | 55.1 | 79.3 | 73.9 | 73 | 74 | — | |
| LoLCATs-8BTokens (B)=0.04, Teacher Model=Llama-3.1-8B2025.05 | 81 | 82.4 | 54.4 | 79.1 | 73.6 | 67.6 | 73 | 0.987 | |
| Llamba-8BTokens (B)=12, Teacher Model=Llama-3.1-8B2025.05 | 80.9 | 82.5 | 54.6 | 77.6 | 73.3 | 69.4 | 73.1 | 0.987 | |
| Llama-3.1-8B-Instr.Tokens (B)=15000+2025.05 | 80.8 | 81.8 | 55.2 | 79.2 | 73.9 | — | 74 | — | |
| LoLCATs-1.3BTokens (B)=0.04, Teacher Model=Phi-1.5-1.3B2025.05 | 76.9 | 77 | 46.9 | 62.3 | 72.7 | — | — | — | |
| Mamba2-Llama3-8BTokens (B)=20, Teacher Model=Llama-3.1-8B Instruct2025.05 | 76.8 | 74.1 | 48 | 70.8 | 58.6 | 43.2 | 61.9 | 0.837 | |
| Phi-1.5-1.3BTokens (B)=1502025.05 | 76.6 | 75.6 | 48 | 62.6 | 73.4 | 53.4 | 64.9 | — | |
| LoLA-1.3BTokens (B)=0.04, Teacher Model=Llama-3.2-1.3B2025.05 | 76.2 | 66.2 | 36.9 | 64.1 | 60.9 | 61.9 | 61 | 1.017 | |
| Phi-Mamba1.5BTokens (B)=3, Teacher Model=Phi-1.5-1.3B2025.05 | 75.5 | 74 | 44.1 | 60.2 | 71.7 | 50.1 | 62.6 | 0.965 | |
| LoLCATs-1.3BTokens (B)=0.04, Teacher Model=Llama-3.2-1.3B2025.05 | 74.6 | 63 | 35.1 | 63.7 | 61.5 | 53.4 | 58.6 | 0.977 | |
| Llama-3.2-1.3BTokens (B)=90002025.05 | 74.4 | 65.5 | 35.8 | 63.7 | 60.5 | 60.1 | 60 | — | |
| Llamba-1.3BTokens (B)=8, Teacher Model=Llama-3.2-1.3B2025.05 | 74 | 69.5 | 37.2 | 61.2 | 60.6 | 48.4 | 58.5 | 0.975 |