Unified Multi-task Language Understanding and Instruction Following on Open LLM Leaderboard v1 (test)
19.84MMLU-P AccuracyTaskPGM (PMI)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| TaskPGM (PMI)Size=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.84 | 28.53 | 42.01 | 1.96 | 46.65 | — | — | 35.95 | |
| LESSSize=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.7 | 26.17 | 41.95 | 1.51 | 43.41 | — | — | 35.04 | |
| TaskPGM (PMI)Size=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.67 | 27.94 | 41.59 | 1.89 | 45.11 | — | — | 35.77 | |
| TaskPGM (JSD)Size=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.27 | 28.02 | 36.24 | 0.98 | 42.46 | — | — | 39.25 | |
| TaskPGM (PMI)Size=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.24 | 27.18 | 40.95 | 1.59 | 42.98 | — | — | 35.61 | |
| TaskPGM (JSD)Size=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.23 | 28.94 | 37.98 | 2.73 | 41.01 | — | — | 41.69 | |
| LESSSize=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 19.06 | 24.75 | 41.33 | 1.36 | 42.26 | — | — | 34.07 | |
| UniformSize=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.95 | 27.85 | 34.8 | 1.61 | 40.57 | — | — | 39.01 | |
| RandomSize=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.93 | 26.21 | 34.58 | 1.13 | 41.01 | — | — | 37.05 | |
| UniformSize=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.9 | 27.1 | 35.69 | 1.89 | 37.3 | — | — | 38.01 | |
| RandomSize=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.77 | 26.26 | 34.82 | 0.98 | 36.77 | — | — | 34.65 | |
| EPMSize=100K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.73 | 28.44 | 34.39 | 0.98 | 42.59 | — | — | 37.17 | |
| EPMSize=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.36 | 26.01 | 35.93 | 1.51 | 42.86 | — | — | 34.05 | |
| LESSSize=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.16 | 24.12 | 40.59 | 0.01 | 41.57 | — | — | 34.17 | |
| RandomSize=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 18.11 | 26.68 | 35.65 | 1.34 | 37.7 | — | — | 35.81 | |
| EPMSize=50K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 17.99 | 26.34 | 35.32 | 1.28 | 42.06 | — | — | 35.07 | |
| TaskPGM (JSD)Size=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 17.9 | 27.85 | 34.54 | 1.51 | 40.21 | — | — | 37.29 | |
| UniformSize=25K, Backbone=Llama-2-7b, beta=20, lambda=10, batch size=82025.07 | 17.68 | 27.01 | 35.01 | 1.51 | 40.27 | — | — | 35.01 | |
| HGRN2Parameters=1.3B, Tokens=100B2026.02 | 11.5 | 25.4 | 29.5 | 0.2 | 35 | 22.3 | 12.9 | — | |
| FEM-GLAParameters=340M, Tokens=15B, Architecture Components=GLA+C,L,T,G2026.02 | 11.5 | 25.5 | 29.7 | 0.9 | 47.3 | 24.1 | 12.3 | — | |
| HGRNParameters=1.3B, Tokens=100B2026.02 | 11.4 | 26.9 | 29.7 | 0.8 | 40.9 | 25.3 | 12.2 | — | |
| Transformer (SMAttn)Parameters=1.3B, Tokens=100B2026.02 | 11.4 | 25.9 | 29.6 | 1.1 | 36.5 | 27 | 14.1 | — | |
| GLAParameters=1.3B, Tokens=100B2026.02 | 11.4 | 25.9 | 29.5 | 0.6 | 42.7 | 27.2 | 15.7 | — | |
| FEM-SMParameters=340M, Tokens=15B, Architecture Components=SM-Attn+C,L,T,G2026.02 | 11.4 | 26.4 | 30 | 1.2 | 43.7 | 27.3 | 14.2 | — | |
| FEM-SMParameters=1.3B, Tokens=100B, Architecture Components=SMAttn+C,L,T,G2026.02 | 11.3 | 26.2 | 30.3 | 1.2 | 45.1 | 32.6 | 19.2 | — | |
| GatedDeltaNetParameters=340M, Tokens=15B2026.02 | 11.3 | 26 | 29.6 | 1 | 42.1 | 25.8 | 13.3 | — | |
| FEM-SM(-G,T,L)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C2026.02 | 11.3 | 25.4 | 29.6 | 0.9 | 38.8 | 24.6 | 12.2 | — | |
| FEM-GLAParameters=1.3B, Tokens=100B, Architecture Components=GLA+C,L,T,G2026.02 | 11.2 | 25.8 | 29.7 | 0.9 | 47.5 | 27.7 | 15.7 | — | |
| DeltaNetParameters=340M, Tokens=15B2026.02 | 11.2 | 26 | 30 | 0.9 | 45.2 | 27.7 | 15 | — | |
| FEM-SM(-G,T)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C,L2026.02 | 11.2 | 25.8 | 29.8 | 0.9 | 40.1 | 25.4 | 12.9 | — | |
| FEM-SM(-G)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C,L,T2026.02 | 11.2 | 26.1 | 29.7 | 1 | 42.1 | 26.6 | 14.4 | — | |
| GSAParameters=1.3B, Tokens=100B2026.02 | 11 | 27 | 29.4 | 1.3 | 43.8 | 30 | 17.9 | — | |
| RetNetParameters=1.3B, Tokens=100B2026.02 | 11 | 25.2 | 29.3 | 0.1 | 38.4 | 5.6 | 2.4 | — | |
| GLAParameters=340M, Tokens=15B2026.02 | 11 | 25.8 | 28.9 | 0.7 | 41.5 | 22.8 | 10.9 | — | |
| DeltaNetParameters=1.3B, Tokens=100B, MMLU-P Shots=5, GPQA Shots=0, BBH Shots=3, MATH Shots=4, MuSR Shots=0, IFE Shots=02026.02 | 10.9 | 26.3 | 30.8 | 1.1 | 41.7 | 28.8 | 16.5 | — | |
| DiffTransParameters=340M, Tokens=15B2026.02 | 10.9 | 25.9 | 29.9 | 0.8 | 39 | 26.6 | 13.3 | — | |
| FEM-SM(-G,T,L,C)Parameters=340M, Tokens=15B, Backbone=SMAttn2026.02 | 10.6 | 26.7 | 29.2 | 1 | 38.6 | 26.9 | 12.6 | — |