Language Model Evaluation on LM Evaluation Harness (test)
74.22MMLUE2E
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| E2EModel=Qwen2.5-7B-Ins, Peak Mem (GB)=75.39, Step Time (s/step)=1.712026.05 | 74.22 | 56.01 | 66.72 | 88.7 | 81.23 | 64.52 | 75.22 | |
| LoPTModel=Qwen2.5-7B-Ins, Peak Mem (GB)=59.66 (↓21%), Step Time (s/step)=1.56 (↓9%)2026.05 | 74.18 | 59.7 | 66.38 | 89.09 | 81.4 | 64.83 | 75.4 | |
| BaseModel=Qwen2.5-7B-Ins2026.05 | 74.17 | 57.3 | 66.72 | 83.47 | 79.35 | 64.75 | 74.51 | |
| LoPTModel=Qwen3-4B-Base, Peak Mem (GB)=32.89 (↓23%), Step Time (s/step)=1.32 (↓7%)2026.05 | 70.1 | 23.84 | 62.29 | 88.63 | 69.95 | 54.62 | 66.85 | |
| BaseModel=Qwen3-4B-Base2026.05 | 70.07 | 23.84 | 57.84 | 85.14 | 65.62 | 54.85 | 67.56 | |
| E2EModel=Qwen3-4B-Base, Peak Mem (GB)=42.55, Step Time (s/step)=1.422026.05 | 70.01 | 24.03 | 62.03 | 88.35 | 68.76 | 54.78 | 67.17 | |
| LoPTModel=Llama-3.1-8B-Ins, Peak Mem (GB)=47.54 (↓24%), Step Time (s/step)=1.65 (↓21%)2026.05 | 68.55 | 47.03 | 61.43 | 82.4 | 80.47 | 54.11 | 77.35 | |
| E2EModel=Llama-3.1-8B-Ins, Peak Mem (GB)=62.62, Step Time (s/step)=2.102026.05 | 68.44 | 47.69 | 61.69 | 82.02 | 80.39 | 54.53 | 76.8 | |
| BaseModel=Llama-3.1-8B-Ins2026.05 | 68.2 | 44.54 | 60.49 | 69.67 | 79.42 | 54.56 | 77.5 |