Language Modeling Evaluation on LM Evaluation Harness (Standard Suite)
60.35AccuracySEFT
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| SEFTModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=4:82025.05 | 60.35 | — | — | — | — | — | — | — | — | — | |
| LoRA*Model Backbone=LLaMA3-8B, N:M Sparsity Pattern=4:82025.05 | 60.04 | — | — | — | — | — | — | — | — | — | |
| SEFTModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=2:42025.05 | 59.63 | — | — | — | — | — | — | — | — | — | |
| SEFTModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=4:82025.05 | 58.65 | — | — | — | — | — | — | — | — | — | |
| SQFTModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=4:82025.05 | 58.28 | — | — | — | — | — | — | — | — | — | |
| LoRA*Model Backbone=LLaMA3-8B, N:M Sparsity Pattern=2:42025.05 | 58.25 | — | — | — | — | — | — | — | — | — | |
| SEFTModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=2:42025.05 | 58.17 | — | — | — | — | — | — | — | — | — | |
| SQFTModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=4:82025.05 | 57.87 | — | — | — | — | — | — | — | — | — | |
| SPPModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=4:82025.05 | 57.81 | — | — | — | — | — | — | — | — | — | |
| LoRA*Model Backbone=LLaMA1-7B, N:M Sparsity Pattern=4:82025.05 | 57.71 | — | — | — | — | — | — | — | — | — | |
| LoRA*Model Backbone=LLaMA1-7B, N:M Sparsity Pattern=2:42025.05 | 57.35 | — | — | — | — | — | — | — | — | — | |
| SQFTModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=2:42025.05 | 56.99 | — | — | — | — | — | — | — | — | — | |
| SQFTModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=2:42025.05 | 56.85 | — | — | — | — | — | — | — | — | — | |
| SPPModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=4:82025.05 | 56.77 | — | — | — | — | — | — | — | — | — | |
| SPPModel Backbone=LLaMA3-8B, N:M Sparsity Pattern=2:42025.05 | 56.55 | — | — | — | — | — | — | — | — | — | |
| SPPModel Backbone=LLaMA1-7B, N:M Sparsity Pattern=2:42025.05 | 55.16 | — | — | — | — | — | — | — | — | — | |
| CARE ONE-SHOTRank=512, Calibration=alpaca-256-32, Token=0B2026.03 | — | 52.73 | 76.3 | 73.98 | 78.73 | 62.17 | 40.6 | 41.53 | 72.61 | 62.33 | |
| GQA (original)Rank=N/A, Calibration=N/A, Token=N/A2026.03 | — | 50.34 | 80.18 | 60.15 | 79.65 | 48.05 | 34.8 | 40.1 | 72.69 | 58.24 | |
| Palu(SVD)Rank=512, Calibration=N/A, Token=0B2026.03 | — | 26.02 | 50.97 | 37.43 | 64.15 | 27.89 | 19.4 | 26.6 | 57.54 | 38.75 | |
| Palu(SVD)Rank=512, Calibration=N/A, Token=1B2026.03 | — | 33.45 | 62.38 | 45.55 | 72.34 | 50.57 | 23 | 31.54 | 61.78 | 47.58 | |
| Palu(SVD)Rank=512, Calibration=N/A, Token=3B2026.03 | — | 44.56 | 74.96 | 52.2 | 76.63 | 61.08 | 30.4 | 45.15 | 65.42 | 56.3 | |
| TransMLARank=512, Calibration=wiki-256-256, Token=0B2026.03 | — | 43.17 | 66.12 | 68.25 | 74.81 | 48.89 | 38.8 | 37.7 | 66.46 | 55.52 | |
| TransMLARank=512, Calibration=wiki-256-256, Token=1B2026.03 | — | 53.04 | 81.07 | 58.75 | 81.04 | 69.13 | 32 | 44.09 | 71.74 | 61.36 | |
| TransMLARank=512, Calibration=wiki-256-256, Token=3B2026.03 | — | 53.77 | 82.34 | 56.44 | 80.7 | 70.23 | 33.3 | 45.61 | 72.47 | 61.86 | |
| TransMLA + CARE(E) InitRank=512, Calibration=alpaca-256-256, Token=0B2026.03 | — | 45.05 | 69.02 | 68.98 | 76.06 | 51.16 | 37.6 | 39.43 | 68.98 | 57.04 | |
| TransMLA + CARE(E) InitRank=512, Calibration=alpaca-256-256, Token=1B2026.03 | — | 52.25 | 82.33 | 62.47 | 80.21 | 70.31 | 32.9 | 45.11 | 75.13 | 62.59 | |
| TransMLA + CARE(E) InitRank=512, Calibration=alpaca-256-256, Token=3B2026.03 | — | 51.75 | 80.73 | 64.45 | 83.23 | 71.57 | 34 | 46.33 | 74.09 | 63.27 |