General Language Evaluation on English lm-evaluation-harness
0.819ARC Easy Acc (Norm)OjaKV
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OjaKVModel=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 0.819 | — | 0.5179 | — | — | 0.5914 | — | — | — | 0.7992 | — | — | 0.7395 | 0.6934 | |
| OjaKVModel=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 0.819 | — | 0.5188 | — | — | 0.5835 | — | — | — | 0.7982 | — | — | 0.7395 | 0.6918 | |
| BaselineModel=Llama-3.1-8B, Compression Ratio=Full2025.09 | 0.8186 | — | 0.5188 | — | — | 0.5904 | — | — | — | 0.8003 | — | — | 0.7388 | 0.6934 | |
| Eigen-NModel=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 0.8165 | — | 0.5154 | — | — | 0.5774 | — | — | — | 0.7965 | — | — | 0.7356 | 0.6883 | |
| StaticPCAModel=Llama-3.1-8B, Compression Ratio=0.8x2025.09 | 0.8165 | — | 0.5154 | — | — | 0.5774 | — | — | — | 0.7965 | — | — | 0.7356 | 0.6883 | |
| Eigen-NModel=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 0.7938 | — | 0.4812 | — | — | 0.5487 | — | — | — | 0.7867 | — | — | 0.6985 | 0.6618 | |
| StaticPCAModel=Llama-3.1-8B, Compression Ratio=0.6x2025.09 | 0.7938 | — | 0.4812 | — | — | 0.5487 | — | — | — | 0.7867 | — | — | 0.6985 | 0.6618 | |
| BaselineModel=Llama-2-7B, Compression Ratio=Full2025.09 | 0.7386 | — | 0.442 | — | — | 0.578 | — | — | — | 0.7639 | — | — | 0.6638 | 0.6373 | |
| OjaKVModel=Llama-2-7B, Compression Ratio=0.8x2025.09 | 0.7386 | — | 0.4437 | — | — | 0.5751 | — | — | — | 0.7579 | — | — | 0.663 | 0.6357 | |
| OjaKVModel=Llama-2-7B, Compression Ratio=0.6x2025.09 | 0.7374 | — | 0.4437 | — | — | 0.5706 | — | — | — | 0.7639 | — | — | 0.663 | 0.6357 | |
| Eigen-NModel=Llama-2-7B, Compression Ratio=0.8x2025.09 | 0.713 | — | 0.4138 | — | — | 0.5629 | — | — | — | 0.7503 | — | — | 0.659 | 0.6198 | |
| StaticPCAModel=Llama-2-7B, Compression Ratio=0.8x2025.09 | 0.713 | — | 0.4138 | — | — | 0.5629 | — | — | — | 0.7503 | — | — | 0.659 | 0.6198 | |
| Eigen-NModel=Llama-2-7B, Compression Ratio=0.6x2025.09 | 0.7003 | — | 0.3993 | — | — | 0.5499 | — | — | — | 0.7454 | — | — | 0.6298 | 0.6049 | |
| StaticPCAModel=Llama-2-7B, Compression Ratio=0.6x2025.09 | 0.7003 | — | 0.3993 | — | — | 0.5499 | — | — | — | 0.7454 | — | — | 0.6298 | 0.6049 | |
| Transformer + Spelling Bee EmbeddingsTotal parameters=918m, Non-embedding parameters=764m2026.01 | 0.5387 | 0.259 | 0.2764 | 0.0237 | 0.049 | 0.4247 | 0.2302 | 0.0169 | 0.31 | 0.6839 | 0.716 | 0.016 | 0.5375 | — | |
| TransformerTotal parameters=918m, Non-embedding parameters=764m2026.01 | 0.5173 | 0.2585 | 0.267 | 0.018 | 0.0382 | 0.4204 | 0.2331 | 0.0166 | 0.296 | 0.6773 | 0.676 | 0.0238 | 0.5295 | — |