Multi-task Language Understanding on MMLU (Accuracy, Std, Speedup)
71MMLU AccuracyTALE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TALEBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=52025.10 | 71 | 19 | -16.6 | |
| BSBABackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=62025.10 | 68.13 | — | -19.9 | |
| BaselineBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 68.1 | — | — | |
| TALEBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=82025.10 | 54 | 35 | -24.1 | |
| TALEBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=12025.10 | 53.8 | 22 | -2.9 | |
| BSBABackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=92025.10 | 50.2 | — | -26.4 | |
| BaselineBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 48.8 | — | — | |
| TALEBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=22025.10 | 40.8 | 20 | -6.2 | |
| BaselineBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 39.4 | — | — | |
| BSBABackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=82025.10 | 39 | — | -24.6 | |
| BSBABackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=222025.10 | 15 | — | -60.2 | |
| BaselineBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 13 | — | — |