Question Answering on ARC Challenge (Accuracy, Std, Speedup)
92AccuracyTALE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TALEBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=22025.10 | 92 | 0.15 | -6.7 | |
| BaselineBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 86.55 | — | — | |
| BSBABackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=62025.10 | 86.55 | — | -19.9 | |
| TALEBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=42025.10 | 80.6 | 0.18 | -11.7 | |
| BaselineBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 79.4 | — | — | |
| TALEBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=62025.10 | 79.1 | 0.18 | -18.5 | |
| BSBABackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=72025.10 | 77.6 | — | -20.5 | |
| BaselineBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 76.2 | — | — | |
| BSBABackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=82025.10 | 76.2 | — | -24.6 | |
| TALEBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=72025.10 | 51.45 | 0.22 | -22.1 | |
| BSBABackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=112025.10 | 48.8 | — | -33.1 | |
| BaselineBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 46 | — | — |