Zero-shot Language Evaluation on WikiText2, BoolQ, PIQA, HellaSwag, WinoGrande, ARC, OBQA, and MTQA (test)
7.43WikiText2 PerplexitySimDiff (MSSD)
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| SimDiff (MSSD)Model=LLaMA2-13B, Layers=82026.04 | 7.43 | 62.23 | 74.21 | 52.23 | 67.72 | 62.88 | 38.31 | 27.2 | 26.6 | 51.42 | |
| SimDiff (MASD)Model=LLaMA2-13B, Layers=82026.04 | 7.45 | 62.42 | 74.92 | 52.44 | 66.77 | 60.73 | 37.8 | 27.8 | 27.24 | 51.26 | |
| ShortGPTModel=LLaMA2-13B, Layers=82026.04 | 8.3 | 62.17 | 73.88 | 50.99 | 65.43 | 61.24 | 36.35 | 26.6 | 26.16 | 50.35 | |
| SimDiffModel=LLaMA2-13B, Layers=102026.04 | 8.87 | 62.26 | 72.03 | 50.18 | 67.01 | 58.21 | 36.18 | 22.6 | 25.33 | 49.22 | |
| SimDiff (MASD)Model=LLaMA2-13B, Layers=122026.04 | 10.44 | 62.45 | 71.82 | 47.3 | 66.61 | 54.88 | 33.96 | 20.8 | 24.32 | 47.77 | |
| SimDiff (MSSD)Model=LLaMA2-13B, Layers=122026.04 | 11.5 | 62.45 | 69.86 | 46.75 | 65.19 | 52.95 | 32.17 | 21 | 24.82 | 46.9 | |
| ShortGPTModel=Qwen2.5-3B, Layers=82026.04 | 13.58 | 62.75 | 74.16 | 44.26 | 53.35 | 70.75 | 34.13 | 29.6 | 29.98 | 49.87 | |
| SimDiffModel=Qwen2.5-3B, Layers=82026.04 | 13.58 | 62.75 | 74.16 | 44.26 | 53.35 | 70.75 | 34.13 | 29.6 | 29.98 | 49.87 | |
| SimDiffModel=Qwen2.5-1.5B, Layers=82026.04 | 18.42 | 62.23 | 70.13 | 37.36 | 53.59 | 62.16 | 26.88 | 25.4 | 24.62 | 45.3 | |
| ShortGPTModel=LLaMA2-13B, Layers=102026.04 | 20.03 | 62.63 | 71.27 | 47.29 | 62.98 | 53.45 | 34.3 | 23.8 | 24.05 | 47.47 | |
| SimDiffModel=Qwen2.5-3B, Layers=122026.04 | 23.14 | 46.18 | 68.66 | 35.94 | 52.17 | 56.27 | 24.91 | 22.4 | 24.86 | 41.42 | |
| SimDiffModel=Qwen2.5-1.5B, Layers=102026.04 | 26.34 | 59.97 | 67.85 | 34.77 | 53.67 | 60.23 | 25.43 | 23.2 | 25.76 | 43.86 | |
| ShortGPTModel=Qwen2.5-3B, Layers=102026.04 | 26.34 | 59.97 | 67.85 | 34.77 | 53.67 | 60.23 | 25.43 | 23.2 | 25.76 | 43.86 | |
| SimDiffModel=Qwen2.5-3B, Layers=102026.04 | 26.34 | 59.97 | 67.85 | 34.77 | 53.67 | 60.23 | 25.43 | 23.2 | 25.76 | 43.86 | |
| ShortGPTModel=LLaMA2-13B, Layers=122026.04 | 34.71 | 62.39 | 68.44 | 45.49 | 62.27 | 48.02 | 31.91 | 20.2 | 24.32 | 45.38 | |
| ShortGPTModel=Qwen2.5-1.5B, Layers=82026.04 | 37.7 | 61.65 | 64.91 | 35.62 | 50.59 | 53.28 | 27.99 | 22.8 | 26.2 | 42.88 | |
| SimDiffModel=Qwen2.5-0.5B, Layers=82026.04 | 38 | 43.18 | 63.17 | 30.26 | 50.43 | 48.82 | 20.48 | 19.6 | 22.45 | 37.3 | |
| SimDiffModel=Qwen2.5-1.5B, Layers=122026.04 | 44.34 | 39.17 | 64.04 | 30.89 | 52.25 | 48.74 | 22.1 | 18.2 | 22.71 | 37.26 | |
| ShortGPTModel=Qwen2.5-3B, Layers=122026.04 | 44.34 | 39.17 | 64.04 | 30.89 | 52.25 | 48.74 | 22.1 | 18.2 | 22.71 | 37.26 | |
| ShortGPTModel=Qwen2.5-1.5B, Layers=102026.04 | 64.45 | 55.57 | 64.15 | 32.9 | 53.43 | 47.52 | 22.35 | 23 | 23.48 | 40.3 | |
| SimDiffModel=Qwen2.5-0.5B, Layers=102026.04 | 87.7 | 41.19 | 60.94 | 28.97 | 51.38 | 44.61 | 18.52 | 17 | 22.58 | 35.65 | |
| ShortGPTModel=Qwen2.5-0.5B, Layers=82026.04 | 110.23 | 47.16 | 59.9 | 30.07 | 51.7 | 41.08 | 20.65 | 18.4 | 22.35 | 36.41 | |
| SimDiff (MSSD)Model=Qwen2.5-0.5B, Layers=122026.04 | 208.23 | 38.47 | 57.78 | 28.26 | 50.28 | 39.94 | 20.22 | 15.4 | 20.77 | 33.89 | |
| SimDiff (MASD)Model=Qwen2.5-0.5B, Layers=122026.04 | 399.39 | 40.24 | 58.16 | 27.5 | 49.57 | 37.25 | 19.8 | 17 | 20.13 | 33.71 | |
| ShortGPTModel=Qwen2.5-1.5B, Layers=122026.04 | 634.94 | 46.61 | 59.25 | 29.34 | 50.28 | 36.24 | 23.29 | 18.2 | 20.8 | 35.5 | |
| ShortGPTModel=Qwen2.5-0.5B, Layers=102026.04 | 858.52 | 42.97 | 58.27 | 28.28 | 49.8 | 36.95 | 21.25 | 16 | 21.94 | 34.43 | |
| ShortGPTModel=Qwen2.5-0.5B, Layers=122026.04 | 10,916.14 | 37.8 | 55.55 | 26.25 | 49.17 | 29.88 | 21.67 | 17.6 | 19.56 | 32.19 |