Identification of inactive attention heads on WinoGrande
20.91Percentage of Zeroed HeadsLast Token Head Output Norm (LN)
Evaluation Results
| Method | Links | |
|---|---|---|
| Last Token Head Output Norm (LN)Model=OLMo-2-1124-7B-Instruct, Score Function=Last Token Head Output Norm (LN)2025.04 | 20.91 | |
| Avg Head Output Norm (LN)Model=Llama-3.2-3B-Instruct, Score Function=Avg Head Output Norm (LN)2025.04 | 20.37 | |
| Avg Head Output Norm (LN)Model=OLMo-2-1124-7B-SFT, Score Function=Avg Head Output Norm (LN)2025.04 | 19 | |
| Avg Head Output Norm (LN)Model=Llama-3.1-8B, Score Function=Avg Head Output Norm (LN)2025.04 | 17.08 | |
| Avg Head Output Norm (LN)Model=OLMo-2-1124-7B-DPO, Score Function=Avg Head Output Norm (LN)2025.04 | 16.94 | |
| Avg Head Output Norm (LN)Model=OLMo-2-1124-7B, Score Function=Avg Head Output Norm (LN)2025.04 | 16.12 | |
| First Token Value Vector Norm (LN)Model=Qwen2.5-7B, Score Function=First Token Value Vector Norm (LN)2025.04 | 15.69 | |
| AWFTModel=OLMo-2-1124-7B, Score Function=AWFT2025.04 | 15.39 | |
| Ours (Aggregate)Model=Average, Score Function=Ours2025.04 | 14.45 | |
| First Token Value Vector NormModel=Qwen2.5-1.5B, Score Function=First Token Value Vector Norm2025.04 | 12.68 | |
| AWFTModel=Llama-3.1-8B-Instruct, Score Function=AWFT2025.04 | 12.44 | |
| Avg Weight of First TokenModel=Llama-3.1-8B-Instruct, Score Function=Avg Weight of First Token2025.04 | 11.89 | |
| Avg Head Output Norm (LN)Model=Llama-3.2-3B, Score Function=Avg Head Output Norm (LN)2025.04 | 11.64 | |
| Avg Head Output Norm (LN)Model=Qwen2.5-0.5B, Score Function=Avg Head Output Norm (LN)2025.04 | 11.35 | |
| AWFTModel=Qwen2.5-14B, Score Function=AWFT2025.04 | 10.8 | |
| First Token Value Vector NormModel=Qwen2.5-3B, Score Function=First Token Value Vector Norm2025.04 | 10.32 | |
| Avg Weight of First TokenModel=Qwen2.5-14B, Score Function=Avg Weight of First Token2025.04 | 10.25 | |
| AWFTModel=Llama-3.2-3B, Score Function=AWFT2025.04 | 9.11 | |
| Avg Value Vector Norm (LN)Model=Qwen2.5-7B-Instruct, Score Function=Avg Value Vector Norm (LN)2025.04 | 8.03 | |
| AWFTModel=Qwen2.5-1.5B, Score Function=AWFT2025.04 | 7.49 | |
| AWFTModel=Average, Score Function=AWFT2025.04 | 6.9 | |
| AWFTModel=Llama-3.1-8B, Score Function=AWFT2025.04 | 6.86 | |
| AWFTModel=Qwen2.5-7B, Score Function=AWFT2025.04 | 6.6 | |
| AWFTModel=Qwen2.5-0.5B, Score Function=AWFT2025.04 | 6.49 | |
| AWFTModel=Qwen2.5-3B, Score Function=AWFT2025.04 | 5.73 | |
| AWFTModel=Llama-3.2-3B-Instruct, Score Function=AWFT2025.04 | 5.57 | |
| AWFTModel=OLMo-2-1124-7B-Instruct, Score Function=AWFT2025.04 | 3.02 | |
| AWFTModel=OLMo-2-1124-7B-DPO, Score Function=AWFT2025.04 | 2.79 | |
| AWFTModel=OLMo-2-1124-7B-SFT, Score Function=AWFT2025.04 | 2.32 | |
| AWFTModel=Qwen2.5-7B-Instruct, Score Function=AWFT2025.04 | 2 |