Inactive Attention Head Identification on PIQA
31.3Percentage of Heads ZeroedAWFT
Evaluation results
| Method | Links | |
|---|---|---|
| AWFTModel=Llama-3.1-8B-Instruct, Score Category=Baseline AWFT2025.04 | 31.3 | |
| First Token Value Vector NormModel=Qwen2.5-7B, Score Category=Best of Proposed Functions2025.04 | 28.66 | |
| Avg Head Output Norm (LN)Model=OLMo-2-1124-7B-Instruct, Score Category=Best of Proposed Functions2025.04 | 26.83 | |
| First Token Value Vector Norm (LN)Model=Qwen2.5-14B, Score Category=Best of Proposed Functions2025.04 | 26.12 | |
| Avg Head Output Norm (LN)Model=Llama-3.2-3B-Instruct, Score Category=Best of Proposed Functions2025.04 | 25.78 | |
| Avg Weight of First TokenModel=Llama-3.1-8B-Instruct, Score Category=Best of Proposed Functions2025.04 | 25.39 | |
| AWFTModel=Qwen2.5-1.5B, Score Category=Baseline AWFT2025.04 | 24.33 | |
| Avg Head Output Norm (LN)Model=OLMo-2-1124-7B, Score Category=Best of Proposed Functions2025.04 | 22.7 | |
| AWFTModel=Llama-3.2-3B, Score Category=Baseline AWFT2025.04 | 21.76 | |
| Avg Head Output NormModel=Qwen2.5-3B, Score Category=Best of Proposed Functions2025.04 | 21.42 | |
| Avg Weight of First TokenModel=Qwen2.5-1.5B, Score Category=Best of Proposed Functions2025.04 | 21.32 | |
| AWFTModel=OLMo-2-1124-7B, Score Category=Baseline AWFT2025.04 | 20.74 | |
| Last Token Head Output Norm (LN)Model=OLMo-2-1124-7B-SFT, Score Category=Best of Proposed Functions2025.04 | 20.12 | |
| Avg Head Output Norm (LN)Model=Qwen2.5-0.5B, Score Category=Best of Proposed Functions2025.04 | 18.36 | |
| First Token Value Vector Norm (LN)Model=Llama-3.1-8B, Score Category=Best of Proposed Functions2025.04 | 17.85 | |
| Avg Weight of First TokenModel=Llama-3.2-3B, Score Category=Best of Proposed Functions2025.04 | 17.37 | |
| AWFTModel=Qwen2.5-3B, Score Category=Baseline AWFT2025.04 | 16.51 | |
| First Token Value Vector Norm (LN)Model=OLMo-2-1124-7B-DPO, Score Category=Best of Proposed Functions2025.04 | 15.87 | |
| AWFTModel=Llama-3.2-3B-Instruct, Score Category=Baseline AWFT2025.04 | 15.32 | |
| Avg Head Output Norm (LN)Model=Qwen2.5-7B-Instruct, Score Category=Best of Proposed Functions2025.04 | 11.45 | |
| AWFTModel=OLMo-2-1124-7B-DPO, Score Category=Baseline AWFT2025.04 | 2.93 | |
| AWFTModel=Qwen2.5-7B-Instruct, Score Category=Baseline AWFT2025.04 | 2.84 | |
| AWFTModel=OLMo-2-1124-7B-Instruct, Score Category=Baseline AWFT2025.04 | 2.54 | |
| AWFTModel=OLMo-2-1124-7B-SFT, Score Category=Baseline AWFT2025.04 | 1.87 | |
| AWFTModel=Llama-3.1-8B, Score Category=Baseline AWFT2025.04 | 0 | |
| AWFTModel=Qwen2.5-0.5B, Score Category=Baseline AWFT2025.04 | 0 | |
| AWFTModel=Qwen2.5-7B, Score Category=Baseline AWFT2025.04 | 0 | |
| AWFTModel=Qwen2.5-14B, Score Category=Baseline AWFT2025.04 | 0 |