List your company
ResearchFormerly WizWand

Identification of inactive attention heads on WinoGrande

20.91Percentage of Zeroed HeadsLast Token Head Output Norm (LN)
1.24366.349311.45516.5607Apr 4, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Last Token Head Output Norm (LN)Model=OLMo-2-1124-7B-Instruct, Score Function=Last Token Head Output Norm (LN)2025.0420.91
Avg Head Output Norm (LN)Model=Llama-3.2-3B-Instruct, Score Function=Avg Head Output Norm (LN)2025.0420.37
Avg Head Output Norm (LN)Model=OLMo-2-1124-7B-SFT, Score Function=Avg Head Output Norm (LN)2025.0419
Avg Head Output Norm (LN)Model=Llama-3.1-8B, Score Function=Avg Head Output Norm (LN)2025.0417.08
Avg Head Output Norm (LN)Model=OLMo-2-1124-7B-DPO, Score Function=Avg Head Output Norm (LN)2025.0416.94
Avg Head Output Norm (LN)Model=OLMo-2-1124-7B, Score Function=Avg Head Output Norm (LN)2025.0416.12
First Token Value Vector Norm (LN)Model=Qwen2.5-7B, Score Function=First Token Value Vector Norm (LN)2025.0415.69
AWFTModel=OLMo-2-1124-7B, Score Function=AWFT2025.0415.39
Ours (Aggregate)Model=Average, Score Function=Ours2025.0414.45
First Token Value Vector NormModel=Qwen2.5-1.5B, Score Function=First Token Value Vector Norm2025.0412.68
AWFTModel=Llama-3.1-8B-Instruct, Score Function=AWFT2025.0412.44
Avg Weight of First TokenModel=Llama-3.1-8B-Instruct, Score Function=Avg Weight of First Token2025.0411.89
Avg Head Output Norm (LN)Model=Llama-3.2-3B, Score Function=Avg Head Output Norm (LN)2025.0411.64
Avg Head Output Norm (LN)Model=Qwen2.5-0.5B, Score Function=Avg Head Output Norm (LN)2025.0411.35
AWFTModel=Qwen2.5-14B, Score Function=AWFT2025.0410.8
First Token Value Vector NormModel=Qwen2.5-3B, Score Function=First Token Value Vector Norm2025.0410.32
Avg Weight of First TokenModel=Qwen2.5-14B, Score Function=Avg Weight of First Token2025.0410.25
AWFTModel=Llama-3.2-3B, Score Function=AWFT2025.049.11
Avg Value Vector Norm (LN)Model=Qwen2.5-7B-Instruct, Score Function=Avg Value Vector Norm (LN)2025.048.03
AWFTModel=Qwen2.5-1.5B, Score Function=AWFT2025.047.49
AWFTModel=Average, Score Function=AWFT2025.046.9
AWFTModel=Llama-3.1-8B, Score Function=AWFT2025.046.86
AWFTModel=Qwen2.5-7B, Score Function=AWFT2025.046.6
AWFTModel=Qwen2.5-0.5B, Score Function=AWFT2025.046.49
AWFTModel=Qwen2.5-3B, Score Function=AWFT2025.045.73
AWFTModel=Llama-3.2-3B-Instruct, Score Function=AWFT2025.045.57
AWFTModel=OLMo-2-1124-7B-Instruct, Score Function=AWFT2025.043.02
AWFTModel=OLMo-2-1124-7B-DPO, Score Function=AWFT2025.042.79
AWFTModel=OLMo-2-1124-7B-SFT, Score Function=AWFT2025.042.32
AWFTModel=Qwen2.5-7B-Instruct, Score Function=AWFT2025.042