Attribution Faithfulness Evaluation on SocialNetwork (test)
-0.1Risk DropFT (Failure Taxonomy)
Evaluation results
| Method | Links | |
|---|---|---|
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=32026.01 | -0.1 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=32026.01 | -0.1 | |
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=102026.01 | 1.5 | |
| RandomModel=DeepSeek-V3.2, Top-k=32026.01 | 1.9 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=102026.01 | 2.1 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=102026.01 | 2.2 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=32026.01 | 2.3 | |
| RandomModel=DeepSeek-V3.2, Top-k=102026.01 | 2.6 | |
| RandomModel=Claude-3-Haiku, Top-k=102026.01 | 2.7 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=32026.01 | 2.7 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=32026.01 | 2.7 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=32026.01 | 2.8 | |
| RandomModel=Qwen-Plus, Top-k=32026.01 | 2.8 | |
| RandomModel=Claude-3-Haiku, Top-k=32026.01 | 2.9 | |
| RandomModel=Qwen-Plus, Top-k=102026.01 | 2.9 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=102026.01 | 3 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=102026.01 | 3.1 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=102026.01 | 3.5 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=32026.01 | 4.6 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=32026.01 | 5.9 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=32026.01 | 7.4 | |
| RandomModel=Llama-3.1-8B, Top-k=32026.01 | 8 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=32026.01 | 8.3 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=32026.01 | 8.3 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=102026.01 | 8.5 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=102026.01 | 8.9 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=32026.01 | 10.4 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=102026.01 | 10.6 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=102026.01 | 10.8 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=102026.01 | 11.1 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=32026.01 | 11.1 | |
| RandomModel=Llama-3.1-8B, Top-k=102026.01 | 11.4 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=102026.01 | 11.6 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=32026.01 | 11.9 | |
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=32026.01 | 11.9 | |
| RandomModel=GPT-4o mini, Top-k=32026.01 | 12.2 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=32026.01 | 12.9 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=32026.01 | 13 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=102026.01 | 14.9 | |
| RandomModel=GPT-4o mini, Top-k=102026.01 | 15.1 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=102026.01 | 16.6 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=32026.01 | 17.9 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=32026.01 | 19 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=102026.01 | 19.2 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=32026.01 | 22.2 | |
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=102026.01 | 23.1 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=102026.01 | 25.5 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=102026.01 | 32.7 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=102026.01 | 39.3 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=102026.01 | 39.7 |