Attribution Faithfulness Evaluation on EconAgent (test)
-86.35Risk DropRandom
Evaluation results
| Method | Links | |
|---|---|---|
| RandomModel=Qwen-Plus, Top-k=32026.01 | -86.35 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=32026.01 | -70.63 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=32026.01 | -69.71 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=32026.01 | -55.09 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=102026.01 | -53.92 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=102026.01 | -33.09 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=32026.01 | -32.09 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=102026.01 | -27.61 | |
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=32026.01 | -24.82 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=32026.01 | -21.57 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=102026.01 | -19.05 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=32026.01 | -18.25 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=32026.01 | -18.2 | |
| RandomModel=Qwen-Plus, Top-k=102026.01 | -16.53 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=32026.01 | -13.99 | |
| RandomModel=Llama-3.1-8B, Top-k=32026.01 | -13.75 | |
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=102026.01 | -9.94 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=32026.01 | -7.39 | |
| RandomModel=DeepSeek-V3.2, Top-k=32026.01 | -5.76 | |
| RandomModel=GPT-4o mini, Top-k=32026.01 | -3.5 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=102026.01 | -3.23 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=102026.01 | -3.06 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=102026.01 | -0.59 | |
| RandomModel=Llama-3.1-8B, Top-k=102026.01 | 0.64 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=102026.01 | 1.32 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=32026.01 | 1.49 | |
| RandomModel=Claude-3-Haiku, Top-k=32026.01 | 1.73 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=32026.01 | 5.32 | |
| RandomModel=GPT-4o mini, Top-k=102026.01 | 7.69 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=102026.01 | 7.73 | |
| RandomModel=DeepSeek-V3.2, Top-k=102026.01 | 10.53 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=32026.01 | 11.56 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=102026.01 | 14.14 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=32026.01 | 20.18 | |
| RandomModel=Claude-3-Haiku, Top-k=102026.01 | 20.39 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=32026.01 | 25.44 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=32026.01 | 25.92 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=102026.01 | 27.49 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=102026.01 | 31.89 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=102026.01 | 32.53 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=32026.01 | 34.95 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=32026.01 | 36.28 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=32026.01 | 36.31 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=102026.01 | 39.17 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=102026.01 | 40.37 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=102026.01 | 41.24 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=102026.01 | 45.42 | |
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=32026.01 | 45.72 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=102026.01 | 47.44 | |
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=102026.01 | 57.08 |