Attribution Faithfulness Evaluation on TwinMarket (test)
7,790Risk DropOurs (Shapley-based framework)
Evaluation results
| Method | Links | |
|---|---|---|
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=102026.01 | 7,790 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=32026.01 | 6,514 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=102026.01 | 6,356 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=102026.01 | 6,005 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=102026.01 | 5,646 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=102026.01 | 5,554 | |
| Ours (Shapley-based framework)Model=Qwen-Plus, Top-k=32026.01 | 5,282 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=102026.01 | 5,088 | |
| Ours (Shapley-based framework)Model=DeepSeek-V3.2, Top-k=32026.01 | 4,621 | |
| AT (Agent Tracer)Model=Qwen-Plus, Top-k=102026.01 | 4,606 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=32026.01 | 4,491 | |
| AT (Agent Tracer)Model=Llama-3.1-8B, Top-k=102026.01 | 4,112 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=102026.01 | 4,072 | |
| Ours (Shapley-based framework)Model=GPT-4o mini, Top-k=32026.01 | 3,820 | |
| Ours (Shapley-based framework)Model=Claude-3-Haiku, Top-k=32026.01 | 3,820 | |
| AT (Agent Tracer)Model=DeepSeek-V3.2, Top-k=32026.01 | 2,550 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=102026.01 | 2,176 | |
| Ours (Shapley-based framework)Model=Llama-3.1-8B, Top-k=32026.01 | 1,700 | |
| AT (Agent Tracer)Model=GPT-4o mini, Top-k=32026.01 | 1,663 | |
| AT (Agent Tracer)Model=Claude-3-Haiku, Top-k=32026.01 | 1,663 | |
| RandomModel=DeepSeek-V3.2, Top-k=102026.01 | 836 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=102026.01 | 476 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=102026.01 | 464 | |
| RandomModel=Claude-3-Haiku, Top-k=102026.01 | 277 | |
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=102026.01 | 203 | |
| RandomModel=GPT-4o mini, Top-k=102026.01 | 139 | |
| RandomModel=Llama-3.1-8B, Top-k=32026.01 | 115 | |
| RandomModel=Qwen-Plus, Top-k=102026.01 | 100 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=32026.01 | 79 | |
| FT (Failure Taxonomy)Model=Qwen-Plus, Top-k=102026.01 | 62 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=102026.01 | 57 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=102026.01 | 55 | |
| RandomModel=GPT-4o mini, Top-k=32026.01 | 49 | |
| RandomModel=Claude-3-Haiku, Top-k=32026.01 | 49 | |
| RandomModel=Qwen-Plus, Top-k=32026.01 | 43 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=32026.01 | 27 | |
| FT (Failure Taxonomy)Model=DeepSeek-V3.2, Top-k=32026.01 | -14 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=32026.01 | -14 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=32026.01 | -17 | |
| RandomModel=DeepSeek-V3.2, Top-k=32026.01 | -27 | |
| FA (Failure Attribution)Model=Llama-3.1-8B, Top-k=102026.01 | -39 | |
| FA (Failure Attribution)Model=Qwen-Plus, Top-k=102026.01 | -51 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=32026.01 | -117 | |
| FT (Failure Taxonomy)Model=GPT-4o mini, Top-k=32026.01 | -134 | |
| FT (Failure Taxonomy)Model=Claude-3-Haiku, Top-k=32026.01 | -134 | |
| FA (Failure Attribution)Model=GPT-4o mini, Top-k=32026.01 | -143 | |
| FA (Failure Attribution)Model=Claude-3-Haiku, Top-k=32026.01 | -143 | |
| FT (Failure Taxonomy)Model=Llama-3.1-8B, Top-k=102026.01 | -338 | |
| RandomModel=Llama-3.1-8B, Top-k=102026.01 | -494 | |
| FA (Failure Attribution)Model=DeepSeek-V3.2, Top-k=102026.01 | -891 |