Failure Attribution on Who&When Hand-Crafted
41.38Step-level AccuracyFamas
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FamasEvaluation Mode=Offline, Method Paradigm=SBFL-Based2026.03 | 41.38 | 62.07 | |
| MASCEvaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct, Ground-truth Guidance=false2026.03 | 20.79 | — | |
| AgenTracerEvaluation Mode=Offline, Method Paradigm=Finetune-Based, Ground-truth Guidance=false2026.03 | 20.68 | 63.82 | |
| AgenTracer (G)Evaluation Mode=Offline, Method Paradigm=Finetune-Based, Ground-truth Guidance=true2026.03 | 20.68 | 69.1 | |
| PROMASEvaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct2026.03 | 19.14 | 27.66 | |
| MASC (G)Evaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct, Ground-truth Guidance=true2026.03 | 18.25 | — | |
| Step-by-StepEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 8.77 | 53.44 | |
| Step-by-Step (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 7.02 | 34.48 | |
| Binary SearchEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 6.9 | 36.21 | |
| Binary Search (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 6.9 | 51.72 | |
| All-at-Once (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 5.27 | 55.17 | |
| RandomEvaluation Mode=N/A2026.03 | 4.16 | 12 | |
| All-at-OnceEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 3.51 | 53.44 |