Failure Attribution on Who&When Algorithm-Generated
42.86Step-level AccuracyAgenTracer (G)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AgenTracer (G)Evaluation Mode=Offline, Method Paradigm=Finetune-Based, Ground-truth Guidance=true2026.03 | 42.86 | 69.62 | |
| AgenTracerEvaluation Mode=Offline, Method Paradigm=Finetune-Based, Ground-truth Guidance=false2026.03 | 37.3 | 63.73 | |
| Step-by-Step (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 25.51 | 35.2 | |
| PROMASEvaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct2026.03 | 24.75 | 46.53 | |
| Binary Search (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 23.98 | 44.13 | |
| FamasEvaluation Mode=Offline, Method Paradigm=SBFL-Based2026.03 | 23.81 | 55.56 | |
| MASCEvaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct, Ground-truth Guidance=false2026.03 | 21.72 | — | |
| MASC (G)Evaluation Mode=Online, Backbone=Meta-Llama-3.1-8B-Instruct, Ground-truth Guidance=true2026.03 | 19.24 | — | |
| RandomEvaluation Mode=N/A2026.03 | 19.06 | 29.1 | |
| Binary SearchEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 16.59 | 30.11 | |
| Step-by-StepEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 15.31 | 26.02 | |
| All-at-OnceEvaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=false2026.03 | 13.52 | 51.12 | |
| All-at-Once (G)Evaluation Mode=Offline, Method Paradigm=LLM-Based, Backbone=GPT-4o, Ground-truth Guidance=true2026.03 | 12.5 | 54.33 |