Summarization Faithfulness on BookSum
39.84SummaC ScoreExtract-to-Abstract (E2A)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Extract-to-Abstract (E2A)System=E2A, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 39.84 | 68.04 | |
| Question-Answer Guided (QAG)System=QAG, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 39.37 | 72.69 | |
| o1System=o1, Model Class=Large Reasoning Models (LRM), Shots=0-shot2025.12 | 38.82 | 68.96 | |
| Chain-of-Thought (CoT)System=COT, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 38.31 | 67.03 | |
| VanillaSystem=Vanilla, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 27.35 | 98.38 | |
| GPT-5System=GPT-5, Model Class=Large Reasoning Models (LRM), Shots=0-shot2025.12 | 27.32 | 99.37 | |
| Iterative Refine (IR)System=IR, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 26.77 | 98.3 | |
| o3System=o3, Model Class=Large Reasoning Models (LRM), Shots=0-shot2025.12 | 25.6 | 98.46 | |
| Cited Summarization (Cite)System=Cite, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 24.81 | 96.99 | |
| Decomposition (Deco)System=Deco, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 23.94 | 98.64 | |
| Self-Consistency (SC)System=SC, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 23.3 | 95.93 | |
| Plan-then-Write (Plan)System=Plan, Model Class=LLM (GPT-4.1), Shots=0-shot2025.12 | 22.21 | 97.83 |