Long Story Evaluation on LongStoryEval Full (test)
27.1Plot ScoreNovelCritique-8B
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| NovelCritique-8BEvaluation Strategy=Summary-Based2025.12 | 27.1 | 27 | 24.1 | 18.3 | 24.3 | 27.8 | 21.1 | 25.5 | 27.7 | |
| Llama 3.1-70BEvaluation Strategy=Aggregation-Based2025.12 | 19.6 | 13.8 | 2.3 | 13.8 | 13.4 | 7.7 | 11.5 | 18.9 | 13.8 | |
| DeepSeek-v2.5Evaluation Strategy=Aggregation-Based2025.12 | 17.2 | 15.8 | 7 | 7.1 | 11 | 14.2 | 11.1 | 16.7 | 15.1 | |
| Llama 3.1-8BEvaluation Strategy=Aggregation-Based2025.12 | 15.5 | 8.5 | -1.4 | 2.8 | 12.3 | 7.5 | 7 | 13.7 | 11.6 | |
| GPT-4oEvaluation Strategy=Summary-Based2025.12 | 15.3 | 17.8 | 4.5 | 5 | 7.2 | 12.6 | 11.8 | 14 | 13.4 | |
| GPT-4oEvaluation Strategy=Aggregation-Based2025.12 | 14.3 | 16.7 | 10.2 | 7.9 | 10.4 | 9.7 | 9.1 | 14.1 | 15.2 | |
| GPT-4o-miniEvaluation Strategy=Aggregation-Based2025.12 | 14.2 | 17.2 | 7.2 | 4.4 | 9.5 | 8.9 | 8.1 | 15.1 | 12.3 | |
| DeepSeek-v2.5Evaluation Strategy=Summary-Based2025.12 | 13.4 | 12.2 | 1.8 | -3.8 | 7.1 | 8.9 | 13.2 | 15.1 | 14.4 | |
| Llama 3.1-70BEvaluation Strategy=Summary-Based2025.12 | 11.2 | 10.8 | -1.6 | 5.3 | 12.4 | 9.2 | 11.4 | 14.5 | 13 | |
| Llama 3.1-8BEvaluation Strategy=Summary-Based2025.12 | 10.4 | 14.1 | 4.9 | 9.1 | 9.6 | 15.3 | 14.5 | 12.3 | 12.4 | |
| Mixtral 8x7BEvaluation Strategy=Aggregation-Based2025.12 | 9.5 | 4 | 2.5 | -0.2 | 8.9 | 9.5 | 10.2 | 6.8 | 9 | |
| Llama 3.1-70BEvaluation Strategy=Incremental-Updated2025.12 | 9.3 | 13.3 | 4.1 | 1.7 | 8.7 | 4.9 | 4.6 | 6.1 | 9.9 | |
| DeepSeek-v2.5Evaluation Strategy=Incremental-Updated2025.12 | 8.9 | 12.2 | 9 | 8.6 | 12.5 | 12.3 | 6.6 | 12.2 | 11.6 | |
| GPT-4o-miniEvaluation Strategy=Summary-Based2025.12 | 8.7 | 7.5 | 5.4 | 4.8 | 11.1 | 11.6 | 8.3 | 7.9 | 9.7 | |
| GPT-4oEvaluation Strategy=Incremental-Updated2025.12 | 8 | 9.1 | 9.1 | 11.7 | 10.5 | 12.3 | 12.1 | 11.5 | 10.9 | |
| GPT-4o-miniEvaluation Strategy=Incremental-Updated2025.12 | 7.9 | 10.8 | 6.7 | 7.4 | 8.5 | 11.6 | 8.5 | 10.7 | 9.3 | |
| Mixtral 8x7BEvaluation Strategy=Summary-Based2025.12 | 7.8 | 7.4 | 7.1 | -0.5 | -4 | 5.6 | 9.4 | 6.7 | 8.3 | |
| Llama 3.1-8BEvaluation Strategy=Incremental-Updated2025.12 | 7 | 7.1 | 4.4 | 2.5 | 1.9 | 8 | 7.8 | 5.1 | 6.7 | |
| Mixtral 8x7BEvaluation Strategy=Incremental-Updated2025.12 | 4.2 | 10.8 | 4.4 | 6.6 | 5.8 | 2.3 | 5.8 | 2.6 | 4.2 |