BookSum
Benchmarks
TaskDatasetSOTA resultTrendResults
BOOKSUM
100TP @ FP=1%
154
BookSum (test)
43.19ROUGE 1
37
BOOKSUM (test)
100Detection Rate (No Attack)
24
BookSum (test)
5Comp Score
24
BookSum
46.62ROUGE-1 Score
22
BOOKSUM (test)
4.38Average Rank
20
BookSum
3.76TPT (s)
20
BookSum
0.955AUC
20
BookSum
98.49AUC
20
BookSum Chapter Level
42.68ROUGE-1
14
BookSum
19.35Perplexity
13
BookSum
39.84SummaC Score
12
BookSum sampled (test)
17.71ROUGE Score
12
BookSum (test)
40.71SummaC
12
BookSum
69.8Detection Accuracy
12
BookSum oracle timing
94Average Score
12
BookSum
26.5F1 Score
10
BookSum
1,224.25Total Time (s)
10
BOOKSUM Mistral-Small-3.1-24B-Base-2503 (test)
0Latency per Token
9
BOOKSUM Mistral-Small-3.1-24B-Base-2503 (test)
0.037Latency per Token (s/tok)
9
BookSum
0.277Reward
6
BookSum Average latest (test)
17.47Average ROUGE
6
BookSum Trun. latest (test)
16.68Avg ROUGE
6
BookSum No Trun. latest (test)
17.74Average ROUGE
6
BOOKSUM (test)
186.7Avg Tokens per Sentence
5