LongBench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
LongBench Gov Report (test)
1.7Avg Accepted Draft Tokens per Round
4
LongBench 32k 2024
57.62HotpotQA Score
4
LongBench
3.08MAT
4
LongBench 20 samples/task
1.91NarrQA Performance
4
LongBench Document-level instruction following v2
27.1Median Score
4
LongBench 15 English
18.26NarrativeQA
4
LongBench English subsets
25.38NarrativeQA
4
LongBench
46.2QA Score
4
LongBench V1
59.63HotpotQA Accuracy
4
LongBench V1
50.48Accuracy (0–4k Context)
4
LongBench
33.9Accuracy
4
LongBench-Chat and LongBench (260 sampled questions)
32.86Fact Count
4
LongBench 1000 (test)
81.3Success Rate
3
LongBench 14-subtask configuration
12.4Average Performance (14 Tasks)
3
LongBench 1M v2
33.5Easy Difficulty Accuracy
3
LongBench-E-QA v1 (test)
55.7Accuracy (0-4k Context)
3
LongBench QA v1 (test)
67.22wikimqa
3
LongBench
87.77TriviaQA Score
3
LongBench v2 (test)
44.07Accuracy (Easy, Short)
3
LongBench Zh out-of-domain (evaluation)
61.2SingleDoc Acc
3
LongBench
49.11FullKV Performance
2
LongBench context-dependent transfer v2
36.4Score (32k Context)
2
LongBench v1
58.8Overall Score
2
LongBench English (14-task average)
11.6LongBench Average Score
2
LongBench QA 8K context
11.6HotpotQA
2