HLE
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HLE Our Settings
44.4Pass@1
15
HLE decontaminated
8.4Accuracy
14
HLE OOD
38.6Accuracy
14
HLE (test)
26Accuracy
14
HLE text-only
40.8Score
14
HLE
18.03Pass@1
14
HLE text-only original (test)
32.9Pass@1
13
HLE
3.94Accuracy
12
HLE
4.45Accuracy
12
HLE-Verified Gold (test)
60Accuracy
12
HLE 500-question ablation
57.3Success Rate (Last)
12
HLE
42.9Success Rate
12
HLE-500
29.5Text Score
12
HLE text
54.2Score
12
HLE
37.7Pass@1
12
HLE Math-text
62.8Pass@1
12
HLE Bio
78Accuracy
11
HLE
51.8Score
11
HLE
23.1Accuracy
11
HLE filtered subset
19.6NComm
10
HLE Full
45.8Pass@1
10
HLE no tools
37.5Score
10
HLE Full
0.502Score (%)
10
HLE
15.5Accuracy
10
HLE Drug Discovery
21.42Exact Match Accuracy
9