Multiple-choice Question Answering on LongBench v2 (val)
37.8Overall Accuracyo1-mini
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| o1-miniChain-of-Thought=false2024.09 | 37.8 | 38.9 | 37.1 | 48.6 | 33.3 | 28.6 | 14 | |
| Mistral LargeSize=123B, Chain-of-Thought=false2024.09 | 34.4 | 38 | 32.2 | 41.7 | 30.7 | 29.6 | 15 | |
| E2LLM-7BSize=7B, Chain-of-Thought=false2024.09 | 31.8 | 33.3 | 30.9 | 37.8 | 28.4 | 28.7 | — | |
| Llama3.1-70BSize=70B, Chain-of-Thought=false2024.09 | 31.6 | 32.3 | 31.2 | 41.1 | 27.4 | 24.1 | 16 | |
| Qwen2.5-7B-ftSize=7B, Chain-of-Thought=false2024.09 | 31.2 | 31.3 | 31.2 | 39.4 | 27 | 25.9 | — | |
| Nemotron-70BSize=70B, Chain-of-Thought=false2024.09 | 31 | 32.8 | 29.9 | 38.3 | 27.9 | 25 | 17 | |
| NExtLong-8BSize=8B, Chain-of-Thought=false2024.09 | 30.8 | 33.9 | 28.9 | 37.8 | 27.4 | 25.9 | 18 | |
| GLM-4-9BSize=9B, Chain-of-Thought=false2024.09 | 30.2 | 30.7 | 29.9 | 33.9 | 29.8 | 25 | 19 | |
| Llama3.1-8BSize=8B, Chain-of-Thought=false2024.09 | 30 | 30.7 | 29.6 | 35 | 27.9 | 25.9 | 20 | |
| Qwen2.5-7BSize=7B, Chain-of-Thought=false2024.09 | 30 | 30.7 | 29.6 | 40.6 | 24.2 | 24.1 | 21 | |
| Llama3.3-70BSize=70B, Chain-of-Thought=false2024.09 | 29.8 | 34.4 | 27 | 36.7 | 27 | 24.1 | 22 | |
| GPT-4o miniChain-of-Thought=false2024.09 | 29.3 | 31.1 | 28.2 | 31.8 | 28.6 | 26.2 | 23 | |
| LLoCOSize=7B, Chain-of-Thought=false2024.09 | 28.2 | 30.2 | 26 | 36.8 | 24.2 | 21.5 | — | |
| Command R+Size=104B, Chain-of-Thought=false2024.09 | 27.8 | 30.2 | 26.4 | 36.7 | 23.7 | 21.3 | 24 | |
| Mistral Large 2Size=123B, Chain-of-Thought=false2024.09 | 26.6 | 29.7 | 24.8 | 37.8 | 19.5 | 22.2 | 25 |