Paper Classification on Fall paper classification corpus 2025
88TP (True Positives)Consensus (Open Large)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Consensus (Open Large)Description=Consensus between all large (>8B) open models: DeepSeek R1 0528, GPT OSS 20B, Llama 4 Scout, Llama 3.3 70B, Qwen32025.10 | 88 | 382 | 7,853 | 0 | 95.41 | 18.72 | 100 | 31.54 | |
| Consensus (Commercial)Description=Consensus between all commercial models: Claude Sonnet 4.5, Gemini 2.5 Flash, GPT 5 (Full/Mini/Nano)2025.10 | 88 | 183 | 8,052 | 0 | 97.8 | 32.47 | 100 | 49.03 | |
| Consensus (Best)Description=Consensus between best combination of models: GPT 5, Claude Sonnet 4.5, Llama 3.3 70B2025.10 | 88 | 166 | 8,069 | 0 | 98.01 | 34.65 | 100 | 51.46 | |
| Consensus (All)Description=Consensus between all 13 models2025.10 | 88 | 653 | 7,582 | 0 | 92.15 | 11.88 | 100 | 21.23 | |
| Llama 4 ScoutParameters=>8B2025.10 | 86 | 341 | 7,894 | 2 | 95.88 | 20.14 | 97.73 | 33.4 | |
| Llama 3.3 70BParameters=70B2025.10 | 85 | 140 | 8,095 | 3 | 98.28 | 37.78 | 96.59 | 54.31 | |
| GPT 5 Mini2025.10 | 85 | 134 | 8,101 | 3 | 98.35 | 38.81 | 96.59 | 55.37 | |
| GPT 5 Nano2025.10 | 84 | 135 | 8,100 | 4 | 98.33 | 38.36 | 95.45 | 54.72 | |
| Consensus (Open 8B)Description=Consensus between all small (8B) open models: Llama 3.1 8B, DeepSeek R1 0528 8B, Qwen3 8B2025.10 | 82 | 426 | 7,809 | 6 | 94.81 | 16.14 | 93.18 | 27.52 | |
| DeepSeek R1 0528 8BParameters=8B2025.10 | 80 | 386 | 7,849 | 8 | 95.27 | 17.17 | 90.91 | 28.88 | |
| GPT 52025.10 | 80 | 58 | 8,177 | 8 | 99.21 | 57.97 | 90.91 | 70.8 | |
| Gemini 2.5 Flash2025.10 | 79 | 68 | 8,167 | 9 | 99.07 | 53.74 | 89.77 | 67.23 | |
| DeepSeek R1 0528Parameters=>8B2025.10 | 77 | 68 | 8,167 | 11 | 99.05 | 53.1 | 87.5 | 66.09 | |
| GPT OSS 20BParameters=20B2025.10 | 77 | 58 | 8,177 | 11 | 99.17 | 57.04 | 87.5 | 69.06 | |
| Claude Sonnet 4.52025.10 | 77 | 39 | 8,196 | 11 | 99.4 | 66.38 | 87.5 | 75.49 | |
| Llama 3.1 8BParameters=8B2025.10 | 72 | 98 | 8,137 | 16 | 98.63 | 42.35 | 81.82 | 55.81 | |
| Qwen3Parameters=>8B2025.10 | 72 | 30 | 8,205 | 16 | 99.45 | 70.59 | 81.82 | 75.79 | |
| Qwen3 8BParameters=8B2025.10 | 56 | 33 | 8,202 | 32 | 99.22 | 62.92 | 63.64 | 63.28 |