Logical Reasoning Machine Reading Comprehension on ReClor
80AccuracyIDOL
Evaluation Results
| Method | Links | |
|---|---|---|
| IDOLSetting=Fine-Tuning2023.06 | 80 | |
| ChatGPTSetting=Chain-of-Thought Prompting2023.06 | 70 | |
| ChatGPTSetting=Zero-Shot Prompting2023.06 | 63.3 | |
| ChatGPTSetting=Few-Shot Prompting2023.06 | 63.3 | |
| GPT-3.5Setting=Zero-Shot Prompting2023.06 | 56.7 | |
| GPT-3.5Setting=Few-Shot Prompting2023.06 | 50 | |
| FullBackbone=Nemotron-Nano-8B2025.12 | 48 | |
| GPT-3.5Setting=Chain-of-Thought Prompting2023.06 | 46.7 | |
| GLM-130BSetting=Zero-Shot Prompting2023.06 | 46.7 | |
| FullBackbone=Deepseek-R1-Distill-Qwen-7B2025.12 | 45 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 45 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 43 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 42 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 42 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 42 | |
| GLM-130BSetting=Few-Shot Prompting2023.06 | 40 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 40 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 40 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 39 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 37 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 35 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 28 | |
| ShadowKVBackbone=Deepseek-R1-Distill-Qwen-7B2025.12 | 27 | |
| ShadowKVBackbone=Nemotron-Nano-8B2025.12 | 27 | |
| GLM-130BSetting=Chain-of-Thought Prompting2023.06 | 23.3 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 22 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 20 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 18 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 14 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 9 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 8 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 8 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 7 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 7 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 7 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 6 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 4 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 4 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 4 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 3 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 3 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 3 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 3 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 2 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 1 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 1 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 1 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 1 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 1 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 1 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 1 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 0 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 0 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 0 |