List your company
ResearchFormerly WizWand

Logical Reasoning Machine Reading Comprehension on ReClor

80AccuracyIDOL
-3.218.44061.6Jun 27, 2023Nov 23, 2023Apr 21, 2024Sep 18, 2024Feb 15, 2025Jul 15, 2025Dec 12, 2025
Updated 8mo ago

Evaluation results

MethodLinks
IDOLSetting=Fine-Tuning2023.0680
ChatGPTSetting=Chain-of-Thought Prompting2023.0670
ChatGPTSetting=Zero-Shot Prompting2023.0663.3
ChatGPTSetting=Few-Shot Prompting2023.0663.3
GPT-3.5Setting=Zero-Shot Prompting2023.0656.7
GPT-3.5Setting=Few-Shot Prompting2023.0650
FullBackbone=Nemotron-Nano-8B2025.1248
GPT-3.5Setting=Chain-of-Thought Prompting2023.0646.7
GLM-130BSetting=Zero-Shot Prompting2023.0646.7
FullBackbone=Deepseek-R1-Distill-Qwen-7B2025.1245
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.1245
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1243
SnapKVBackbone=Nemotron-Nano-8B, Budget=1282025.1242
SnapKVBackbone=Nemotron-Nano-8B, Budget=2562025.1242
SnapKVBackbone=Nemotron-Nano-8B, Budget=3842025.1242
GLM-130BSetting=Few-Shot Prompting2023.0640
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1240
H2OBackbone=Nemotron-Nano-8B, Budget=5122025.1240
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.1239
SnapKVBackbone=Nemotron-Nano-8B, Budget=5122025.1237
H2OBackbone=Nemotron-Nano-8B, Budget=3842025.1235
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1228
ShadowKVBackbone=Deepseek-R1-Distill-Qwen-7B2025.1227
ShadowKVBackbone=Nemotron-Nano-8B2025.1227
GLM-130BSetting=Chain-of-Thought Prompting2023.0623.3
H2OBackbone=Nemotron-Nano-8B, Budget=2562025.1222
H2OBackbone=Nemotron-Nano-8B, Budget=1282025.1220
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1218
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=5122025.1214
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=3842025.129
RKVBackbone=Nemotron-Nano-8B, Budget=2562025.128
RKVBackbone=Nemotron-Nano-8B, Budget=3842025.128
KnormBackbone=Nemotron-Nano-8B, Budget=3842025.127
KnormBackbone=Nemotron-Nano-8B, Budget=5122025.127
RKVBackbone=Nemotron-Nano-8B, Budget=5122025.127
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=2562025.126
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.124
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.124
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.124
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.123
KnormBackbone=Nemotron-Nano-8B, Budget=2562025.123
RKVBackbone=Nemotron-Nano-8B, Budget=1282025.123
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=1282025.123
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.122
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.121
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.121
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.121
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.121
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.121
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.121
KnormBackbone=Nemotron-Nano-8B, Budget=1282025.121
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.120
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.120
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.120