Logical Reasoning on LogiQA (val)
58.37AccuracyGPT-4-0125-preview
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT-4-0125-previewCategory=LLM2024.05 | 58.37 | |
| PaLMv2Category=LLM, Extra data=true2024.05 | 48 | |
| PathReasoner2024.05 | 43.16 | |
| Qwen1.5-7BTraining Dataset=Base Model, Model=Qwen1.5-7B2024.05 | 42.7 | |
| LogiformerCategory=Graph2024.05 | 42.24 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 41.94 | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 41.94 | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 41.93 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 41.78 | |
| LLaMA3-8BTraining Dataset=Base Model, Model=LLaMA3-8B2024.05 | 41.47 | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 41.17 | |
| FocalReasonerCategory=Graph2024.05 | 41.01 | |
| text-davinci-003Category=LLM2024.05 | 41 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 40.55 | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 40.4 | |
| GPT-3.5-turboCategory=LLM2024.05 | 40.25 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 40.09 | |
| AdaLoGNCategory=Graph2024.05 | 39.94 | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 39.63 | |
| MERITCategory=Sequence, Extra data=true2024.05 | 39.5 | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 38.71 | |
| LReasonerCategory=Sequence2024.05 | 38.1 | |
| LoRATraining Dataset=Alpaca-GPT4, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 37.63 | |
| LoRATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 37.33 | |
| DAGNCategory=Graph2024.05 | 36.87 | |
| MOLORATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 36.26 | |
| LoRATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 35.64 | |
| ROBERTa-LargeCategory=Sequence2024.05 | 35.02 | |
| LoRATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 35.02 | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 34.56 | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 34.41 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 34.25 | |
| BERT-LargeCategory=Sequence2024.05 | 34.1 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 34.1 | |
| LLaMA2-7BTraining Dataset=Base Model, Model=LLaMA2-7B2024.05 | 33.95 | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 33.33 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 33.03 | |
| TAIATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=TAIA2024.05 | 32.87 | |
| MOLORATraining Dataset=Alpaca-GPT4, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 32.72 | |
| TAIATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 32.57 | |
| Qwen1.5-1.8BTraining Dataset=Base Model, Model=Qwen1.5-1.8B2024.05 | 32.41 | |
| TAIATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=TAIA2024.05 | 31.8 | |
| LoRATraining Dataset=CoT-Collection, Model=LLaMA2-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 30.57 | |
| MOLORATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 28.27 | |
| LoRATraining Dataset=CoT-Collection, Model=LLaMA3-8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 27.19 | |
| Random2024.05 | 25 | |
| MOLORATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 22.73 | |
| MOLORATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=MOLORA, Infer Mode=Vanilla2024.05 | 22.43 | |
| LoRATraining Dataset=CoT-Collection, Model=Qwen1.5-1.8B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 21.97 | |
| LoRATraining Dataset=CoT-Collection, Model=Qwen1.5-7B, FT Method=LoRA, Infer Mode=Vanilla2024.05 | 21.35 |