Compositional Generalization on Evaluation Dataset (Fold 1 Seen)
0.6191ScoreMistral-7B
Evaluation Results
| Method | Links | |
|---|---|---|
| Mistral-7BFine-tuning Strategy=LoRA, Model Backbone=Mistral-7B2026.01 | 0.6191 | |
| Mistral-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Mistral-7B, Method Components=COGLM2026.01 | 0.5794 | |
| Gemma-7BFine-tuning Strategy=LoRA, Model Backbone=Gemma-7B2026.01 | 0.5747 | |
| LLaMA-3-8B + COGLMFine-tuning Strategy=LoRA, Model Backbone=LLaMA-3-8B, Method Components=COGLM2026.01 | 0.5678 | |
| Gemma-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Gemma-7B, Method Components=COGLM2026.01 | 0.5626 | |
| Qwen2.5-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Qwen2.5-7B, Method Components=COGLM2026.01 | 0.5465 | |
| Qwen2.5-7BFine-tuning Strategy=LoRA, Model Backbone=Qwen2.5-7B2026.01 | 0.5203 | |
| LLaMA-3-8BFine-tuning Strategy=LoRA, Model Backbone=LLaMA-3-8B2026.01 | 0.5095 | |
| DeepSeek-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=DeepSeek-7B, Method Components=COGLM2026.01 | 0.4425 | |
| Claude 3.5 Sonnet + FS*Evaluation Protocol=Few-shot, Model Backbone=Claude 3.5 Sonnet2026.01 | 0.4062 | |
| GPT-4o + FS*Evaluation Protocol=Few-shot, Model Backbone=GPT-4o2026.01 | 0.3908 | |
| DeepSeek V3 + FS*Evaluation Protocol=Few-shot, Model Backbone=DeepSeek V32026.01 | 0.3885 | |
| DeepSeek V3Evaluation Protocol=Zero-shot, Model Backbone=DeepSeek V32026.01 | 0.3806 | |
| Claude 3.5 SonnetEvaluation Protocol=Zero-shot, Model Backbone=Claude 3.5 Sonnet2026.01 | 0.3671 | |
| DeepSeek-7BFine-tuning Strategy=LoRA, Model Backbone=DeepSeek-7B2026.01 | 0.3261 | |
| GPT-4oEvaluation Protocol=Zero-shot, Model Backbone=GPT-4o2026.01 | 0.3161 | |
| Llama 3 (70B) + FS*Evaluation Protocol=Few-shot, Model Backbone=Llama 3 (70B)2026.01 | 0.2503 | |
| Llama 3 (70B)Evaluation Protocol=Zero-shot, Model Backbone=Llama 3 (70B)2026.01 | 0.1797 |