Compositional Generalization on Evaluation Dataset (Unseen Average)
42.86ScoreMistral-7B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Mistral-7BFine-tuning Strategy=LoRA, Model Backbone=Mistral-7B2026.01 | 42.86 | -0.1948 | |
| Qwen2.5-7BFine-tuning Strategy=LoRA, Model Backbone=Qwen2.5-7B2026.01 | 42.46 | -0.1578 | |
| Claude 3.5 Sonnet + FS*Evaluation Protocol=Few-shot, Model Backbone=Claude 3.5 Sonnet2026.01 | 42.03 | -0.0092 | |
| LLaMA-3-8BFine-tuning Strategy=LoRA, Model Backbone=LLaMA-3-8B2026.01 | 41.76 | -0.1573 | |
| LLaMA-3-8B + COGLMFine-tuning Strategy=LoRA, Model Backbone=LLaMA-3-8B, Method Components=COGLM2026.01 | 41.13 | -0.2118 | |
| DeepSeek V3Evaluation Protocol=Zero-shot, Model Backbone=DeepSeek V32026.01 | 41.08 | -0.0088 | |
| GPT-4o + FS*Evaluation Protocol=Few-shot, Model Backbone=GPT-4o2026.01 | 40.41 | 0.0099 | |
| Gemma-7BFine-tuning Strategy=LoRA, Model Backbone=Gemma-7B2026.01 | 40.25 | -0.1817 | |
| Mistral-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Mistral-7B, Method Components=COGLM2026.01 | 40.04 | -0.2203 | |
| Gemma-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Gemma-7B, Method Components=COGLM2026.01 | 39.57 | -0.2096 | |
| Qwen2.5-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=Qwen2.5-7B, Method Components=COGLM2026.01 | 39.1 | -0.2031 | |
| Claude 3.5 SonnetEvaluation Protocol=Zero-shot, Model Backbone=Claude 3.5 Sonnet2026.01 | 38.18 | -0.0162 | |
| DeepSeek V3 + FS*Evaluation Protocol=Few-shot, Model Backbone=DeepSeek V32026.01 | 36.37 | -0.0685 | |
| GPT-4oEvaluation Protocol=Zero-shot, Model Backbone=GPT-4o2026.01 | 35.95 | 0.0022 | |
| DeepSeek-7B + COGLMFine-tuning Strategy=LoRA, Model Backbone=DeepSeek-7B, Method Components=COGLM2026.01 | 32.92 | -0.1433 | |
| Llama 3 (70B) + FS*Evaluation Protocol=Few-shot, Model Backbone=Llama 3 (70B)2026.01 | 32.69 | 0.0213 | |
| DeepSeek-7BFine-tuning Strategy=LoRA, Model Backbone=DeepSeek-7B2026.01 | 31.88 | -0.085 | |
| Llama 3 (70B)Evaluation Protocol=Zero-shot, Model Backbone=Llama 3 (70B)2026.01 | 20.67 | -0.0064 |