Sequential Decision Making on ALFWorld (test)
96.27Success RateSMaRT
Evaluation Results
| Method | Links | |
|---|---|---|
| SMaRTLLM=GPT-4, Shot setup=3-shot2025.10 | 96.27 | |
| LLM-as-a-JudgeLLM=GPT-4, Shot setup=3-shot2025.10 | 91.79 | |
| CoTLLM=GPT-4, Shot setup=3-shot2025.10 | 88.81 | |
| SMaRTLLM=Gemini-1.5, Shot setup=3-shot2025.10 | 83.58 | |
| LLM-as-a-JudgeLLM=Gemini-1.5, Shot setup=3-shot2025.10 | 80.59 | |
| DirectLLM=GPT-4, Shot setup=3-shot2025.10 | 76.87 | |
| OPTRole=Teacher, Size=13B2025.05 | 76.5 | |
| LLaMARole=Teacher, Size=13B2025.05 | 75.3 | |
| CoTLLM=Gemini-1.5, Shot setup=3-shot2025.10 | 70.89 | |
| Structured Agent DistillationTeacher=LLaMA-13B, Student Size=7B2025.05 | 68 | |
| Structured Agent DistillationTeacher=OPT-13B, Student Size=6.7B2025.05 | 67.1 | |
| Token-level baselineTeacher=LLaMA-13B, Student Size=7B2025.05 | 64.2 | |
| Token-level baselineTeacher=OPT-13B, Student Size=6.7B2025.05 | 62.8 | |
| SeqKDTeacher=LLaMA-13B, Student Size=7B2025.05 | 62 | |
| KDTeacher=LLaMA-13B, Student Size=7B2025.05 | 60.1 | |
| Structured Agent DistillationTeacher=OPT-13B, Student Size=2.7B2025.05 | 59.2 | |
| SeqKDTeacher=OPT-13B, Student Size=6.7B2025.05 | 59 | |
| KDTeacher=OPT-13B, Student Size=6.7B2025.05 | 57.2 | |
| DirectLLM=Gemini-1.5, Shot setup=3-shot2025.10 | 55.97 | |
| Token-level baselineTeacher=OPT-13B, Student Size=2.7B2025.05 | 55.6 | |
| Structured Agent DistillationTeacher=OPT-13B, Student Size=1.3B2025.05 | 52.3 | |
| SeqKDTeacher=OPT-13B, Student Size=2.7B2025.05 | 51 | |
| KDTeacher=OPT-13B, Student Size=2.7B2025.05 | 49.6 | |
| Token-level baselineTeacher=OPT-13B, Student Size=1.3B2025.05 | 47.8 | |
| SeqKDTeacher=OPT-13B, Student Size=1.3B2025.05 | 44 | |
| KDTeacher=OPT-13B, Student Size=1.3B2025.05 | 42.5 |