Temporal Reasoning on Time-Dialog (test)
88.9Location AccuracyGPT-4
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4Oracle Evidence=true, Setting=Oracle setting using gold test evidence2025.12 | 88.9 | 78.3 | 54.9 | 95 | 66.7 | 100 | 88.9 | 93.8 | 83.3 | 100 | 35.4 | 86.2 | |
| GPT-4Prompting Strategy=ReAct2025.12 | 66.7 | 43.5 | 39.2 | 75 | 32.2 | 67.1 | 72.2 | 70.8 | 68.5 | 84.3 | 29.2 | 62.8 | |
| GPT-4Prompting Strategy=Full Prompt2025.12 | 61.1 | 87 | 46.7 | 85 | 22.2 | 64.1 | 55.6 | 81.3 | 50 | 77.8 | 27.1 | 64.8 | |
| Qwen2.5-7BModel Size=7B, Training Strategy=Instruct2025.12 | 61.1 | 52.2 | 0 | 75 | 55.6 | 63.3 | 38.9 | 54.2 | 50 | 72.2 | 16.7 | 53.2 | |
| MemoryT1Model Size=7B2025.12 | 61.1 | 52.2 | 8.6 | 65 | 56.7 | 71.8 | 83.3 | 87.5 | 88.9 | 94.4 | 27.1 | 67 | |
| MemAgent-7BModel Size=7B2025.12 | 55.6 | 47.8 | 10.2 | 55 | 40.7 | 61.5 | 38.9 | 62.5 | 38.9 | 72.2 | 27.1 | 49.9 | |
| Memory-T1Model Size=3B2025.12 | 50 | 52.2 | 7.1 | 75 | 55.6 | 82.1 | 66.7 | 87.5 | 88.9 | 94.4 | 12.4 | 66.9 | |
| Qwen2.5-3BModel Size=3B, Training Strategy=SFT2025.12 | 22.2 | 56.5 | 7.9 | 65 | 44.4 | 66.7 | 33.3 | 56.3 | 55.6 | 77.8 | 16.7 | 50.6 | |
| Llama-3-8BModel Size=8B, Training Strategy=Instruct2025.12 | 22.2 | 43.5 | 5.6 | 75 | 33.3 | 79.5 | 27.8 | 56.3 | 72.2 | 27.8 | 14.6 | 48.4 | |
| Qwen2.5-14BModel Size=14B, Training Strategy=Instruct2025.12 | 16.7 | 47.8 | 4.4 | 70 | 55.6 | 84.6 | 66.7 | 75 | 69.7 | 94.4 | 20.8 | 60.7 | |
| Time-R12025.12 | 11.1 | 47.8 | 13.9 | 65 | 55.6 | 76.9 | 27.8 | 44.4 | 55.6 | 66.7 | 25 | 49.4 | |
| Gemma-4B-itModel Size=4B, Strategy=it2025.12 | 5.6 | 60.9 | 12.7 | 55 | 33.3 | 62.2 | 38.9 | 50 | 44.4 | 61.1 | 15.3 | 45 | |
| Qwen2.5-3BModel Size=3B, Training Strategy=Instruct2025.12 | 5.6 | 56.5 | 7.2 | 70 | 44.4 | 66.7 | 33.3 | 56.3 | 55.6 | 77.8 | 12.5 | 49.4 |