Atomic Memory Recall on Super-Natural Instructions (test)
75.8ROUGE-L (10 tasks)Fine-Tuing
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Fine-TuingModel=Llama 3.1 8B2025.10 | 75.8 | 64.3 | 63.2 | 58.7 | 61.6 | 64.7 | |
| Replay MemoryModel=Llama 3.1 8B2025.10 | 75.8 | 65.2 | 64.5 | 63.4 | 63.6 | 66.5 | |
| TokMem+DCModel=Llama 3.1 8B2025.10 | 75.6 | 65.8 | 63.7 | 64.2 | 64.4 | 66.7 | |
| TokMemModel=Llama 3.1 8B2025.10 | 75.4 | 65.5 | 65.1 | 64.4 | 64.8 | 67 | |
| TokMem+DCModel=Llama 3.2 3B2025.10 | 68.8 | 62.5 | 58.7 | 61.7 | 61.1 | 62.6 | |
| TokMemModel=Llama 3.2 3B2025.10 | 68 | 62.3 | 61.2 | 61.5 | 61.5 | 62.9 | |
| Fine-TuingModel=Llama 3.2 3B2025.10 | 67.1 | 59.1 | 59.5 | 58.4 | 57.9 | 60.4 | |
| Replay MemoryModel=Llama 3.2 3B2025.10 | 67.1 | 61.1 | 60.6 | 61.4 | 60 | 62 | |
| RAGModel=Llama 3.1 8B2025.10 | 63.8 | 53.9 | 49.1 | 45.3 | 42.6 | 50.9 | |
| RAGModel=Llama 3.2 3B2025.10 | 60 | 48.7 | 45.8 | 42.3 | 39.9 | 47.3 | |
| TokMem+DCModel=Qwen 2.5 0.5B2025.10 | 53.8 | 50.5 | 50.2 | 50.9 | 50 | 51.1 | |
| TokMemModel=Qwen 2.5 0.5B2025.10 | 52.8 | 51.3 | 49.3 | 50.2 | 50 | 50.7 | |
| Fine-TuingModel=Qwen 2.5 0.5B2025.10 | 52.4 | 48 | 40.6 | 41.7 | 43.2 | 45.2 | |
| Replay MemoryModel=Qwen 2.5 0.5B2025.10 | 52.4 | 49.5 | 47.2 | 47.7 | 46.7 | 48.7 | |
| RAGModel=Qwen 2.5 0.5B2025.10 | 50.4 | 43.2 | 38.8 | 36.2 | 34.7 | 40.7 | |
| BaseModel=Qwen 2.5 0.5B2025.10 | 33.9 | 39 | 38.8 | 39.1 | 38.5 | 37.9 | |
| BaseModel=Llama 3.1 8B2025.10 | 27.2 | 27.8 | 30.4 | 29.6 | 29.5 | 28.9 | |
| BaseModel=Llama 3.2 3B2025.10 | 16.6 | 19.9 | 20 | 18.7 | 18.2 | 18.7 |