Memory Capability Evaluation on BEAM 10M tokens
5Abstention RateLlama Maverick (Vanilla)
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama Maverick (Vanilla)Model=Llama Maverick, Setup=Vanilla2025.10 | 5 | 2.5 | 19 | 7.5 | 25 | 10 | 0 | 29.1 | 6.5 | 0 | 10.4 | |
| Qwen 2.5 (Vanilla)Model=Qwen 2.5, Setup=Vanilla2025.10 | 25 | 5 | 18 | 10 | 17.5 | 10 | 12.5 | 24.1 | 11.4 | 0 | 13.3 | |
| LIGHTModel=GPT-4.1-nano, Setup=LIGHT2025.10 | 40 | 2.5 | 17.3 | 35 | 25 | 30 | 13.5 | 42.5 | 17.9 | 2.5 | 22.6 | |
| LIGHTModel=Llama Maverick, Setup=LIGHT2025.10 | 45 | 0 | 17.6 | 30 | 50 | 32.5 | 12.5 | 48.3 | 27.7 | 2.5 | 26.6 | |
| GPT-4.1-nano (Vanilla)Model=GPT-4.1-nano, Setup=Vanilla2025.10 | 45 | 0 | 21.5 | 5 | 7.5 | 5 | 1.2 | 17.5 | 2 | 5 | 10.9 | |
| LIGHTModel=Qwen 2.5, Setup=LIGHT2025.10 | 55 | 1.2 | 19.7 | 35 | 35 | 27.5 | 12.5 | 30.8 | 22 | 0 | 23.8 | |
| Qwen 2.5 (RAG)Model=Qwen 2.5, Setup=RAG2025.10 | 60 | 0 | 22.1 | 35 | 20 | 30 | 5 | 29.1 | 10.6 | 0 | 21.1 | |
| Gemini 2 Flash (RAG)Model=Gemini 2 Flash, Setup=RAG2025.10 | 65 | 2.5 | 26.6 | 27.5 | 12.5 | 32.5 | 12.5 | 30 | 4.5 | 2.5 | 21.6 | |
| LIGHTModel=Gemini 2 Flash, Setup=LIGHT2025.10 | 65 | 0 | 19.3 | 15 | 25 | 20 | 12.5 | 15 | 13.6 | 7.5 | 19.2 | |
| GPT-4.1-nano (RAG)Model=GPT-4.1-nano, Setup=RAG2025.10 | 65 | 1.2 | 20.1 | 30 | 17.5 | 32.5 | 9.1 | 36.6 | 6.3 | 0 | 21.8 | |
| Llama Maverick (RAG)Model=Llama Maverick, Setup=RAG2025.10 | 70 | 0 | 22 | 37.5 | 35 | 37.5 | 7.5 | 31.6 | 5.3 | 2.5 | 24.9 | |
| Gemini 2 Flash (Vanilla)Model=Gemini 2 Flash, Setup=Vanilla2025.10 | 75 | 0 | 22 | 7.5 | 2.5 | 5 | 0 | 7.5 | 0 | 2.5 | 12.2 |