Memory Capability Evaluation on BEAM 1M tokens
22.1AbstentionLlama Maverick (Vanilla)
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama Maverick (Vanilla)Model=Llama Maverick, Setup=Vanilla2025.10 | 22.1 | 4.6 | 21.4 | 48.9 | 44 | 16.4 | 17.4 | 53.5 | 20.7 | 9.7 | 25.9 | — | |
| Qwen 2.5 (Vanilla)Model=Qwen 2.5, Setup=Vanilla2025.10 | 34.2 | 3.5 | 18.3 | 13.8 | 38.3 | 6.4 | 10.2 | 48.6 | 12.2 | 7.3 | 19.3 | — | |
| LIGHTModel=Llama Maverick, Setup=LIGHT2025.10 | 43.5 | 4.2 | 19.3 | 47.4 | 43.3 | 41.4 | 27 | 61 | 31.5 | 17.6 | 33.6 | — | |
| GPT-4.1-nano (Vanilla)Model=GPT-4.1-nano, Setup=Vanilla2025.10 | 49.2 | 5 | 19.1 | 15.3 | 22.6 | 15 | 9.1 | 43.5 | 6 | 6.1 | 19.1 | — | |
| LIGHTModel=Qwen 2.5, Setup=LIGHT2025.10 | 50 | 2.1 | 20 | 36.6 | 41.9 | 35.7 | 20.9 | 55.1 | 31.6 | 15.4 | 30.9 | — | |
| Gemini 2 Flash (Vanilla)Model=Gemini 2 Flash, Setup=Vanilla2025.10 | 64.2 | 1 | 19 | 37.4 | 12 | 10.7 | 8.3 | 27.3 | 9.1 | 10.4 | 19.9 | — | |
| Qwen 2.5 (RAG)Model=Qwen 2.5, Setup=RAG2025.10 | 65 | 3.5 | 19.5 | 40.7 | 30 | 37.8 | 16.3 | 49.1 | 15.7 | 7.8 | 28.5 | — | |
| LIGHTModel=GPT-4.1-nano, Setup=LIGHT2025.10 | 67.8 | 2.1 | 21.1 | 41 | 39.4 | 39.2 | 27.8 | 57.6 | 29 | 10.7 | 33.6 | — | |
| LIGHTModel=Gemini 2 Flash, Setup=LIGHT2025.10 | 73.5 | 0.7 | 18.5 | 34.1 | 38 | 26.4 | 14.7 | 47.2 | 22.4 | 8.5 | 28.4 | — | |
| Llama Maverick (RAG)Model=Llama Maverick, Setup=RAG2025.10 | 74.2 | 2.8 | 17.9 | 43.1 | 33.8 | 34.2 | 24.5 | 51.4 | 14.5 | 10.7 | 30.7 | — | |
| Gemini 2 Flash (RAG)Model=Gemini 2 Flash, Setup=RAG2025.10 | 75 | 2.8 | 19.8 | 38 | 29 | 27.8 | 13.4 | 47 | 12.5 | 5.7 | 27.1 | — | |
| GPT-4.1-nano (RAG)Model=GPT-4.1-nano, Setup=RAG2025.10 | 77.8 | 2.8 | 17.9 | 39.9 | 27.1 | 34.2 | 29.3 | 51.3 | 15.2 | 6.4 | 30.2 | — | |
| HindsightAnswer model=proprietary2026.05 | — | — | — | — | — | — | — | — | — | — | — | 73.9 | |
| TM ProAggregation=3-run mean, Answer model=gpt-4.1-mini2026.05 | — | — | — | — | — | — | — | — | — | — | — | 76.6 |