Memory Capability Evaluation on BEAM 500K tokens
82.8Abstention RateGPT-4.1-nano (RAG)
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4.1-nano (RAG)Model=GPT-4.1-nano, Setup=RAG2025.10 | 82.8 | 2.5 | 18 | 38.2 | 28.6 | 28.8 | 23.3 | 57.7 | 18.4 | 16.1 | 31.4 | |
| Gemini 2 Flash (RAG)Model=Gemini 2 Flash, Setup=RAG2025.10 | 80 | 2.1 | 18.9 | 34.3 | 33.4 | 18 | 13.5 | 42.7 | 16.5 | 7.8 | 26.7 | |
| Llama Maverick (RAG)Model=Llama Maverick, Setup=RAG2025.10 | 78.5 | 2.8 | 18.6 | 40.2 | 44.7 | 33.8 | 31.3 | 52.5 | 19.7 | 7.8 | 33 | |
| Qwen 2.5 (RAG)Model=Qwen 2.5, Setup=RAG2025.10 | 72.8 | 1.7 | 22.1 | 40 | 35 | 22.6 | 18.7 | 47.7 | 18.7 | 11.4 | 29.1 | |
| Gemini 2 Flash (Vanilla)Model=Gemini 2 Flash, Setup=Vanilla2025.10 | 71.4 | 1 | 21.5 | 46.9 | 13.3 | 17.1 | 19.8 | 37.9 | 13.6 | 15 | 25.7 | |
| LIGHTModel=Gemini 2 Flash, Setup=LIGHT2025.10 | 68.5 | 2.1 | 20 | 47.8 | 28 | 22.3 | 15.7 | 53.2 | 25 | 9.2 | 29.2 | |
| LIGHTModel=Llama Maverick, Setup=LIGHT2025.10 | 62.8 | 4.2 | 19.7 | 53.5 | 39 | 26.4 | 35 | 62.3 | 37.3 | 19 | 35.9 | |
| LIGHTModel=GPT-4.1-nano, Setup=LIGHT2025.10 | 60 | 3.5 | 20.4 | 49.1 | 34.2 | 24 | 26.6 | 68.4 | 33.4 | 15.4 | 33.5 | |
| LIGHTModel=Qwen 2.5, Setup=LIGHT2025.10 | 57.1 | 1.7 | 24.4 | 50.6 | 29.5 | 27.8 | 21.4 | 57.1 | 34.4 | 12.1 | 31.6 | |
| GPT-4.1-nano (Vanilla)Model=GPT-4.1-nano, Setup=Vanilla2025.10 | 55.7 | 1.7 | 18.8 | 14.2 | 24.4 | 10.7 | 7 | 45 | 10.9 | 5.7 | 19.4 | |
| Qwen 2.5 (Vanilla)Model=Qwen 2.5, Setup=Vanilla2025.10 | 31.4 | 5.3 | 18.5 | 16.6 | 30.4 | 11.1 | 12.5 | 56.7 | 13.7 | 3.5 | 20 | |
| Llama Maverick (Vanilla)Model=Llama Maverick, Setup=Vanilla2025.10 | 18.5 | 3.5 | 20.9 | 60.8 | 40.3 | 27.6 | 21.9 | 56 | 26.6 | 6.4 | 28.3 |