Test-Time Learning (TTL) suite
87Bank77 AccuracyUMA
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| UMAIs Proposed Full Method=true2026.02 | 87 | 91 | 82 | 67.2 | 95 | 75 | — | |
| UMA2026.02 | 87 | 91 | 82 | 67.2 | 95 | 75 | 65.4 | |
| UMA2026.02 | 87 | 91 | 82 | 67.2 | 95 | 75 | 77.36 | |
| UMAStages=22026.02 | 85.07 | 84.67 | 70.17 | 45.47 | 77 | 43 | — | |
| UMA (2 Stage)Processing Stages=22026.02 | 85.07 | 84.67 | 70.17 | 45.47 | 77 | 43 | 54.63 | |
| UMAPhase I=false2026.02 | 84.67 | 79.25 | 75 | 47.35 | 81 | 25 | — | |
| UMA (w/o Phase I)Phase I=false2026.02 | 84.67 | 79.25 | 75 | 47.35 | 81 | 25 | 52.34 | |
| UMAPhase I (sequential memory maintenance)=false2026.02 | 84 | 79 | 74 | 47.7 | 81 | 25 | 63.81 | |
| UMAtraining stages=2 Stage2026.02 | 84 | 84 | 69 | 45 | 78 | 43 | 66.29 | |
| MemAlpha2026.02 | 83 | 78 | 71 | 57.4 | 78 | 60 | 64.5 | |
| RAGk=202026.02 | 81 | 70 | 65 | 53.3 | 10 | 10 | 54.09 | |
| UMAReinforcement Learning=false2026.02 | 78.12 | 72 | 61.67 | 39.64 | 69 | 20 | — | |
| UMA (w/o RL)Reinforcement Learning=false2026.02 | 78.12 | 72 | 61.67 | 39.64 | 69 | 20 | 49.47 | |
| UMARL training=false2026.02 | 76 | 72 | 61 | 38.8 | 70 | 20 | 61.38 | |
| Concat2026.02 | 72 | 79 | 71 | 57.8 | 71 | 18 | 51.87 | |
| UMAtask grouping strategy=Global Group2026.02 | 72 | 79 | 82 | 74 | 89 | 55 | 73.58 | |
| Concat2026.02 | 70 | 79 | 71 | 57.8 | 70 | 18 | — | |
| Concat2026.02 | 70 | 79 | 71 | 57.8 | 70 | 18 | 46.13 | |
| UMAGrouping=Global2026.02 | 69.01 | 77 | 82.01 | 69.11 | 87.01 | 47.03 | — | |
| UMA (Global Group)Grouping Strategy=Global2026.02 | 69.01 | 77 | 82.01 | 69.11 | 87.01 | 47.03 | 56.37 | |
| UMARL training=false, Phase I (sequential memory maintenance)=false2026.02 | 34 | 28 | 24 | 40.2 | 23 | 15 | 45.26 | |
| MemAgent-woqwithout question=true2026.02 | 27 | 41 | 34 | 0.82 | 69.67 | 42.04 | — | |
| MemAgent-woq2026.02 | 27 | 41 | 34 | 0.83 | 69.67 | 42.04 | 19.25 | |
| MemAgent-woq2026.02 | 27 | 41 | 34 | 47 | 70 | 44 | 30.94 | |
| UMAReinforcement Learning=false, Phase I=false2026.02 | 26 | 27.5 | 22 | 40.2 | 12 | 7 | — | |
| UMA (w/o RL & Phase I)Reinforcement Learning=false, Phase I=false2026.02 | 26 | 27.5 | 22 | 40.2 | 12 | 7 | 31.68 | |
| MemAgent2026.02 | 26 | 14 | 24 | 51.2 | 61 | 37 | 50.09 | |
| RAGk=202026.02 | 25.13 | 30.82 | 20.9 | 18.92 | 3.63 | 2.04 | 17.6 | |
| RAGk=202026.02 | 25.12 | 30.82 | 20.9 | 18.92 | 3.63 | 2.04 | — | |
| MemAgent2026.02 | 24.4 | 13 | 23 | 26.01 | 61 | 37 | — | |
| MemAgent2026.02 | 24.4 | 13 | 23 | 26.04 | 61 | 37 | 26.1 | |
| Mem12026.02 | 0 | 4 | 0 | 0.44 | 7 | 0 | — | |
| MemAlpha2026.02 | 0 | 0 | 2.61 | 0.53 | 0.02 | 0 | — | |
| Mem12026.02 | 0 | 4 | 0 | 0.45 | 7 | 0 | 3.17 | |
| MemAlpha2026.02 | 0 | 0 | 2.61 | 0.54 | 0.02 | 0 | 4.53 | |
| Mem12026.02 | 0 | 4 | 0 | 45.4 | 7 | 0 | 10.03 |