General language understanding and reasoning on TRACE
59C-STANCE AccuracyMagMax
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| MagMax# Mem. Size=0.6, # of θ stored=t, Runtime (min.)=469’92025.02 | 59 | 45 | 0.37 | 0.32 | 43 | 42 | 45 | 0.4 | 0.42 | |
| Task-Arith.# Mem. Size=0.2, # of θ stored=t, Runtime (min.)=374’62025.02 | 51 | 2 | 0.39 | 0.56 | 29 | 40 | 52 | 0.38 | 0.38 | |
| Avg. (Offline)# Mem. Size=0.8, # of θ stored=t, Runtime (min.)=525’42025.02 | 50 | 21 | 0.38 | 0.59 | 35 | 38 | 49 | 0.37 | 0.41 | |
| Task-Arith.# Mem. Size=0.6, # of θ stored=t, Runtime (min.)=474’92025.02 | 50 | 47 | 0.38 | 0.6 | 32 | 42 | 52 | 0.4 | 0.45 | |
| Replay# Mem. Size=0.4, Runtime (min.)=351’22025.02 | 49 | 21 | 0.38 | 0.5 | 47 | 35 | 41 | 0.39 | 0.4 | |
| Replay# Mem. Size=0.8, Runtime (min.)=449’42025.02 | 49 | 13 | 0.37 | 0.57 | 21 | 38 | 56 | 0.37 | 0.38 | |
| Task-Arith.# Mem. Size=0.4, # of θ stored=t, Runtime (min.)=427’62025.02 | 49 | 1 | 0.39 | 0.59 | 32 | 40 | 53 | 0.39 | 0.39 | |
| Task-Arith.# Mem. Size=0.8, # of θ stored=t, Runtime (min.)=523’12025.02 | 49 | 28 | 0.39 | 0.57 | 40 | 44 | 48 | 0.38 | 0.43 | |
| MagMax# Mem. Size=0.2, # of θ stored=t, Runtime (min.)=373’32025.02 | 49 | 44 | 0.25 | 0.26 | 37 | 31 | 48 | 0.39 | 0.37 | |
| Avg. (Offline)# Mem. Size=0.2, # of θ stored=t, Runtime (min.)=377’12025.02 | 48 | 2 | 0.37 | 0.58 | 29 | 34 | 51 | 0.37 | 0.36 | |
| Replay# Mem. Size=0.6, Runtime (min.)=394’72025.02 | 47 | 13 | 0.38 | 0.55 | 18 | 40 | 56 | 0.37 | 0.38 | |
| MagMax# Mem. Size=0.4, # of θ stored=t, Runtime (min.)=424’82025.02 | 46 | 41 | 0.39 | 0.26 | 35 | 43 | 43 | 0.4 | 0.39 | |
| Weight Space Consolidation# Mem. Size=0.8, # of θ stored=1, Runtime (min.)=522’92025.02 | 46 | 47 | 0.37 | 0.55 | 46 | 36 | 53 | 0.4 | 0.45 | |
| SLERP# Mem. Size=0.4, # of θ stored=1, Runtime (min.)=422’42025.02 | 45 | 42 | 0.26 | 0.53 | 33 | 29 | 48 | 0.43 | 0.4 | |
| SLERP# Mem. Size=0.8, # of θ stored=1, Runtime (min.)=526’42025.02 | 45 | 18 | 0.37 | 0.58 | 39 | 37 | 53 | 0.38 | 0.41 | |
| Avg. (Online)# Mem. Size=0.6, # of θ stored=1, Runtime (min.)=471’42025.02 | 45 | 50 | 0.41 | 0.55 | 35 | 34 | 52 | 0.39 | 0.44 | |
| Avg. (Online)# Mem. Size=0.8, # of θ stored=1, Runtime (min.)=524’52025.02 | 45 | 49 | 0.35 | 0.52 | 16 | 35 | 47 | 0.36 | 0.39 | |
| Avg. (Offline)# Mem. Size=0.4, # of θ stored=t, Runtime (min.)=430’62025.02 | 45 | 3 | 0.38 | 0.58 | 33 | 40 | 48 | 0.37 | 0.38 | |
| Avg. (Online)# Mem. Size=0.2, # of θ stored=1, Runtime (min.)=374’52025.02 | 44 | 46 | 0.33 | 0.52 | 39 | 31 | 37 | 0.37 | 0.4 | |
| Avg. (Offline)# Mem. Size=0.6, # of θ stored=t, Runtime (min.)=479’32025.02 | 44 | 56 | 0.38 | 0.59 | 36 | 38 | 53 | 0.36 | 0.45 | |
| Weight Space Consolidation# Mem. Size=0.6, # of θ stored=1, Runtime (min.)=465’82025.02 | 44 | 57 | 0.35 | 0.52 | 47 | 38 | 55 | 0.39 | 0.46 | |
| Replay# Mem. Size=0.2, Runtime (min.)=314’72025.02 | 43 | 1 | 0.35 | 0.5 | 29 | 32 | 38 | 0.37 | 0.33 | |
| Fine-tuneRuntime (min.)=137’52025.02 | 42 | 25 | 0.33 | 0.49 | 21 | 28 | 51 | 0.37 | 0.35 | |
| SLERP# Mem. Size=0.2, # of θ stored=1, Runtime (min.)=375’92025.02 | 42 | 45 | 0.31 | 0.54 | 36 | 29 | 42 | 0.4 | 0.39 | |
| SLERP# Mem. Size=0.6, # of θ stored=1, Runtime (min.)=472’92025.02 | 42 | 56 | 0.38 | 0.57 | 30 | 34 | 54 | 0.38 | 0.43 | |
| Avg. (Online)# Mem. Size=0.4, # of θ stored=1, Runtime (min.)=421’72025.02 | 42 | 41 | 0.35 | 0.55 | 43 | 28 | 43 | 0.44 | 0.41 | |
| Weight Space Consolidation# Mem. Size=0.4, # of θ stored=1, Runtime (min.)=422’62025.02 | 41 | 48 | 0.38 | 0.55 | 45 | 41 | 30 | 0.39 | 0.42 | |
| Weight Space Consolidation# Mem. Size=0.2, # of θ stored=1, Runtime (min.)=370’92025.02 | 39 | 13 | 0.36 | 0.51 | 32 | 40 | 56 | 0.39 | 0.38 | |
| MagMax# Mem. Size=0.8, # of θ stored=t, Runtime (min.)=525’62025.02 | 32 | 54 | 0.23 | 0.25 | 32 | 29 | 51 | 0.39 | 0.36 |