Natural Language Understanding on SuperGLUE (BoolQ, WiC, RTE, CB, ReCoRD, WSC, MultiRC, COPA)
59.36BoolQ AccuracySCAPE (d = 0.1)
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| SCAPE (d = 0.1)Protocol=Zero-shot, Model=GPT-345M, d=0.12026.07 | 59.36 | 51.72 | 52.35 | 32.14 | 77.38 | 41.35 | 55.18 | 73 | 55.31 | |
| SCAPEBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Sparsity (d)=0.012026.07 | 57.92 | 50 | 52.71 | 32.14 | 69.66 | 36.54 | 56.48 | 68 | 52.93 | |
| SCAPESparsity (d)=0.01, Backbone=Llama-500M, Evaluation Protocol=Zero-shot2026.07 | 57.92 | 50 | 52.71 | 32.14 | 69.66 | 36.54 | 56.48 | 68 | 52.93 | |
| ADAMWBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Communication Strategy=DENSE all-reduce2026.07 | 57.65 | 49.69 | 53.43 | 35.71 | 69.81 | 41.35 | 57.03 | 65 | 53.71 | |
| ADAMWSynchronization=DENSE all-reduce, Backbone=Llama-500M, Evaluation Protocol=Zero-shot2026.07 | 57.65 | 49.69 | 53.43 | 35.71 | 69.81 | 41.35 | 57.03 | 65 | 53.71 | |
| ADAMSBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Communication Strategy=DENSE all-reduce2026.07 | 57.55 | 50.63 | 53.07 | 41.07 | 69.54 | 34.62 | 49.86 | 68 | 53.04 | |
| ADAMSSynchronization=DENSE all-reduce, Backbone=Llama-500M, Evaluation Protocol=Zero-shot2026.07 | 57.55 | 50.63 | 53.07 | 41.07 | 69.54 | 34.62 | 49.86 | 68 | 53.04 | |
| SCAPE (d = 0.01)Protocol=Zero-shot, Model=GPT-345M, d=0.012026.07 | 56.42 | 49.84 | 50.18 | 35.71 | 76.7 | 36.54 | 54.83 | 69 | 53.65 | |
| ADAMS (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 55.72 | 50 | 53.43 | 50 | 78.06 | 36.54 | 54.02 | 70 | 55.97 | |
| ADAMW (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 53.12 | 49.53 | 52.71 | 28.57 | 76.24 | 59.62 | 57.24 | 74 | 56.38 | |
| SCAPEBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Sparsity (d)=0.12026.07 | 51.9 | 49.06 | 57.4 | 28.57 | 70 | 39.42 | 53.11 | 68 | 52.18 | |
| SCAPESparsity (d)=0.1, Backbone=Llama-500M, Evaluation Protocol=Zero-shot2026.07 | 51.9 | 49.06 | 57.4 | 28.57 | 70 | 39.42 | 53.11 | 68 | 52.18 |