Long-context language understanding on SCROLLS (dev)
57.4GovRep ROUGE-1BARTlarge-SLED
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BARTlarge-SLEDChunk/Input=256/16K, #Params=406M2022.08 | 57.4 | 26.3 | 27.5 | 35.3 | 8.8 | 19.5 | 36.3 | 12.2 | 23.3 | 42.5 | 23.6 | 37.2 | 85.3 | |
| LEDbaseChunk/Input=256/16K, #Params=162M2022.08 | 57.3 | 27.9 | 30 | 30.7 | 6.3 | 17.9 | 32.5 | 9 | 21.1 | 30.4 | 20.2 | 30.9 | 82.3 | |
| BARTbase-SLEDChunk/Input=256/16K, #Params=139M2022.08 | 55.7 | 24.8 | 25.8 | 33.6 | 8.5 | 19.2 | 34.4 | 11.5 | 22.7 | 35.8 | 21.3 | 33.7 | 85.3 | |
| BARTlargeChunk/Input=1K/1K, #Params=406M2022.08 | 50.6 | 19.8 | 23.5 | 32.1 | 7.4 | 18.7 | 33.3 | 9.4 | 21.6 | 24.5 | 17.9 | 36.1 | 79.3 | |
| BARTbaseChunk/Input=1K/1K, #Params=139M2022.08 | 47.7 | 18.5 | 22.3 | 30.1 | 7 | 18.3 | 32.2 | 9.3 | 21.1 | 23.3 | 15.9 | 33.8 | 78.4 | |
| T5base-SLEDChunk/Input=256/16K, #Params=220M2022.08 | 47 | 20.2 | 25.2 | 25.3 | 5 | 16.6 | 29.9 | 8.7 | 21.4 | 38.2 | 18.2 | 34.6 | 82.4 | |
| T5baseChunk/Input=1K/1K, #Params=220M2022.08 | 32.8 | 11.7 | 20.2 | 22.2 | 3.7 | 15.3 | 26.1 | 6.6 | 19.8 | 13.2 | 14.9 | 35.1 | 76.8 |