Long-context Summarization on BookSum (Efficiency Measured)
3.76TPT (s)SWIFT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SWIFTBackbone=Llama3.1-70B, Layers (L)=802026.02 | 3.76 | 0.84 | 49.3 | |
| CLaSpBackbone=Llama3.1-70B, Layers (L)=802026.02 | 4.57 | 1.01 | 81.3 | |
| ARBackbone=Llama3.1-70B, Layers (L)=802026.02 | 4.76 | 1 | — | |
| DELBackbone=Llama3.1-70B, Layers (L)=802026.02 | 4.84 | 0.87 | 0 | |
| KnapSpecBackbone=Llama3.1-70B, Layers (L)=802026.02 | 6.99 | 1.14 | 84 | |
| SWIFTBackbone=Llama3.1-8B, Layers (L)=322026.02 | 8.25 | 0.72 | 31.6 | |
| CLaSpBackbone=Llama3.1-8B, Layers (L)=322026.02 | 8.79 | 1.05 | 85.3 | |
| ARBackbone=Llama3.1-8B, Layers (L)=322026.02 | 10.77 | 1 | — | |
| SWIFTBackbone=Llama3.2-3B, Layers (L)=282026.02 | 11.09 | 0.89 | 42.1 | |
| DELBackbone=Llama3.1-8B, Layers (L)=322026.02 | 11.49 | 0.74 | 0.1 | |
| CLaSpBackbone=Llama3.2-3B, Layers (L)=282026.02 | 12.22 | 1.06 | 82.3 | |
| KnapSpecBackbone=Llama3.1-8B, Layers (L)=322026.02 | 15.48 | 1.18 | 91.5 | |
| ARBackbone=Llama3.2-3B, Layers (L)=282026.02 | 15.71 | 1 | — | |
| DELBackbone=Llama3.2-3B, Layers (L)=282026.02 | 16.37 | 0.75 | 20.1 | |
| SWIFTBackbone=Llama3.2-1B, Layers (L)=162026.02 | 21.31 | 0.96 | 69.5 | |
| CLaSpBackbone=Llama3.2-1B, Layers (L)=162026.02 | 23.42 | 1.04 | 82.8 | |
| KnapSpecBackbone=Llama3.2-3B, Layers (L)=282026.02 | 23.48 | 1.18 | 87 | |
| ARBackbone=Llama3.2-1B, Layers (L)=162026.02 | 30.43 | 1 | — | |
| DELBackbone=Llama3.2-1B, Layers (L)=162026.02 | 32.97 | 0.76 | 31.4 | |
| KnapSpecBackbone=Llama3.2-1B, Layers (L)=162026.02 | 38.23 | 1.13 | 91.2 |