Long-context understanding on LongBench (32k context length)
19.55Score (32k)SSA
Evaluation Results
| Method | Links | |
|---|---|---|
| SSATraining Protocol=Continual Trained to 32k, Inference Mode=Full Attention, Receptive Field=Full2025.11 | 19.55 | |
| SSATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 19.05 | |
| FullAttnTraining Protocol=Continual Trained to 32k, Inference Mode=Full Attention, Receptive Field=Full2025.11 | 18.97 | |
| MoBATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 18.68 | |
| SSATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 18.58 | |
| SSATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Full Attention, Receptive Field=Full2025.11 | 18.32 | |
| SSATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 18.25 | |
| MoBATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 18.23 | |
| NSATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 18.08 | |
| SSATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 17.89 | |
| FullAttnTraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Full Attention, Receptive Field=Full2025.11 | 17.71 | |
| NSATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 17.45 | |
| NSATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 17.36 | |
| NSATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 17.25 | |
| MoBATraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 17.17 | |
| MoBATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 16.88 | |
| FullAttnTraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 16.84 | |
| FullAttnTraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=10242025.11 | 16.72 | |
| FullAttnTraining Protocol=Continual Trained to 32k, Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 14.96 | |
| FullAttnTraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Sparse Attention, Receptive Field=2562025.11 | 14.75 | |
| MoBATraining Protocol=Continual Trained to 32k, Inference Mode=Full Attention, Receptive Field=Full2025.11 | 14.47 | |
| MoBATraining Protocol=Length Extrapolation (RoPE Scaling), Inference Mode=Full Attention, Receptive Field=Full2025.11 | 13.62 |