Long-context understanding on LongBench (Specific Metric Subset)
27.7Qasper ScoreLongChat-7B-v1.5
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| LongChat-7B-v1.5Comp (%)=02026.06 | 27.7 | 22.7 | 82.3 | 43 | 68.5 | 26.1 | 15.4 | 40.81 | |
| LLaMA-3.1-8B-InstComp (%)=02026.06 | 25.19 | 23.2 | 92 | 39.9 | 72.5 | 26.9 | 15.91 | 42.23 | |
| PaluBase Model=LongChat-7B-v1.5, Comp (%)=302026.06 | 23.81 | 22.64 | 80.61 | 44.15 | 64.5 | 25.4 | 14.08 | 39.31 | |
| STAR-KVBase Model=LLaMA-3.1-8B-Inst, Comp (%)=602026.06 | 23.2 | 22.4 | 89.06 | 36.34 | 67 | 25.89 | 13.2 | 39.58 | |
| STAR-KVBase Model=LongChat-7B-v1.5, Comp (%)=602026.06 | 22.58 | 22.4 | 81.3 | 41.04 | 65 | 25.7 | 15.6 | 39.09 | |
| PaluBase Model=LongChat-7B-v1.5, Comp (%)=502026.06 | 21.1 | 22.4 | 75.81 | 40.78 | 62.5 | 22.6 | 12.58 | 36.8 | |
| PaluBase Model=LLaMA-3.1-8B-Inst, Comp (%)=502026.06 | 15.38 | 22.1 | 73.36 | 27.58 | 63.5 | 21.66 | 1.95 | 32.21 | |
| PaluBase Model=LLaMA-3.1-8B-Inst, Comp (%)=302026.06 | 14.47 | 23.3 | 86.71 | 26.57 | 73 | 26.19 | 8.33 | 36.93 |