List your company

Long-context Language Understanding on LongBench-e (test)

68.42LCC (Language Comprehension Score)HATA
25.7836.8547.9258.99Jun 3, 2025

Evaluation results

MethodLinks
HATABackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0668.4210.6115.6585.8354.9239.6169.3422.3913.4431.90.3425.0612.3134.6
DenseBackbone=Llama-2-7B-32K-Instruct, Token Budget=Full Context2025.0667.5311.8915.385.0355.0339.326922.4413.1332.011.1724.5111.7634.47
HATABackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0667.2599.6760.1991.9451.7242.3571.6655.1743.8235.0212.4426.0743.9553.94
DenseBackbone=Llama-3.1-8B-Instruct, Token Budget=Full Context2025.0667.2499.6760.2191.6452.3642.5571.6654.8244.0835.0313.1926.1944.6854.1
SnapKVBackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0666.4999.6759.9391.9848.640.5860.3352.8243.6326.2913.0423.236.4251
MagicPIGBackbone=Llama-2-7B-32K-Instruct, Token Budget=2-3% of sequence length2025.0666.4310.0114.6986.1755.8138.946921.713.2931.291.0823.7411.0634.09
QuestBackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0665.1415.5313.6485.1852.5739.2467.5719.3312.5124.831.216.6110.9332.64
H2OBackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0664.999457.8992.0646.741.236545.1240.9129.412.8224.6433.7549.89
StreamingLLMBackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0664.994.3348.5279.2445.5840.151.6734.1337.8122.7313.1521.5124.9144.51
LokiBackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0661.2999.6759.4891.4548.4741.9972.3354.4744.3334.7412.7425.8545.1553.23
QuestBackbone=Llama-3.1-8B-Instruct, Token Budget=5122025.0658.8199.6760.0390.7946.7239.7571.3351.543.933.6413.1625.6943.5252.19
LokiBackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0658.6811.9714.9185.344.4138.956922.1113.0930.510.5223.8212.8232.78
MagicPIGBackbone=Llama-3.1-8B-Instruct, Token Budget=2-3% of sequence length2025.0653.3998.8356.2877.942.3534.2863.6749.137.8432.589.9624.5738.247.61
SnapKVBackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0652.749.4411.7866.945.1337.563717.4412.6513.540.0812.987.2424.96
StreamingLLMBackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0646.914.848.2260.6735.4319.52815.517.5419.39018.095.4320.73
H2OBackbone=Llama-2-7B-32K-Instruct, Token Budget=5122025.0627.422.414.1920.0716.336.7424.333.062.199.6107.830.239.57