List your company

Long-context Language Understanding on LongBench (7-Category Performance Summary)

80.29Math Performance16-bit Baseline
0.646821.32344262.6766Mar 25, 2025

Evaluation results

MethodLinks
16-bit BaselineModel=Phi-3-mini-128k-instruct, KV Cache Precision=16-bit2025.0380.2955.9752.5833.5542.4717.5648
16-bit BaselineModel=llama-3.1-8B-Instruct, KV Cache Precision=16-bit2025.0371.4259.7861.2139.9547.7118.0767.78
16-bit BaselineModel=Qwen1.5-14B-Chat-AWQ, KV Cache Precision=16-bit2025.0370.2857.4759.0239.7242.4817.2161.33
LogQuantModel=Qwen1.5-14B-Chat-AWQ, KV Cache Precision=2-bit2025.0363.3149.3758.2538.0141.3717.2452.17
KiViModel=Qwen1.5-14B-Chat-AWQ, KV Cache Precision=2-bit2025.0359.8237.4857.537.9140.3917.1746.85
16-bit BaselineModel=Qwen1.5-7B-Chat-AWQ, KV Cache Precision=16-bit2025.0356.1852.4653.8833.0539.2617.1126.5
16-bit BaselineModel=Qwen2-7B-Instruct, KV Cache Precision=16-bit2025.0352.9958.2361.933.3544.6616.3343
LogQuantModel=Phi-3-mini-128k-instruct, KV Cache Precision=2-bit2025.0351.8640.8439.3621.723.639.895.39
LogQuantModel=Qwen1.5-7B-Chat-AWQ, KV Cache Precision=2-bit2025.0349.2840.6852.5432.0437.2217.3813.5
LogQuantModel=llama-3.1-8B-Instruct, KV Cache Precision=2-bit2025.0340.4152.0956.4236.0841.916.6252.51
KiViModel=Qwen1.5-7B-Chat-AWQ, KV Cache Precision=2-bit2025.0339.2734.7951.3231.0835.817.1610
LogQuantModel=Qwen2-7B-Instruct, KV Cache Precision=2-bit2025.0334.3448.7151.2328.2834.8413.1322.83
KiViModel=llama-3.1-8B-Instruct, KV Cache Precision=2-bit2025.0318.0443.0652.534.0138.8916.145.02
KiViModel=Phi-3-mini-128k-instruct, KV Cache Precision=2-bit2025.0312.5933.9736.1718.1919.589.14.83
KiViModel=Qwen2-7B-Instruct, KV Cache Precision=2-bit2025.033.7135.9135.2612.3520.529.3111.42