List your company

Long-context language modeling on RULER 1.0 (test)

0.977Accuracy (4K Context)MInference
-0.018280.240110.49850.75689Jul 2, 2024

Evaluation results

MethodLinks
MInferenceClaimed context length=-, Effective context length=32K, Backbone=LLAMA-3-8B-262K2024.070.9770.9120.8850.850.8230.7760.87
LLAMA-3-8B-262KClaimed context length=262K, Effective context length=16K, Backbone=LLAMA-3-8B-262K2024.070.9720.9180.8730.8080.7740.7220.844
StreamingLLMClaimed context length=-, Effective context length=4K, Backbone=LLAMA-3-8B-262K2024.070.9720.3810.3750.1720.1420.0940.35
InfLLMClaimed context length=-, Effective context length=8K, Backbone=GLM-4-9B-1M2024.070.9470.8950.7640.6650.5680.5350.729
MInferenceClaimed context length=-, Effective context length=64K, Backbone=GLM-4-9B-1M2024.070.9460.9310.910.8960.8550.840.896
GLM-4-9B-1MClaimed context length=1M, Effective context length=64K, Backbone=GLM-4-9B-1M2024.070.9380.9160.8930.8740.8520.8080.88
StreamingLLMClaimed context length=-, Effective context length=4K, Backbone=GLM-4-9B-1M2024.070.9380.6690.5850.5140.4590.3910.593
MInferenceClaimed context length=-, Effective context length=8K, Backbone=Yi-9B-200K2024.070.9230.8970.790.7380.6470.5690.747
Yi-9B-200KClaimed context length=200K, Effective context length=8K, Backbone=Yi-9B-200K2024.070.9190.9020.7880.7630.6810.6290.781
StreamingLLMClaimed context length=-, Effective context length=4K, Backbone=Yi-9B-200K2024.070.9190.3780.3390.1860.130.1280.343
InfLLMClaimed context length=-, Effective context length=4K, Backbone=LLAMA-3-8B-262K2024.070.8940.7980.7010.5560.430.3950.629
InfLLMClaimed context length=-, Effective context length=<4K, Backbone=Yi-9B-200K2024.070.8030.8390.6070.4520.3860.3020.565
StreamingLLM w/ dilatedClaimed context length=-, Effective context length=<4K, Backbone=Yi-9B-200K2024.070.4480.4280.3850.2980.2680.2390.344
StreamingLLM w/ dilatedClaimed context length=-, Effective context length=<4K, Backbone=LLAMA-3-8B-262K2024.070.2340.0070.0140.1880.1650.1560.127
StreamingLLM w/ stridedClaimed context length=-, Effective context length=<4K, Backbone=Yi-9B-200K2024.070.0260.0070.0060.0060.0120.0050.011
StreamingLLM w/ stridedClaimed context length=-, Effective context length=<4K, Backbone=LLAMA-3-8B-262K2024.070.020.0070.0060.0060.0070.0130.01