Long-context language modeling on RULER
0.9142RULER ScoreDASH-3
Evaluation results
| Method | Links | |
|---|---|---|
| DASH-3Type=Diff. search, Budget=182026.05 | 0.9142 | |
| DASH-2Type=Diff. search, Budget=182026.05 | 0.9114 | |
| GA-S2Teacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.911 | |
| GA-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.9061 | |
| GA-S2Type=GA (S2-KL), Budget=182026.05 | 0.9061 | |
| AVG-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.9051 | |
| Avg-S2Type=Avg. (S2-KL), Budget=182026.05 | 0.9051 | |
| SMARTTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.8949 | |
| GR-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8869 | |
| GR-S2Type=GR (S2-KL), Budget=182026.05 | 0.8869 | |
| DASH-3Type=Diff. search, Budget=92026.05 | 0.8763 | |
| GA-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.8733 | |
| ARBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8717 | |
| ARType=Task probe, Budget=182026.05 | 0.8717 | |
| AVG-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.8704 | |
| DASH-2Type=Diff. search, Budget=92026.05 | 0.8688 | |
| GA-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.8631 | |
| GA-S2Type=GA (S2-KL), Budget=92026.05 | 0.8631 | |
| GA-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8619 | |
| GA-S2Teacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.8584 | |
| GR-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.8541 | |
| AR-MHTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.8533 | |
| CWEBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8444 | |
| CWEType=Task probe, Budget=182026.05 | 0.8444 | |
| ARTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.8423 | |
| CWEBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8394 | |
| ARBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8347 | |
| GR-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.8259 | |
| GR-S2Type=GR (S2-KL), Budget=92026.05 | 0.8259 | |
| KVBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8257 | |
| KVType=Task probe, Budget=182026.05 | 0.8257 | |
| AR-MHBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8231 | |
| AVG-S1Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8226 | |
| Avg-S1Type=Avg. (S1-MSE), Budget=182026.05 | 0.8226 | |
| GR-S1Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8209 | |
| GR-S1Type=GR (S1-MSE), Budget=182026.05 | 0.8209 | |
| AVG-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.8205 | |
| Avg-S2Type=Avg. (S2-KL), Budget=92026.05 | 0.8205 | |
| SMARTBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.819 | |
| SMARTType=Prior, Budget=182026.05 | 0.819 | |
| SMARTTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.8158 | |
| SMARTBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.8126 | |
| GA-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.8118 | |
| AVG-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8111 | |
| UNIFORMBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.8031 | |
| UniformType=Fixed, Budget=182026.05 | 0.8031 | |
| AR-MHBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7983 | |
| AR-MHType=Task probe, Budget=182026.05 | 0.7983 | |
| GA-S1Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7873 | |
| GA-S1Type=GA (S1-MSE), Budget=182026.05 | 0.7873 | |
| SMARTBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7728 | |
| GR-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7662 | |
| KVBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.7552 | |
| KVBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.7539 | |
| GA-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=25%2025.12 | 0.7539 | |
| KVType=Task probe, Budget=92026.05 | 0.7539 | |
| KVBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7538 | |
| VTBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7538 | |
| UNIFORMTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.7516 | |
| ARTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.7453 | |
| VTBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7409 | |
| VTType=Task probe, Budget=182026.05 | 0.7409 | |
| DASH-3Type=Diff. search, Budget=52026.05 | 0.7358 | |
| UNIFORMTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.7357 | |
| AR-MHTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.7322 | |
| AR-MHBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.7318 | |
| DASH-2Type=Diff. search, Budget=52026.05 | 0.7299 | |
| UNIFORMBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7262 | |
| ACT-MSETeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.7227 | |
| ACT-MSEBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7175 | |
| Act-MSEType=Model proxy, Budget=182026.05 | 0.7175 | |
| AR-MHBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.716 | |
| AR-MHType=Task probe, Budget=92026.05 | 0.716 | |
| LM-PPLBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.7152 | |
| AVG-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=12.5%2025.12 | 0.7075 | |
| Avg-S2Type=Avg. (S2-KL), Budget=52026.05 | 0.7075 | |
| CWEBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.7065 | |
| LM-PPLBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=50%2025.12 | 0.6997 | |
| LM-PPLType=Model proxy, Budget=182026.05 | 0.6997 | |
| GR-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6991 | |
| GA-S2Teacher model=Qwen2.5-1.5B-Instruct, Softmax ratio=33%2025.12 | 0.6953 | |
| AdaKV w/ CriticalKVBackbone=Mistral-7B, Cache budget=40%2025.02 | 0.6917 | |
| LM-PPLBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6907 | |
| UNIFORMBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=25%2025.12 | 0.6904 | |
| UniformType=Fixed, Budget=92026.05 | 0.6904 | |
| AVG-S2Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6786 | |
| CWEBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6752 | |
| ARBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6749 | |
| GR-S1Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6743 | |
| ARBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6707 | |
| GA-S1Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6657 | |
| KVBackbone=Llama-3.2-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6626 | |
| GA-S2Backbone=Qwen2.5-3B-Instruct, Softmax Ratio=12.5%2025.12 | 0.6617 | |
| GA-S2Type=GA (S2-KL), Budget=52026.05 | 0.6617 | |
| LM-PPLTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=33%2025.12 | 0.659 | |
| GA-S1Backbone=Llama-3.2-3B-Instruct, Softmax Ratio=50%2025.12 | 0.6569 | |
| LM-PPLTeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.6544 | |
| SMARTTeacher model=Qwen2.5-1.5B-Instruct, Softmax ratio=33%2025.12 | 0.6479 | |
| ACT-MSETeacher model=Qwen2.5-7B-Instruct, Softmax ratio=25%2025.12 | 0.6469 | |
| UNIFORMBackbone=Qwen2.5-3B-Instruct, Softmax Ratio=33%2025.12 | 0.6464 |