Commonsense Reasoning on Hella
49.6AccuracyRoPE++EC
Evaluation results
| Method | Links | |
|---|---|---|
| RoPE++ECModel Scale=1.5B, Context Length Setting=Long2025.12 | 49.6 | |
| RoPEModel Scale=1.5B, Context Length Setting=Long2025.12 | 49.3 | |
| RoPE++EHModel Scale=1.5B, Context Length Setting=Long2025.12 | 48 | |
| RoPE++ECModel Scale=1.5B, Context Length Setting=Short2025.12 | 46.3 | |
| RoPEModel Scale=1.5B, Context Length Setting=Short2025.12 | 46.2 | |
| RoPE++EHModel Scale=1.5B, Context Length Setting=Short2025.12 | 45.6 | |
| weight decayBackbone=Qwen-0.6B2025.12 | 45.4 | |
| baselineBackbone=Qwen-0.6B2025.12 | 45.1 | |
| NEFTuneBackbone=Qwen-0.6B2025.12 | 44.6 | |
| EntroDropBackbone=Qwen-0.6B2025.12 | 44.5 | |
| hidden dropoutBackbone=Qwen-0.6B2025.12 | 44.4 | |
| vanilla token dropoutBackbone=Qwen-0.6B2025.12 | 44.4 |